Repository: spack/spack

Issue:
Defaults in `package_base` unconditionally deactivate package checks for multiple build systems
### Steps to reproduce

* Attempt to install a package with `--test=root` for a package using any of the build systems:
  * autotools
  * cmake
  * makefile
  * meson
  * nmake
  * python
  * qmake
  * racket
  * scons
  * sip
  * waf
* **IF** the recipe/build system defines checks at build or install time **AND** the relevant `XXX_time_test_callbacks` attribute is mentioned in the `legacy_attributes` list of the build system **AND** the recipe **DOES NOT** override same in (specifically) the Package class...
* **THEN** those checks will not be executed. 
 

### Error message

**N/A**

### Information on your system

* **Spack:** 0.19.0
* **Python:** 3.6.8
* **Platform:** linux-scientific7-skylake_avx512
* **Concretizer:** clingo


### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have searched the issues of this repo and believe this is not a duplicate
- [X] I have run the failing commands in debug mode and reported the output


Discussion:
This issue appears to be the result of the defaulting of `build_time_test_callbacks` and `install_time_test_callbacks` to `None` in `PackageBase` in conjunction with the listing of either or both of these items as `legacy_attributes`.

In the case where the recipe's package class does not override the default, then the `None` default from `PackageBase` overrides anything set by the builder in either the recipe's in-use builder or any of its bases.

Incoming PR with fix.
This issue and #33942 are both applicable to `develop` and `releases/v0.19`

Resolving patch:
diff --git a/lib/spack/spack/audit.py b/lib/spack/spack/audit.py
--- a/lib/spack/spack/audit.py
+++ b/lib/spack/spack/audit.py
@@ -287,7 +287,7 @@ def _check_build_test_callbacks(pkgs, error_cls):
     errors = []
     for pkg_name in pkgs:
         pkg_cls = spack.repo.path.get_pkg_class(pkg_name)
-        test_callbacks = pkg_cls.build_time_test_callbacks
+        test_callbacks = getattr(pkg_cls, "build_time_test_callbacks", None)
 
         if test_callbacks and "test" in test_callbacks:
             msg = '{0} package contains "test" method in ' "build_time_test_callbacks"
diff --git a/lib/spack/spack/cmd/info.py b/lib/spack/spack/cmd/info.py
--- a/lib/spack/spack/cmd/info.py
+++ b/lib/spack/spack/cmd/info.py
@@ -241,8 +241,8 @@ def print_tests(pkg):
     # So the presence of a callback in Spack does not necessarily correspond
     # to the actual presence of built-time tests for a package.
     for callbacks, phase in [
-        (pkg.build_time_test_callbacks, "Build"),
-        (pkg.install_time_test_callbacks, "Install"),
+        (getattr(pkg, "build_time_test_callbacks", None), "Build"),
+        (getattr(pkg, "install_time_test_callbacks", None), "Install"),
     ]:
         color.cprint("")
         color.cprint(section_title("Available {0} Phase Test Methods:".format(phase)))
diff --git a/lib/spack/spack/package_base.py b/lib/spack/spack/package_base.py
--- a/lib/spack/spack/package_base.py
+++ b/lib/spack/spack/package_base.py
@@ -528,10 +528,6 @@ class PackageBase(WindowsRPath, PackageViewMixin, metaclass=PackageMeta):
     # These are default values for instance variables.
     #
 
-    #: A list or set of build time test functions to be called when tests
-    #: are executed or 'None' if there are no such test functions.
-    build_time_test_callbacks = None  # type: Optional[List[str]]
-
     #: By default, packages are not virtual
     #: Virtual packages override this attribute
     virtual = False
@@ -540,10 +536,6 @@ class PackageBase(WindowsRPath, PackageViewMixin, metaclass=PackageMeta):
     #: those that do not can be used to install a set of other Spack packages.
     has_code = True
 
-    #: A list or set of install time test functions to be called when tests
-    #: are executed or 'None' if there are no such test functions.
-    install_time_test_callbacks = None  # type: Optional[List[str]]
-
     #: By default we build in parallel.  Subclasses can override this.
     parallel = True
 


Test patch:
diff --git a/lib/spack/spack/test/builder.py b/lib/spack/spack/test/builder.py
--- a/lib/spack/spack/test/builder.py
+++ b/lib/spack/spack/test/builder.py
@@ -121,3 +121,17 @@ def test_old_style_compatibility_with_super(spec_str, method_name, expected):
     builder = spack.builder.create(s.package)
     value = getattr(builder, method_name)()
     assert value == expected
+
+
+@pytest.mark.regression("33928")
+@pytest.mark.usefixtures("builder_test_repository", "config", "working_env")
+@pytest.mark.disable_clean_stage_check
+def test_build_time_tests_are_executed_from_default_builder():
+    s = spack.spec.Spec("old-style-autotools").concretized()
+    builder = spack.builder.create(s.package)
+    builder.pkg.run_tests = True
+    for phase_fn in builder:
+        phase_fn.execute()
+
+    assert os.environ.get("CHECK_CALLED") == "1", "Build time tests not executed"
+    assert os.environ.get("INSTALLCHECK_CALLED") == "1", "Install time tests not executed"
diff --git a/var/spack/repos/builder.test/packages/old-style-autotools/package.py b/var/spack/repos/builder.test/packages/old-style-autotools/package.py
--- a/var/spack/repos/builder.test/packages/old-style-autotools/package.py
+++ b/var/spack/repos/builder.test/packages/old-style-autotools/package.py
@@ -48,3 +48,9 @@ def after_autoreconf_1(self):
     @run_after("autoreconf", when="@2.0")
     def after_autoreconf_2(self):
         os.environ["AFTER_AUTORECONF_2_CALLED"] = "1"
+
+    def check(self):
+        os.environ["CHECK_CALLED"] = "1"
+
+    def installcheck(self):
+        os.environ["INSTALLCHECK_CALLED"] = "1"