Repository: spack/spack

Issue:
Bugfix: support package.test as install-time callback
Addresses issue reported in #34440 

Is this sufficient to handle tests that require the package build environment?

TBD:
- Add unit test  (Update: I'm struggling to add a unit test to exercise the test method of the package (and ideally parent).)
package_base: Fix Python package tests
Running `spack install --test=root py-nipype` results in the following error:
```
'super' object has no attribute 'test_log_file'
```
Replacing `test_log_file` with `test_install_log_path` (as seems to be intended) still causes the following error:
```
'NoneType' object has no attribute 'append'
```
This can be fixed by making sure that `test_failures` is not `None`.


Discussion:
@alalazo I'm having problems adding a unit test that can go through the builder to run the package's `test` method (ideally to include that of the parent class as well so similar to the `PythonPackage` import tests plus the package's tests.  I must be missing something.  Advice would be appreciated.
cc @tldahlgren (who has refactored this recently)
Other packages where this bug happened were:
- `py-reportlab@3.4.0` (but strangely not version 3.6.12)
- `py-scikit-image`
Interesting.  Tried to reproduce the issue.  

I could not build `py-nipype` or `py-scikit-image`.

Did not have the problem with `py-reportlab@3.4.0`, in fact all of the import checks passed for me.

```
* **Spack:** 0.20.0.dev0 (94bac8d6dd9140820dbde9d1dda4ea6c5736ca43)
```
I gave it a try on another system and could also reproduce it there for `py-nipype`:
* **Spack:** 0.20.0.dev0 (94bac8d6dd9140820dbde9d1dda4ea6c5736ca43)
* **Python:** 3.6.8
* **Platform:** linux-centos8-x86_64_v3
* **Concretizer:** clingo
I'm also able to reproduce this:
```
$ spack install --test=root py-nipype
...
==> Testing package py-nipype-1.7.0-rs3buym
==> [2022-12-12-17:11:14.767790] Running install-time tests
==> [2022-12-12-17:11:14.767996] RUN-TESTS: install-time tests [test]
==> [2022-12-12-17:11:24.485358] RUN-TESTS: method not implemented [test]
==> Error: TestFailure: 1 tests failed.


'super' object has no attribute 'test_log_file'

RUN-TESTS: method not implemented [test]

See build log for details:
  /var/folders/j1/68dlgpr91vlgs26vty2c8xk80000gn/T/ajstewart/spack-stage/spack-stage-py-nipype-1.7.0-rs3buymexzw2ckcn5uj7ugmnndzrgzll/spack-build-out.txt

```
* **Spack:** 0.20.0.dev0 (94bac8d6dd9140820dbde9d1dda4ea6c5736ca43)
* **Python:** 3.10.8
* **Platform:** darwin-monterey-m1
* **Concretizer:** clingo
I could still reproduce it for all 3 packages (`py-reportlab@3.4.0` is the fastest to check because the others have a lot of dependencies)

```
$ spack install --test=root py-nipype
...
==> Testing package py-nipype-1.7.0-m3xunci
==> [2022-12-13-00:20:57.074414] Running install-time tests
==> [2022-12-13-00:20:57.074461] RUN-TESTS: install-time tests [test]
==> [2022-12-13-00:21:04.163874] RUN-TESTS: method not implemented [test]
==> Error: TestFailure: 1 tests failed.


'super' object has no attribute 'test_log_file'

RUN-TESTS: method not implemented [test]

See build log for details:
  $spack-stage/spack-stage-py-nipype-1.7.0-m3xuncimklvzzafjrmpgcyzwku3zrujf/spack-build-out.txt
```
```
$ spack install --test=root py-scikit-image
...
==> Testing package py-scikit-image-0.18.3-kzswzk2
==> [2022-12-13-00:18:21.985301] Running install-time tests
==> [2022-12-13-00:18:21.985348] RUN-TESTS: install-time tests [test]
==> [2022-12-13-00:18:22.433831] RUN-TESTS: method not implemented [test]
==> Error: TestFailure: 1 tests failed.


'super' object has no attribute 'test_log_file'

RUN-TESTS: method not implemented [test]

See build log for details:
  $spack-stage/spack-stage-py-scikit-image-0.18.3-kzswzk2xvkjrut6oipsxnhdh5ijky6dk/spack-build-out.txt
```
```
$ spack install --test=root py-reportlab@3.4.0
...
==> Testing package py-reportlab-3.4.0-xekzjyn
==> [2022-12-13-00:21:42.089951] Running install-time tests
==> [2022-12-13-00:21:42.089997] RUN-TESTS: install-time tests [test]
==> [2022-12-13-00:21:42.134144] RUN-TESTS: method not implemented [test]
==> Error: TestFailure: 1 tests failed.


'super' object has no attribute 'test_log_file'

RUN-TESTS: method not implemented [test]

See build log for details:
  $spack-stage/spack-stage-py-reportlab-3.4.0-xekzjyndvfhpyzal3dj77mrycssz2caa/spack-build-out.txt
```

* **Spack:** 0.20.0.dev0 (94bac8d6dd9140820dbde9d1dda4ea6c5736ca43)
* **Python:** 3.11.0
* **Platform:** linux-fedora37-skylake
* **Concretizer:** clingo
Likely possible to reproduce by adding `import_modules = ['foo']` to any Python package.
Thank you all for the additional information.  

I have not liked the idea of only defining that property when calling certain test method(s) so it looks like this demonstrates a case where that approach fails.  The refactor should address this issue but there should definitely be a test to confirm.
> Thank you all for the additional information.
> 
> I have not liked the idea of only defining that property when calling certain test method(s) so it looks like this demonstrates a case where that approach fails. The refactor should address this issue but there should definitely be a test to confirm.

Correction.  This is an issue with changes related to the builder and #33942 in particular.

Resolving patch:
diff --git a/lib/spack/spack/builder.py b/lib/spack/spack/builder.py
--- a/lib/spack/spack/builder.py
+++ b/lib/spack/spack/builder.py
@@ -128,7 +128,12 @@ def __init__(self, wrapped_pkg_object, root_builder):
             new_cls = type(
                 new_cls_name,
                 bases,
-                {"run_tests": property(lambda x: x.wrapped_package_object.run_tests)},
+                {
+                    "run_tests": property(lambda x: x.wrapped_package_object.run_tests),
+                    "test_log_file": property(lambda x: x.wrapped_package_object.test_log_file),
+                    "test_failures": property(lambda x: x.wrapped_package_object.test_failures),
+                    "test_suite": property(lambda x: x.wrapped_package_object.test_suite),
+                },
             )
             new_cls.__module__ = package_cls.__module__
             self.__class__ = new_cls
diff --git a/lib/spack/spack/package_base.py b/lib/spack/spack/package_base.py
--- a/lib/spack/spack/package_base.py
+++ b/lib/spack/spack/package_base.py
@@ -651,10 +651,12 @@ class PackageBase(WindowsRPath, PackageViewMixin, metaclass=PackageMeta):
     #: List of test failures encountered during a smoke/install test run.
     test_failures = None
 
-    #: TestSuite instance used to manage smoke/install tests for one or more
-    #: specs.
+    #: TestSuite instance used to manage smoke/install tests for one or more specs.
     test_suite = None
 
+    #: Path to the log file used for tests
+    test_log_file = None
+
     def __init__(self, spec):
         # this determines how the package should be built.
         self.spec = spec
@@ -1884,7 +1886,10 @@ def _setup_test(self, verbose, externals):
             pkg_id = self.test_suite.test_pkg_id(self.spec)
         else:
             self.test_log_file = fsys.join_path(self.stage.path, _spack_install_test_log)
+            self.test_suite = TestSuite([self.spec])
+            self.test_suite.stage = self.stage.path
             pkg_id = self.spec.format("{name}-{version}-{hash:7}")
+
         fsys.touch(self.test_log_file)  # Otherwise log_parse complains
 
         with tty.log.log_output(self.test_log_file, verbose) as logger:
@@ -2441,6 +2446,11 @@ def run_test_callbacks(builder, method_names, callback_type="install"):
         with builder.pkg._setup_test(verbose=False, externals=False) as logger:
             # Report running each of the methods in the build log
             print_test_message(logger, "Running {0}-time tests".format(callback_type), True)
+            builder.pkg.test_suite.current_test_spec = builder.pkg.spec
+            builder.pkg.test_suite.current_base_spec = builder.pkg.spec
+
+            if "test" in method_names:
+                _copy_cached_test_files(builder.pkg, builder.pkg.spec)
 
             for name in method_names:
                 try:
@@ -2489,6 +2499,25 @@ def print_test_message(logger, msg, verbose):
         tty.msg(msg)
 
 
+def _copy_cached_test_files(pkg, spec):
+    """Copy any cached stand-alone test-related files."""
+
+    # copy installed test sources cache into test cache dir
+    if spec.concrete:
+        cache_source = spec.package.install_test_root
+        cache_dir = pkg.test_suite.current_test_cache_dir
+        if os.path.isdir(cache_source) and not os.path.exists(cache_dir):
+            fsys.install_tree(cache_source, cache_dir)
+
+    # copy test data into test data dir
+    data_source = Prefix(spec.package.package_dir).test
+    data_dir = pkg.test_suite.current_test_data_dir
+    if os.path.isdir(data_source) and not os.path.exists(data_dir):
+        # We assume data dir is used read-only
+        # maybe enforce this later
+        shutil.copytree(data_source, data_dir)
+
+
 def test_process(pkg, kwargs):
     verbose = kwargs.get("verbose", False)
     externals = kwargs.get("externals", False)
@@ -2527,20 +2556,7 @@ def test_process(pkg, kwargs):
                     except spack.repo.UnknownPackageError:
                         continue
 
-                    # copy installed test sources cache into test cache dir
-                    if spec.concrete:
-                        cache_source = spec_pkg.install_test_root
-                        cache_dir = pkg.test_suite.current_test_cache_dir
-                        if os.path.isdir(cache_source) and not os.path.exists(cache_dir):
-                            fsys.install_tree(cache_source, cache_dir)
-
-                    # copy test data into test data dir
-                    data_source = Prefix(spec_pkg.package_dir).test
-                    data_dir = pkg.test_suite.current_test_data_dir
-                    if os.path.isdir(data_source) and not os.path.exists(data_dir):
-                        # We assume data dir is used read-only
-                        # maybe enforce this later
-                        shutil.copytree(data_source, data_dir)
+                    _copy_cached_test_files(pkg, spec)
 
                     # grab the function for each method so we can call
                     # it with the package
diff --git a/var/spack/repos/builtin/packages/py-libensemble/package.py b/var/spack/repos/builtin/packages/py-libensemble/package.py
--- a/var/spack/repos/builtin/packages/py-libensemble/package.py
+++ b/var/spack/repos/builtin/packages/py-libensemble/package.py
@@ -80,7 +80,7 @@ def run_tutorial_tests(self, exe):
         )
 
         if not os.path.isfile(join_path(test_dir, exe)):
-            print("Skipping {0} test".format(exe))
+            print("SKIPPED: {0} test does not exist".format(exe))
             return
 
         self.run_test(
@@ -91,4 +91,6 @@ def run_tutorial_tests(self, exe):
         )
 
     def test(self):
-        self.run_tutorial_tests("test_uniform_sampling.py")
+        super(__class__, self).test()
+        for tutorial in ["test_uniform_sampling.py", "test_1d_sampling.py"]:
+            self.run_tutorial_tests(tutorial)


Test patch:
diff --git a/lib/spack/spack/install_test.py b/lib/spack/spack/install_test.py
--- a/lib/spack/spack/install_test.py
+++ b/lib/spack/spack/install_test.py
@@ -13,9 +13,9 @@
 
 import spack.error
 import spack.paths
-import spack.util.prefix
 import spack.util.spack_json as sjson
 from spack.spec import Spec
+from spack.util.prefix import Prefix
 
 test_suite_filename = "test_suite.lock"
 results_filename = "results.txt"
@@ -145,6 +145,7 @@ def __init__(self, specs, alias=None):
 
         self.alias = alias
         self._hash = None
+        self._stage = None
 
         self.fails = 0
 
@@ -247,8 +248,19 @@ def ensure_stage(self):
 
     @property
     def stage(self):
-        """The root test suite stage directory."""
-        return spack.util.prefix.Prefix(os.path.join(get_test_stage_dir(), self.content_hash))
+        """The root test suite stage directory.
+
+        Returns:
+            str: the spec's test stage directory path
+        """
+        if not self._stage:
+            self._stage = Prefix(fs.join_path(get_test_stage_dir(), self.content_hash))
+        return self._stage
+
+    @stage.setter
+    def stage(self, value):
+        """Set the value of a non-default stage directory."""
+        self._stage = value if isinstance(value, Prefix) else Prefix(value)
 
     @property
     def results_file(self):
@@ -299,7 +311,7 @@ def test_dir_for_spec(self, spec):
         Returns:
             str: the spec's test stage directory path
         """
-        return self.stage.join(self.test_pkg_id(spec))
+        return Prefix(self.stage.join(self.test_pkg_id(spec)))
 
     @classmethod
     def tested_file_name(cls, spec):
@@ -322,7 +334,7 @@ def tested_file_for_spec(self, spec):
         Returns:
             str: the spec's test status file path
         """
-        return self.stage.join(self.tested_file_name(spec))
+        return fs.join_path(self.stage, self.tested_file_name(spec))
 
     @property
     def current_test_cache_dir(self):
diff --git a/lib/spack/spack/test/builder.py b/lib/spack/spack/test/builder.py
--- a/lib/spack/spack/test/builder.py
+++ b/lib/spack/spack/test/builder.py
@@ -154,3 +154,17 @@ def test_monkey_patching_wrapped_pkg():
     s.package.run_tests = True
     assert builder.pkg.run_tests is True
     assert builder.pkg_with_dispatcher.run_tests is True
+
+
+@pytest.mark.regression("34440")
+@pytest.mark.usefixtures("builder_test_repository", "config", "working_env")
+def test_monkey_patching_test_log_file():
+    s = spack.spec.Spec("old-style-autotools").concretized()
+    builder = spack.builder.create(s.package)
+    assert s.package.test_log_file is None
+    assert builder.pkg.test_log_file is None
+    assert builder.pkg_with_dispatcher.test_log_file is None
+
+    s.package.test_log_file = "/some/file"
+    assert builder.pkg.test_log_file == "/some/file"
+    assert builder.pkg_with_dispatcher.test_log_file == "/some/file"