Repository: spack/spack

Issue:
py-numpy: add headers attribute
Copy-n-pasted this from the `PythonPackage` base class. For some reason, it doesn't work if it's in the base class, see https://github.com/spack/spack/pull/28527#issuecomment-1172046128


Discussion:
The `headers` property of a package defines where to look for headers for that package. So with this PR, `spec['py-numpy'].headers` will tell you where the headers can be found. We _could_ define this in the package that needs to find the headers, but if there are more than one, we would need to copy-n-paste it every time. So the idea is to define it only once and let packages reuse it. It was _supposed_ to work by defining it in the base class, but it isn't working and I don't know why.

Btw, `py-petsc4py` is the package that has trouble finding these headers without adding them to `CPATH` @balay
Ah okay, that is helpful context info. In that case, recursively searching `platlib` seems watertight. The numpy headers should never be landing in `<prefix>/include`.
@spackbot run pipeline
I've started that [pipeline](https://gitlab.spack.io//spack/spack/-/pipelines/177377) for you!
> Btw, py-petsc4py is the package that has trouble finding these headers without adding them to CPATH @balay

Looks like petsc4py is querying numpy directly to get this info
```python
>>> import numpy
>>> numpy.get_include()
'/usr/lib64/python3.10/site-packages/numpy/core/include'
```
So assuming petsc4py will be unaffected by this change..
I'm seeing this pattern used in the `py-gpaw`, `vapor`, and `precice` packages to get a particular `numpy/core/include` subdirectory of `platlib`, which in `precice`'s case at least is used to populate a specific `-DNumPy_INCLUDE_DIR` CMake var: https://github.com/spack/spack/blob/977c89cee19b0c589f3ffba3f600892789acebf5/var/spack/repos/builtin/packages/precice/package.py#L153-L169

Outside of `py-matplotlib` and `py-torchvision`, I wasn't able to see any packages specifically requesting `py-numpy`'s `.headers` (which I guess makes sense if #28527 was just merged). The reason I mention this is: if multiple packages are trying to get at that *precise* subdirectory of numpy's includes, would it make sense to add that particular platlib subdirectory as a property of the `py-numpy` package as well here?
@adamjstewart: I don't fully understand what function that `inspect.getmodule(self).include` (added in #28527) is expected to invoke, and by inspecting the callstack, `inspect.getmodule(self).include` seemed to be calling `_headers_default_handler()` in `spec.py` (which I still do not understand whatsoever), which is why you saw both the default and the `build_systems/python.py` method being called in order.

However, I was regardless able to make `py-scipy` find the headers for `py-numpy` by taking the exact code you used in *this* PR, and instead moving it into `build_systems/python.py`'s `def headers(self)` method. The diff is here https://github.com/adamjstewart/spack/compare/packages/py-numpy...cosmicexplorer:spack:packages/py-numpy, but it's just one commit that does exactly that. Please let me know if this also works for you.
> would it make sense to add that particular platlib subdirectory as a property of the `py-numpy` package as well here?

That's the point of using `spec['py-numpy'].headers.directories[0]`, we should replace the hacks in those package with this if we can get it working.

> I don't fully understand what function that `inspect.getmodule(self).include` (added in https://github.com/spack/spack/pull/28527) is expected to invoke

It's supposed to access `spec['python'].package.include`, but I guess it's actually named `python_include`, not `include`. Let me try that. I'll try your patch as well.
Yep, that seems to work, nice find! I'll open a separate PR for this.

Resolving patch:
diff --git a/lib/spack/spack/build_systems/python.py b/lib/spack/spack/build_systems/python.py
--- a/lib/spack/spack/build_systems/python.py
+++ b/lib/spack/spack/build_systems/python.py
@@ -243,8 +243,8 @@ def headers(self):
         """Discover header files in platlib."""
 
         # Headers may be in either location
-        include = inspect.getmodule(self).include
-        platlib = inspect.getmodule(self).platlib
+        include = self.prefix.join(self.spec["python"].package.include)
+        platlib = self.prefix.join(self.spec["python"].package.platlib)
         headers = find_all_headers(include) + find_all_headers(platlib)
 
         if headers:
@@ -259,7 +259,7 @@ def libs(self):
 
         # Remove py- prefix in package name
         library = "lib" + self.spec.name[3:].replace("-", "?")
-        root = inspect.getmodule(self).platlib
+        root = self.prefix.join(self.spec["python"].package.platlib)
 
         for shared in [True, False]:
             libs = find_libraries(library, root, shared=shared, recursive=True)
diff --git a/var/spack/repos/builtin/packages/py-torch/package.py b/var/spack/repos/builtin/packages/py-torch/package.py
--- a/var/spack/repos/builtin/packages/py-torch/package.py
+++ b/var/spack/repos/builtin/packages/py-torch/package.py
@@ -321,37 +321,6 @@ class PyTorch(PythonPackage, CudaPackage, ROCmPackage):
         when="@:1.9.1 ^cuda@11.4.100:",
     )
 
-    @property
-    def headers(self):
-        """Discover header files in platlib."""
-
-        # Headers may be in either location
-        include = join_path(self.prefix, self.spec["python"].package.include)
-        platlib = join_path(self.prefix, self.spec["python"].package.platlib)
-        headers = find_all_headers(include) + find_all_headers(platlib)
-
-        if headers:
-            return headers
-
-        msg = "Unable to locate {} headers in {} or {}"
-        raise NoHeadersError(msg.format(self.spec.name, include, platlib))
-
-    @property
-    def libs(self):
-        """Discover libraries in platlib."""
-
-        # Remove py- prefix in package name
-        library = "lib" + self.spec.name[3:].replace("-", "?")
-        root = join_path(self.prefix, self.spec["python"].package.platlib)
-
-        for shared in [True, False]:
-            libs = find_libraries(library, root, shared=shared, recursive=True)
-            if libs:
-                return libs
-
-        msg = "Unable to recursively locate {} libraries in {}"
-        raise NoLibrariesError(msg.format(self.spec.name, root))
-
     @when("@1.5.0:")
     def patch(self):
         # https://github.com/pytorch/pytorch/issues/52208


Test patch: