Repository: spack/spack

Issue:
py-pybind11 not usable with system-provided python due to PYTHONPATH problem
### Steps to reproduce the issue

py-pybind11 builds fine, but is unusable because spack doesn't correctly make PYTHONPATH point to it when using a system-provided python installation. I don't know if it's a problem with spack or with the pybind11 package itself.

To reproduce, first make sure you have a system-installed python, listed in your packages.yaml file, then do the following:

```console
$ spack install py-pybind11
$ spack load -r py-pybind11
$ python3 -c "import pybind11"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'pybind11'
```

Upon examining `PYTHONPATH`, I see this:
```
/projects/spack/opt/spack/linux-ubuntu21.04-sandybridge/gcc-10.3.0/py-pybind11-2.5.0-tolbywo4itr2xvmg6ttre3ltuu6ucmcl/lib/python3/dist-packages
```

However this `lib/python3/dist-packages` folder is empty. If I want to be able to import pybind11, I need to add the `lib/python3.9/site-packages` folder to `PYTHONPATH`. The problem makes all the packages that depend on py-pybind11 fail to build.

All the above was done done with a system-provided python 3.9 (installed in /usr and listed in my packages.yaml file). However if I do the following:

```console
$ spack install py-pybind11 ^python@3.8.2    # force installing a python package other than the one provided by the system
$ spack load -r  py-pybind11 ^python@3.8.2
```
the correct `site-packages` folder is added to PYTHONPATH and I can successfully import the module.

@ax3l

### Information on your system

* **Spack:** 0.16.2-4011-5823a9b302
* **Python:** 3.9.5
* **Platform:** linux-ubuntu21.04-sandybridge
* **Concretizer:** original

### Additional information

_No response_

### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have run `spack maintainers <name-of-the-package>` and **@mentioned** any maintainers
- [X] I have uploaded the build log and environment files
- [X] I have searched the issues of this repo and believe this is not a duplicate


Discussion:
@ax3l the issue is that `py-pybind11` is a `CMakePackage` and doesn't contain the same logic as `PythonPackage` for installing libraries to the correct directory. Does it have to be a `CMakePackage`? If so, we'll need to copy the logic from `PythonPackage` into `py-pybind11`.
Uff, I have to admit we reworked the CMake logic a ton in 2.6+.

Let's see if that vanishes with the latest release? (#25519)

> Does it have to be a CMakePackage? If so, we'll need to copy the logic from PythonPackage into py-pybind11.

Not necessarily, but usually downstream consumers are C++ projects that use CMake's `find_package` to find it.
I think if we install it as a PythonPackage we should still be able to find it from C++, we can test with `openPMD-api` or other projects if we change it.
I wonder if we can subclass both `CMakePackage` and `PythonPackage` and call the install methods for both in the package. That would help prevent the setup.py logic from becoming outdated when it gets updated in `PythonPackage`.

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/py-pybind11/package.py b/var/spack/repos/builtin/packages/py-pybind11/package.py
--- a/var/spack/repos/builtin/packages/py-pybind11/package.py
+++ b/var/spack/repos/builtin/packages/py-pybind11/package.py
@@ -8,7 +8,7 @@
 from spack import *
 
 
-class PyPybind11(CMakePackage):
+class PyPybind11(CMakePackage, PythonPackage):
     """pybind11 -- Seamless operability between C++11 and Python.
 
     pybind11 is a lightweight header-only library that exposes C++ types in
@@ -43,13 +43,13 @@ class PyPybind11(CMakePackage):
     depends_on('py-setuptools', type='build')
     depends_on('py-pytest', type='test')
 
-    extends('python')
-
     # compiler support
     conflicts('%gcc@:4.7')
     conflicts('%clang@:3.2')
     conflicts('%intel@:16')
 
+    build_directory = '.'
+
     def cmake_args(self):
         args = []
         args.append('-DPYTHON_EXECUTABLE:FILEPATH=%s'
@@ -72,9 +72,8 @@ def patch(self):
                     string=True)
 
     def install(self, spec, prefix):
-        super(PyPybind11, self).install(spec, prefix)
-        setup_py('install', '--single-version-externally-managed', '--root=/',
-                 '--prefix={0}'.format(prefix))
+        CMakePackage.install(self, spec, prefix)
+        PythonPackage.install(self, spec, prefix)
 
     @run_after('install')
     @on_package_attributes(run_tests=True)


Test patch: