Repository: spack/spack

Issue:
vtk8.2 python modules do not get loaded in PYTHONPATH
Maintainers: @chuckatkins @danlipsa

It appears `vtk` restructured their python bindings starting `8.2.0` (https://blog.kitware.com/vtk-8-2-0/). However, this change is apparently not reflected in `spack`, and therefore, I am not getting the correct `PYTHONPATH`.

### Steps to reproduce the issue

I can install version `8.2.0` successfully, and it puts python bindings in `lib64`. Upon loading the package, `spack` puts `lib` in the `PYTHONPATH`, and I cannot import the python package.

```console
$ spack install vtk@8.2.0 +python # i installed with +mpi+osmesa, but that should not matter here
$ spack cd -i vtk@8.2.0
$ ls lib/python3.7/site-packages/          # returns empty
$ ls lib64/python3.7/site-packages/
vtk.py	vtkmodules

$ spack load -r vtk@8.2.0
$ echo $PYTHONPATH
$SPACK_ROOT/opt/spack/linux-rhel7-broadwell/gcc-7.3.0/vtk-8.2.0-75xr5zuecmun6av2rmgbrzjammyh62sz/lib/python3.7/site-packages: .....
```


Here is the behavior for `vtk@8.1.2` (works correctly)
```console
$ spack install vtk@8.1.2 +python # i installed with +mpi+osmesa, but that should not matter here
$  spack cd -i vtk@8.1.2
$ ls lib/python3.7/site-packages/
vtk               # python package
$ ls lib64/python3.7/site-packages/          # does not exist

$ spack load -r vtk@8.1.2
$ echo $PYTHONPATH
$SPACK_ROOT/opt/spack/linux-rhel7-broadwell/gcc-7.3.0/vtk-8.1.2-6emg5zldy6rw5qgcf33x7s3unh55s6z6/lib/python3.7/site-packages: ....
```

### Information on your system

Tried this on `linux-rhel7-broadwell` architecture. 

### General information

<!-- These boxes can be checked by replacing [ ] with [x] or by clicking them after submitting the issue. -->
- [ ] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [x] I have run `spack maintainers <name-of-the-package>` and @mentioned any maintainers
- [x] I have uploaded the build log and environment files
- [x] I have searched the issues of this repo and believe this is not a duplicate



Discussion:
We could modify the `python` package to check both `lib/pythonX.Y/site-packages` and `lib64/pythonX.Y/site-packages`, but I've literally never seen a package that uses the latter. Was this a mistake made by the VTK team or do they know more than me?
lib64 is picked up from the host's preferred library directory. Spack can pass `CMAKE_INSTALL_LIBDIR=lib` to always use it instead.

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/python/package.py b/var/spack/repos/builtin/packages/python/package.py
--- a/var/spack/repos/builtin/packages/python/package.py
+++ b/var/spack/repos/builtin/packages/python/package.py
@@ -919,11 +919,14 @@ def setup_dependent_build_environment(self, env, dependent_spec):
             env.prepend_path('PATH', path)
 
         python_paths = []
-        for d in dependent_spec.traverse(
-                deptype=('build', 'run', 'test')):
+        for d in dependent_spec.traverse(deptype=('build', 'run', 'test')):
             if d.package.extends(self.spec):
-                python_paths.append(join_path(d.prefix,
-                                              self.site_packages_dir))
+                # Python libraries may be installed in lib or lib64
+                # See issues #18520 and #17126
+                for lib in ['lib', 'lib64']:
+                    python_paths.append(join_path(
+                        d.prefix, lib, 'python' + str(self.version.up_to(2)),
+                        'site-packages'))
 
         pythonpath = ':'.join(python_paths)
         env.set('PYTHONPATH', pythonpath)
@@ -932,8 +935,10 @@ def setup_dependent_run_environment(self, env, dependent_spec):
         # For run time environment set only the path for
         # dependent_spec and prepend it to PYTHONPATH
         if dependent_spec.package.extends(self.spec):
-            env.prepend_path('PYTHONPATH', join_path(
-                dependent_spec.prefix, self.site_packages_dir))
+            for lib in ['lib', 'lib64']:
+                env.prepend_path('PYTHONPATH', join_path(
+                    dependent_spec.prefix, lib,
+                    'python' + str(self.version.up_to(2)), 'site-packages'))
 
     def setup_dependent_package(self, module, dependent_spec):
         """Called before python modules' install() methods.


Test patch: