Repository: spack/spack

Issue:
PythonPackage: add RPATHs, parallel builds
This is a reboot of #4545 

Some Python packages depend on non-Python dependencies. For example, `py-pillow` depends on several imaging libraries. When building these Python packages, Python uses whatever compiler it was built with. Since we filter the compiler wrappers out of the Python installation, this means that we are not using Spack's compiler wrappers when installing Python packages, and hence we are not RPATHing dependencies. Even worse, some packages like `py-torch` will use a mix of `CC` and the compiler Python was built with, meaning that some parts of the build can find dependencies while others can't.

Previously, we were building with:
```
$ python setup.py build
$ python setup.py install --prefix=...
```
Now, we build with:
```
$ python setup.py build_py
$ python setup.py build_ext --parallel=... --include-dirs=... --library-dirs=... --rpath=...
$ python setup.py build_clib
$ python setup.py build_scripts
$ python setup.py install --prefix=...
```
@robertu94 @mjwoods @trws @skosukhin 


Discussion:
Problems I've encountered so far:

1. **Some packages can't be built in parallel**

   This seems to be a problem particularly due to the Python 3.8 + macOS multiprocessing fiasco. Packages affected include cython (https://github.com/cython/cython/issues/3973) and possibly pandas (https://github.com/pandas-dev/pandas/issues/30873)

2. **install phase reruns build phases**

   When you run `python setup.py install`, it actually re-runs `python setup.py build`. For most packages, this is a no-op, as the build has already finished. But I've encountered many packages that have been hacked to re-run build. This also re-runs build without the flags we added, undoing our work and making the build take twice as long. `py-cdat-lite` is an example of a package where this happens. I haven't tried this yet, but I wonder if replacing `install` with `install_(lib|headers|scripts|data)` will prevent it from re-running these steps. Another possibility is to write the config flags into `setup.cfg` instead of supplying them on the command line.

3. **`--rpath` may not even make a difference?**

   With this PR, when I build `py-cartopy`, the libraries are correctly RPATHed as evidenced by `otool -L`, but the module still fails to import unless `(DY)?LD_LIBRARY_PATH` is set. Cartopy uses `ctypes.util.find_library` to locate libraries, which uses the env var instead of the RPATH.

I'm hoping that most of these problems are package-specific and infrequent enough that we can fix those libraries by upstreaming patches. Definitely needs more testing to see what other issues this could cause.
@adamjstewart great work!

this might affect the Cartopy example, but if you look at the implementation of `find__library`, python doesn't seem to directly ask the loader which libraries to load, instead it uses custom logic per OS.  In the case of MacOS, it first searches the "override path" which comes from `DYLD_FRAMEWORK_PATH` and `DYLD_LIBRARY_PATH`.  Next it searches for libraries that have a special prefix `@executable_path/` in their name and looks for them at a path relative to the executable.  Finally it searches `DYLD_FALLBACK_FRAMEWORK_PATH`, `DYLD_FALLBACK_LIBRARY_PATH`,  then a specific list of fall back directories: `"~/Library/Frameworks"`, `"/Library/Frameworks"`,  `"/Network/Library/Frameworks"`,  `"/System/Library/Frameworks"`, `"~/lib"`, `"/usr/local/lib"`, `"/lib"`, `"/usr/lib"`.

On Linux, we ask  `/sbin/ldconfig` then the `gcc` first in in `PATH` and finally `ld`

> This seems to be a problem particularly due to the Python 3.8 + macOS multiprocessing fiasco. 

"The Python 3.8 + macOS multiprocessing fiasco" makes me think of https://github.com/spack/spack/issues/14102, which was resolved by https://github.com/spack/spack/pull/18205. Are you seeing other issues with creating processes on Mac?

It sounds like the issue is with `spack install -j x ...` for `x > 1`: i.e. individual package installs cannot run in parallel. If you point out some examples I could check and see if this is isolated to Mac.
@scheibelp https://github.com/cython/cython/issues/3973 is the main multiprocessing problem I've encountered so far. Basically, Cython implements its own multiprocessing, so when you do `-j4`, a subprocess tries to spawn its own subprocess.

So this isn't a Spack bug. The only remaining Spack + Python 3.8 + macOS bug that I know of is #20025.
@adamjstewart using this branch `py-cython`, `py-pandas`, and `py-numpy` all build successfully in parallel on my linux box.
At this point I'm pretty confident in the parallel build logic. Cython is the only build failure I've seen so far, but we can add a workaround for that. What I'm less confident in is the RPATH logic. If that doesn't actually work, there's no point in adding this additional complexity. We can still use `--parallel` for the current build/install phases without changing the phases. I'm not sure how much more time I plan to devote to this PR, but it would be good to know whether the RPATH logic actually helps or not, at least for some packages.
@adamjstewart how can we test the rpath logic? I’m happy to help testing if I know how to test it. I’d never used RPATH before spack, so I don’t know how to test it at all. 
@robertu94 In my mind, there's 3 different kinds of tests. 

1. The first test is to see whether or not the flags I'm using in `build_ext` are sufficient to build the package. First, make sure you remove any custom hacks I've added to packages in the past. You can see where I did this for `py-matplotlib`. Then, make sure the package actually builds successfully. So far `py-pillow` doesn't build at all without those custom hacks, so the RPATH logic doesn't seem to help with that package.
2. The second test is whether or not the installed libraries can be imported successfully. This can be done at build-time using `spack install --test=root py-matplotlib`. You can also run import tests post-installation using `spack test run py-matplotlib`. These two sets of tests have different env vars set, so it's important to test both. For example, `py-cartopy` builds fine, but the import tests fail for some reason.
3. The third test is whether or not any shared/dynamic libs can find their dependencies. On Linux, you can use `ldd -r /path/to/libfoo.so`, and on macOS you can use `otool -L /path/to/libfoo.dylib`. On macOS, you may see libraries named with `.dylib` or with `.so`, on Linux it's usually only `.so`.
Before this can be merged, I want to make sure a sizeable portion of Python packages don't break. Personally, I use the following `spack.yaml`:
```yaml
spack:
  specs:
  - py-black
  - py-cartopy
  - py-cmocean
  - py-codecov
  - py-fiona
  - py-flake8
  - py-geopandas
  - py-geoplot
  - py-jupyterlab
  - py-matplotlib
  - py-mypy
  - py-numpy
  - py-pandas
  - py-pygeos
  - py-pyinstrument
  - py-pytest
  - py-pytest-cov
  - py-pytest-mock
  - py-pyyaml
  - py-rasterio
  - py-scikit-learn
  - py-scipy
  - py-setuptools
  - py-shapely
  - py-sphinx
  - py-sphinx-rtd-theme
  - py-sphinxcontrib-programoutput
  - py-tables
  - py-torch
  - py-torchvision
  - py-twine
  - py-vermin
```
Other users may have more packages they care about, but this is probably enough for testing.
Ok, I've setup a mass build and test script with this information to run.  I'll let you know how it goes.
@adamjstewart  Here are the high level results:

- the build log for the new feature branch is much longer which generally doesn't bode well
- several of the packages took longer to build with this patch such as `py-fiona`
- almost all of the runtime import failures where caused by loading libgeos_c.so.
- cartopy and geopandas did not pass runtime import tests with the feature branch or with develop
- I believe shapely passed more a one or two more tests on the feature branch.

I'll have to dig into the results more carefully to know exactly what worked/didn't.

[spack_build.tar.zip](https://github.com/spack/spack/files/5817400/spack_build.tar.zip)



Hmm, so there are some packages that this doesn't help. Are there any packages that this hurts? Are there any packages that this does help?
@adamjstewart  I've had more time to read the build logs, this change breaks the build for matplotolib and py-tables and a possibly a few of their dependents; scipy's build time nearly doubled with this patch.
Let me investigate some of these examples a bit. In the meantime, I may open a separate PR that only includes the parallel build changes.
> scipy's build time nearly doubled with this patch.

@robertu94 I was unable to reproduce this locally.

`py-scipy` build time on `develop`:
```
  Fetch: 0.11s.  Build: 19m 45.79s.  Total: 19m 45.90s.
```
`py-scipy` build time on `features/build-ext`:
```
  Fetch: 0.15s.  Build: 19m 30.79s.  Total: 19m 30.94s.
```

I do notice the following warning message:
```
NOTE: -j build option not supportd. Set NPY_NUM_BUILD_JOBS=4 for parallel build.
```
If I set `NPY_NUM_BUILD_JOBS` to `str(make_jobs)` (which happens to be 4 on my ancient laptop), the new build time is:
```
  Fetch: 0.14s.  Build: 19m 55.85s.  Total: 19m 56.00s.
```
@adamjstewart in my testing it was about 11m for the parallel build, and 7m40s for the serial build on a 40core node on my schools cluster.  The file system is served to the nodes over NFS, so perhaps I hit several load spikes on the file system that pushed the build time higher.
Hmm, so even building in parallel might be problematic. Do you encounter that same issue with non-Python packages? Does limiting the parallelism to 4 help?
I do occasionally hit that with non python packages as well — it seems to be a feature of our cluster from time to time. I don’t blame spack for NFS being slow. 

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/python/package.py b/var/spack/repos/builtin/packages/python/package.py
--- a/var/spack/repos/builtin/packages/python/package.py
+++ b/var/spack/repos/builtin/packages/python/package.py
@@ -3,18 +3,14 @@
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
-import ast
 import os
 import platform
 import re
 
 import llnl.util.tty as tty
 from llnl.util.lang import match_predicate
-from llnl.util.filesystem import (force_remove, get_filetype,
-                                  path_contains_subdirectory)
+from llnl.util.filesystem import get_filetype, path_contains_subdirectory
 
-import spack.store
-import spack.util.spack_json as sjson
 from spack.util.environment import is_system_path
 from spack.util.prefix import Prefix
 from spack import *
@@ -221,10 +217,6 @@ class Python(AutotoolsPackage):
 
     conflicts('%nvhpc')
 
-    _DISTUTIL_VARS_TO_SAVE = ['LDSHARED']
-    _DISTUTIL_CACHE_FILENAME = 'sysconfig.json'
-    _distutil_vars = None
-
     # Used to cache home locations, since computing them might be expensive
     _homes = {}
 
@@ -481,90 +473,6 @@ def configure_args(self):
 
         return config_args
 
-    @run_after('install')
-    def _save_distutil_vars(self):
-        """
-        Run before changing automatically generated contents of the
-        _sysconfigdata.py, which is used by distutils to figure out what
-        executables to use while compiling and linking extensions. If we build
-        extensions with spack those executables should be spack's wrappers.
-        Spack partially covers this by setting environment variables that
-        are also accounted for by distutils. Currently there is one more known
-        variable that must be set, which is LDSHARED, so the method saves its
-        autogenerated value to pass it to the dependent package's setup script.
-        """
-
-        self._distutil_vars = {}
-
-        input_filename = self.get_sysconfigdata_name()
-        input_dict = None
-        try:
-            with open(input_filename) as input_file:
-                match = re.search(r'build_time_vars\s*=\s*(?P<dict>{.*})',
-                                  input_file.read(),
-                                  flags=re.DOTALL)
-
-                if match:
-                    input_dict = ast.literal_eval(match.group('dict'))
-        except (IOError, SyntaxError):
-            pass
-
-        if not input_dict:
-            tty.warn("Failed to find 'build_time_vars' dictionary in file "
-                     "'%s'. This might cause the extensions that are "
-                     "installed with distutils to call compilers directly "
-                     "avoiding Spack's wrappers." % input_filename)
-            return
-
-        for var_name in Python._DISTUTIL_VARS_TO_SAVE:
-            if var_name in input_dict:
-                self._distutil_vars[var_name] = input_dict[var_name]
-            else:
-                tty.warn("Failed to find key '%s' in 'build_time_vars' "
-                         "dictionary in file '%s'. This might cause the "
-                         "extensions that are installed with distutils to "
-                         "call compilers directly avoiding Spack's wrappers."
-                         % (var_name, input_filename))
-
-        if len(self._distutil_vars) > 0:
-            output_filename = None
-            try:
-                output_filename = join_path(
-                    spack.store.layout.metadata_path(self.spec),
-                    Python._DISTUTIL_CACHE_FILENAME)
-                with open(output_filename, 'w') as output_file:
-                    sjson.dump(self._distutil_vars, output_file)
-            except Exception:
-                tty.warn("Failed to save metadata for distutils. This might "
-                         "cause the extensions that are installed with "
-                         "distutils to call compilers directly avoiding "
-                         "Spack's wrappers.")
-                # We make the cache empty if we failed to save it to file
-                # to provide the same behaviour as in the case when the cache
-                # is initialized by the method load_distutils_data().
-                self._distutil_vars = {}
-                if output_filename:
-                    force_remove(output_filename)
-
-    def _load_distutil_vars(self):
-        # We update and keep the cache unchanged only if the package is
-        # installed.
-        if not self._distutil_vars and self.installed:
-            try:
-                input_filename = join_path(
-                    spack.store.layout.metadata_path(self.spec),
-                    Python._DISTUTIL_CACHE_FILENAME)
-                if os.path.isfile(input_filename):
-                    with open(input_filename) as input_file:
-                        self._distutil_vars = sjson.load(input_file)
-            except Exception:
-                pass
-
-            if not self._distutil_vars:
-                self._distutil_vars = {}
-
-        return self._distutil_vars
-
     @run_after('install')
     def filter_compilers(self):
         """Run after install to tell the configuration files and Makefiles
@@ -946,6 +854,61 @@ def setup_dependent_build_environment(self, env, dependent_spec):
         pythonpath = ':'.join(python_paths)
         env.set('PYTHONPATH', pythonpath)
 
+        # We need to make sure that the extensions are compiled and linked with
+        # the Spack wrapper. Paths to the executables that are used for these
+        # operations are normally taken from the sysconfigdata file, which we
+        # modify after the installation (see method filter compilers). The
+        # modified file contains paths to the real compilers, not the wrappers.
+        # The values in the file, however, can be overridden with environment
+        # variables. The first variable, CC (CXX), which is used for
+        # compilation, is set by Spack for the dependent package by default.
+        # That is not 100% correct because the value for CC (CXX) in the
+        # sysconfigdata file often contains additional compiler flags (e.g.
+        # -pthread), which we lose by simply setting CC (CXX) to the path to the
+        # Spack wrapper. Moreover, the user might try to build an extension with
+        # a compiler that is different from the one that was used to build
+        # Python itself, which might have unexpected side effects. However, the
+        # experience shows that none of the above is a real issue and we will
+        # not try to change the default behaviour. Given that, we will simply
+        # try to modify LDSHARED (LDCXXSHARED), the second variable, which is
+        # used for linking, in a consistent manner.
+
+        for compile_var, link_var in [('CC', 'LDSHARED'),
+                                      ('CXX', 'LDCXXSHARED')]:
+            # First, we get the values from the sysconfigdata:
+            config_compile = self.get_config_var(compile_var)
+            config_link = self.get_config_var(link_var)
+
+            # The dependent environment will have the compilation command set to
+            # the following:
+            new_compile = join_path(
+                spack.paths.build_env_path,
+                dependent_spec.package.compiler.link_paths[compile_var.lower()])
+
+            # Normally, the link command starts with the compilation command:
+            if config_link.startswith(config_compile):
+                new_link = new_compile + config_link[len(config_compile):]
+            else:
+                # Otherwise, we try to replace the compiler command if it
+                # appears "in the middle" of the link command; to avoid
+                # mistaking some substring of a path for the compiler (e.g. to
+                # avoid replacing "gcc" in "-L/path/to/gcc/"), we require that
+                # the compiler command be surrounded by spaces. Note this may
+                # leave "config_link" unchanged if the compilation command does
+                # not appear in the link command at all, for example if "ld" is
+                # invoked directly (no change would be required in that case
+                # because Spack arranges for the Spack ld wrapper to be the
+                # first instance of "ld" in PATH).
+                new_link = config_link.replace(" {0} ".format(config_compile),
+                                               " {0} ".format(new_compile))
+
+            # There is logic in the sysconfig module that is sensitive to the
+            # fact that LDSHARED is set in the environment, therefore we export
+            # the variable only if the new value is different from what we got
+            # from the sysconfigdata file:
+            if config_link != new_link:
+                env.set(link_var, new_link)
+
     def setup_dependent_run_environment(self, env, dependent_spec):
         python_paths = []
         for d in dependent_spec.traverse(deptype='run'):
@@ -973,12 +936,6 @@ def setup_dependent_package(self, module, dependent_spec):
         module.setup_py = Executable(
             self.command.path + ' setup.py --no-user-cfg')
 
-        distutil_vars = self._load_distutil_vars()
-
-        if distutil_vars:
-            for key, value in distutil_vars.items():
-                module.setup_py.add_default_env(key, value)
-
         # Add variables for lib/pythonX.Y and lib/pythonX.Y/site-packages dirs.
         module.python_lib_dir = join_path(dependent_spec.prefix,
                                           self.python_lib_dir)


Test patch: