Repository: spack/spack

Issue:
GCC rpath installation in specs has unintended consequences
### Steps to reproduce

Spack's GCC install hardwires (through the `specs` file) the linker to adding an rpath entry to GCC's library dir, so that compiled binaries get the right `libstdc++` etc. This is mostly necessary because Spack doesn't currently treat spack-compiled compilers as part of the spec (you can't install `zlib %gcc@8.3 % gcc@4.8` where 4.8 is the system compiler and 8.3 is built by spack): otherwise, you could have GCC add a runtime environment flag to append rpaths to the LDFLAGS.

The problem I'm having is that CMake isn't able to detect this sneaky compiler-added rpath entry when attempting to set the install rpath, so when I ask for the install rpaths to be set to `$ORIGIN/../lib` it actually ends up as `/spack/gcc/lib:$ORIGIN/../lib`. So what I'd *like* to do is add the ability to build *without* forcibly adding this rpath.

I think what I will do is have the GCC package *move* the default `specs` to `specs.norpath` and then write a new `specs` file:
```
%include <specs.norpath>

%rename link orig_link

*link:
-rpath ${GCC}/lib:${GCC}/lib64 $(orig_link)
```

See [kitware/cmake/cmake#22730](https://gitlab.kitware.com/cmake/cmake/-/issues/22730)

### Error message

```
$ patchelf --print-rpath bin/foo
/projects/spack/opt/spack/gcc-4.4.7/gcc/punpbku/lib:/projects/spack/opt/spack/gcc-4.4.7/gcc/punpbku/lib64:$ORIGIN/../lib64
```

### Information on your system

* **Spack:** 0.16.2-3016-1fd1f1c93f
* **Python:** 3.8.10
* **Platform:** linux-rhel6-broadwell
* **Concretizer:** clingo

### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have searched the issues of this repo and believe this is not a duplicate
- [X] I have run the failing commands in debug mode and reported the output


Discussion:
It turns out that `%include` doesn't work from the main `specs` file: had to [read the GCC source](https://github.com/gcc-mirror/gcc/blob/0e90799071ee78f712f3b58fca7000bc0a258ade/gcc/gcc.c#L2377) to figure this out.

Other spec file modifications for rpaths:
- [GCC configure-time adjustment](https://www.py4u.net/discuss/105305)
- [Homebrew GCC](https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/gcc.rb) 

Updated working solution:
- Instead of patching `*link` directly, let's patch into something easier and more appropriate (`*link_libgcc`) and introduce our own variable `*link_libgcc_rpath` that can be trivially overridden
- Append ` %(link_libgcc_rpath)` to the `link_libgcc` section and add the typical spack rpath to `*link_libgcc_rpath`. This reproduces the current spack behavior.
- To override, define a spec file `norpath.spec` with the contents `*link_libgcc_rpath:` followed by two blank lines; and pass `--specs=norpath.spec` to GCC. This prevents the GCC rpaths from propagating through the compiler options!

I will submit a patch to do Spack's GCC to do this.

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/gcc/package.py b/var/spack/repos/builtin/packages/gcc/package.py
--- a/var/spack/repos/builtin/packages/gcc/package.py
+++ b/var/spack/repos/builtin/packages/gcc/package.py
@@ -12,6 +12,7 @@
 
 import spack.platforms
 import spack.util.executable
+from spack.build_environment import dso_suffix
 from spack.operating_systems.mac_os import macos_sdk_path, macos_version
 
 
@@ -643,22 +644,74 @@ def spec_dir(self):
     @run_after('install')
     def write_rpath_specs(self):
         """Generate a spec file so the linker adds a rpath to the libs
-           the compiler used to build the executable."""
+           the compiler used to build the executable.
+
+           .. caution::
+
+              The custom spec file by default with *always* pass ``-Wl,-rpath
+              ...`` to the linker, which will cause the linker to *ignore* the
+              value of ``LD_RUN_PATH``, which otherwise would be saved to the
+              binary as the default rpath. See the mitigation below for how to
+              temporarily disable this behavior.
+
+           Structure the specs file so that users can define a custom spec file
+           to suppress the spack-linked rpaths to facilitate rpath adjustment
+           for relocatable binaries. The custom spec file
+           :file:`{norpath}.spec` will have a single
+           line followed by two blanks lines::
+
+               *link_libgcc_rpath:
+
+
+
+           It can be passed to the GCC linker using the argument
+           ``--specs=norpath.spec`` to disable the automatic rpath and restore
+           the behavior of ``LD_RUN_PATH``."""
         if not self.spec_dir:
             tty.warn('Could not install specs for {0}.'.format(
                      self.spec.format('{name}{@version}')))
             return
 
         gcc = self.spec['gcc'].command
-        lines = gcc('-dumpspecs', output=str).strip().split('\n')
+        lines = gcc('-dumpspecs', output=str).splitlines(True)
         specs_file = join_path(self.spec_dir, 'specs')
+
+        # Save a backup
+        with open(specs_file + '.orig', 'w') as out:
+            out.writelines(lines)
+
+        # Find which directories have shared libraries
+        rpath_libdirs = []
+        for dir in ['lib', 'lib64']:
+            libdir = join_path(self.prefix, dir)
+            if glob.glob(join_path(libdir, "*." + dso_suffix)):
+                rpath_libdirs.append(libdir)
+
+        if not rpath_libdirs:
+            # No shared libraries
+            tty.warn('No dynamic libraries found in lib/lib64')
+            return
+
+        # Overwrite the specs file
         with open(specs_file, 'w') as out:
             for line in lines:
-                out.write(line + '\n')
-                if line.startswith('*link:'):
-                    out.write('-rpath {0}:{1} '.format(
-                              self.prefix.lib, self.prefix.lib64))
+                out.write(line)
+                if line.startswith('*link_libgcc:'):
+                    # Insert at start of line following link_libgcc, which gets
+                    # inserted into every call to the linker
+                    out.write('%(link_libgcc_rpath) ')
+
+            # Add easily-overridable rpath string at the end
+            out.write('*link_libgcc_rpath:\n')
+            if 'platform=darwin' in self.spec:
+                # macOS linker requires separate rpath commands
+                out.write(' '.join('-rpath ' + lib for lib in rpath_libdirs))
+            else:
+                # linux linker uses colon-separated rpath
+                out.write('-rpath ' + ':'.join(rpath_libdirs))
+            out.write('\n')
         set_install_permissions(specs_file)
+        tty.info('Wrote new spec file to {0}'.format(specs_file))
 
     def setup_run_environment(self, env):
         # Search prefix directory for possibly modified compiler names


Test patch: