Repository: spack/spack

Issue:
binary relocation: libtool control files: hardcoded paths to libstdc++.la are ploblematic
Libtool control files can contain hardcoded paths to `libstdc++.la`. This can break builds when packages that contain these hardcoded paths get cached in a build cache, and are then subsequently installed into a target environment where `libstdc++.la` is not in the same location. 

For instance, if I build the dependencies of `hpctoolkit` in an environment where `libstdc++.la` is at
```
/spack-build/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-8.1.0-glpr3bujg7j5hxvesdbis4cjqzwlncth/lib/../lib64/libstdc++.la
``` 

Then the following files end up with this path hardcoded:
```
/spack-build/opt/spack/linux-centos7-x86_64/gcc-8.1.0/berkeley-db-18.1.40-uasyknrjfdzy2xfoswjqn36w7yzovyts/lib/libdb_cxx-18.1.la
/spack-build/opt/spack/linux-centos7-x86_64/gcc-8.1.0/berkeley-db-18.1.40-uasyknrjfdzy2xfoswjqn36w7yzovyts/lib/libdb_stl-18.1.la
/spack-build/opt/spack/linux-centos7-x86_64/gcc-8.1.0/gettext-0.21-md2v74tvbcqvsu56jmlre3p6mnms5tzm/lib/libasprintf.la
/spack-build/opt/spack/linux-centos7-x86_64/gcc-8.1.0/mpich-3.2.1-y2wrnnftiawtynzg4qzugcfqktatnh64/lib/libmpicxx.la
/spack-build/opt/spack/linux-centos7-x86_64/gcc-8.1.0/xerces-c-3.2.2-ba55khya5d3b6526s2lpzxadl24figaq/lib/libxerces-c.la
```

If I then cache all the dependencies of `hpctoolkit`, and subsequently install those dependencies into an environment where `gcc@8.1.0` is installed to `/packages/gcc/8.1`,  this is what happens when you try to build `hpctoolkit`:

```
==> Installing hpctoolkit
==> hpctoolkit: Executing phase: 'autoreconf'
==> hpctoolkit: Executing phase: 'configure'
==> hpctoolkit: Executing phase: 'build'
==> Error: ProcessError: Command exited with status 2:
    'make' '-j16'

5 errors found in build log:
...
     1250    /bin/sh ../../../libtool  --tag=CXX   --mode=link /spack-test/lib/spack/env/gcc/g++ -g -O3 -Wall  -faligned-new -I/tmp/eugeneswalker/sp
             ack-stage/spack-stage-hpctoolkit-2020.08.03-h7pd5znfeg25vx5d7pijtvkrmqqnljrr/spack-src/src -I/tmp/eugeneswalker/spack-stage/spack-stage-hpctoolki
             t-2020.08.03-h7pd5znfeg25vx5d7pijtvkrmqqnljrr/spack-src/src -I/spack-test/opt/spack/linux-centos7-x86_64/gcc-8.1.0/binutils-2.33.1
             -qmxhsmfvocs5u4qfad3prpixgjtjocgo/include -I/spack-test/opt/spack/linux-centos7-x86_64/gcc-8.1.0/xerces-c-3.2.2-ba55khya5d3b6526s2
             lpzxadl24figaq/include  -g -O3 -Wall  -faligned-new -L/spack-test/opt/spack/linux-centos7-x86_64/gcc-8.1.0/xerces-c-3.2.2-ba55khya
             5d3b6526s2lpzxadl24figaq/lib -L/spack-test/opt/spack/linux-centos7-x86_64/gcc-8.1.0/xz-5.2.5-fqzrczlylvl333jhjlasdsc5n2ydugxo/lib
             -llzma  -o hpcprof-bin hpcprof_bin-main.o hpcprof_bin-Args.o ../../../src/lib/analysis/libHPCanalysis.la ../../../src/lib/banal/libHPCbanal_
             simple.la ../../../src/lib/profxml/libHPCprofxml.la ../../../src/lib/prof/libHPCprof.la ../../../src/lib/binutils/libHPCbinutils.la ../../..
             /src/lib/prof-lean/libHPCprof-lean.la ../../../src/lib/isa/libHPCisa.la -L/spack-test/opt/spack/linux-centos7-x86_64/gcc-8.1.0/int
             el-xed-2019.03.01-ivkiqjbbhifszwlm3ncetcnztjqr5gjy/lib -lxed ../../../src/lib/xml/libHPCxml.la ../../../src/lib/support/libHPCsupport.la ../
             ../../src/lib/support-lean/libHPCsupport-lean.la -L/spack-test/opt/spack/linux-centos7-x86_64/gcc-8.1.0/elfutils-0.180-s4ib3ctvhay
             jk7l3r2diozomettjzdwj/lib -ldw -lelf /spack-test/opt/spack/linux-centos7-x86_64/gcc-8.1.0/xz-5.2.5-fqzrczlylvl333jhjlasdsc5n2ydugx
             o/lib/liblzma.a -lxerces-c -lxerces-c /spack-test/opt/spack/linux-centos7-x86_64/gcc-8.1.0/binutils-2.33.1-qmxhsmfvocs5u4qfad3prpi
             xgjtjocgo/lib/libbfd.a /spack-test/opt/spack/linux-centos7-x86_64/gcc-8.1.0/binutils-2.33.1-qmxhsmfvocs5u4qfad3prpixgjtjocgo/lib64
             /libiberty.a -ldl /spack-test/opt/spack/linux-centos7-x86_64/gcc-8.1.0/zlib-1.2.11-sh7mgsynw6uq2lkaqsv4ty5zerjfu7ww/lib/libz.a -lm
  >> 1251    libtool:   error: cannot find the library '/spack-test/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-8.1.0-glpr3bujg7j5hxvesdbis4cj
             qzwlncth/lib/../lib64/libstdc++.la' or unhandled argument '/spack-test/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-8.1.0-glpr3buj
             g7j5hxvesdbis4cjqzwlncth/lib/../lib64/libstdc++.la'
  >> 1252    make[3]: *** [hpcprof-bin] Error 1
     1253    make[3]: Leaving directory `/tmp/eugeneswalker/spack-stage/spack-stage-hpctoolkit-2020.08.03-h7pd5znfeg25vx5d7pijtvkrmqqnljrr/spack-src/src/tool/
             hpcprof'
  >> 1254    make[2]: *** [all-recursive] Error 1
     1255    make[2]: Leaving directory `/tmp/eugeneswalker/spack-stage/spack-stage-hpctoolkit-2020.08.03-h7pd5znfeg25vx5d7pijtvkrmqqnljrr/spack-src/src/tool'
  >> 1256    make[1]: *** [all-recursive] Error 1
     1257    make[1]: Leaving directory `/tmp/eugeneswalker/spack-stage/spack-stage-hpctoolkit-2020.08.03-h7pd5znfeg25vx5d7pijtvkrmqqnljrr/spack-src/src'
  >> 1258    make: *** [all-recursive] Error 1
...
```

Notice in the error output that although the Spack `install_tree` was re-written, where the original install tree of `/spack-build/opt/spack` was replaced with `/spack-test/opt/spack`, you still have these paths:

```
  >> 1251    libtool:   error: cannot find the library '/spack-test/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-8.1.0-glpr3bujg7j5hxvesdbis4cj
             qzwlncth/lib/../lib64/libstdc++.la' or unhandled argument '/spack-test/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-8.1.0-glpr3buj
             g7j5hxvesdbis4cjqzwlncth/lib/../lib64/libstdc++.la'
```

@gartung @alalazo @tgamblin @becker33 @scottwittenburg 


Discussion:
Confirmed that from a clean checkout of Spack :
```console
$ spack install gcc
[ ... ]
$ spack compiler find $(spack location -i gcc)
$ spack install berkeley-db
```
and inspecting the file:
```
$(spack location -i berkeley-db %gcc@10.2.0)/lib/libdb_stl-18.1.la
```
one can find hard-coded compiler paths. Creating a tarball and relocating to a different place has the effects reported in the issue. For the records, the version I am currently using is:

* **Spack:** 0.15.4-1022-80fe51046
* **Python:** 3.6.9
* **Platform:** linux-ubuntu18.04-broadwell

I have been looking into the simple strategy to:
1. Mark compiler provided *.la files with a placeholder when we build a tarball
2. Replace the placeholder with the appropriate path on relocation

but this is getting more complicated than I initially thought due to the different layout that system installed compilers and Spack installed compilers have, and the fact that they might or might not have said *.la files. For instance this is what a Spack installed GCC 10 looks like:

<details>
 <summary>Spack installed GCC 10.2.0</summary>

```console
$ tree -L 4 /home/culpo/production/spack/store/linux-ubuntu18.04-broadwell/gcc-10.1.0/gcc-10.2.0-dl56nz6vuan7v5rdtylkvhki6fjlyl7c/{lib,lib64}
/home/culpo/production/spack/store/linux-ubuntu18.04-broadwell/gcc-10.1.0/gcc-10.2.0-dl56nz6vuan7v5rdtylkvhki6fjlyl7c/lib
└── gcc
    └── x86_64-pc-linux-gnu
        └── 10.2.0
            ├── crtbegin.o
            ├── crtbeginS.o
            ├── crtbeginT.o
            ├── crtend.o
            ├── crtendS.o
            ├── crtfastmath.o
            ├── crtprec32.o
            ├── crtprec64.o
            ├── crtprec80.o
            ├── finclude
            ├── include
            ├── include-fixed
            ├── install-tools
            ├── libcaf_single.a
            ├── libcaf_single.la
            ├── libgcc.a
            ├── libgcc_eh.a
            ├── libgcov.a
            ├── plugin
            └── specs
/home/culpo/production/spack/store/linux-ubuntu18.04-broadwell/gcc-10.1.0/gcc-10.2.0-dl56nz6vuan7v5rdtylkvhki6fjlyl7c/lib64
├── libasan.a
├── libasan.la
├── libasan_preinit.o
├── libasan.so -> libasan.so.6.0.0
├── libasan.so.6 -> libasan.so.6.0.0
├── libasan.so.6.0.0
├── libatomic.a
├── libatomic.la
├── libatomic.so -> libatomic.so.1.2.0
├── libatomic.so.1 -> libatomic.so.1.2.0
├── libatomic.so.1.2.0
├── libcc1.la
├── libcc1.so -> libcc1.so.0.0.0
├── libcc1.so.0 -> libcc1.so.0.0.0
├── libcc1.so.0.0.0
├── libgcc_s.so
├── libgcc_s.so.1
├── libgfortran.a
├── libgfortran.la
├── libgfortran.so -> libgfortran.so.5.0.0
├── libgfortran.so.5 -> libgfortran.so.5.0.0
├── libgfortran.so.5.0.0
├── libgfortran.spec
├── libgomp.a
├── libgomp.la
├── libgomp.so -> libgomp.so.1.0.0
├── libgomp.so.1 -> libgomp.so.1.0.0
├── libgomp.so.1.0.0
├── libgomp.spec
├── libitm.a
├── libitm.la
├── libitm.so -> libitm.so.1.0.0
├── libitm.so.1 -> libitm.so.1.0.0
├── libitm.so.1.0.0
├── libitm.spec
├── liblsan.a
├── liblsan.la
├── liblsan_preinit.o
├── liblsan.so -> liblsan.so.0.0.0
├── liblsan.so.0 -> liblsan.so.0.0.0
├── liblsan.so.0.0.0
├── libquadmath.a
├── libquadmath.la
├── libquadmath.so -> libquadmath.so.0.0.0
├── libquadmath.so.0 -> libquadmath.so.0.0.0
├── libquadmath.so.0.0.0
├── libsanitizer.spec
├── libssp.a
├── libssp.la
├── libssp_nonshared.a
├── libssp_nonshared.la
├── libssp.so -> libssp.so.0.0.0
├── libssp.so.0 -> libssp.so.0.0.0
├── libssp.so.0.0.0
├── libstdc++.a
├── libstdc++fs.a
├── libstdc++fs.la
├── libstdc++.la
├── libstdc++.so -> libstdc++.so.6.0.28
├── libstdc++.so.6 -> libstdc++.so.6.0.28
├── libstdc++.so.6.0.28
├── libstdc++.so.6.0.28-gdb.py
├── libsupc++.a
├── libsupc++.la
├── libtsan.a
├── libtsan.la
├── libtsan_preinit.o
├── libtsan.so -> libtsan.so.0.0.0
├── libtsan.so.0 -> libtsan.so.0.0.0
├── libtsan.so.0.0.0
├── libubsan.a
├── libubsan.la
├── libubsan.so -> libubsan.so.1.0.0
├── libubsan.so.1 -> libubsan.so.1.0.0
└── libubsan.so.1.0.0

8 directories, 90 files
```

</details>

while a system installed GCC is laid out like:
<details>
<summary>System installed GCC 10.1.0 </summary>

```console
$ ls  /usr/lib/gcc/x86_64-linux-gnu/10/
cc1          crtfastmath.o      finclude           libcaf_single.a  libgfortran.spec  liblsan_preinit.o       libssp_nonshared.a  libubsan.so
cc1plus      crtoffloadbegin.o  include            libcc1.so        libgomp.a         liblsan.so              libstdc++.a         lto1
collect2     crtoffloadend.o    libasan.a          libgcc.a         libgomp.so        liblto_plugin.so        libstdc++fs.a       lto-wrapper
crtbegin.o   crtoffloadtable.o  libasan_preinit.o  libgcc_eh.a      libgomp.spec      liblto_plugin.so.0      libstdc++.so        plugin
crtbeginS.o  crtprec32.o        libasan.so         libgcc_s.so      libitm.a          liblto_plugin.so.0.0.0  libsupc++.a
crtbeginT.o  crtprec64.o        libatomic.a        libgcov.a        libitm.so         libquadmath.a           libtsan.a
crtend.o     crtprec80.o        libatomic.so       libgfortran.a    libitm.spec       libquadmath.so          libtsan.so
crtendS.o    f951               libbacktrace.a     libgfortran.so   liblsan.a         libsanitizer.spec       libubsan.a
```

</details>

and has no *.la files at all. This might be an issue for packages whose tarball has been generated with a system compiler and get relocated to use a Spack installed compiler at the same version, or vice versa.
A possible variation of the solution above:

1. Mark compiler provided *.la files with a placeholder when we build a tarball
2. Replace the placeholder with the appropriate path on relocation OR remove the placeholder if the target compiler doesn't have the required .la file

This leaves in principle a case uncovered, i.e. the case where we build tarballs from a compiler that doesn't have .la files and we relocate to a compiler that has them. I'm currently trying to check if I can reproduce the entire `hpctoolkit` case, and what happens if I use buildcaches in this situation.
It seems that most (all?) distributions don't package the .la files, at least for GCC. I just tried to:
1. Build GCC with Spack, delete all the .la files manually, register that GCC as a valid compiler
2. Build all the dependencies of `hpctoolkit` and package them with Spack (they don't have any reference to libstdc++ etc. anymore in their .la files)
3. Install the binaries created at point 2 in another instance of Spack, with the same GCC in a different folder

and I was  able to install `hpctoolkit` correctly. Looking at what CentOS does [here](https://git.stg.centos.org/rpms/gcc/blob/master/f/gcc.spec#_1303):
```
find %{buildroot} -name \*.la | xargs rm -f
```
I would be for solving this particular issue by just doing the same and remove `.la` files after having installed GCC. I'll think a bit if this should be generalized and pushed to `AutotoolsPackage`.
> I would be for solving this particular issue by just doing the same and remove `.la` files after having installed GCC. I'll think a bit if this should be generalized and pushed to `AutotoolsPackage`.

I like this solution.



Resolving patch:
diff --git a/lib/spack/spack/build_systems/autotools.py b/lib/spack/spack/build_systems/autotools.py
--- a/lib/spack/spack/build_systems/autotools.py
+++ b/lib/spack/spack/build_systems/autotools.py
@@ -2,8 +2,6 @@
 # Spack Project Developers. See the top-level COPYRIGHT file for details.
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
-
-
 import inspect
 import os
 import os.path
@@ -80,10 +78,25 @@ class AutotoolsPackage(PackageBase):
     #: Options to be passed to autoreconf when using the default implementation
     autoreconf_extra_args = []
 
+    #: If False deletes all the .la files in the prefix folder
+    #: after the installation. If True instead it installs them.
+    install_libtool_archives = False
+
+    @property
+    def _removed_la_files_log(self):
+        """File containing the list of remove libtool archives"""
+        build_dir = self.build_directory
+        if not os.path.isabs(self.build_directory):
+            build_dir = os.path.join(self.stage.path, build_dir)
+        return os.path.join(build_dir, 'removed_la_files.txt')
+
     @property
     def archive_files(self):
         """Files to archive for packages based on autotools"""
-        return [os.path.join(self.build_directory, 'config.log')]
+        files = [os.path.join(self.build_directory, 'config.log')]
+        if not self.install_libtool_archives:
+            files.append(self._removed_la_files_log)
+        return files
 
     @run_after('autoreconf')
     def _do_patch_config_files(self):
@@ -524,3 +537,19 @@ def installcheck(self):
 
     # Check that self.prefix is there after installation
     run_after('install')(PackageBase.sanity_check_prefix)
+
+    @run_after('install')
+    def remove_libtool_archives(self):
+        """Remove all .la files in prefix sub-folders if the package sets
+        ``install_libtool_archives`` to be False.
+        """
+        # If .la files are to be installed there's nothing to do
+        if self.install_libtool_archives:
+            return
+
+        # Remove the files and create a log of what was removed
+        libtool_files = fs.find(str(self.prefix), '*.la', recursive=True)
+        with fs.safe_remove(*libtool_files):
+            fs.mkdirp(os.path.dirname(self._removed_la_files_log))
+            with open(self._removed_la_files_log, mode='w') as f:
+                f.write('\n'.join(libtool_files))
diff --git a/var/spack/repos/builtin.mock/packages/libtool-deletion/package.py b/var/spack/repos/builtin.mock/packages/libtool-deletion/package.py
new file mode 100644
--- /dev/null
+++ b/var/spack/repos/builtin.mock/packages/libtool-deletion/package.py
@@ -0,0 +1,35 @@
+# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+import os.path
+
+
+class LibtoolDeletion(AutotoolsPackage):
+    """Mock AutotoolsPackage to check proper deletion
+    of libtool archives.
+    """
+    homepage = "https://www.gnu.org/software/make/"
+    url = "http://www.example.com/libtool-deletion-1.0.tar.gz"
+    version('4.2.1', sha256='e40b8f018c1da64edd1cc9a6fce5fa63b2e707e404e20cad91fbae337c98a5b7')
+
+    def do_stage(self):
+        mkdirp(self.stage.source_path)
+
+    def autoreconf(self, spec, prefix):
+        mkdirp(os.path.dirname(self.configure_abs_path))
+        touch(self.configure_abs_path)
+
+    def configure(self, spec, prefix):
+        pass
+
+    def build(self, spec, prefix):
+        pass
+
+    def install(self, spec, prefix):
+        mkdirp(os.path.dirname(self.libtool_archive_file))
+        touch(self.libtool_archive_file)
+
+    @property
+    def libtool_archive_file(self):
+        return os.path.join(str(self.prefix.lib), 'libfoo.la')


Test patch:
diff --git a/lib/spack/spack/test/build_systems.py b/lib/spack/spack/test/build_systems.py
--- a/lib/spack/spack/test/build_systems.py
+++ b/lib/spack/spack/test/build_systems.py
@@ -7,8 +7,8 @@
 import os
 import pytest
 
+import llnl.util.filesystem as fs
 import spack.repo
-from llnl.util.filesystem import working_dir
 from spack.build_environment import get_std_cmake_args, setup_package
 from spack.spec import Spec
 from spack.util.executable import which
@@ -30,7 +30,7 @@ def test_affirmative_make_check(directory, config, mock_packages, working_env):
     pkg = spack.repo.get(s)
     setup_package(pkg, False)
 
-    with working_dir(directory):
+    with fs.working_dir(directory):
         assert pkg._has_make_target('check')
 
         pkg._if_make_target_execute('check')
@@ -50,7 +50,7 @@ def test_negative_make_check(directory, config, mock_packages, working_env):
     pkg = spack.repo.get(s)
     setup_package(pkg, False)
 
-    with working_dir(directory):
+    with fs.working_dir(directory):
         assert not pkg._has_make_target('check')
 
         pkg._if_make_target_execute('check')
@@ -71,7 +71,7 @@ def test_affirmative_ninja_check(
     pkg = spack.repo.get(s)
     setup_package(pkg, False)
 
-    with working_dir(directory):
+    with fs.working_dir(directory):
         assert pkg._has_ninja_target('check')
 
         pkg._if_ninja_target_execute('check')
@@ -96,7 +96,7 @@ def test_negative_ninja_check(directory, config, mock_packages, working_env):
     pkg = spack.repo.get(s)
     setup_package(pkg, False)
 
-    with working_dir(directory):
+    with fs.working_dir(directory):
         assert not pkg._has_ninja_target('check')
 
         pkg._if_ninja_target_execute('check')
@@ -182,6 +182,36 @@ def test_none_is_allowed(self):
         assert '--without-baz' in options
         assert '--no-fee' in options
 
+    def test_libtool_archive_files_are_deleted_by_default(
+            self, mutable_database
+    ):
+        # Install a package that creates a mock libtool archive
+        s = spack.spec.Spec('libtool-deletion')
+        s.concretize()
+        s.package.do_install(explicit=True)
+
+        # Assert the libtool archive is not there and we have
+        # a log of removed files
+        assert not os.path.exists(s.package.libtool_archive_file)
+        search_directory = os.path.join(s.prefix, '.spack')
+        libtool_deletion_log = fs.find(
+            search_directory, 'removed_la_files.txt', recursive=True
+        )
+        assert libtool_deletion_log
+
+    def test_libtool_archive_files_might_be_installed_on_demand(
+            self, mutable_database, monkeypatch
+    ):
+        # Install a package that creates a mock libtool archive,
+        # patch its package to preserve the installation
+        s = spack.spec.Spec('libtool-deletion')
+        s.concretize()
+        monkeypatch.setattr(s.package, 'install_libtool_archives', True)
+        s.package.do_install(explicit=True)
+
+        # Assert libtool archives are installed
+        assert os.path.exists(s.package.libtool_archive_file)
+
 
 @pytest.mark.usefixtures('config', 'mock_packages')
 class TestCMakePackage(object):