Repository: spack/spack

Issue:
Ship latest `config.{guess,sub}` with Spack
### Summary

In autotools packages we have some logic to replace outdated `config.{guess,sub}` files with recent ones, so that `./configure` works on architectures unknown at the point where the configure script was generated.

This logic requires the package to depend on `automake`, so it pulls these files from there.


### Rationale


Unfortunately automake pulls in a lot of dependencies:

![Screenshot from 2021-09-03 17-09-02](https://user-images.githubusercontent.com/194764/132028539-504777ea-1489-44b6-8b7e-3684a87ca91f.png)

and the whole point of autotools is not to have automake as a dependency.

### Description

We can do 2 things:

- Put a recent `config.guess` and `config.sub` in `spack/share/spack` and pull the files from there.
- Create a Spack package for https://git.savannah.gnu.org/cgit/config.git/tree/, and make packages depend on this as a build dependency, and have our autotools build system use that instead. Unfortunately, it seems like they don't provide proper versioned tarballs for this (but I might be looking wrong?)

This reduces build times for autotools with ./configure scripts on new platforms.

### Additional information

_No response_

### General information

- [X] I have run `spack --version` and reported the version of Spack
- [X] I have searched the issues of this repo and believe this is not a duplicate


Discussion:
~@tgamblin mentioned that in some cases an exotic machine may provide a system `automake` which provides customized `config.{guess,sub}`, and the current approach allows one to mark `automake` as external, so that spack picks up these custom files (iiuc).~

~Shipping a recent config.guess/config.sub with Spack would not really support that case :(~

~Unless we make the paths to `config.guess` / `config.sub` configurable, but well... that's awkward too.~

Okay, I got this wrong as was pointed out by @joennlae: spack actually explicitly looks in system paths for config.sub and config.guess, which is very unexpected to me, since we try so hard to NOT rely on system paths at all:

https://github.com/spack/spack/blob/c309adb4b3a3d278fef89b32e6ad6fe677748ecb/lib/spack/spack/build_systems/autotools.py#L154

Instead, can't we ship our own versioned package with config.guess / config.sub files, simply redistributing the ones from upstream in a proper package instead of a git repo. Then *if* a user really needs a system version of config.guess / config.sub, they can always mark this package as external, and spack will find the config files from the external prefix = system.
I can tell you how Debian-based distros do it:

They have a package `autotools-dev` which provides 
```
/usr/share/misc/config.guess
/usr/share/misc/config.sub
```
Until some time ago, it also had the script to update these files in the source tree of a package.
This script has since been moved to the package `debhelper`: `/usr/bin/dh_update_autotools_config`
```perl
for my $basename (qw(config.guess config.sub)) {
	my $new_version = "/usr/share/misc/$basename";
	open(my $fd, '-|', 'find', '-mindepth', '1',
```
I don't know, but maybe you can make it so that for destination architecture aarch64 and ppc64le,
the calss AutotoolsPackage has a depends on the new `autotools-dev`-like package with the `config.{sub,guess}`

The automake tarball has the latest released `config.{sub,guess}` files, and to not pull the entire automake dependency tree into the build, we could add a BaseClass to `var/spack/repos/builtin/packages/automake/package.py`.

The BaseClass(derived from GNUMirrorPackage) would have the homepage, gnu_mirror_path and the versions, and "class Automake" derives from it, adds the depends_on lines and all the other properties and methods.

Then the class autotools-dev (or in this case likely automake-dev, or rather automake-files) package could derive from the BaseClass as well and just provide the `config.{sub,guess}` directly without adding any dependencies.
Yeah, it's highly similar then. So our `autotools-dev` would be `gnuconfig` https://github.com/spack/spack/blob/59a2594d153325e47cfac546212f6dd7e71956c3/var/spack/repos/builtin/packages/gnuconfig/package.py which just pulls sources from a mirror of the gnu config.git repo from tags I've added manually when they bump the timestamp.

The `dh_update_autotools_config` is written in python in `lib/spack/spack/build_systems/autotools.py`.

And I don't think it hurts to add `depends_on('gnuconfig', when='target=ppc64le:', type='build')` unconditionally (and the same for aarch64:).

Resolving patch:
diff --git a/lib/spack/spack/build_systems/autotools.py b/lib/spack/spack/build_systems/autotools.py
--- a/lib/spack/spack/build_systems/autotools.py
+++ b/lib/spack/spack/build_systems/autotools.py
@@ -3,7 +3,6 @@
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
 import inspect
-import itertools
 import os
 import os.path
 import stat
@@ -14,6 +13,8 @@
 import llnl.util.tty as tty
 from llnl.util.filesystem import force_remove, working_dir
 
+from spack.build_environment import InstallError
+from spack.directives import depends_on
 from spack.package import PackageBase, run_after, run_before
 from spack.util.executable import Executable
 
@@ -54,9 +55,22 @@ class AutotoolsPackage(PackageBase):
     #: This attribute is used in UI queries that need to know the build
     #: system base class
     build_system_class = 'AutotoolsPackage'
-    #: Whether or not to update ``config.guess`` and ``config.sub`` on old
-    #: architectures
-    patch_config_files = True
+
+    @property
+    def patch_config_files(self):
+        """
+        Whether or not to update old ``config.guess`` and ``config.sub`` files
+        distributed with the tarball. This currently only applies to ``ppc64le:``
+        and ``aarch64:`` target architectures. The substitutes are taken from the
+        ``gnuconfig`` package, which is automatically added as a build dependency
+        for these architectures. In case system versions of these config files are
+        required, the ``gnuconfig`` package can be marked external with a prefix
+        pointing to the directory containing the system ``config.guess`` and
+        ``config.sub`` files.
+        """
+        return (self.spec.satisfies('target=ppc64le:')
+                or self.spec.satisfies('target=aarch64:'))
+
     #: Whether or not to update ``libtool``
     #: (currently only for Arm/Clang/Fujitsu compilers)
     patch_libtool = True
@@ -83,6 +97,9 @@ class AutotoolsPackage(PackageBase):
     #: after the installation. If True instead it installs them.
     install_libtool_archives = False
 
+    depends_on('gnuconfig', type='build', when='target=ppc64le:')
+    depends_on('gnuconfig', type='build', when='target=aarch64:')
+
     @property
     def _removed_la_files_log(self):
         """File containing the list of remove libtool archives"""
@@ -106,10 +123,7 @@ def _do_patch_config_files(self):
         In particular, config.guess fails for PPC64LE for version prior
         to a 2013-06-10 build date (automake 1.13.4) and for ARM (aarch64).
         """
-        if not self.patch_config_files or (
-                not self.spec.satisfies('target=ppc64le:') and
-                not self.spec.satisfies('target=aarch64:')
-        ):
+        if not self.patch_config_files:
             return
 
         # TODO: Expand this to select the 'config.sub'-compatible architecture
@@ -138,39 +152,69 @@ def runs_ok(script_abs_path):
 
             return True
 
-        # Compute the list of files that needs to be patched
-        search_dir = self.stage.path
-        to_be_patched = fs.find(
-            search_dir, files=['config.sub', 'config.guess'], recursive=True
-        )
+        # Get the list of files that needs to be patched
+        to_be_patched = fs.find(self.stage.path, files=['config.sub', 'config.guess'])
         to_be_patched = [f for f in to_be_patched if not runs_ok(f)]
 
         # If there are no files to be patched, return early
         if not to_be_patched:
             return
 
-        # Directories where to search for files to be copied
-        # over the failing ones
-        good_file_dirs = ['/usr/share']
-        if 'automake' in self.spec:
-            good_file_dirs.insert(0, self.spec['automake'].prefix)
+        # Otherwise, require `gnuconfig` to be a build dependency
+        self._require_build_deps(
+            pkgs=['gnuconfig'],
+            spec=self.spec,
+            err="Cannot patch config files")
 
-        # List of files to be found in the directories above
+        # Get the config files we need to patch (config.sub / config.guess).
         to_be_found = list(set(os.path.basename(f) for f in to_be_patched))
+        gnuconfig = self.spec['gnuconfig']
+        gnuconfig_dir = gnuconfig.prefix
+
+        # An external gnuconfig may not not have a prefix.
+        if gnuconfig_dir is None:
+            raise InstallError("Spack could not find substitutes for GNU config "
+                               "files because no prefix is available for the "
+                               "`gnuconfig` package. Make sure you set a prefix "
+                               "path instead of modules for external `gnuconfig`.")
+
+        candidates = fs.find(gnuconfig_dir, files=to_be_found, recursive=False)
+
+        # For external packages the user may have specified an incorrect prefix.
+        # otherwise the installation is just corrupt.
+        if not candidates:
+            msg = ("Spack could not find `config.guess` and `config.sub` "
+                   "files in the `gnuconfig` prefix `{0}`. This means the "
+                   "`gnuconfig` package is broken").format(gnuconfig_dir)
+            if gnuconfig.external:
+                msg += (" or the `gnuconfig` package prefix is misconfigured as"
+                        " an external package")
+            raise InstallError(msg)
+
+        # Filter working substitutes
+        candidates = [f for f in candidates if runs_ok(f)]
         substitutes = {}
-        for directory in good_file_dirs:
-            candidates = fs.find(directory, files=to_be_found, recursive=True)
-            candidates = [f for f in candidates if runs_ok(f)]
-            for name, good_files in itertools.groupby(
-                    candidates, key=os.path.basename
-            ):
-                substitutes[name] = next(good_files)
-                to_be_found.remove(name)
+        for candidate in candidates:
+            config_file = os.path.basename(candidate)
+            substitutes[config_file] = candidate
+            to_be_found.remove(config_file)
 
         # Check that we found everything we needed
         if to_be_found:
-            msg = 'Failed to find suitable substitutes for {0}'
-            raise RuntimeError(msg.format(', '.join(to_be_found)))
+            msg = """\
+Spack could not find working replacements for the following autotools config
+files: {0}.
+
+To resolve this problem, please try the following:
+1. Try to rebuild with `patch_config_files = False` in the package `{1}`, to
+   rule out that Spack tries to replace config files not used by the build.
+2. Verify that the `gnuconfig` package is up-to-date.
+3. On some systems you need to use system-provided `config.guess` and `config.sub`
+   files. In this case, mark `gnuconfig` as an non-buildable external package,
+   and set the prefix to the directory containing the `config.guess` and
+   `config.sub` files.
+"""
+            raise InstallError(msg.format(', '.join(to_be_found), self.name))
 
         # Copy the good files over the bad ones
         for abs_path in to_be_patched:
@@ -252,30 +296,40 @@ def delete_configure_to_force_update(self):
         if self.force_autoreconf:
             force_remove(self.configure_abs_path)
 
-    def _autoreconf_warning(self, spec, missing):
-        msg = ("Cannot generate configure: missing dependencies {0}.\n\nPlease add "
-               "the following lines to the package:\n\n".format(", ".join(missing)))
+    def _require_build_deps(self, pkgs, spec, err):
+        """Require `pkgs` to be direct build dependencies of `spec`. Raises a
+        RuntimeError with a helpful error messages when any dep is missing."""
 
-        for dep in missing:
+        build_deps = [d.name for d in spec.dependencies(deptype='build')]
+        missing_deps = [x for x in pkgs if x not in build_deps]
+
+        if not missing_deps:
+            return
+
+        # Raise an exception on missing deps.
+        msg = ("{0}: missing dependencies: {1}.\n\nPlease add "
+               "the following lines to the package:\n\n"
+               .format(err, ", ".join(missing_deps)))
+
+        for dep in missing_deps:
             msg += ("    depends_on('{0}', type='build', when='@{1}')\n"
                     .format(dep, spec.version))
 
         msg += "\nUpdate the version (when='@{0}') as needed.".format(spec.version)
-
-        return msg
+        raise RuntimeError(msg)
 
     def autoreconf(self, spec, prefix):
         """Not needed usually, configure should be already there"""
+
         # If configure exists nothing needs to be done
         if os.path.exists(self.configure_abs_path):
             return
-        # Else try to regenerate it
-        needed_dependencies = ['autoconf', 'automake', 'libtool']
-        build_deps = [d.name for d in spec.dependencies(deptype='build')]
-        missing = [x for x in needed_dependencies if x not in build_deps]
 
-        if missing:
-            raise RuntimeError(self._autoreconf_warning(spec, missing))
+        # Else try to regenerate it, which reuquires a few build dependencies
+        self._require_build_deps(
+            pkgs=['autoconf', 'automake', 'libtool'],
+            spec=spec,
+            err="Cannot generate configure")
 
         tty.msg('Configure script not found: trying to generate it')
         tty.warn('*********************************************************')
diff --git a/var/spack/repos/builtin.mock/packages/autotools-config-replacement/package.py b/var/spack/repos/builtin.mock/packages/autotools-config-replacement/package.py
new file mode 100644
--- /dev/null
+++ b/var/spack/repos/builtin.mock/packages/autotools-config-replacement/package.py
@@ -0,0 +1,88 @@
+# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+import os
+
+from spack import *
+
+
+class AutotoolsConfigReplacement(AutotoolsPackage):
+    """
+    This package features broken and working config.sub and config.guess files,
+    that should be replaced by the ones provided by gnuconfig. It allows testing
+    with / without patches and with / without substitutes available.
+    """
+
+    has_code = False
+
+    version('1.0.0')
+    variant('patch_config_files', default=False)
+    variant('gnuconfig', default=False)
+
+    depends_on('gnuconfig', type='build', when='+gnuconfig')
+
+    @property
+    def patch_config_files(self):
+        return self.spec.satisfies('+patch_config_files')
+
+    def autoreconf(self, spec, prefix):
+        pass
+
+    def configure(self, spec, prefix):
+        pass
+
+    def build(self, spec, prefix):
+        pass
+
+    def install(self, spec, prefix):
+        broken = os.path.join(self.stage.source_path, 'broken')
+        working = os.path.join(self.stage.source_path, 'working')
+        install_tree(broken, self.prefix.broken)
+        install_tree(working, self.prefix.working)
+
+    @run_before('autoreconf')
+    def create_the_package_sources(self):
+        # Creates the following file structure:
+        # ./broken/config.sub    -- not executable
+        # ./broken/config.guess  -- exectuable & exit code 1
+        # ./working/config.sub   -- executable & exit code 0
+        # ./working/config.guess -- executable & exit code 0
+        # Automatic config helper script substitution should replace the two
+        # broken scripts with those from the gnuconfig package.
+
+        broken = os.path.join(self.stage.source_path, 'broken')
+        working = os.path.join(self.stage.source_path, 'working')
+
+        mkdirp(broken)
+        mkdirp(working)
+
+        # a configure script is required
+        configure_script = join_path(self.stage.source_path, 'configure')
+        with open(configure_script, 'w') as f:
+            f.write("#!/bin/sh\nexit 0")
+        os.chmod(configure_script, 0o775)
+
+        # broken config.sub (not executable)
+        broken_config_sub = join_path(broken, 'config.sub')
+        with open(broken_config_sub, 'w') as f:
+            f.write("#!/bin/sh\nexit 0")
+
+        # broken config.guess (exectuable but with error return code)
+        broken_config_guess = join_path(broken, 'config.guess')
+        with open(broken_config_guess, 'w') as f:
+            f.write("#!/bin/sh\nexit 1")
+        os.chmod(broken_config_guess, 0o775)
+
+        # working config.sub
+        working_config_sub = join_path(working, 'config.sub')
+        with open(working_config_sub, 'w') as f:
+            f.write("#!/bin/sh\nexit 0")
+        os.chmod(working_config_sub, 0o775)
+
+        # working config.guess
+        working_config_guess = join_path(working, 'config.guess')
+        with open(working_config_guess, 'w') as f:
+            f.write("#!/bin/sh\nexit 0")
+        os.chmod(working_config_guess, 0o775)
diff --git a/var/spack/repos/builtin.mock/packages/gnuconfig/package.py b/var/spack/repos/builtin.mock/packages/gnuconfig/package.py
new file mode 100644
--- /dev/null
+++ b/var/spack/repos/builtin.mock/packages/gnuconfig/package.py
@@ -0,0 +1,35 @@
+# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+import os
+
+from spack import *
+
+
+class Gnuconfig(Package):
+    """
+    The GNU config.guess and config.sub scripts versioned by timestamp.
+    This package can be used as a build dependency for autotools packages that
+    ship a tarball with outdated config.guess and config.sub files.
+    """
+
+    has_code = False
+
+    version('2021-08-14')
+
+    def install(self, spec, prefix):
+        config_sub = join_path(prefix, 'config.sub')
+        config_guess = join_path(prefix, 'config.guess')
+
+        # Create files
+        with open(config_sub, 'w') as f:
+            f.write("#!/bin/sh\necho gnuconfig version of config.sub")
+
+        with open(config_guess, 'w') as f:
+            f.write("#!/bin/sh\necho gnuconfig version of config.guess")
+
+        # Make executable
+        os.chmod(config_sub, 0o775)
+        os.chmod(config_guess, 0o775)


Test patch:
diff --git a/lib/spack/spack/test/build_systems.py b/lib/spack/spack/test/build_systems.py
--- a/lib/spack/spack/test/build_systems.py
+++ b/lib/spack/spack/test/build_systems.py
@@ -10,8 +10,10 @@
 
 import llnl.util.filesystem as fs
 
+import spack.architecture
+import spack.environment
 import spack.repo
-from spack.build_environment import get_std_cmake_args, setup_package
+from spack.build_environment import ChildError, get_std_cmake_args, setup_package
 from spack.spec import Spec
 from spack.util.executable import which
 
@@ -190,7 +192,7 @@ def test_libtool_archive_files_are_deleted_by_default(
             self, mutable_database
     ):
         # Install a package that creates a mock libtool archive
-        s = spack.spec.Spec('libtool-deletion')
+        s = Spec('libtool-deletion')
         s.concretize()
         s.package.do_install(explicit=True)
 
@@ -208,7 +210,7 @@ def test_libtool_archive_files_might_be_installed_on_demand(
     ):
         # Install a package that creates a mock libtool archive,
         # patch its package to preserve the installation
-        s = spack.spec.Spec('libtool-deletion')
+        s = Spec('libtool-deletion')
         s.concretize()
         monkeypatch.setattr(s.package, 'install_libtool_archives', True)
         s.package.do_install(explicit=True)
@@ -216,6 +218,89 @@ def test_libtool_archive_files_might_be_installed_on_demand(
         # Assert libtool archives are installed
         assert os.path.exists(s.package.libtool_archive_file)
 
+    def test_autotools_gnuconfig_replacement(self, mutable_database):
+        """
+        Tests whether only broken config.sub and config.guess are replaced with
+        files from working alternatives from the gnuconfig package.
+        """
+        s = Spec('autotools-config-replacement +patch_config_files +gnuconfig')
+        s.concretize()
+        s.package.do_install()
+
+        with open(os.path.join(s.prefix.broken, 'config.sub')) as f:
+            assert "gnuconfig version of config.sub" in f.read()
+
+        with open(os.path.join(s.prefix.broken, 'config.guess')) as f:
+            assert "gnuconfig version of config.guess" in f.read()
+
+        with open(os.path.join(s.prefix.working, 'config.sub')) as f:
+            assert "gnuconfig version of config.sub" not in f.read()
+
+        with open(os.path.join(s.prefix.working, 'config.guess')) as f:
+            assert "gnuconfig version of config.guess" not in f.read()
+
+    def test_autotools_gnuconfig_replacement_disabled(self, mutable_database):
+        """
+        Tests whether disabling patch_config_files
+        """
+        s = Spec('autotools-config-replacement ~patch_config_files +gnuconfig')
+        s.concretize()
+        s.package.do_install()
+
+        with open(os.path.join(s.prefix.broken, 'config.sub')) as f:
+            assert "gnuconfig version of config.sub" not in f.read()
+
+        with open(os.path.join(s.prefix.broken, 'config.guess')) as f:
+            assert "gnuconfig version of config.guess" not in f.read()
+
+        with open(os.path.join(s.prefix.working, 'config.sub')) as f:
+            assert "gnuconfig version of config.sub" not in f.read()
+
+        with open(os.path.join(s.prefix.working, 'config.guess')) as f:
+            assert "gnuconfig version of config.guess" not in f.read()
+
+    @pytest.mark.disable_clean_stage_check
+    def test_autotools_gnuconfig_replacement_no_gnuconfig(self, mutable_database):
+        """
+        Tests whether a useful error message is shown when patch_config_files is
+        enabled, but gnuconfig is not listed as a direct build dependency.
+        """
+        s = Spec('autotools-config-replacement +patch_config_files ~gnuconfig')
+        s.concretize()
+
+        msg = "Cannot patch config files: missing dependencies: gnuconfig"
+        with pytest.raises(ChildError, match=msg):
+            s.package.do_install()
+
+    @pytest.mark.disable_clean_stage_check
+    def test_broken_external_gnuconfig(self, mutable_database, tmpdir):
+        """
+        Tests whether we get a useful error message when gnuconfig is marked
+        external, but the install prefix is misconfigured and no config.guess
+        and config.sub substitute files are found in the provided prefix.
+        """
+        env_dir = str(tmpdir.ensure('env', dir=True))
+        gnuconfig_dir = str(tmpdir.ensure('gnuconfig', dir=True))  # empty dir
+        with open(os.path.join(env_dir, 'spack.yaml'), 'w') as f:
+            f.write("""\
+spack:
+  specs:
+  - 'autotools-config-replacement +patch_config_files +gnuconfig'
+  packages:
+    gnuconfig:
+      buildable: false
+      externals:
+      - spec: gnuconfig@1.0.0
+        prefix: {0}
+""".format(gnuconfig_dir))
+
+        msg = ("Spack could not find `config.guess`.*misconfigured as an "
+               "external package")
+        with spack.environment.Environment(env_dir) as e:
+            e.concretize()
+            with pytest.raises(ChildError, match=msg):
+                e.install_all()
+
 
 @pytest.mark.usefixtures('config', 'mock_packages')
 class TestCMakePackage(object):