Repository: spack/spack

Issue:
Major slowdown when installing binaries from binary cache due to #18482
Bisected to  #18482 bugfix: fix config merge order for OrderdDicts

My CI script that ran `spack install [spec]` using a binary cache slowed down after #18482 to the point where building from sources is almost equally fast. Previously the CI job ran in 11 minutes (installing ~100 packages), now it's taking > 90 minutes and it is not even finished yet pulling binaries from the binary cache.

Ping @tgamblin.

The CI script builds a docker image as follows (adding a mirror / binary cache, copying an environment file, and then installing):

```docker
# "Install" compilers
COPY "$COMPILER_CONFIG" /opt/spack/etc/spack/compilers.yaml

# Add our custom spack repo from here
COPY ./spack /user_repo

RUN spack repo add --scope site /user_repo

# Set up the binary cache and trust the public part of our signing key
COPY ./ci/spack/public_key.asc ./public_key.asc
RUN spack mirror add --scope site minio [some url] && \
    spack gpg trust ./public_key.asc

# Copy over the environment file
COPY $SPACK_ENVIRONMENT /spack_environment/spack.yaml

# Build dependencies
# 1. Create a spack environment named `ci` from the input spack.yaml file
# 2. Install only the dependencies of this (top level is our package)
RUN spack --color=always env create --without-view ci /spack_environment/spack.yaml

RUN spack --color=always -e ci install --only=dependencies
```


Discussion:
Restarting the CI job to make sure all binaries were in fact cached in the mirror already:

CI job went from 11 minutes to 67 minutes (after #18482) installing roughly 100 small packages inside a docker image

This is too slow to make sense... The blocker seems really to be Python running on 100% cpu. The total download size is < 500MB.
@haampie I think the cause of the slowdown is the `deepcopy` see https://github.com/spack/spack/pull/18482#discussion_r512763660 Still need to check how to improve the situation.
For the record, I'm trying to understand if:
```diff
--- a/lib/spack/spack/config.py
+++ b/lib/spack/spack/config.py
@@ -552,6 +552,10 @@ def get_config(self, section, scope=None):
            }
 
         """
+        return self._get_config_memoized(section, scope)
+
+    @llnl.util.lang.memoized
+    def _get_config_memoized(self, section, scope):
         _validate_section_name(section)

```
is a valid solution to this issue, as it seems it could be from some simple manual tests. For instance on:

* **Spack:** 0.15.4-1674-bb00b1a7c9
* **Python:** 3.8.5
* **Platform:** linux-ubuntu18.04-broadwell

before this patch I have:
```console
$ time spack spec -l hdf5
Input spec
--------------------------------
hdf5

Concretized
--------------------------------
lnm5tdy  hdf5@1.10.7%gcc@10.2.0~cxx~debug~fortran~hl~java+mpi+pic+shared~szip~threadsafe api=none arch=linux-ubuntu18.04-broadwell
aojanih      ^openmpi@3.1.6%gcc@10.2.0~atomics~cuda~cxx~cxx_exceptions+gpfs~java~legacylaunchers~lustre~memchecker~pmi~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=none schedulers=none arch=linux-ubuntu18.04-broadwell
pq4rxuv          ^hwloc@1.11.11%gcc@10.2.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-ubuntu18.04-broadwell
go5mgat              ^libpciaccess@0.16%gcc@10.2.0 arch=linux-ubuntu18.04-broadwell
4swz6du                  ^libtool@2.4.6%gcc@10.2.0 arch=linux-ubuntu18.04-broadwell
fvibkic                      ^m4@1.4.18%gcc@10.2.0+sigsegv patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 arch=linux-ubuntu18.04-broadwell
b6faazt                          ^libsigsegv@2.12%gcc@10.2.0 arch=linux-ubuntu18.04-broadwell
z3getxj                  ^pkgconf@1.7.3%gcc@10.2.0 arch=linux-ubuntu18.04-broadwell
fylewdu                  ^util-macros@1.19.1%gcc@10.2.0 arch=linux-ubuntu18.04-broadwell
o56eham              ^libxml2@2.9.10%gcc@10.2.0~python arch=linux-ubuntu18.04-broadwell
upvkp7k                  ^libiconv@1.16%gcc@10.2.0 arch=linux-ubuntu18.04-broadwell
r5xn4ra                  ^xz@5.2.5%gcc@10.2.0~pic arch=linux-ubuntu18.04-broadwell
azie5vs                  ^zlib@1.2.11%gcc@10.2.0+optimize+pic+shared arch=linux-ubuntu18.04-broadwell
jclx53j              ^numactl@2.0.14%gcc@10.2.0 patches=4e1d78cbbb85de625bad28705e748856033eaafab92a66dffd383a3d7e00cc94 arch=linux-ubuntu18.04-broadwell
q3gkqfy                  ^autoconf@2.69%gcc@10.2.0 arch=linux-ubuntu18.04-broadwell
zseosv7                      ^perl@5.30.3%gcc@10.2.0+cpanm+shared+threads arch=linux-ubuntu18.04-broadwell
hvbjbd4                          ^berkeley-db@18.1.40%gcc@10.2.0 arch=linux-ubuntu18.04-broadwell
wl5k65m                          ^gdbm@1.18.1%gcc@10.2.0 patches=7f569b9287e9e5a4236636eef32ff3428f0462ec17aea359e0ebee07649f1f16 arch=linux-ubuntu18.04-broadwell
ciaqmp5                              ^readline@8.0%gcc@10.2.0 arch=linux-ubuntu18.04-broadwell
vkv6uu4                                  ^ncurses@6.2%gcc@10.2.0~symlinks+termlib arch=linux-ubuntu18.04-broadwell
bx5xku3                  ^automake@1.16.2%gcc@10.2.0 arch=linux-ubuntu18.04-broadwell


real    0m8,717s
user    0m8,220s
sys     0m0,203s
```
and after the patch:
```console
$ time spack spec -l hdf5
Input spec
--------------------------------
hdf5

Concretized
--------------------------------
lnm5tdy  hdf5@1.10.7%gcc@10.2.0~cxx~debug~fortran~hl~java+mpi+pic+shared~szip~threadsafe api=none arch=linux-ubuntu18.04-broadwell
aojanih      ^openmpi@3.1.6%gcc@10.2.0~atomics~cuda~cxx~cxx_exceptions+gpfs~java~legacylaunchers~lustre~memchecker~pmi~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=none schedulers=none arch=linux-ubuntu18.04-broadwell
pq4rxuv          ^hwloc@1.11.11%gcc@10.2.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-ubuntu18.04-broadwell
go5mgat              ^libpciaccess@0.16%gcc@10.2.0 arch=linux-ubuntu18.04-broadwell
4swz6du                  ^libtool@2.4.6%gcc@10.2.0 arch=linux-ubuntu18.04-broadwell
fvibkic                      ^m4@1.4.18%gcc@10.2.0+sigsegv patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 arch=linux-ubuntu18.04-broadwell
b6faazt                          ^libsigsegv@2.12%gcc@10.2.0 arch=linux-ubuntu18.04-broadwell
z3getxj                  ^pkgconf@1.7.3%gcc@10.2.0 arch=linux-ubuntu18.04-broadwell
fylewdu                  ^util-macros@1.19.1%gcc@10.2.0 arch=linux-ubuntu18.04-broadwell
o56eham              ^libxml2@2.9.10%gcc@10.2.0~python arch=linux-ubuntu18.04-broadwell
upvkp7k                  ^libiconv@1.16%gcc@10.2.0 arch=linux-ubuntu18.04-broadwell
r5xn4ra                  ^xz@5.2.5%gcc@10.2.0~pic arch=linux-ubuntu18.04-broadwell
azie5vs                  ^zlib@1.2.11%gcc@10.2.0+optimize+pic+shared arch=linux-ubuntu18.04-broadwell
jclx53j              ^numactl@2.0.14%gcc@10.2.0 patches=4e1d78cbbb85de625bad28705e748856033eaafab92a66dffd383a3d7e00cc94 arch=linux-ubuntu18.04-broadwell
q3gkqfy                  ^autoconf@2.69%gcc@10.2.0 arch=linux-ubuntu18.04-broadwell
zseosv7                      ^perl@5.30.3%gcc@10.2.0+cpanm+shared+threads arch=linux-ubuntu18.04-broadwell
hvbjbd4                          ^berkeley-db@18.1.40%gcc@10.2.0 arch=linux-ubuntu18.04-broadwell
wl5k65m                          ^gdbm@1.18.1%gcc@10.2.0 patches=7f569b9287e9e5a4236636eef32ff3428f0462ec17aea359e0ebee07649f1f16 arch=linux-ubuntu18.04-broadwell
ciaqmp5                              ^readline@8.0%gcc@10.2.0 arch=linux-ubuntu18.04-broadwell
vkv6uu4                                  ^ncurses@6.2%gcc@10.2.0~symlinks+termlib arch=linux-ubuntu18.04-broadwell
bx5xku3                  ^automake@1.16.2%gcc@10.2.0 arch=linux-ubuntu18.04-broadwell


real    0m3,132s
user    0m2,218s
sys     0m0,302s
```
It's likely I need to invalidate the cache explicitly on config set, but I wonder if there are more issues than just that. @haampie Can you give it a try (while I inspect failing unit tests)?
Thanks @alalazo, just tested it, seems a lot faster for sure! Downloading / installing took < 4min; in the CI job there is some additional pulling / pushing of images, but I think with this patch things might run faster than before?

Resolving patch:
diff --git a/lib/spack/spack/config.py b/lib/spack/spack/config.py
--- a/lib/spack/spack/config.py
+++ b/lib/spack/spack/config.py
@@ -2,7 +2,6 @@
 # Spack Project Developers. See the top-level COPYRIGHT file for details.
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
-
 """This module implements Spack's configuration file handling.
 
 This implements Spack's configuration system, which handles merging
@@ -29,9 +28,9 @@
 schemas are in submodules of :py:mod:`spack.schema`.
 
 """
-
 import collections
 import copy
+import functools
 import os
 import re
 import sys
@@ -157,7 +156,7 @@ def get_section(self, section):
             self.sections[section] = data
         return self.sections[section]
 
-    def write_section(self, section):
+    def _write_section(self, section):
         filename = self.get_section_filename(section)
         data = self.get_section(section)
 
@@ -253,7 +252,7 @@ def get_section(self, section):
                 self.sections[section_key] = {section_key: data}
         return self.sections.get(section, None)
 
-    def write_section(self, section):
+    def _write_section(self, section):
         data_to_write = self._raw_data
 
         # If there is no existing data, this section SingleFileScope has never
@@ -305,7 +304,7 @@ class ImmutableConfigScope(ConfigScope):
     This is used for ConfigScopes passed on the command line.
     """
 
-    def write_section(self, section):
+    def _write_section(self, section):
         raise ConfigError("Cannot write to immutable scope %s" % self)
 
     def __repr__(self):
@@ -341,7 +340,7 @@ def get_section(self, section):
             self.sections[section] = None
         return self.sections[section]
 
-    def write_section(self, section):
+    def _write_section(self, section):
         """This only validates, as the data is already in memory."""
         data = self.get_section(section)
         if data is not None:
@@ -371,6 +370,18 @@ def _process_dict_keyname_overrides(data):
         return result
 
 
+def _config_mutator(method):
+    """Decorator to mark all the methods in the Configuration class
+    that mutate the underlying configuration. Used to clear the
+    memoization cache.
+    """
+    @functools.wraps(method)
+    def _method(self, *args, **kwargs):
+        self._get_config_memoized.cache.clear()
+        return method(self, *args, **kwargs)
+    return _method
+
+
 class Configuration(object):
     """A full Spack configuration, from a hierarchy of config files.
 
@@ -390,6 +401,7 @@ def __init__(self, *scopes):
             self.push_scope(scope)
         self.format_updates = collections.defaultdict(list)
 
+    @_config_mutator
     def push_scope(self, scope):
         """Add a higher precedence scope to the Configuration."""
         cmd_line_scope = None
@@ -404,11 +416,13 @@ def push_scope(self, scope):
         if cmd_line_scope:
             self.scopes['command_line'] = cmd_line_scope
 
+    @_config_mutator
     def pop_scope(self):
         """Remove the highest precedence scope and return it."""
         name, scope = self.scopes.popitem(last=True)
         return scope
 
+    @_config_mutator
     def remove_scope(self, scope_name):
         return self.scopes.pop(scope_name)
 
@@ -472,6 +486,7 @@ def get_config_filename(self, scope, section):
         scope = self._validate_scope(scope)
         return scope.get_section_filename(section)
 
+    @_config_mutator
     def clear_caches(self):
         """Clears the caches for configuration files,
 
@@ -479,6 +494,7 @@ def clear_caches(self):
         for scope in self.scopes.values():
             scope.clear()
 
+    @_config_mutator
     def update_config(self, section, update_data, scope=None, force=False):
         """Update the configuration file for a particular scope.
 
@@ -526,7 +542,7 @@ def update_config(self, section, update_data, scope=None, force=False):
                     yaml.comments.Comment.attrib,
                     comments)
 
-        scope.write_section(section)
+        scope._write_section(section)
 
     def get_config(self, section, scope=None):
         """Get configuration settings for a section.
@@ -552,6 +568,10 @@ def get_config(self, section, scope=None):
            }
 
         """
+        return self._get_config_memoized(section, scope)
+
+    @llnl.util.lang.memoized
+    def _get_config_memoized(self, section, scope):
         _validate_section_name(section)
 
         if scope is None:
@@ -619,6 +639,7 @@ def get(self, path, default=None, scope=None):
 
         return value
 
+    @_config_mutator
     def set(self, path, value, scope=None):
         """Convenience function for setting single values in config files.
 
@@ -784,6 +805,22 @@ def _config():
 config = llnl.util.lang.Singleton(_config)
 
 
+def replace_config(configuration):
+    """Replace the current global configuration with the instance passed as
+    argument.
+
+    Args:
+        configuration (Configuration): the new configuration to be used.
+
+    Returns:
+        The old configuration that has been removed
+    """
+    global config
+    config.clear_caches(), configuration.clear_caches()
+    old_config, config = config, configuration
+    return old_config
+
+
 def get(path, default=None, scope=None):
     """Module-level wrapper for ``Configuration.get()``."""
     return config.get(path, default, scope)


Test patch:
diff --git a/lib/spack/spack/test/cmd/common/arguments.py b/lib/spack/spack/test/cmd/common/arguments.py
--- a/lib/spack/spack/test/cmd/common/arguments.py
+++ b/lib/spack/spack/test/cmd/common/arguments.py
@@ -20,6 +20,7 @@ def parser():
     arguments.add_common_arguments(p, ['jobs'])
     yield p
     # Cleanup the command line scope if it was set during tests
+    spack.config.config.clear_caches()
     if 'command_line' in spack.config.config.scopes:
         spack.config.config.scopes['command_line'].clear()
 
diff --git a/lib/spack/spack/test/config.py b/lib/spack/spack/test/config.py
--- a/lib/spack/spack/test/config.py
+++ b/lib/spack/spack/test/config.py
@@ -766,7 +766,7 @@ def test_immutable_scope(tmpdir):
     assert data['config']['install_tree'] == {'root': 'dummy_tree_value'}
 
     with pytest.raises(spack.config.ConfigError):
-        scope.write_section('config')
+        scope._write_section('config')
 
 
 def test_single_file_scope(tmpdir, config):
@@ -839,7 +839,7 @@ def test_write_empty_single_file_scope(tmpdir):
     env_schema = spack.schema.env.schema
     scope = spack.config.SingleFileScope(
         'test', str(tmpdir.ensure('config.yaml')), env_schema, ['spack'])
-    scope.write_section('config')
+    scope._write_section('config')
     # confirm we can write empty config
     assert not scope.get_section('config')
 
diff --git a/lib/spack/spack/test/conftest.py b/lib/spack/spack/test/conftest.py
--- a/lib/spack/spack/test/conftest.py
+++ b/lib/spack/spack/test/conftest.py
@@ -318,8 +318,7 @@ def _skip_if_missing_executables(request):
 @contextlib.contextmanager
 def use_configuration(config):
     """Context manager to swap out the global Spack configuration."""
-    saved = spack.config.config
-    spack.config.config = config
+    saved = spack.config.replace_config(config)
 
     # Avoid using real spack configuration that has been cached by other
     # tests, and avoid polluting the cache with spack test configuration
@@ -329,7 +328,7 @@ def use_configuration(config):
 
     yield
 
-    spack.config.config = saved
+    spack.config.replace_config(saved)
     spack.compilers._cache_config_file = saved_compiler_cache