Repository: spack/spack

Issue:
ASP-based solver: external packages in environment scope are not considered during solve
This error has been reported by @bvanessen If the ASP-based solver is used to concretize a Spack environment, external packages that are declared in the environment scope are not taken into account (while scopes outside the environment instead are).

### Steps to reproduce the issue

Using the following `spack.yaml`:
```yaml
spack:
  specs:
  - openjpeg
  config:
    concretizer: clingo
  view: false
  packages:
    cmake:
      externals:
      - spec: cmake@3.19.5
        prefix: /usr
```
and ensuring that there is no external cmake declared in any other scope, one obtains:
```console
$ spack -e . concretize -f
==> Concretized openjpeg
 -   atepy7l  openjpeg@2.3.1%gcc@10.1.0~ipo build_type=RelWithDebInfo arch=linux-ubuntu18.04-broadwell
 -   to7mk5p      ^cmake@3.19.7%gcc@10.1.0~doc+ncurses+openssl+ownlibs~qt build_type=Release patches=b48396c0e4f61756248156b6cebe9bc0d7a22228639b47b5aa77c9330588ce88 arch=linux-ubuntu18.04-broadwell
 -   m73bzjn          ^ncurses@6.2%gcc@10.1.0~symlinks+termlib arch=linux-ubuntu18.04-broadwell
 -   kzr452j              ^pkgconf@1.7.3%gcc@10.1.0 arch=linux-ubuntu18.04-broadwell
 -   7wm24a6          ^openssl@1.1.1j%gcc@10.1.0~docs+systemcerts arch=linux-ubuntu18.04-broadwell
 -   rdmotez              ^perl@5.32.1%gcc@10.1.0+cpanm+shared+threads arch=linux-ubuntu18.04-broadwell
 -   6madour                  ^berkeley-db@18.1.40%gcc@10.1.0~docs patches=b231fcc4d5cff05e5c3a4814f6a5af0e9a966428dc2176540d2c05aff41de522 arch=linux-ubuntu18.04-broadwell
 -   6p234pa                  ^gdbm@1.19%gcc@10.1.0 arch=linux-ubuntu18.04-broadwell
 -   nsh6fwm                      ^readline@8.0%gcc@10.1.0 arch=linux-ubuntu18.04-broadwell
 -   kepdmuv              ^zlib@1.2.11%gcc@10.1.0+optimize+pic+shared arch=linux-ubuntu18.04-broadwell
```
where externals are not taken into account. When the externals are declared outside of the environment, using their own `packages.yaml`:
```yaml
packages:
  cmake:
    externals:
    - spec: cmake@3.19.5
      prefix: /usr
```
then they are suddenly taken into account:
```console
$ spack -C . -e . concretize -f
==> Concretized openjpeg
 -   atepy7l  openjpeg@2.3.1%gcc@10.1.0~ipo build_type=RelWithDebInfo arch=linux-ubuntu18.04-broadwell
 -   jtt4geb      ^cmake@3.19.5%gcc@10.1.0~doc+ncurses+openssl+ownlibs~qt build_type=Release patches=b48396c0e4f61756248156b6cebe9bc0d7a22228639b47b5aa77c9330588ce88 arch=linux-ubuntu18.04-broadwell
 -   kepdmuv      ^zlib@1.2.11%gcc@10.1.0+optimize+pic+shared arch=linux-ubuntu18.04-broadwell
```

### Error Message

No error message, just the unexpected behavior described above.

### Information on your system

* **Spack:** 0.16.1-1888-f5b9f86bdb
* **Python:** 3.6.9
* **Platform:** linux-ubuntu18.04-broadwell
* **Concretizer:** clingo

### Additional information

- [x] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [x] I have searched the issues of this repo and believe this is not a duplicate
- [x] I have run the failing commands in debug mode and reported the output



Discussion:
This is related to the `spack.config.use_configuration` context manager that is used when bootstrapping clingo. If clingo is already present in the environment the issue cannot be reproduced.
This diff seems to fix the issue for me:
```diff
diff --git a/lib/spack/spack/config.py b/lib/spack/spack/config.py
index f5fc488f52..a52fa31924 100644
--- a/lib/spack/spack/config.py
+++ b/lib/spack/spack/config.py
@@ -241,11 +241,15 @@ def get_section(self, section):
         #      }
         #   }
         # }
+
+        # This bit ensures we have read the file
         if self._raw_data is None:
             self._raw_data = read_config_file(self.path, self.schema)
             if self._raw_data is None:
                 return None
 
+        # Here we recompute sections is it's empty
+        if not self.sections:
             section_data = self._raw_data
             for key in self.yaml_path:
                 if section_data is None:
@@ -254,6 +258,7 @@ def get_section(self, section):
 
             for section_key, data in section_data.items():
                 self.sections[section_key] = {section_key: data}
+
         return self.sections.get(section, None)
 
     def _write_section(self, section):

```
The bug was particularly tricky and boils down to a bug in the `SingleFileScope` class. An object of such a class has a `clear` method to clear its cache, but with the previous implementation the cache was never recomputed once cleared. Having to swap configurations cleared the environment cache when bootstrapping and after that whatever was in the `spack.yaml` file was not recomputed.

Resolving patch:
diff --git a/lib/spack/spack/config.py b/lib/spack/spack/config.py
--- a/lib/spack/spack/config.py
+++ b/lib/spack/spack/config.py
@@ -241,11 +241,18 @@ def get_section(self, section):
         #      }
         #   }
         # }
+
+        # This bit ensures we have read the file and have
+        # the raw data in memory
         if self._raw_data is None:
             self._raw_data = read_config_file(self.path, self.schema)
             if self._raw_data is None:
                 return None
 
+        # Here we know we have the raw data and ensure we
+        # populate the sections dictionary, which may be
+        # cleared by the clear() method
+        if not self.sections:
             section_data = self._raw_data
             for key in self.yaml_path:
                 if section_data is None:
@@ -254,6 +261,7 @@ def get_section(self, section):
 
             for section_key, data in section_data.items():
                 self.sections[section_key] = {section_key: data}
+
         return self.sections.get(section, None)
 
     def _write_section(self, section):


Test patch:
diff --git a/lib/spack/spack/test/config.py b/lib/spack/spack/test/config.py
--- a/lib/spack/spack/test/config.py
+++ b/lib/spack/spack/test/config.py
@@ -73,6 +73,25 @@ def _write(config, data, scope):
     return _write
 
 
+@pytest.fixture()
+def env_yaml(tmpdir):
+    """Return a sample env.yaml for test purposes"""
+    env_yaml = str(tmpdir.join("env.yaml"))
+    with open(env_yaml, 'w') as f:
+        f.write("""\
+env:
+    config:
+        verify_ssl: False
+        dirty: False
+    packages:
+        libelf:
+            compiler: [ 'gcc@4.5.3' ]
+    repos:
+        - /x/y/z
+""")
+    return env_yaml
+
+
 def check_compiler_config(comps, *compiler_names):
     """Check that named compilers in comps match Spack's config."""
     config = spack.config.get('compilers')
@@ -861,23 +880,10 @@ def test_immutable_scope(tmpdir):
         scope._write_section('config')
 
 
-def test_single_file_scope(tmpdir, config):
-    env_yaml = str(tmpdir.join("env.yaml"))
-    with open(env_yaml, 'w') as f:
-        f.write("""\
-env:
-    config:
-        verify_ssl: False
-        dirty: False
-    packages:
-        libelf:
-            compiler: [ 'gcc@4.5.3' ]
-    repos:
-        - /x/y/z
-""")
-
+def test_single_file_scope(config, env_yaml):
     scope = spack.config.SingleFileScope(
-        'env', env_yaml, spack.schema.env.schema, ['env'])
+        'env', env_yaml, spack.schema.env.schema, ['env']
+    )
 
     with spack.config.override(scope):
         # from the single-file config
@@ -1109,3 +1115,20 @@ def test_bad_path_double_override(config):
                        match='Meaningless second override'):
         with spack.config.override('bad::double:override::directive', ''):
             pass
+
+
+@pytest.mark.regression('22547')
+def test_single_file_scope_cache_clearing(env_yaml):
+    scope = spack.config.SingleFileScope(
+        'env', env_yaml, spack.schema.env.schema, ['env']
+    )
+    # Check that we can retrieve data from the single file scope
+    before = scope.get_section('config')
+    assert before
+    # Clear the cache of the Single file scope
+    scope.clear()
+    # Check that the section can be retireved again and it's
+    # the same as before
+    after = scope.get_section('config')
+    assert after
+    assert before == after