Repository: spack/spack

Issue:
External packages modules are cached in opt/spack/.spack-db without being updated on configuration changes
### Steps to reproduce

Let's say you have an external package provided via modules like this

```yaml
packages:
  cray-mpich:
    externals:
    - spec: cray-mpich@8.1.21
      modules:
      - cray/22.11
      - cray-mpich/8.1.21
```

and you install it in your spack instance.

Later, you want to update the config because this package now requires an additional module (e.g. `libfabric`)

```yaml
packages:
  cray-mpich:
    externals:
    - spec: cray-mpich@8.1.21
      modules:
      - cray/22.11
      - cray-mpich/8.1.21
      - libfabric
```

### Error message

Unfortunately, the change in the config is not going to be reflected in the behaviour, because the configuration used will always be the one that existed at the time the package was installed the first time.

Indeed, the configuration is stored in the spack db, i.e. `$SPACK_ROOT/opt/spack/.spack-db/index.json` and does not get "updated".

The only way to reflect the change was to uninstall the package for which I made the configuration change (together with all the dependents), and re-install it so that it's the "first" installation and it gather the new information from the configuration file.

### Information on your system

* **Spack:** 0.21.0.dev0 (33cb8c988f3fd9afb364b403eda3aaaabe130729)
* **Python:** 3.6.15
* **Platform:** linux-sles15-zen2
* **Concretizer:** clingo

### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have searched the issues of this repo and believe this is not a duplicate
- [X] I have run the failing commands in debug mode and reported the output


Discussion:
Thanks to @haampie for the support in debugging this problem. 

Probably worth pointing out that https://github.com/spack/spack/pull/35975 might be useful for trying solving this.
The idea of #35975 is to only reuse those externals that match config:

```python
reusables = [s for s in specs if not s.external or any(s.satisfies(t) and s.prefix == t.prefix and s.modules == t.modules for t in package_py_externals]
```

that way any config update to modules takes effect immediately.
There is one thing that I don't understand in this conversation. Given that installing the first `cray-mpich` right now will leave the following entry in the DB:

<details><summary> cray-mpich </summary>
<p>

```json
{
  "spec": {
    "_meta": {
      "version": 4
    },
    "nodes": [
      {
        "name": "cray-mpich",
        "version": "8.1.21",
        "arch": {
          "platform": "linux",
          "platform_os": "ubuntu20.04",
          "target": {
            "name": "icelake",
            "vendor": "GenuineIntel",
            "features": [
              "adx",
              "aes",
              "avx",
              "avx2",
              "avx512_bitalg",
              "avx512_vbmi2",
              "avx512_vnni",
              "avx512_vpopcntdq",
              "avx512bw",
              "avx512cd",
              "avx512dq",
              "avx512f",
              "avx512ifma",
              "avx512vbmi",
              "avx512vl",
              "bmi1",
              "bmi2",
              "clflushopt",
              "clwb",
              "f16c",
              "fma",
              "gfni",
              "mmx",
              "movbe",
              "pclmulqdq",
              "popcnt",
              "rdpid",
              "rdrand",
              "rdseed",
              "sha_ni",
              "sse",
              "sse2",
              "sse4_1",
              "sse4_2",
              "ssse3",
              "vaes",
              "vpclmulqdq",
              "xsavec",
              "xsaveopt"
            ],
            "generation": 0,
            "parents": [
              "cascadelake",
              "cannonlake"
            ]
          }
        },
        "compiler": {
          "name": "gcc",
          "version": "9.4.0"
        },
        "namespace": "builtin",
        "parameters": {
          "build_system": "generic",
          "wrappers": true,
          "cflags": [],
          "cppflags": [],
          "cxxflags": [],
          "fflags": [],
          "ldflags": [],
          "ldlibs": []
        },
        "external": {
          "path": null,
          "module": [
            "cray/22.11",
            "cray-mpich/8.1.21"
          ],
          "extra_attributes": {}
        },
        "package_hash": "rodj3475q2dbljuz5g5qa6btl36vw2ek6ubhip4i2oylayabwhdq====",
        "hash": "la6bdrwowheiuqiybkanvmpp4j5226c4"
      }
    ]
  }
}
```

</p>
</details> 

which includes the modules being loaded, how can we ensure that:

> any config update to modules takes effect immediately.

without either having splicing, or disregarding the hash of all the specs built on top of `cray-mpich`? If we don't do one of the two things, wouldn't that mean that we have to reinstall everything built on top of `cray-mpich` because of the change in configuration?
The solution to that is the extension of #35975

Resolving patch:
diff --git a/lib/spack/spack/solver/asp.py b/lib/spack/spack/solver/asp.py
--- a/lib/spack/spack/solver/asp.py
+++ b/lib/spack/spack/solver/asp.py
@@ -3134,14 +3134,38 @@ def _develop_specs_from_env(spec, env):
     spec.constrain(dev_info["spec"])
 
 
-def _is_reusable_external(packages, spec: spack.spec.Spec) -> bool:
-    """Returns true iff spec is an external that can be reused.
+def _is_reusable(spec: spack.spec.Spec, packages, local: bool) -> bool:
+    """A spec is reusable if it's not a dev spec, it's imported from the cray manifest, it's not
+    external, or it's external with matching packages.yaml entry. The latter prevents two issues:
+
+    1. Externals in build caches: avoid installing an external on the build machine not
+       available on the target machine
+    2. Local externals: avoid reusing an external if the local config changes. This helps in
+       particular when a user removes an external from packages.yaml, and expects that that
+       takes effect immediately.
 
     Arguments:
-        packages: the packages configuration
         spec: the spec to check
+        packages: the packages configuration
     """
-    for name in {spec.name, *(p.name for p in spec.package.provided)}:
+    if "dev_path" in spec.variants:
+        return False
+
+    if not spec.external:
+        return True
+
+    # Cray external manifest externals are always reusable
+    if local:
+        _, record = spack.store.STORE.db.query_by_spec_hash(spec.dag_hash())
+        if record and record.origin == "external-db":
+            return True
+
+    try:
+        provided = [p.name for p in spec.package.provided]
+    except spack.repo.RepoError:
+        provided = []
+
+    for name in {spec.name, *provided}:
         for entry in packages.get(name, {}).get("externals", []):
             if (
                 spec.satisfies(entry["spec"])
@@ -3188,29 +3212,21 @@ def _check_input_and_extract_concrete_specs(specs):
     def _reusable_specs(self, specs):
         reusable_specs = []
         if self.reuse:
+            packages = spack.config.get("packages")
             # Specs from the local Database
             with spack.store.STORE.db.read_transaction():
                 reusable_specs.extend(
-                    [
-                        s
-                        for s in spack.store.STORE.db.query(installed=True)
-                        if not s.satisfies("dev_path=*")
-                    ]
+                    s
+                    for s in spack.store.STORE.db.query(installed=True)
+                    if _is_reusable(s, packages, local=True)
                 )
 
             # Specs from buildcaches
             try:
-                # Specs in a build cache that depend on externals are reusable as long as local
-                # config has matching externals. This should guard against picking up binaries
-                # linked against externals not available locally, while still supporting the use
-                # case of distributing binaries across machines with similar externals.
-                packages = spack.config.get("packages")
                 reusable_specs.extend(
-                    [
-                        s
-                        for s in spack.binary_distribution.update_cache_and_get_specs()
-                        if not s.external or _is_reusable_external(packages, s)
-                    ]
+                    s
+                    for s in spack.binary_distribution.update_cache_and_get_specs()
+                    if _is_reusable(s, packages, local=False)
                 )
             except (spack.binary_distribution.FetchCacheError, IndexError):
                 # this is raised when no mirrors had indices.


Test patch:
diff --git a/lib/spack/spack/test/concretize.py b/lib/spack/spack/test/concretize.py
--- a/lib/spack/spack/test/concretize.py
+++ b/lib/spack/spack/test/concretize.py
@@ -1817,12 +1817,14 @@ def test_git_ref_version_succeeds_with_unknown_version(self, git_ref):
 
     @pytest.mark.regression("31484")
     @pytest.mark.only_clingo("Use case not supported by the original concretizer")
-    def test_installed_externals_are_reused(self, mutable_database, repo_with_changing_recipe):
+    def test_installed_externals_are_reused(
+        self, mutable_database, repo_with_changing_recipe, tmp_path
+    ):
         """Test that external specs that are in the DB can be reused."""
         external_conf = {
             "changing": {
                 "buildable": False,
-                "externals": [{"spec": "changing@1.0", "prefix": "/usr"}],
+                "externals": [{"spec": "changing@1.0", "prefix": str(tmp_path)}],
             }
         }
         spack.config.set("packages", external_conf)
@@ -1847,12 +1849,12 @@ def test_installed_externals_are_reused(self, mutable_database, repo_with_changi
 
     @pytest.mark.regression("31484")
     @pytest.mark.only_clingo("Use case not supported by the original concretizer")
-    def test_user_can_select_externals_with_require(self, mutable_database):
+    def test_user_can_select_externals_with_require(self, mutable_database, tmp_path):
         """Test that users have means to select an external even in presence of reusable specs."""
         external_conf = {
             "mpi": {"buildable": False},
             "multi-provider-mpi": {
-                "externals": [{"spec": "multi-provider-mpi@2.0.0", "prefix": "/usr"}]
+                "externals": [{"spec": "multi-provider-mpi@2.0.0", "prefix": str(tmp_path)}]
             },
         }
         spack.config.set("packages", external_conf)
@@ -2434,7 +2436,8 @@ def test_reusable_externals_match(mock_packages, tmpdir):
     spec.external_path = tmpdir.strpath
     spec.external_modules = ["mpich/4.1"]
     spec._mark_concrete()
-    assert spack.solver.asp._is_reusable_external(
+    assert spack.solver.asp._is_reusable(
+        spec,
         {
             "mpich": {
                 "externals": [
@@ -2442,7 +2445,7 @@ def test_reusable_externals_match(mock_packages, tmpdir):
                 ]
             }
         },
-        spec,
+        local=False,
     )
 
 
@@ -2451,7 +2454,8 @@ def test_reusable_externals_match_virtual(mock_packages, tmpdir):
     spec.external_path = tmpdir.strpath
     spec.external_modules = ["mpich/4.1"]
     spec._mark_concrete()
-    assert spack.solver.asp._is_reusable_external(
+    assert spack.solver.asp._is_reusable(
+        spec,
         {
             "mpi": {
                 "externals": [
@@ -2459,7 +2463,7 @@ def test_reusable_externals_match_virtual(mock_packages, tmpdir):
                 ]
             }
         },
-        spec,
+        local=False,
     )
 
 
@@ -2468,7 +2472,8 @@ def test_reusable_externals_different_prefix(mock_packages, tmpdir):
     spec.external_path = "/other/path"
     spec.external_modules = ["mpich/4.1"]
     spec._mark_concrete()
-    assert not spack.solver.asp._is_reusable_external(
+    assert not spack.solver.asp._is_reusable(
+        spec,
         {
             "mpich": {
                 "externals": [
@@ -2476,7 +2481,7 @@ def test_reusable_externals_different_prefix(mock_packages, tmpdir):
                 ]
             }
         },
-        spec,
+        local=False,
     )
 
 
@@ -2486,7 +2491,8 @@ def test_reusable_externals_different_modules(mock_packages, tmpdir, modules):
     spec.external_path = tmpdir.strpath
     spec.external_modules = modules
     spec._mark_concrete()
-    assert not spack.solver.asp._is_reusable_external(
+    assert not spack.solver.asp._is_reusable(
+        spec,
         {
             "mpich": {
                 "externals": [
@@ -2494,7 +2500,7 @@ def test_reusable_externals_different_modules(mock_packages, tmpdir, modules):
                 ]
             }
         },
-        spec,
+        local=False,
     )
 
 
@@ -2502,6 +2508,8 @@ def test_reusable_externals_different_spec(mock_packages, tmpdir):
     spec = Spec("mpich@4.1%gcc@13.1.0~debug build_system=generic arch=linux-ubuntu23.04-zen2")
     spec.external_path = tmpdir.strpath
     spec._mark_concrete()
-    assert not spack.solver.asp._is_reusable_external(
-        {"mpich": {"externals": [{"spec": "mpich@4.1 +debug", "prefix": tmpdir.strpath}]}}, spec
+    assert not spack.solver.asp._is_reusable(
+        spec,
+        {"mpich": {"externals": [{"spec": "mpich@4.1 +debug", "prefix": tmpdir.strpath}]}},
+        local=False,
     )
diff --git a/lib/spack/spack/test/cray_manifest.py b/lib/spack/spack/test/cray_manifest.py
--- a/lib/spack/spack/test/cray_manifest.py
+++ b/lib/spack/spack/test/cray_manifest.py
@@ -19,6 +19,7 @@
 import spack.compilers
 import spack.config
 import spack.cray_manifest as cray_manifest
+import spack.solver.asp
 import spack.spec
 import spack.store
 from spack.cray_manifest import compiler_from_entry, entries_to_specs
@@ -488,3 +489,23 @@ def test_find_external_nonempty_default_manifest_dir(
     spack.cmd.external._collect_and_consume_cray_manifest_files(ignore_default_dir=False)
     specs = spack.store.STORE.db.query("hwloc")
     assert any(x.dag_hash() == "hwlocfakehashaaa" for x in specs)
+
+
+def test_reusable_externals_cray_manifest(
+    tmpdir, mutable_config, mock_packages, temporary_store, manifest_content
+):
+    """The concretizer should be able to reuse specs imported from a manifest without a
+    externals config entry in packages.yaml"""
+    with tmpdir.as_cwd():
+        with open("external-db.json", "w") as f:
+            json.dump(manifest_content, f)
+        cray_manifest.read(path="external-db.json", apply_updates=True)
+
+        # Get any imported spec
+        spec = temporary_store.db.query_local()[0]
+
+        # Reusable if imported locally
+        assert spack.solver.asp._is_reusable(spec, packages={}, local=True)
+
+        # If cray manifest entries end up in a build cache somehow, they are not reusable
+        assert not spack.solver.asp._is_reusable(spec, packages={}, local=False)