Repository: spack/spack

Issue:
Concretizer: reuse broken for external packages
### Steps to reproduce

First, blow away `~/.spack`, clone a fresh copy of Spack, and check for external packages:
```console
$ git clone https://github.com/spack/spack.git
$ . spack/share/spack/setup-env.sh
$ spack external find --not-buildable
```
This will register all external packages it finds on the system in `packages.yaml`. We'll focus on git since it's guaranteed to be found, but this affects all external packages.

Next, install git:
```console
$ spack install git
[+] /usr (external git-2.32.1-lweosdwarje645t4pqta7becf7n5dwbc)
$ spack find -l git
==> 1 installed package
-- darwin-monterey-m1 / apple-clang@13.1.6 ----------------------
lweosdw git@2.32.1
[12:52:36] ajstewart@ajstewart-mbp:~
$ spack spec -Il --reuse git
Input spec
--------------------------------
 -   git

Concretized
--------------------------------
[+]  lweosdw  git@2.32.1%apple-clang@13.1.6+man+nls+perl+subtree~svn~tcltk arch=darwin-monterey-m1
```
Next, modify the git package. For example, add a new variable:
```diff
diff --git a/var/spack/repos/builtin/packages/git/package.py b/var/spack/repos/builtin/packages/git/package.py
index 2d9a7a06f1..71d8d3e6c3 100644
--- a/var/spack/repos/builtin/packages/git/package.py
+++ b/var/spack/repos/builtin/packages/git/package.py
@@ -18,6 +18,7 @@ class Git(AutotoolsPackage):
     homepage = "http://git-scm.com"
     url      = "https://mirrors.edge.kernel.org/pub/software/scm/git/git-2.12.0.tar.gz"
     maintainers = ['jennfshr']
+    foo = 'bar'
 
     tags = ['build-tools']
```
Now, Spack will no longer be able to reuse the existing git installation:
```console
$ spack spec -Il --reuse git
Input spec
--------------------------------
 -   git

Concretized
--------------------------------
 -   634kih5  git@2.32.1%apple-clang@13.1.6+man+nls+perl+subtree~svn~tcltk arch=darwin-monterey-m1

$ spack install --reuse git
[+] /usr (external git-2.32.1-634kih5iv3oxsqem6mrsz75t3zkdwxml)
```
This introduces a lot of inconsistencies in the database. `spack find` claims to only find a single installation, but `spack uninstall` begs to differ:
```
$ spack find -l git
==> 1 installed package
-- darwin-monterey-m1 / apple-clang@13.1.6 ----------------------
lweosdw git@2.32.1
$ spack uninstall git
==> Error: git matches multiple packages:

    -- darwin-monterey-m1 / apple-clang@13.1.6 ----------------------
    lweosdw git@2.32.1  634kih5 git@2.32.1

==> Error: You can either:
    a) use a more specific spec, or
    b) specify the spec by its hash (e.g. `spack uninstall /hash`), or
    c) use `spack uninstall --all` to uninstall ALL matching specs.
```
This also has a recursive effect. If a single package in the DAG is modifed and another package in the DAG is external, Spack will reinstall the modified package and anything that depends on it. Effectively, this means that `--reuse` no longer works if you use external packages and update Spack on occasion.

This is particularly problematic on macOS because many packages depend on `unwind` and `uuid` and Spack ships with a `packages.yaml` that registers these as external. Since Python depends on `uuid`, Spack reinstalls all of my Python libraries any time any file is modified, just like in the days before `--reuse` was added, but worse because we now use a full hash.

### Error message

_No response_

### Information on your system

* **Spack:** 0.19.0.dev0 (a12ce9fd59c32e650b43610925e454322995a81d)
* **Python:** 3.9.12
* **Platform:** darwin-monterey-m1
* **Concretizer:** clingo

@tgamblin @alalazo 

### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have searched the issues of this repo and believe this is not a duplicate
- [X] I have run the failing commands in debug mode and reported the output


Discussion:
Confirmed the bug
The issue stems from how we derive the `external_spec_selected` atom. Currently that doesn't take into account if a spec is reused by hash:
https://github.com/spack/spack/blob/0e783b2c47ec1dc36e3356b5c1038c1c014f6478/lib/spack/spack/solver/concretize.lp#L479-L481

As it stands, this rule never allow reuse of "installed" external specs. Even without the change in the `package.py`, that causes a change in the hash and makes everything evident, one can see that for `clingo` an external is always "built" and never "reused". 

This diff seems to solve the issue:
```diff
diff --git a/lib/spack/spack/solver/concretize.lp b/lib/spack/spack/solver/concretize.lp
index 3b264d7bfd..e70aef6d4f 100644
--- a/lib/spack/spack/solver/concretize.lp
+++ b/lib/spack/spack/solver/concretize.lp
@@ -478,7 +478,8 @@ external(Package) :- external_spec_selected(Package, _).
 % determine if an external spec has been selected
 external_spec_selected(Package, LocalIndex) :-
     external_conditions_hold(Package, LocalIndex),
-    node(Package).
+    node(Package),
+    not hash(Package, _).
 
 external_conditions_hold(Package, LocalIndex) :-
     possible_external(ID, Package, LocalIndex), condition_holds(ID).
@@ -1197,7 +1198,7 @@ opt_criterion(5, "version badness").
 #minimize{ 0@5: #true }.
 #minimize{
     Weight@5+Priority,Package
-    : version_weight(Package, Weight),
+    : version_weight(Package, Weight), not root(Package),
       build_priority(Package, Priority)
 }.
```
even though a better approach long term would be to treat externals differently (ideally, register them in the DB directly and avoid all the custom logic we have now)

Resolving patch:
diff --git a/lib/spack/spack/solver/asp.py b/lib/spack/spack/solver/asp.py
--- a/lib/spack/spack/solver/asp.py
+++ b/lib/spack/spack/solver/asp.py
@@ -1210,10 +1210,11 @@ def external_packages(self):
 
             self.gen.h2("External package: {0}".format(pkg_name))
             # Check if the external package is buildable. If it is
-            # not then "external(<pkg>)" is a fact.
+            # not then "external(<pkg>)" is a fact, unless we can
+            # reuse an already installed spec.
             external_buildable = data.get("buildable", True)
             if not external_buildable:
-                self.gen.fact(fn.external_only(pkg_name))
+                self.gen.fact(fn.buildable_false(pkg_name))
 
             # Read a list of all the specs for this package
             externals = data.get("externals", [])


Test patch:
diff --git a/lib/spack/spack/test/concretize.py b/lib/spack/spack/test/concretize.py
--- a/lib/spack/spack/test/concretize.py
+++ b/lib/spack/spack/test/concretize.py
@@ -1767,3 +1767,90 @@ def test_git_ref_version_errors_if_unknown_version(self, git_ref):
             match="The reference version 'main' for package 'develop-branch-version'",
         ):
             s.concretized()
+
+    @pytest.mark.regression("31484")
+    def test_installed_externals_are_reused(self, mutable_database, repo_with_changing_recipe):
+        """Test that external specs that are in the DB can be reused."""
+        if spack.config.get("config:concretizer") == "original":
+            pytest.xfail("Use case not supported by the original concretizer")
+
+        # Configuration to be added to packages.yaml
+        external_conf = {
+            "changing": {
+                "buildable": False,
+                "externals": [{"spec": "changing@1.0", "prefix": "/usr"}],
+            }
+        }
+        spack.config.set("packages", external_conf)
+
+        # Install the external spec
+        external1 = Spec("changing@1.0").concretized()
+        external1.package.do_install(fake=True, explicit=True)
+        assert external1.external
+
+        # Modify the package.py file
+        repo_with_changing_recipe.change({"delete_variant": True})
+
+        # Try to concretize the external without reuse and confirm the hash changed
+        with spack.config.override("concretizer:reuse", False):
+            external2 = Spec("changing@1.0").concretized()
+        assert external2.dag_hash() != external1.dag_hash()
+
+        # ... while with reuse we have the same hash
+        with spack.config.override("concretizer:reuse", True):
+            external3 = Spec("changing@1.0").concretized()
+        assert external3.dag_hash() == external1.dag_hash()
+
+    @pytest.mark.regression("31484")
+    def test_user_can_select_externals_with_require(self, mutable_database):
+        """Test that users have means to select an external even in presence of reusable specs."""
+        if spack.config.get("config:concretizer") == "original":
+            pytest.xfail("Use case not supported by the original concretizer")
+
+        # Configuration to be added to packages.yaml
+        external_conf = {
+            "mpi": {"buildable": False},
+            "multi-provider-mpi": {
+                "externals": [{"spec": "multi-provider-mpi@2.0.0", "prefix": "/usr"}]
+            },
+        }
+        spack.config.set("packages", external_conf)
+
+        # mpich and others are installed, so check that
+        # fresh use the external, reuse does not
+        with spack.config.override("concretizer:reuse", False):
+            mpi_spec = Spec("mpi").concretized()
+            assert mpi_spec.name == "multi-provider-mpi"
+
+        with spack.config.override("concretizer:reuse", True):
+            mpi_spec = Spec("mpi").concretized()
+            assert mpi_spec.name != "multi-provider-mpi"
+
+        external_conf["mpi"]["require"] = "multi-provider-mpi"
+        spack.config.set("packages", external_conf)
+
+        with spack.config.override("concretizer:reuse", True):
+            mpi_spec = Spec("mpi").concretized()
+            assert mpi_spec.name == "multi-provider-mpi"
+
+    @pytest.mark.regression("31484")
+    def test_installed_specs_disregard_conflicts(self, mutable_database, monkeypatch):
+        """Test that installed specs do not trigger conflicts. This covers for the rare case
+        where a conflict is added on a package after a spec matching the conflict was installed.
+        """
+        if spack.config.get("config:concretizer") == "original":
+            pytest.xfail("Use case not supported by the original concretizer")
+
+        # Add a conflict to "mpich" that match an already installed "mpich~debug"
+        pkg_cls = spack.repo.path.get_pkg_class("mpich")
+        monkeypatch.setitem(pkg_cls.conflicts, "~debug", [(spack.spec.Spec(), None)])
+
+        # If we concretize with --fresh the conflict is taken into account
+        with spack.config.override("concretizer:reuse", False):
+            s = Spec("mpich").concretized()
+            assert s.satisfies("+debug")
+
+        # If we concretize with --reuse it is not, since "mpich~debug" was already installed
+        with spack.config.override("concretizer:reuse", True):
+            s = Spec("mpich").concretized()
+            assert s.satisfies("~debug")