Repository: spack/spack

Issue:
Incorrect concretization when a package is found with the wrong provider in a cache
### Steps to reproduce

Have something installed in a cache with a provider:

```console
spack install wrf@3.9.1.1 ^intel-mpi
spack buildcache create ....
```

Try to concretize an environment with a different provider:
```yaml
spack:
  definitions:
  - compilers:
    - gcc @8.2.0
  - mpis:
    - openmpi @4.1.3 +legacylaunchers +pmi +cxx schedulers=slurm
  - mpi_packages:
    - wrf @3.9.1.1 build_type=dm+sm compile_type=em_real nesting=basic ~pnetcdf
  concretizer:
    unify: true
  specs:
  - matrix:
    - - $mpis
    - - $%compilers
  - matrix:
    - - $mpi_packages
    - - $%compilers
    - - $^mpis
  modules:
    default:
      lmod:
        core_compilers:
        - gcc@4.8.5
```
```console
spack -e <env_path> concretize -f
```

(As an aside here, adding `- $compilers` under specs fixes the issue as well oddly)

Results in:
```console
==> [2022-08-29-16:05:11.625385] Concretized openmpi@4.1.3%gcc@8.2.0+cxx+legacylaunchers+pmi schedulers=slurm
 -   433oahj  openmpi@4.1.3%gcc@8.2.0~atomics~cuda+cxx~cxx_exceptions~gpfs~internal-hwloc~java+legacylaunchers~lustre~memchecker+pmi+romio+rsh~singularity+static+vt+wrapper-rpath fabrics=none schedulers=slurm arch=linux-centos7-skylake_avx512
 -   c25vs2b      ^hwloc@2.8.0%gcc@8.2.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml~oneapi-level-zero~opencl+pci~rocm+shared arch=linux-centos7-skylake_avx512
 -   4ii6zcg          ^libpciaccess@0.16%gcc@8.2.0 arch=linux-centos7-skylake_avx512
 -   3ax6pf6              ^libtool@2.4.7%gcc@8.2.0 arch=linux-centos7-skylake_avx512
 -   bftjq73                  ^m4@1.4.19%gcc@8.2.0+sigsegv patches=9dc5fbd,bfdffa7 arch=linux-centos7-skylake_avx512
 -   t35vg7z                      ^libsigsegv@2.13%gcc@8.2.0 arch=linux-centos7-skylake_avx512
 -   zdbgivs              ^pkgconf@1.8.0%gcc@8.2.0 arch=linux-centos7-skylake_avx512
 -   ghk2q7n              ^util-macros@1.19.3%gcc@8.2.0 arch=linux-centos7-skylake_avx512
 -   5qhjl35          ^libxml2@2.9.13%gcc@8.2.0~python arch=linux-centos7-skylake_avx512
 -   geierzd              ^libiconv@1.16%gcc@8.2.0 libs=shared,static arch=linux-centos7-skylake_avx512
 -   yhrdkkk              ^xz@5.2.5%gcc@8.2.0~pic libs=shared,static arch=linux-centos7-skylake_avx512
 -   snis4or              ^zlib@1.2.12%gcc@8.2.0+optimize+pic+shared patches=0d38234 arch=linux-centos7-skylake_avx512
 -   p4zwpol          ^ncurses@6.2%gcc@8.2.0~symlinks+termlib abi=none arch=linux-centos7-skylake_avx512
 -   yencj3g      ^numactl@2.0.14%gcc@8.2.0 patches=4e1d78c,62fc8a8,ff37630 arch=linux-centos7-skylake_avx512
 -   pxx4e4c          ^autoconf@2.69%gcc@8.2.0 patches=35c4492,7793209,a49dd5b arch=linux-centos7-skylake_avx512
 -   cilmspn              ^perl@5.34.1%gcc@8.2.0+cpanm+shared+threads arch=linux-centos7-skylake_avx512
 -   gfxvuvy                  ^berkeley-db@18.1.40%gcc@8.2.0+cxx~docs+stl patches=b231fcc arch=linux-centos7-skylake_avx512
 -   qncsfzp                  ^bzip2@1.0.8%gcc@8.2.0~debug~pic+shared arch=linux-centos7-skylake_avx512
 -   stbtqv5                      ^diffutils@3.8%gcc@8.2.0 arch=linux-centos7-skylake_avx512
 -   ox5gi3s                  ^gdbm@1.19%gcc@8.2.0 arch=linux-centos7-skylake_avx512
 -   gxq3uel                      ^readline@8.1%gcc@8.2.0 arch=linux-centos7-skylake_avx512
 -   54jh6li          ^automake@1.16.5%gcc@8.2.0 arch=linux-centos7-skylake_avx512
 -   knv7qtv      ^openssh@9.0p1%gcc@8.2.0 arch=linux-centos7-skylake_avx512
 -   lhyicyu          ^libedit@3.1-20210216%gcc@8.2.0 arch=linux-centos7-skylake_avx512
 -   ht77s2p          ^openssl@1.1.1o%gcc@8.2.0~docs~shared certs=system arch=linux-centos7-skylake_avx512
 -   cfgxzzm      ^pmix@4.1.2%gcc@8.2.0~docs+pmi_backwards_compatibility~restful arch=linux-centos7-skylake_avx512
 -   j3h7bh4          ^libevent@2.1.12%gcc@8.2.0+openssl arch=linux-centos7-skylake_avx512
 -   gvjqvy5      ^slurm@21.08.4%gcc@8.2.0~gtk~hdf5~hwloc~mariadb~pmix+readline~restd sysconfdir=PREFIX/etc arch=linux-centos7-skylake_avx
```

Where wrf is omitted from the concretized specs.

(Here is a lot more debugging info than we started with https://gist.github.com/douglasjacobsen/86bf90f668f6d356bd6631b3af5620fc/ae331962c42d7603666d8f836a3d907c41a6d642)

### Error message

No errors are reported

### Information on your system

$ spack debug report
* **Spack:** 0.19.0.dev0
* **Python:** 3.7.13
* **Platform:** linux-centos7-cascadelake
* **Concretizer:** clingo


### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have searched the issues of this repo and believe this is not a duplicate
- [X] I have run the failing commands in debug mode and reported the output


Discussion:

Resolving patch:
diff --git a/lib/spack/spack/solver/asp.py b/lib/spack/spack/solver/asp.py
--- a/lib/spack/spack/solver/asp.py
+++ b/lib/spack/spack/solver/asp.py
@@ -1463,6 +1463,12 @@ class Body(object):
                         if concrete_build_deps or dtype != "build":
                             clauses.append(fn.depends_on(spec.name, dep.name, dtype))
 
+                            # Ensure Spack will not coconcretize this with another provider
+                            # for the same virtual
+                            for virtual in dep.package.virtuals_provided:
+                                clauses.append(fn.virtual_node(virtual.name))
+                                clauses.append(fn.provider(dep.name, virtual.name))
+
                     # imposing hash constraints for all but pure build deps of
                     # already-installed concrete specs.
                     if concrete_build_deps or dspec.deptypes != ("build",):


Test patch:
diff --git a/lib/spack/spack/test/concretize.py b/lib/spack/spack/test/concretize.py
--- a/lib/spack/spack/test/concretize.py
+++ b/lib/spack/spack/test/concretize.py
@@ -1695,6 +1695,28 @@ def test_best_effort_coconcretize_preferences(self, specs, expected_spec, occura
                 counter += 1
         assert counter == occurances, concrete_specs
 
+    def test_coconcretize_reuse_and_virtuals(self):
+        import spack.solver.asp
+
+        if spack.config.get("config:concretizer") == "original":
+            pytest.skip("Original concretizer cannot reuse")
+
+        reusable_specs = []
+        for s in ["mpileaks ^mpich", "zmpi"]:
+            reusable_specs.extend(spack.spec.Spec(s).concretized().traverse(root=True))
+
+        root_specs = [spack.spec.Spec("mpileaks"), spack.spec.Spec("zmpi")]
+
+        import spack.solver.asp
+
+        with spack.config.override("concretizer:reuse", True):
+            solver = spack.solver.asp.Solver()
+            setup = spack.solver.asp.SpackSolverSetup()
+            result, _, _ = solver.driver.solve(setup, root_specs, reuse=reusable_specs)
+
+        for spec in result.specs:
+            assert "zmpi" in spec
+
     @pytest.mark.regression("30864")
     def test_misleading_error_message_on_version(self, mutable_database):
         # For this bug to be triggered we need a reusable dependency