Repository: spack/spack

Issue:
concretizer does not take a unique provider
### Steps to reproduce

```yaml
spack:
  specs:
  - sirius ^intel-mkl
  "packages:":
    all:
      compiler: [gcc, intel, pgi, clang, xl, nag, fj, aocc]
      providers:
        blas: [openblas, amdblis]
        fftw-api: [fftw, amdfftw]
        lapack: [openblas, amdlibflame]
        mkl: [intel-mkl]
        mpi: [openmpi, mpich]
        scalapack: [netlib-scalapack, amdscalapack]
```

`spack -e . concretize` shows `openblas@0.3.18`, `intel-mkl@2020.4.304`, `fftw@3.3.9`

meaning, repeated providers for blas, lapack (openblas / intel-mkl) and fftw-api (fftw, intel-mkl).

I can't find a Spack version where this does *not* happen right now.



Discussion:
The bug is confirmed:

<details>

<summary> spack solve sirius ^intel-mkl </summary>

```console
==> Best of 9 considered solutions.
==> Optimization Criteria:
  Priority  Criterion                                           Value
  1         deprecated versions used                                0
  2         version weight                                          0
  3         number of non-default variants (roots)                  0
  4         preferred providers for roots                           0
  5         number of values in multi valued variants (root)       -4
  6         number of non-default variants (non-roots)              0
  7         preferred providers (non-roots)                         0
  8         compiler mismatches                                     0
  9         version badness                                         2
  10        number of values in multi valued variants (non-root)  -13
  11        non-preferred compilers                                 0
  12        target mismatches                                       0
  13        non-preferred targets                                   0

sirius@7.2.7%gcc@9.3.0+apps~boost_filesystem~cuda~elpa~fortran~ipo~magma+memory_pool~nlcglib+openmp+profiler~python~rocm~scalapack+shared~single_precision~tests~vdwxc amdgpu_target=gfx803,gfx900,gfx906 build_type=Release cuda_arch=none arch=linux-ubuntu20.04-icelake
    ^cmake@3.21.3%gcc@9.3.0~doc+ncurses+openssl+ownlibs~qt build_type=Release arch=linux-ubuntu20.04-icelake
        ^ncurses@6.2%gcc@9.3.0~symlinks+termlib abi=none arch=linux-ubuntu20.04-icelake
            ^pkgconf@1.8.0%gcc@9.3.0 arch=linux-ubuntu20.04-icelake
        ^openssl@1.1.1l%gcc@9.3.0~docs certs=system arch=linux-ubuntu20.04-icelake
            ^perl@5.34.0%gcc@9.3.0+cpanm+shared+threads arch=linux-ubuntu20.04-icelake
                ^berkeley-db@18.1.40%gcc@9.3.0+cxx~docs+stl patches=b231fcc4d5cff05e5c3a4814f6a5af0e9a966428dc2176540d2c05aff41de522 arch=linux-ubuntu20.04-icelake
                ^bzip2@1.0.8%gcc@9.3.0~debug~pic+shared arch=linux-ubuntu20.04-icelake
                    ^diffutils@3.8%gcc@9.3.0 arch=linux-ubuntu20.04-icelake
                        ^libiconv@1.16%gcc@9.3.0 libs=shared,static arch=linux-ubuntu20.04-icelake
                ^gdbm@1.19%gcc@9.3.0 arch=linux-ubuntu20.04-icelake
                    ^readline@8.1%gcc@9.3.0 arch=linux-ubuntu20.04-icelake
                ^zlib@1.2.11%gcc@9.3.0+optimize+pic+shared arch=linux-ubuntu20.04-icelake
    ^fftw@3.3.9%gcc@9.3.0+mpi~openmp~pfft_patches precision=double,float arch=linux-ubuntu20.04-icelake
        ^openmpi@4.1.1%gcc@9.3.0~atomics~cuda~cxx~cxx_exceptions+gpfs~internal-hwloc~java~legacylaunchers~lustre~memchecker~pmi~pmix~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=none schedulers=none arch=linux-ubuntu20.04-icelake
            ^hwloc@2.5.0%gcc@9.3.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml~opencl+pci~rocm+shared arch=linux-ubuntu20.04-icelake
                ^libpciaccess@0.16%gcc@9.3.0 arch=linux-ubuntu20.04-icelake
                    ^libtool@2.4.6%gcc@9.3.0 arch=linux-ubuntu20.04-icelake
                        ^m4@1.4.19%gcc@9.3.0+sigsegv patches=9dc5fbd0d5cb1037ab1e6d0ecc74a30df218d0a94bdd5a02759a97f62daca573,bfdffa7c2eb01021d5849b36972c069693654ad826c1a20b53534009a4ec7a89 arch=linux-ubuntu20.04-icelake
                            ^libsigsegv@2.13%gcc@9.3.0 arch=linux-ubuntu20.04-icelake
                    ^util-macros@1.19.3%gcc@9.3.0 arch=linux-ubuntu20.04-icelake
                ^libxml2@2.9.12%gcc@9.3.0~python arch=linux-ubuntu20.04-icelake
                    ^xz@5.2.5%gcc@9.3.0~pic libs=shared,static arch=linux-ubuntu20.04-icelake
            ^libevent@2.1.12%gcc@9.3.0+openssl arch=linux-ubuntu20.04-icelake
            ^numactl@2.0.14%gcc@9.3.0 patches=4e1d78cbbb85de625bad28705e748856033eaafab92a66dffd383a3d7e00cc94,62fc8a8bf7665a60e8f4c93ebbd535647cebf74198f7afafec4c085a8825c006,ff37630df599cfabf0740518b91ec8daaf18e8f288b19adaae5364dc1f6b2296 arch=linux-ubuntu20.04-icelake
                ^autoconf@2.69%gcc@9.3.0 patches=35c449281546376449766f92d49fc121ca50e330e60fefcfc9be2af3253082c2,7793209b33013dc0f81208718c68440c5aae80e7a1c4b8d336e382525af791a7,a49dd5bac3b62daa0ff688ab4d508d71dbd2f4f8d7e2a02321926346161bf3ee arch=linux-ubuntu20.04-icelake
                ^automake@1.16.3%gcc@9.3.0 arch=linux-ubuntu20.04-icelake
            ^openssh@8.7p1%gcc@9.3.0 arch=linux-ubuntu20.04-icelake
                ^libedit@3.1-20210216%gcc@9.3.0 arch=linux-ubuntu20.04-icelake
    ^gsl@2.7%gcc@9.3.0~external-cblas arch=linux-ubuntu20.04-icelake
    ^hdf5@1.10.7%gcc@9.3.0~cxx~fortran+hl~ipo~java+mpi+shared~szip~threadsafe+tools api=default build_type=RelWithDebInfo arch=linux-ubuntu20.04-icelake
    ^intel-mkl@2020.4.304%gcc@9.3.0~ilp64+shared threads=openmp arch=linux-ubuntu20.04-icelake
        ^cpio@2.13%gcc@9.3.0 arch=linux-ubuntu20.04-icelake
    ^libxc@5.1.5%gcc@9.3.0~cuda+shared cuda_arch=none arch=linux-ubuntu20.04-icelake
    ^openblas@0.3.18%gcc@9.3.0~bignuma~consistent_fpcsr~ilp64+locking+pic+shared threads=openmp arch=linux-ubuntu20.04-icelake
    ^python@3.8.11%gcc@9.3.0+bz2+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib patches=0d98e93189bc278fbc37a50ed7f183bd8aaf249a8e1670a465f0db6bb4f8cf87,4c2457325f2b608b1b6a2c63087df8c26e07db3e3d493caf36a56f0ecf6fb768,f2fd060afc4b4618fe8104c4c5d771f36dc55b1db5a4623785a4ea707ec72fb4 arch=linux-ubuntu20.04-icelake
        ^expat@2.4.1%gcc@9.3.0+libbsd arch=linux-ubuntu20.04-icelake
            ^libbsd@0.11.3%gcc@9.3.0 arch=linux-ubuntu20.04-icelake
                ^libmd@1.0.3%gcc@9.3.0 arch=linux-ubuntu20.04-icelake
        ^gettext@0.21%gcc@9.3.0+bzip2+curses+git~libunistring+libxml2+tar+xz arch=linux-ubuntu20.04-icelake
            ^tar@1.34%gcc@9.3.0 arch=linux-ubuntu20.04-icelake
        ^libffi@3.3%gcc@9.3.0 patches=26f26c6f29a7ce9bf370ad3ab2610f99365b4bdd7b82e7c31df41a3370d685c0 arch=linux-ubuntu20.04-icelake
        ^sqlite@3.36.0%gcc@9.3.0+column_metadata+fts~functions~rtree arch=linux-ubuntu20.04-icelake
        ^util-linux-uuid@2.36.2%gcc@9.3.0 arch=linux-ubuntu20.04-icelake
    ^spfft@1.0.4%gcc@9.3.0~cuda~fortran~gpu_direct~ipo+mpi+openmp~rocm~single_precision~static amdgpu_target=gfx803,gfx900,gfx906 build_type=Release cuda_arch=none arch=linux-ubuntu20.04-icelake
    ^spglib@1.16.1%gcc@9.3.0~ipo build_type=RelWithDebInfo arch=linux-ubuntu20.04-icelake
    ^spla@1.5.1%gcc@9.3.0~cuda~fortran~ipo+openmp~rocm~static build_type=RelWithDebInfo arch=linux-ubuntu20.04-icelake

```

</details>
In the solve showing the bug `intel-mkl` is in the DAG (without triggering conflicts), but is not providing any virtual package:
```
provider("fftw","fftw-api") provider("openblas","lapack") provider("openmpi","mpi") provider("pkgconf","pkgconfig") provider("util-linux-uuid","uuid") provider("openblas","blas") provider("libiconv","iconv")
```
The condition for `intel-mkl` to be a possible provider are met and in fact:
```
[ ... ] possible_provider("intel-mkl","blas") possible_provider("intel-mkl","fftw-api") possible_provider("intel-mkl","lapack") possible_provider("intel-mkl","mkl") possible_provider("intel-mkl","scalapack") [ ... ]
```
I'm also confused about `intel-mkl` being a provider for `mkl` by the way in packages.yaml... may be unrelated, but... what does that even mean? I hope virtuals are not providing virtuals :p 

edit: didn't read you second comment before posting.
I think that is to distinguish the provider for other part of the MKL library, like `pardiso` etc. ?
This
```diff
diff --git a/lib/spack/spack/solver/concretize.lp b/lib/spack/spack/solver/concretize.lp
index b85d2f798a..552d9118cd 100644
--- a/lib/spack/spack/solver/concretize.lp
+++ b/lib/spack/spack/solver/concretize.lp
@@ -196,6 +196,11 @@ provides_virtual(Provider, Virtual) :-
    provider(Package, V1), not provider(Package, V2),
    virtual_node(V1), virtual_node(V2).
 
+:- provides_virtual(P1, Virtual), provides_virtual(P2, Virtual), P1 != P2,
+   provider(P1, Virtual), not provider(P2, Virtual),
+   virtual_node(Virtual).
+
+
 #defined possible_provider/2.
 
 %-----------------------------------------------------------------------------
```
seem to solve this issue. I am checking right now if it causes regressions. Note that by default it tries to use an old `intel-mkl` to bring in `fftw`.
Cool.

> it tries to use an old intel-mkl to bring in fftw

Yeah, I can live with that! Pretty sure than soon :tm: someone will complain about it, but oh.
I'll tag as "workaround" given the diff above, but still trying to check why this happens in the first place. I think one of the causes is this pattern:
```python
depends_on('intel-mkl threads=openmp', when='+openmp ^intel-mkl')
```
because `^intel-mkl` is true by default given the root spec, regardless of the virtuals that might or might not be provided by `intel-mkl` in the DAG. Currently we don't have a way to say "sirius depends on X with ... if X provides any virtual". This is not needed right now, but will likely be needed to turn compiler into proper dependencies when we resuscitate #15569
Pinging @tgamblin @becker33 as they might also be interested in the issue.
Hm, but that pattern is a workaround in the first place... if I use conflicts instead, spack will default to using non-preferred providers like amdblis, because for those it doesn't have to change defaults like +openmp.

In fact cp2k is an example where they *don't* want to have openmp flag propagation to amdblis, because apparently it's broken. So they used to specify `conflicts('^openblas threads=none')`. This causes bad default concretization, requiring the user to repeat their preferred provider for blas `^openblas`. That's not great user experience.

So instead, `depends_on('openblas threads=openmp', when='+openmp ^openblas')`
That pattern makes sense with the current semantics, so that's fine. Currently (modulo bugs like this one) if a package provides some virtual, it is also the provider of all the other possible virtuals provided by the same spec that may be used in the DAG. 

That won't be true anymore with #15569 where the intent is to e.g. cherry-pick using lapack and blas from `intel-mkl` and `fftw-api` from `fftw`. So that PR imo needs also to introduce a way to express:
```python
depends_on('intel-mkl threads=openmp', when='+openmp (intel-mkl is a provider of at least a virtual)')
```
Yeah, actually this ties in with that point raised above, where spack concretizes to fftw and old intel-mkl, because fftw is preferred but intel-mkl is required, and old intel-mkl does not provide fftw-api.

The *user intent* with ^intel-mkl is to say: use intel-mkl for blas / lapack / fftw-api. But that's not what is technically expressed, what is expressed is: a dependency on intel-mkl is required.

The problem would be solved when the user could write `sirius ^fftw-api=intel-mkl` or something along those lines.
@alalazo if it helps, try and remove that depends_on statement for intel-mkl in sirius, and note what happens: intel-mkl becomes a provider of fftw-api as dependency of spfft, but sirius gets openblas and fftw for its virtuals... Does that help with debugging?
I'm afraid my brain is not advanced enough to follow `concretize.lp` :brain: 
So these are the providers according to `clingo` if  I remove the `depends_on` for `intel-mkl`:
```
provider("fftw","fftw-api") provider("intel-mkl","lapack") provider("openmpi","mpi") provider("pkgconf","pkgconfig") provider("util-linux-uuid","uuid") provider("intel-mkl","blas") provider("libiconv","iconv")
```
I think `openblas` is there because of the self-fulfilling `depends_on` mentioned above:
```python
depends_on('openblas threads=openmp', when='+openmp ^openblas')
```
For `clingo`, without the rule in the diff above, this directive is a choice i.e. "when +openmp you can add openblas with threads=openmp" as opposed to:
```python
depends_on('openblas threads=openmp', when='+openmp')
```
which imposes the dependency.


So, turns out writing a test is what takes most time :laughing: 

Resolving patch:
diff --git a/var/spack/repos/builtin.mock/packages/low-priority-provider/package.py b/var/spack/repos/builtin.mock/packages/low-priority-provider/package.py
new file mode 100644
--- /dev/null
+++ b/var/spack/repos/builtin.mock/packages/low-priority-provider/package.py
@@ -0,0 +1,18 @@
+# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+from spack import *
+
+
+class LowPriorityProvider(Package):
+    """Provides multiple virtuals but is low in the priority of clingo"""
+
+    homepage = "http://www.example.com"
+    url = "http://www.example.com/a-1.0.tar.gz"
+
+    version('1.0', '0123456789abcdef0123456789abcdef')
+
+    provides('lapack')
+    provides('mpi')
diff --git a/var/spack/repos/builtin.mock/packages/many-virtual-consumer/package.py b/var/spack/repos/builtin.mock/packages/many-virtual-consumer/package.py
new file mode 100644
--- /dev/null
+++ b/var/spack/repos/builtin.mock/packages/many-virtual-consumer/package.py
@@ -0,0 +1,18 @@
+# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+class ManyVirtualConsumer(Package):
+    """PAckage that depends on many virtual packages"""
+    url = "http://www.example.com/"
+    url = "http://www.example.com/2.0.tar.gz"
+
+    version('1.0', 'abcdef1234567890abcdef1234567890')
+
+    depends_on('mpi')
+    depends_on('lapack')
+
+    # This directive is an example of imposing a constraint on a
+    # dependency is that dependency is in the DAG. This pattern
+    # is mainly used with virtual providers.
+    depends_on('low-priority-provider@1.0', when='^low-priority-provider')


Test patch:
diff --git a/lib/spack/spack/test/cmd/dependencies.py b/lib/spack/spack/test/cmd/dependencies.py
--- a/lib/spack/spack/test/cmd/dependencies.py
+++ b/lib/spack/spack/test/cmd/dependencies.py
@@ -14,7 +14,9 @@
 
 dependencies = SpackCommand('dependencies')
 
-mpis = ['mpich', 'mpich2', 'multi-provider-mpi', 'zmpi']
+mpis = [
+    'low-priority-provider', 'mpich', 'mpich2', 'multi-provider-mpi', 'zmpi'
+]
 mpi_deps = ['fake']
 
 
diff --git a/lib/spack/spack/test/concretize.py b/lib/spack/spack/test/concretize.py
--- a/lib/spack/spack/test/concretize.py
+++ b/lib/spack/spack/test/concretize.py
@@ -1274,3 +1274,17 @@ def test_versions_in_virtual_dependencies(self):
         # package doesn't end up using a later implementation
         s = spack.spec.Spec('hpcviewer@2019.02').concretized()
         assert s['java'].satisfies('virtual-with-versions@1.8.0')
+
+    @pytest.mark.regression('26866')
+    def test_non_default_provider_of_multiple_virtuals(self):
+        s = spack.spec.Spec(
+            'many-virtual-consumer ^low-priority-provider'
+        ).concretized()
+        assert s['mpi'].name == 'low-priority-provider'
+        assert s['lapack'].name == 'low-priority-provider'
+
+        for virtual_pkg in ('mpi', 'lapack'):
+            for pkg in spack.repo.path.providers_for(virtual_pkg):
+                if pkg.name == 'low-priority-provider':
+                    continue
+                assert pkg not in s
diff --git a/lib/spack/spack/test/data/config/packages.yaml b/lib/spack/spack/test/data/config/packages.yaml
--- a/lib/spack/spack/test/data/config/packages.yaml
+++ b/lib/spack/spack/test/data/config/packages.yaml
@@ -1,7 +1,8 @@
 packages:
   all:
     providers:
-      mpi: [openmpi, mpich]
+      mpi: [openmpi, mpich, zmpi]
+      lapack: [openblas-with-lapack]
       blas: [openblas]
   externaltool:
     buildable: False
diff --git a/lib/spack/spack/test/package_class.py b/lib/spack/spack/test/package_class.py
--- a/lib/spack/spack/test/package_class.py
+++ b/lib/spack/spack/test/package_class.py
@@ -30,6 +30,7 @@ def mpi_names(mock_repo_path):
 def mpileaks_possible_deps(mock_packages, mpi_names):
     possible = {
         'callpath': set(['dyninst'] + mpi_names),
+        'low-priority-provider': set(),
         'dyninst': set(['libdwarf', 'libelf']),
         'fake': set(),
         'libdwarf': set(['libelf']),