Repository: spack/spack

Issue:
Preferences are not taken into account when installing virtual packages from command line
### Steps to reproduce

```console
$ echo 'packages:
  all:
    providers:
      mpi: [mpich]' > etc/spack/packages.yaml
$ spack spec -I mpi
```

### Error message

`spack spec -I mpi` results in:
```
 -   openmpi@4.1.1%gcc@11.2.1~atomics~cuda~cxx~cxx_exceptions+gpfs~internal-hwloc~java~legacylaunchers~lustre~memchecker~pmi~pmix~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=none schedulers=none arch=linux-fedora34-zen2
...
```
It works correctly if I switch back to the old concretizer.

### Information on your system

* **Spack:** 0.16.3-4764-da31c7e894
* **Python:** 3.9.7
* **Platform:** linux-fedora34-zen2
* **Concretizer:** clingo

### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have searched the issues of this repo and believe this is not a duplicate
- [X] I have run the failing commands in debug mode and reported the output


Discussion:
can you run `spack solve mpi` and show the scores?
```console
$ spack solve mpi  
==> Best of 13 considered solutions.
==> Optimization Criteria:
  Priority  Criterion                                 Value
  1         deprecated versions used                      0
  2         version weight                                0
  3         number of non-default variants (roots)        0
  4         multi-valued variants                        -2
  5         preferred providers for roots                 1
  6         number of non-default variants (non-roots)    0
  7         preferred providers (non-roots)               0
  8         compiler mismatches                           0
  9         version badness                               2
  10        count of non-root multi-valued variants      -4
  11        non-preferred compilers                       0
  12        target mismatches                             0
  13        non-preferred targets                         0

openmpi@4.1.1%gcc@11.2.1~atomics~cuda~cxx~cxx_exceptions+gpfs~internal-hwloc~java~legacylaunchers~lustre~memchecker~pmi~pmix~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=none schedulers=none arch=linux-fedora34-zen2
...
```
As a side note, this is what happens when I remove `openmpi` from the providers in `etc/spack/defaults/packages.yaml`:
```console
$ spack solve mpi               
==> Best of 11 considered solutions.
==> Optimization Criteria:
  Priority  Criterion                                 Value
  1         deprecated versions used                      0
  2         version weight                                0
  3         number of non-default variants (roots)        0
  4         multi-valued variants                        -2
  5         preferred providers for roots               100
  6         number of non-default variants (non-roots)    0
  7         preferred providers (non-roots)               0
  8         compiler mismatches                           0
  9         version badness                               2
  10        count of non-root multi-valued variants      -4
  11        non-preferred compilers                       0
  12        target mismatches                             0
  13        non-preferred targets                         0

mvapich2@2.3.6%gcc@11.2.1~alloca~cuda~debug+regcache+wrapperrpath ch3_rank_bits=32 fabrics=mrail file_systems=auto process_managers=auto threads=multiple arch=linux-fedora34-zen2
...
```
I guess the new concretizer just hates `mpich`. :laughing:
@alalazo negative score for multivalued variants seems like a modelling problem
Negative scores are maximizations. Changing the title since:
```console
$ spack solve hdf5
==> Best of 8 considered solutions.
==> Optimization Criteria:
  Priority  Criterion                                           Value
  1         deprecated versions used                                0
  2         version weight                                          0
  3         number of non-default variants (roots)                  0
  4         preferred providers for roots                           0
  5         number of values in multi valued variants (root)        0
  6         number of non-default variants (non-roots)              0
  7         preferred providers (non-roots)                         0
  8         compiler mismatches                                     0
  9         version badness                                         2
  10        number of values in multi valued variants (non-root)   -7
  11        non-preferred compilers                                 0
  12        target mismatches                                       0
  13        non-preferred targets                                   0

hdf5@1.10.7%gcc@9.3.0~cxx~fortran~hl~ipo~java+mpi+shared~szip~threadsafe+tools api=default build_type=RelWithDebInfo arch=linux-ubuntu20.04-icelake
    ^cmake@3.21.3%gcc@9.3.0~doc+ncurses+openssl+ownlibs~qt build_type=Release arch=linux-ubuntu20.04-icelake
        ^ncurses@6.2%gcc@9.3.0~symlinks+termlib abi=none arch=linux-ubuntu20.04-icelake
            ^pkgconf@1.8.0%gcc@9.3.0 arch=linux-ubuntu20.04-icelake
        ^openssl@1.1.1l%gcc@9.3.0~docs certs=system arch=linux-ubuntu20.04-icelake
            ^perl@5.34.0%gcc@9.3.0+cpanm+shared+threads arch=linux-ubuntu20.04-icelake
                ^berkeley-db@18.1.40%gcc@9.3.0+cxx~docs+stl patches=b231fcc4d5cff05e5c3a4814f6a5af0e9a966428dc2176540d2c05aff41de522 arch=linux-ubuntu20.04-icelake
                ^bzip2@1.0.8%gcc@9.3.0~debug~pic+shared arch=linux-ubuntu20.04-icelake
                    ^diffutils@3.8%gcc@9.3.0 arch=linux-ubuntu20.04-icelake
                        ^libiconv@1.16%gcc@9.3.0 libs=shared,static arch=linux-ubuntu20.04-icelake
                ^gdbm@1.19%gcc@9.3.0 arch=linux-ubuntu20.04-icelake
                    ^readline@8.1%gcc@9.3.0 arch=linux-ubuntu20.04-icelake
                ^zlib@1.2.11%gcc@9.3.0+optimize+pic+shared arch=linux-ubuntu20.04-icelake
    ^mpich@3.4.2%gcc@9.3.0~argobots+fortran+hwloc+hydra+libxml2+pci+romio~slurm~verbs+wrapperrpath device=ch4 netmod=ofi pmi=pmi arch=linux-ubuntu20.04-icelake
        ^findutils@4.8.0%gcc@9.3.0 arch=linux-ubuntu20.04-icelake
        ^hwloc@2.5.0%gcc@9.3.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml~opencl+pci~rocm+shared arch=linux-ubuntu20.04-icelake
            ^libpciaccess@0.16%gcc@9.3.0 arch=linux-ubuntu20.04-icelake
                ^libtool@2.4.6%gcc@9.3.0 arch=linux-ubuntu20.04-icelake
                    ^m4@1.4.19%gcc@9.3.0+sigsegv patches=9dc5fbd0d5cb1037ab1e6d0ecc74a30df218d0a94bdd5a02759a97f62daca573,bfdffa7c2eb01021d5849b36972c069693654ad826c1a20b53534009a4ec7a89 arch=linux-ubuntu20.04-icelake
                        ^libsigsegv@2.13%gcc@9.3.0 arch=linux-ubuntu20.04-icelake
                ^util-macros@1.19.3%gcc@9.3.0 arch=linux-ubuntu20.04-icelake
            ^libxml2@2.9.12%gcc@9.3.0~python arch=linux-ubuntu20.04-icelake
                ^xz@5.2.5%gcc@9.3.0~pic libs=shared,static arch=linux-ubuntu20.04-icelake
        ^libfabric@1.13.1%gcc@9.3.0~debug~kdreg fabrics=sockets,tcp,udp arch=linux-ubuntu20.04-icelake
```
so this seems to happen only when installing virtual packages as root.
Ok, deleted previous message since it was a red-herring. I am checking a few lines fix for this right now. The fix might be swapping target 4 with target 5 in the optimization. Checking if some other use case breaks because of that.

Resolving patch:
diff --git a/var/spack/repos/builtin.mock/packages/some-virtual-mv/package.py b/var/spack/repos/builtin.mock/packages/some-virtual-mv/package.py
new file mode 100644
--- /dev/null
+++ b/var/spack/repos/builtin.mock/packages/some-virtual-mv/package.py
@@ -0,0 +1,22 @@
+# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+from spack import *
+
+
+class SomeVirtualMv(Package):
+    """Package providing a virtual dependency and with a multivalued variant."""
+
+    homepage = "http://www.example.com"
+    url = "http://www.example.com/foo-1.0.tar.gz"
+
+    version('1.0', '0123456789abcdef0123456789abcdef')
+
+    provides('somevirtual')
+
+    # This multi valued variant is needed to trigger an optimization
+    # criteria for clingo
+    variant('libs', default='shared,static', values=('shared', 'static'),
+            multi=True, description='Build shared libs, static libs or both')
diff --git a/var/spack/repos/builtin.mock/packages/some-virtual-preferred/package.py b/var/spack/repos/builtin.mock/packages/some-virtual-preferred/package.py
new file mode 100644
--- /dev/null
+++ b/var/spack/repos/builtin.mock/packages/some-virtual-preferred/package.py
@@ -0,0 +1,17 @@
+# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+from spack import *
+
+
+class SomeVirtualPreferred(Package):
+    """Package providing a virtual dependency with a preference in packages.yaml"""
+
+    homepage = "http://www.example.com"
+    url = "http://www.example.com/foo-1.0.tar.gz"
+
+    version('1.0', '0123456789abcdef0123456789abcdef')
+
+    provides('somevirtual')


Test patch:
diff --git a/lib/spack/spack/test/concretize_preferences.py b/lib/spack/spack/test/concretize_preferences.py
--- a/lib/spack/spack/test/concretize_preferences.py
+++ b/lib/spack/spack/test/concretize_preferences.py
@@ -396,3 +396,23 @@ def test_dependencies_cant_make_version_parent_score_better(self):
 
         assert s.satisfies('^version-test-pkg@2.4.6')
         assert 'version-test-dependency-preferred' not in s
+
+    @pytest.mark.regression('26598')
+    def test_multivalued_variants_are_lower_priority_than_providers(self):
+        """Test that the rule to maximize the number of values for multivalued
+        variants is considered at lower priority than selecting the default
+        provider for virtual dependencies.
+
+        This ensures that we don't e.g. select openmpi over mpich even if we
+        specified mpich as the default mpi provider, just because openmpi supports
+        more fabrics by default.
+        """
+        with spack.config.override(
+            'packages:all', {
+                'providers': {
+                    'somevirtual': ['some-virtual-preferred']
+                }
+            }
+        ):
+            s = Spec('somevirtual').concretized()
+            assert s.name == 'some-virtual-preferred'