Repository: spack/spack

Issue:
Unclear which provider will be used, when preferences are set in `packages.yaml` at the package level
### Steps to reproduce

I have found an issue with the concretizer not picking up the correct providers when described in `packages.yaml`. If providers are aded to the command line, e.g. `^intel-mpi ^intel-mkl`, all works as expected. Here is one example of intel-mpi not being picked up.

Tagging @tgamblin as discussed yesterday.

The default MPI provider after pulling the `develop` branch is Open MPI:
```console
[ec2-user@ip-172-31-1-92 spack]$ spack solve gromacs
==> Best of 9 considered solutions.
==> Optimization Criteria:
  Priority  Criterion                                           Value
  1         deprecated versions used                                0
  2         version weight                                          0
  3         number of non-default variants (roots)                  0
  4         preferred providers for roots                           0
  5         number of values in multi valued variants (root)        0
  6         number of non-default variants (non-roots)              0
  7         preferred providers (non-roots)                         0
  8         compiler mismatches                                     0
  9         version badness                                         2
  10        number of values in multi valued variants (non-root)   -8
  11        non-preferred compilers                                 0
  12        target mismatches                                       0
  13        non-preferred targets                                   0

gromacs@2021.3%gcc@7.3.1~blas~cuda~cycle_subcounters~double+hwloc~ipo~lapack~mdrun_only+mpi~nosuffix~opencl+openmp~plumed~relaxed_double_precision+shared~sycl build_type=RelWithDebInfo arch=linux-amzn2-skylake_avx512
    ^cmake@3.21.4%gcc@7.3.1~doc+ncurses+openssl+ownlibs~qt build_type=Release arch=linux-amzn2-skylake_avx512
        ^ncurses@6.2%gcc@7.3.1~symlinks+termlib abi=none arch=linux-amzn2-skylake_avx512
            ^pkgconf@1.8.0%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
        ^openssl@1.1.1l%gcc@7.3.1~docs certs=system arch=linux-amzn2-skylake_avx512
            ^perl@5.34.0%gcc@7.3.1+cpanm+shared+threads arch=linux-amzn2-skylake_avx512
                ^berkeley-db@18.1.40%gcc@7.3.1+cxx~docs+stl patches=b231fcc4d5cff05e5c3a4814f6a5af0e9a966428dc2176540d2c05aff41de522 arch=linux-amzn2-skylake_avx512
                ^bzip2@1.0.8%gcc@7.3.1~debug~pic+shared arch=linux-amzn2-skylake_avx512
                    ^diffutils@3.8%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
                        ^libiconv@1.16%gcc@7.3.1 libs=shared,static arch=linux-amzn2-skylake_avx512
                ^gdbm@1.19%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
                    ^readline@8.1%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
                ^zlib@1.2.11%gcc@7.3.1+optimize+pic+shared arch=linux-amzn2-skylake_avx512
    ^fftw@3.3.10%gcc@7.3.1+mpi~openmp~pfft_patches precision=double,float arch=linux-amzn2-skylake_avx512
        ^openmpi@4.1.1%gcc@7.3.1~atomics~cuda~cxx~cxx_exceptions+gpfs~internal-hwloc~java~legacylaunchers~lustre~memchecker~pmi~pmix~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=none schedulers=none arch=linux-amzn2-skylake_avx512
            ^hwloc@2.5.0%gcc@7.3.1~cairo~cuda~gl~libudev+libxml2~netloc~nvml~opencl+pci~rocm+shared arch=linux-amzn2-skylake_avx512
                ^libpciaccess@0.16%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
                    ^libtool@2.4.6%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
                        ^m4@1.4.19%gcc@7.3.1+sigsegv patches=9dc5fbd0d5cb1037ab1e6d0ecc74a30df218d0a94bdd5a02759a97f62daca573,bfdffa7c2eb01021d5849b36972c069693654ad826c1a20b53534009a4ec7a89 arch=linux-amzn2-skylake_avx512
                            ^libsigsegv@2.13%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
                    ^util-macros@1.19.3%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
                ^libxml2@2.9.12%gcc@7.3.1~python arch=linux-amzn2-skylake_avx512
                    ^xz@5.2.5%gcc@7.3.1~pic libs=shared,static arch=linux-amzn2-skylake_avx512
            ^libevent@2.1.12%gcc@7.3.1+openssl arch=linux-amzn2-skylake_avx512
            ^numactl@2.0.14%gcc@7.3.1 patches=4e1d78cbbb85de625bad28705e748856033eaafab92a66dffd383a3d7e00cc94,62fc8a8bf7665a60e8f4c93ebbd535647cebf74198f7afafec4c085a8825c006,ff37630df599cfabf0740518b91ec8daaf18e8f288b19adaae5364dc1f6b2296 arch=linux-amzn2-skylake_avx512
                ^autoconf@2.69%gcc@7.3.1 patches=35c449281546376449766f92d49fc121ca50e330e60fefcfc9be2af3253082c2,7793209b33013dc0f81208718c68440c5aae80e7a1c4b8d336e382525af791a7,a49dd5bac3b62daa0ff688ab4d508d71dbd2f4f8d7e2a02321926346161bf3ee arch=linux-amzn2-skylake_avx512
                ^automake@1.16.3%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
            ^openssh@8.7p1%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
                ^libedit@3.1-20210216%gcc@7.3.1 arch=linux-amzn2-skylake_avx512

```

The default provider can be overwritten by adding a dependency to a different MPI on the command line:
```console
[ec2-user@ip-172-31-1-92 spack]$ spack solve gromacs ^intel-mpi
==> Bootstrapping clingo from pre-built binaries
==> Best of 9 considered solutions.
==> Optimization Criteria:
  Priority  Criterion                                           Value
  1         deprecated versions used                                0
  2         version weight                                          0
  3         number of non-default variants (roots)                  0
  4         preferred providers for roots                           0
  5         number of values in multi valued variants (root)        0
  6         number of non-default variants (non-roots)              0
  7         preferred providers (non-roots)                       100
  8         compiler mismatches                                     0
  9         version badness                                         2
  10        number of values in multi valued variants (non-root)   -6
  11        non-preferred compilers                                 0
  12        target mismatches                                       0
  13        non-preferred targets                                   0

gromacs@2021.3%gcc@7.3.1~blas~cuda~cycle_subcounters~double+hwloc~ipo~lapack~mdrun_only+mpi~nosuffix~opencl+openmp~plumed~relaxed_double_precision+shared~sycl build_type=RelWithDebInfo arch=linux-amzn2-skylake_avx512
    ^cmake@3.21.4%gcc@7.3.1~doc+ncurses+openssl+ownlibs~qt build_type=Release arch=linux-amzn2-skylake_avx512
        ^ncurses@6.2%gcc@7.3.1~symlinks+termlib abi=none arch=linux-amzn2-skylake_avx512
            ^pkgconf@1.8.0%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
        ^openssl@1.1.1l%gcc@7.3.1~docs certs=system arch=linux-amzn2-skylake_avx512
            ^perl@5.34.0%gcc@7.3.1+cpanm+shared+threads arch=linux-amzn2-skylake_avx512
                ^berkeley-db@18.1.40%gcc@7.3.1+cxx~docs+stl patches=b231fcc4d5cff05e5c3a4814f6a5af0e9a966428dc2176540d2c05aff41de522 arch=linux-amzn2-skylake_avx512
                ^bzip2@1.0.8%gcc@7.3.1~debug~pic+shared arch=linux-amzn2-skylake_avx512
                    ^diffutils@3.8%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
                        ^libiconv@1.16%gcc@7.3.1 libs=shared,static arch=linux-amzn2-skylake_avx512
                ^gdbm@1.19%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
                    ^readline@8.1%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
                ^zlib@1.2.11%gcc@7.3.1+optimize+pic+shared arch=linux-amzn2-skylake_avx512
    ^fftw@3.3.10%gcc@7.3.1+mpi~openmp~pfft_patches precision=double,float arch=linux-amzn2-skylake_avx512
        ^intel-mpi@2019.10.317%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
    ^hwloc@2.5.0%gcc@7.3.1~cairo~cuda~gl~libudev+libxml2~netloc~nvml~opencl+pci~rocm+shared arch=linux-amzn2-skylake_avx512
        ^libpciaccess@0.16%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
            ^libtool@2.4.6%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
                ^m4@1.4.19%gcc@7.3.1+sigsegv patches=9dc5fbd0d5cb1037ab1e6d0ecc74a30df218d0a94bdd5a02759a97f62daca573,bfdffa7c2eb01021d5849b36972c069693654ad826c1a20b53534009a4ec7a89 arch=linux-amzn2-skylake_avx512
                    ^libsigsegv@2.13%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
            ^util-macros@1.19.3%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
        ^libxml2@2.9.12%gcc@7.3.1~python arch=linux-amzn2-skylake_avx512
            ^xz@5.2.5%gcc@7.3.1~pic libs=shared,static arch=linux-amzn2-skylake_avx512
```

However, if I try ot overwrite the provider in `packages.yaml` it is not picked up:
```console
[ec2-user@ip-172-31-1-92 spack]$ echo "packages:
>   gromacs:
>     providers:
>       mpi: [intel-mpi]
>   fftw:
>     providers:
>       mpi: [intel-mpi]
> " > etc/spack/packages.yaml
[ec2-user@ip-172-31-1-92 spack]$ spack solve gromacs
==> Best of 9 considered solutions.
==> Optimization Criteria:
  Priority  Criterion                                           Value
  1         deprecated versions used                                0
  2         version weight                                          0
  3         number of non-default variants (roots)                  0
  4         preferred providers for roots                           0
  5         number of values in multi valued variants (root)        0
  6         number of non-default variants (non-roots)              0
  7         preferred providers (non-roots)                         0
  8         compiler mismatches                                     0
  9         version badness                                         2
  10        number of values in multi valued variants (non-root)   -8
  11        non-preferred compilers                                 0
  12        target mismatches                                       0
  13        non-preferred targets                                   0

gromacs@2021.3%gcc@7.3.1~blas~cuda~cycle_subcounters~double+hwloc~ipo~lapack~mdrun_only+mpi~nosuffix~opencl+openmp~plumed~relaxed_double_precision+shared~sycl build_type=RelWithDebInfo arch=linux-amzn2-skylake_avx512
    ^cmake@3.21.4%gcc@7.3.1~doc+ncurses+openssl+ownlibs~qt build_type=Release arch=linux-amzn2-skylake_avx512
        ^ncurses@6.2%gcc@7.3.1~symlinks+termlib abi=none arch=linux-amzn2-skylake_avx512
            ^pkgconf@1.8.0%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
        ^openssl@1.1.1l%gcc@7.3.1~docs certs=system arch=linux-amzn2-skylake_avx512
            ^perl@5.34.0%gcc@7.3.1+cpanm+shared+threads arch=linux-amzn2-skylake_avx512
                ^berkeley-db@18.1.40%gcc@7.3.1+cxx~docs+stl patches=b231fcc4d5cff05e5c3a4814f6a5af0e9a966428dc2176540d2c05aff41de522 arch=linux-amzn2-skylake_avx512
                ^bzip2@1.0.8%gcc@7.3.1~debug~pic+shared arch=linux-amzn2-skylake_avx512
                    ^diffutils@3.8%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
                        ^libiconv@1.16%gcc@7.3.1 libs=shared,static arch=linux-amzn2-skylake_avx512
                ^gdbm@1.19%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
                    ^readline@8.1%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
                ^zlib@1.2.11%gcc@7.3.1+optimize+pic+shared arch=linux-amzn2-skylake_avx512
    ^fftw@3.3.10%gcc@7.3.1+mpi~openmp~pfft_patches precision=double,float arch=linux-amzn2-skylake_avx512
        ^openmpi@4.1.1%gcc@7.3.1~atomics~cuda~cxx~cxx_exceptions+gpfs~internal-hwloc~java~legacylaunchers~lustre~memchecker~pmi~pmix~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=none schedulers=none arch=linux-amzn2-skylake_avx512
            ^hwloc@2.5.0%gcc@7.3.1~cairo~cuda~gl~libudev+libxml2~netloc~nvml~opencl+pci~rocm+shared arch=linux-amzn2-skylake_avx512
                ^libpciaccess@0.16%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
                    ^libtool@2.4.6%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
                        ^m4@1.4.19%gcc@7.3.1+sigsegv patches=9dc5fbd0d5cb1037ab1e6d0ecc74a30df218d0a94bdd5a02759a97f62daca573,bfdffa7c2eb01021d5849b36972c069693654ad826c1a20b53534009a4ec7a89 arch=linux-amzn2-skylake_avx512
                            ^libsigsegv@2.13%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
                    ^util-macros@1.19.3%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
                ^libxml2@2.9.12%gcc@7.3.1~python arch=linux-amzn2-skylake_avx512
                    ^xz@5.2.5%gcc@7.3.1~pic libs=shared,static arch=linux-amzn2-skylake_avx512
            ^libevent@2.1.12%gcc@7.3.1+openssl arch=linux-amzn2-skylake_avx512
            ^numactl@2.0.14%gcc@7.3.1 patches=4e1d78cbbb85de625bad28705e748856033eaafab92a66dffd383a3d7e00cc94,62fc8a8bf7665a60e8f4c93ebbd535647cebf74198f7afafec4c085a8825c006,ff37630df599cfabf0740518b91ec8daaf18e8f288b19adaae5364dc1f6b2296 arch=linux-amzn2-skylake_avx512
                ^autoconf@2.69%gcc@7.3.1 patches=35c449281546376449766f92d49fc121ca50e330e60fefcfc9be2af3253082c2,7793209b33013dc0f81208718c68440c5aae80e7a1c4b8d336e382525af791a7,a49dd5bac3b62daa0ff688ab4d508d71dbd2f4f8d7e2a02321926346161bf3ee arch=linux-amzn2-skylake_avx512
                ^automake@1.16.3%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
            ^openssh@8.7p1%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
                ^libedit@3.1-20210216%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
```

The weird part is that for some packages it does pick up the correct MPI and I cannot find out when. One working example is fftw:
```console
[ec2-user@ip-172-31-1-92 spack]$ echo "packages:
  gromacs:
    providers:
      mpi: [intel-mpi]
  fftw:
    providers:
      mpi: [intel-mpi]
" > etc/spack/packages.yaml
[ec2-user@ip-172-31-1-92 spack]$ spack solve fftw
==> Best of 8 considered solutions.
==> Optimization Criteria:
  Priority  Criterion                                           Value
  1         deprecated versions used                                0
  2         version weight                                          0
  3         number of non-default variants (roots)                  0
  4         preferred providers for roots                           0
  5         number of values in multi valued variants (root)       -2
  6         number of non-default variants (non-roots)              0
  7         preferred providers (non-roots)                         0
  8         compiler mismatches                                     0
  9         version badness                                         0
  10        number of values in multi valued variants (non-root)    0
  11        non-preferred compilers                                 0
  12        target mismatches                                       0
  13        non-preferred targets                                   0

fftw@3.3.10%gcc@7.3.1+mpi~openmp~pfft_patches precision=double,float arch=linux-amzn2-skylake_avx512
    ^intel-mpi@2019.10.317%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
[ec2-user@ip-172-31-1-92 spack]$ rm etc/spack/packages.yaml
[ec2-user@ip-172-31-1-92 spack]$ spack solve fftw
==> Best of 9 considered solutions.
==> Optimization Criteria:
  Priority  Criterion                                           Value
  1         deprecated versions used                                0
  2         version weight                                          0
  3         number of non-default variants (roots)                  0
  4         preferred providers for roots                           0
  5         number of values in multi valued variants (root)       -2
  6         number of non-default variants (non-roots)              0
  7         preferred providers (non-roots)                         0
  8         compiler mismatches                                     0
  9         version badness                                         2
  10        number of values in multi valued variants (non-root)   -6
  11        non-preferred compilers                                 0
  12        target mismatches                                       0
  13        non-preferred targets                                   0

fftw@3.3.10%gcc@7.3.1+mpi~openmp~pfft_patches precision=double,float arch=linux-amzn2-skylake_avx512
    ^openmpi@4.1.1%gcc@7.3.1~atomics~cuda~cxx~cxx_exceptions+gpfs~internal-hwloc~java~legacylaunchers~lustre~memchecker~pmi~pmix~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=none schedulers=none arch=linux-amzn2-skylake_avx512
        ^hwloc@2.5.0%gcc@7.3.1~cairo~cuda~gl~libudev+libxml2~netloc~nvml~opencl+pci~rocm+shared arch=linux-amzn2-skylake_avx512
            ^libpciaccess@0.16%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
                ^libtool@2.4.6%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
                    ^m4@1.4.19%gcc@7.3.1+sigsegv patches=9dc5fbd0d5cb1037ab1e6d0ecc74a30df218d0a94bdd5a02759a97f62daca573,bfdffa7c2eb01021d5849b36972c069693654ad826c1a20b53534009a4ec7a89 arch=linux-amzn2-skylake_avx512
                        ^libsigsegv@2.13%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
                ^pkgconf@1.8.0%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
                ^util-macros@1.19.3%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
            ^libxml2@2.9.12%gcc@7.3.1~python arch=linux-amzn2-skylake_avx512
                ^libiconv@1.16%gcc@7.3.1 libs=shared,static arch=linux-amzn2-skylake_avx512
                ^xz@5.2.5%gcc@7.3.1~pic libs=shared,static arch=linux-amzn2-skylake_avx512
                ^zlib@1.2.11%gcc@7.3.1+optimize+pic+shared arch=linux-amzn2-skylake_avx512
            ^ncurses@6.2%gcc@7.3.1~symlinks+termlib abi=none arch=linux-amzn2-skylake_avx512
        ^libevent@2.1.12%gcc@7.3.1+openssl arch=linux-amzn2-skylake_avx512
            ^openssl@1.1.1l%gcc@7.3.1~docs certs=system arch=linux-amzn2-skylake_avx512
                ^perl@5.34.0%gcc@7.3.1+cpanm+shared+threads arch=linux-amzn2-skylake_avx512
                    ^berkeley-db@18.1.40%gcc@7.3.1+cxx~docs+stl patches=b231fcc4d5cff05e5c3a4814f6a5af0e9a966428dc2176540d2c05aff41de522 arch=linux-amzn2-skylake_avx512
                    ^bzip2@1.0.8%gcc@7.3.1~debug~pic+shared arch=linux-amzn2-skylake_avx512
                        ^diffutils@3.8%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
                    ^gdbm@1.19%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
                        ^readline@8.1%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
        ^numactl@2.0.14%gcc@7.3.1 patches=4e1d78cbbb85de625bad28705e748856033eaafab92a66dffd383a3d7e00cc94,62fc8a8bf7665a60e8f4c93ebbd535647cebf74198f7afafec4c085a8825c006,ff37630df599cfabf0740518b91ec8daaf18e8f288b19adaae5364dc1f6b2296 arch=linux-amzn2-skylake_avx512
            ^autoconf@2.69%gcc@7.3.1 patches=35c449281546376449766f92d49fc121ca50e330e60fefcfc9be2af3253082c2,7793209b33013dc0f81208718c68440c5aae80e7a1c4b8d336e382525af791a7,a49dd5bac3b62daa0ff688ab4d508d71dbd2f4f8d7e2a02321926346161bf3ee arch=linux-amzn2-skylake_avx512
            ^automake@1.16.3%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
        ^openssh@8.7p1%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
            ^libedit@3.1-20210216%gcc@7.3.1 arch=linux-amzn2-skylake_avx512

```

### Error message

_No response_

### Information on your system

[ec2-user@ip-172-31-1-92 spack]$ spack debug report
* **Spack:** 0.16.3
* **Python:** 3.7.10
* **Platform:** linux-amzn2-skylake_avx512
* **Concretizer:** clingo


### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have searched the issues of this repo and believe this is not a duplicate
- [X] I have run the failing commands in debug mode and reported the output


Discussion:
Is this syntax supposed to work? 

```yaml
packages:
  gromacs:
    providers:
      mpi: [...]
```

It looks like an ill-defined minimization problem :thinking: if there are two packages depending on mpi, with different preferences, what determines optimality?
> Is this syntax supposed to work?

Yes: for now the specifications in `packages.yaml` are not hard constraints, they are preferences that can be ignored. 

I think @tgamblin has suggested that these be treated the same as command-line constraints (e.g. `foo ^openmpi`). Even in that case, it could still make sense to allow different packages to specify different providers for a virtual (as long as they are not concretized together).
The main question for me is how to make the command-line constraints persistent. `spack solve gromacs ^intel-mpi` does exactly what I want it to do, so how can I add this to `packages.yaml` to make `spack solve gromacs` behave the same?
Notice that `^intel-mpi` does not do exactly what you want to do. It means requiring a dependency on intel-mpi, which as a side-effect forces the provider for mpi to intel-mpi, because intel-mpi unconditionally provides mpi. There are packages that conditionally provide a virtual, in which case requiring that package as a dependency is not the same as forcing a provider (notably intel-mkl, which provides fftw-api only from a certain version -- clingo rather gives you an old intel-mkl so that it can give you your preferred fftw-api provider from say fftw).

Isn't setting `packages:all:providers:mpi: [intel-mpi]` an option? Or if that's not an option in user config scope, how about creating an environment with a `packages` section?

> I think @tgamblin has suggested that these be treated the same as command-line constraints

I don't think it's necessary to resolve this issue, it looks more like a bug.
> Notice that `^intel-mpi` does not do exactly what you want to do. It means requiring a dependency on intel-mpi, which as a side-effect forces the provider for mpi to intel-mpi, because intel-mpi unconditionally provides mpi. There are packages that conditionally provide a virtual, in which case requiring that package as a dependency is not the same as forcing a provider (notably intel-mkl, which provides fftw-api only from a certain version -- clingo rather gives you an old intel-mkl so that it can give you your preferred fftw-api provider from say fftw).

Yes, you are correct. I want to force a provider rather than add a dependency.

> Isn't setting `packages:all:providers:mpi: [intel-mpi]` an option? Or if that's not an option in user config scope, how about creating an environment with a `packages` section?

Setting providers for `all` is not an option as I want to include multiple packages with different providers per package. I have tried the environment route with adding `spack:packages:gromacs::providers:mpi:[intel-mpi]` via `spack config edit`, but no luck. Intel MPI was not pocked up by `spack solve`.

This will be solved, very likely, by a reworking of the rules for multi-valued variants in #25310 In the meanwhile you can set preferences as:
```yaml
packages:
  all:
    providers:
      mpi:: [intel-mpi]
```
to avoid `openmpi` having the same score as `intel-mpi` when used as an mpi provider. Note the double `:` which is key to discard Spack's default provider list.
This again only works if set for `all`. If I set it only for `gromacs`, then Open MPI is picked up.
```console
[ec2-user@ip-172-31-1-92 spack]$ echo "" > etc/spack/packages.yaml
[ec2-user@ip-172-31-1-92 spack]$ spack solve gromacs | grep mpi@
        ^openmpi@4.1.1%gcc@7.3.1~atomics~cuda~cxx~cxx_exceptions+gpfs~internal-hwloc~java~legacylaunchers~lustre~memchecker~pmi~pmix~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=none schedulers=none arch=linux-amzn2-skylake_avx512
[ec2-user@ip-172-31-1-92 spack]$ echo "packages:
  all:
    providers:
      mpi:: [intel-mpi]
" > etc/spack/packages.yaml
[ec2-user@ip-172-31-1-92 spack]$ spack solve gromacs | grep mpi@
        ^intel-mpi@2019.10.317%gcc@7.3.1 arch=linux-amzn2-skylake_avx512
[ec2-user@ip-172-31-1-92 spack]$ echo "packages:
  gromacs:
    providers:
      mpi:: [intel-mpi]
" > etc/spack/packages.yaml
[ec2-user@ip-172-31-1-92 spack]$ spack solve gromacs | grep mpi@
        ^openmpi@4.1.1%gcc@7.3.1~atomics~cuda~cxx~cxx_exceptions+gpfs~internal-hwloc~java~legacylaunchers~lustre~memchecker~pmi~pmix~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=none schedulers=none arch=linux-amzn2-skylake_avx512
```

I guess for now the workaround is to use one environment per application and set providers of `all` inside each environment.
seeing the same thing with `intel-parallel-studio` as mpi/fftw/blas/lapack provider, but in my case setting it for `all` did not help
Is this still an issue after https://github.com/spack/spack/pull/25310?

Resolving patch:
diff --git a/lib/spack/spack/cmd/config.py b/lib/spack/spack/cmd/config.py
--- a/lib/spack/spack/cmd/config.py
+++ b/lib/spack/spack/cmd/config.py
@@ -407,7 +407,9 @@ def config_prefer_upstream(args):
     pkgs = {}
     for spec in pref_specs:
         # Collect all the upstream compilers and versions for this package.
-        pkg = pkgs.get(spec.name, {"version": [], "compiler": []})
+        pkg = pkgs.get(spec.name, {"version": []})
+        all = pkgs.get("all", {"compiler": []})
+        pkgs["all"] = all
         pkgs[spec.name] = pkg
 
         # We have no existing variant if this is our first added version.
@@ -418,8 +420,8 @@ def config_prefer_upstream(args):
             pkg["version"].append(version)
 
         compiler = str(spec.compiler)
-        if compiler not in pkg["compiler"]:
-            pkg["compiler"].append(compiler)
+        if compiler not in all["compiler"]:
+            all["compiler"].append(compiler)
 
         # Get and list all the variants that differ from the default.
         variants = []
diff --git a/lib/spack/spack/schema/packages.py b/lib/spack/spack/schema/packages.py
--- a/lib/spack/spack/schema/packages.py
+++ b/lib/spack/spack/schema/packages.py
@@ -8,6 +8,66 @@
    :lines: 13-
 """
 
+permissions = {
+    "type": "object",
+    "additionalProperties": False,
+    "properties": {
+        "read": {"type": "string", "enum": ["user", "group", "world"]},
+        "write": {"type": "string", "enum": ["user", "group", "world"]},
+        "group": {"type": "string"},
+    },
+}
+
+variants = {"oneOf": [{"type": "string"}, {"type": "array", "items": {"type": "string"}}]}
+
+requirements = {
+    "oneOf": [
+        # 'require' can be a list of requirement_groups.
+        # each requirement group is a list of one or more
+        # specs. Either at least one or exactly one spec
+        # in the group must be satisfied (depending on
+        # whether you use "any_of" or "one_of",
+        # repectively)
+        {
+            "type": "array",
+            "items": {
+                "oneOf": [
+                    {
+                        "type": "object",
+                        "additionalProperties": False,
+                        "properties": {
+                            "one_of": {"type": "array", "items": {"type": "string"}},
+                            "any_of": {"type": "array", "items": {"type": "string"}},
+                            "spec": {"type": "string"},
+                            "message": {"type": "string"},
+                            "when": {"type": "string"},
+                        },
+                    },
+                    {"type": "string"},
+                ]
+            },
+        },
+        # Shorthand for a single requirement group with
+        # one member
+        {"type": "string"},
+    ]
+}
+
+permissions = {
+    "type": "object",
+    "additionalProperties": False,
+    "properties": {
+        "read": {"type": "string", "enum": ["user", "group", "world"]},
+        "write": {"type": "string", "enum": ["user", "group", "world"]},
+        "group": {"type": "string"},
+    },
+}
+
+package_attributes = {
+    "type": "object",
+    "additionalProperties": False,
+    "patternProperties": {r"\w+": {}},
+}
 
 #: Properties for inclusion in other schemas
 properties = {
@@ -15,57 +75,14 @@
         "type": "object",
         "default": {},
         "additionalProperties": False,
-        "patternProperties": {
-            r"\w[\w-]*": {  # package name
+        "properties": {
+            "all": {  # package name
                 "type": "object",
                 "default": {},
                 "additionalProperties": False,
                 "properties": {
-                    "require": {
-                        "oneOf": [
-                            # 'require' can be a list of requirement_groups.
-                            # each requirement group is a list of one or more
-                            # specs. Either at least one or exactly one spec
-                            # in the group must be satisfied (depending on
-                            # whether you use "any_of" or "one_of",
-                            # repectively)
-                            {
-                                "type": "array",
-                                "items": {
-                                    "oneOf": [
-                                        {
-                                            "type": "object",
-                                            "additionalProperties": False,
-                                            "properties": {
-                                                "one_of": {
-                                                    "type": "array",
-                                                    "items": {"type": "string"},
-                                                },
-                                                "any_of": {
-                                                    "type": "array",
-                                                    "items": {"type": "string"},
-                                                },
-                                                "spec": {"type": "string"},
-                                                "message": {"type": "string"},
-                                                "when": {"type": "string"},
-                                            },
-                                        },
-                                        {"type": "string"},
-                                    ]
-                                },
-                            },
-                            # Shorthand for a single requirement group with
-                            # one member
-                            {"type": "string"},
-                        ]
-                    },
-                    "version": {
-                        "type": "array",
-                        "default": [],
-                        # version strings (type should be string, number is still possible
-                        # but deprecated. this is to avoid issues with e.g. 3.10 -> 3.1)
-                        "items": {"anyOf": [{"type": "string"}, {"type": "number"}]},
-                    },
+                    "require": requirements,
+                    "version": {},  # Here only to warn users on ignored properties
                     "target": {
                         "type": "array",
                         "default": [],
@@ -78,22 +95,10 @@
                         "items": {"type": "string"},
                     },  # compiler specs
                     "buildable": {"type": "boolean", "default": True},
-                    "permissions": {
-                        "type": "object",
-                        "additionalProperties": False,
-                        "properties": {
-                            "read": {"type": "string", "enum": ["user", "group", "world"]},
-                            "write": {"type": "string", "enum": ["user", "group", "world"]},
-                            "group": {"type": "string"},
-                        },
-                    },
+                    "permissions": permissions,
                     # If 'get_full_repo' is promoted to a Package-level
                     # attribute, it could be useful to set it here
-                    "package_attributes": {
-                        "type": "object",
-                        "additionalProperties": False,
-                        "patternProperties": {r"\w+": {}},
-                    },
+                    "package_attributes": package_attributes,
                     "providers": {
                         "type": "object",
                         "default": {},
@@ -106,12 +111,40 @@
                             }
                         },
                     },
-                    "variants": {
-                        "oneOf": [
-                            {"type": "string"},
-                            {"type": "array", "items": {"type": "string"}},
-                        ]
+                    "variants": variants,
+                },
+                "deprecatedProperties": {
+                    "properties": ["version"],
+                    "message": "setting version preferences in the 'all' section of packages.yaml "
+                    "is deprecated and will be removed in v0.22\n\n\tThese preferences "
+                    "will be ignored by Spack. You can set them only in package specific sections "
+                    "of the same file.\n",
+                    "error": False,
+                },
+            }
+        },
+        "patternProperties": {
+            r"(?!^all$)(^\w[\w-]*)": {  # package name
+                "type": "object",
+                "default": {},
+                "additionalProperties": False,
+                "properties": {
+                    "require": requirements,
+                    "version": {
+                        "type": "array",
+                        "default": [],
+                        # version strings
+                        "items": {"anyOf": [{"type": "string"}, {"type": "number"}]},
                     },
+                    "target": {},  # Here only to warn users on ignored properties
+                    "compiler": {},  # Here only to warn users on ignored properties
+                    "buildable": {"type": "boolean", "default": True},
+                    "permissions": permissions,
+                    # If 'get_full_repo' is promoted to a Package-level
+                    # attribute, it could be useful to set it here
+                    "package_attributes": package_attributes,
+                    "providers": {},  # Here only to warn users on ignored properties
+                    "variants": variants,
                     "externals": {
                         "type": "array",
                         "items": {
@@ -127,6 +160,14 @@
                         },
                     },
                 },
+                "deprecatedProperties": {
+                    "properties": ["target", "compiler", "providers"],
+                    "message": "setting compiler, target or provider preferences in a package "
+                    "specific section of packages.yaml is deprecated, and will be removed in "
+                    "v0.22.\n\n\tThese preferences will be ignored by Spack. You "
+                    "can set them only in the 'all' section of the same file.\n",
+                    "error": False,
+                },
             }
         },
     }
diff --git a/lib/spack/spack/solver/asp.py b/lib/spack/spack/solver/asp.py
--- a/lib/spack/spack/solver/asp.py
+++ b/lib/spack/spack/solver/asp.py
@@ -1258,32 +1258,9 @@ def compiler_facts(self):
         matches = sorted(indexed_possible_compilers, key=lambda x: ppk(x[1].spec))
 
         for weight, (compiler_id, cspec) in enumerate(matches):
-            f = fn.default_compiler_preference(compiler_id, weight)
+            f = fn.compiler_weight(compiler_id, weight)
             self.gen.fact(f)
 
-    def package_compiler_defaults(self, pkg):
-        """Facts about packages' compiler prefs."""
-
-        packages = spack.config.get("packages")
-        pkg_prefs = packages.get(pkg.name)
-        if not pkg_prefs or "compiler" not in pkg_prefs:
-            return
-
-        compiler_list = self.possible_compilers
-        compiler_list = sorted(compiler_list, key=lambda x: (x.name, x.version), reverse=True)
-        ppk = spack.package_prefs.PackagePrefs(pkg.name, "compiler", all=False)
-        matches = sorted(compiler_list, key=lambda x: ppk(x.spec))
-
-        for i, compiler in enumerate(reversed(matches)):
-            self.gen.fact(
-                fn.pkg_fact(
-                    pkg.name,
-                    fn.node_compiler_preference(
-                        compiler.spec.name, compiler.spec.version, -i * 100
-                    ),
-                )
-            )
-
     def package_requirement_rules(self, pkg):
         rules = self.requirement_rules_from_package_py(pkg)
         rules.extend(self.requirement_rules_from_packages_yaml(pkg))
@@ -1375,9 +1352,6 @@ def pkg_rules(self, pkg, tests):
         # conflicts
         self.conflict_rules(pkg)
 
-        # default compilers for this package
-        self.package_compiler_defaults(pkg)
-
         # virtuals
         self.package_provider_rules(pkg)
 
@@ -1673,6 +1647,7 @@ def virtual_preferences(self, pkg_name, func):
             for i, provider in enumerate(providers):
                 provider_name = spack.spec.Spec(provider).name
                 func(vspec, provider_name, i)
+            self.gen.newline()
 
     def provider_defaults(self):
         self.gen.h2("Default virtual providers")
@@ -1865,8 +1840,8 @@ def preferred_variants(self, pkg_name):
                     fn.variant_default_value_from_packages_yaml(pkg_name, variant.name, value)
                 )
 
-    def target_preferences(self, pkg_name):
-        key_fn = spack.package_prefs.PackagePrefs(pkg_name, "target")
+    def target_preferences(self):
+        key_fn = spack.package_prefs.PackagePrefs("all", "target")
 
         if not self.target_specs_cache:
             self.target_specs_cache = [
@@ -1876,17 +1851,25 @@ def target_preferences(self, pkg_name):
 
         package_targets = self.target_specs_cache[:]
         package_targets.sort(key=key_fn)
-
-        offset = 0
-        best_default = self.default_targets[0][1]
         for i, preferred in enumerate(package_targets):
-            if str(preferred.architecture.target) == best_default and i != 0:
-                offset = 100
-            self.gen.fact(
-                fn.pkg_fact(
-                    pkg_name, fn.target_weight(str(preferred.architecture.target), i + offset)
-                )
-            )
+            self.gen.fact(fn.target_weight(str(preferred.architecture.target), i))
+
+    def flag_defaults(self):
+        self.gen.h2("Compiler flag defaults")
+
+        # types of flags that can be on specs
+        for flag in spack.spec.FlagMap.valid_compiler_flags():
+            self.gen.fact(fn.flag_type(flag))
+        self.gen.newline()
+
+        # flags from compilers.yaml
+        compilers = all_compilers_in_config()
+        for compiler in compilers:
+            for name, flags in compiler.flags.items():
+                for flag in flags:
+                    self.gen.fact(
+                        fn.compiler_version_flag(compiler.name, compiler.version, name, flag)
+                    )
 
     def spec_clauses(self, *args, **kwargs):
         """Wrap a call to `_spec_clauses()` into a try/except block that
@@ -2340,6 +2323,8 @@ def target_defaults(self, specs):
 
         self.default_targets = list(sorted(set(self.default_targets)))
 
+        self.target_preferences()
+
     def virtual_providers(self):
         self.gen.h2("Virtual providers")
         msg = (
@@ -2661,7 +2646,6 @@ def setup(
             self.pkg_rules(pkg, tests=self.tests)
             self.gen.h2("Package preferences: %s" % pkg)
             self.preferred_variants(pkg)
-            self.target_preferences(pkg)
 
         self.gen.h1("Develop specs")
         # Inject dev_path from environment


Test patch:
diff --git a/lib/spack/spack/test/cmd/config.py b/lib/spack/spack/test/cmd/config.py
--- a/lib/spack/spack/test/cmd/config.py
+++ b/lib/spack/spack/test/cmd/config.py
@@ -215,10 +215,10 @@ def test_config_add_override_leaf(mutable_empty_config):
 
 
 def test_config_add_update_dict(mutable_empty_config):
-    config("add", "packages:all:version:[1.0.0]")
+    config("add", "packages:hdf5:version:[1.0.0]")
     output = config("get", "packages")
 
-    expected = "packages:\n  all:\n    version: [1.0.0]\n"
+    expected = "packages:\n  hdf5:\n    version: [1.0.0]\n"
     assert output == expected
 
 
@@ -352,8 +352,7 @@ def test_config_add_update_dict_from_file(mutable_empty_config, tmpdir):
     contents = """spack:
   packages:
     all:
-      version:
-      - 1.0.0
+      target: [x86_64]
 """
 
     # create temp file and add it to config
@@ -368,8 +367,7 @@ def test_config_add_update_dict_from_file(mutable_empty_config, tmpdir):
     # added config comes before prior config
     expected = """packages:
   all:
-    version:
-    - 1.0.0
+    target: [x86_64]
     compiler: [gcc]
 """
 
@@ -381,7 +379,7 @@ def test_config_add_invalid_file_fails(tmpdir):
     # invalid because version requires a list
     contents = """spack:
   packages:
-    all:
+    hdf5:
       version: 1.0.0
 """
 
@@ -631,14 +629,11 @@ def test_config_prefer_upstream(
     packages = syaml.load(open(cfg_file))["packages"]
 
     # Make sure only the non-default variants are set.
-    assert packages["boost"] == {
-        "compiler": ["gcc@=10.2.1"],
-        "variants": "+debug +graph",
-        "version": ["1.63.0"],
-    }
-    assert packages["dependency-install"] == {"compiler": ["gcc@=10.2.1"], "version": ["2.0"]}
+    assert packages["all"] == {"compiler": ["gcc@=10.2.1"]}
+    assert packages["boost"] == {"variants": "+debug +graph", "version": ["1.63.0"]}
+    assert packages["dependency-install"] == {"version": ["2.0"]}
     # Ensure that neither variant gets listed for hdf5, since they conflict
-    assert packages["hdf5"] == {"compiler": ["gcc@=10.2.1"], "version": ["2.3"]}
+    assert packages["hdf5"] == {"version": ["2.3"]}
 
     # Make sure a message about the conflicting hdf5's was given.
     assert "- hdf5" in output
diff --git a/lib/spack/spack/test/cmd/env.py b/lib/spack/spack/test/cmd/env.py
--- a/lib/spack/spack/test/cmd/env.py
+++ b/lib/spack/spack/test/cmd/env.py
@@ -2621,7 +2621,7 @@ def test_env_write_only_non_default_nested(tmpdir):
   - matrix:
     - [mpileaks]
   packages:
-    mpileaks:
+    all:
       compiler: [gcc]
   view: true
 """
diff --git a/lib/spack/spack/test/concretize_preferences.py b/lib/spack/spack/test/concretize_preferences.py
--- a/lib/spack/spack/test/concretize_preferences.py
+++ b/lib/spack/spack/test/concretize_preferences.py
@@ -105,17 +105,13 @@ def test_preferred_variants_from_wildcard(self):
 
     @pytest.mark.parametrize(
         "compiler_str,spec_str",
-        [("gcc@4.5.0", "mpileaks"), ("clang@12.0.0", "mpileaks"), ("gcc@4.5.0", "openmpi")],
+        [("gcc@=4.5.0", "mpileaks"), ("clang@=12.0.0", "mpileaks"), ("gcc@=4.5.0", "openmpi")],
     )
     def test_preferred_compilers(self, compiler_str, spec_str):
         """Test preferred compilers are applied correctly"""
-        spec = Spec(spec_str)
-        update_packages(spec.name, "compiler", [compiler_str])
-        spec.concretize()
-        # note: lhs has concrete compiler version, rhs still abstract.
-        # Could be made more strict by checking for equality with `gcc@=4.5.0`
-        # etc.
-        assert spec.compiler.satisfies(CompilerSpec(compiler_str))
+        update_packages("all", "compiler", [compiler_str])
+        spec = spack.spec.Spec(spec_str).concretized()
+        assert spec.compiler == CompilerSpec(compiler_str)
 
     @pytest.mark.only_clingo("Use case not supported by the original concretizer")
     def test_preferred_target(self, mutable_mock_repo):
@@ -124,7 +120,7 @@ def test_preferred_target(self, mutable_mock_repo):
         default = str(spec.target)
         preferred = str(spec.target.family)
 
-        update_packages("mpich", "target", [preferred])
+        update_packages("all", "target", [preferred])
         spec = concretize("mpich")
         assert str(spec.target) == preferred
 
@@ -132,7 +128,7 @@ def test_preferred_target(self, mutable_mock_repo):
         assert str(spec["mpileaks"].target) == preferred
         assert str(spec["mpich"].target) == preferred
 
-        update_packages("mpileaks", "target", [default])
+        update_packages("all", "target", [default])
         spec = concretize("mpileaks")
         assert str(spec["mpileaks"].target) == default
         assert str(spec["mpich"].target) == default
diff --git a/lib/spack/spack/test/config.py b/lib/spack/spack/test/config.py
--- a/lib/spack/spack/test/config.py
+++ b/lib/spack/spack/test/config.py
@@ -78,7 +78,7 @@ def env_yaml(tmpdir):
         verify_ssl: False
         dirty: False
     packages:
-        libelf:
+        all:
             compiler: [ 'gcc@4.5.3' ]
     repos:
         - /x/y/z
@@ -942,7 +942,7 @@ def test_single_file_scope(config, env_yaml):
         # from the single-file config
         assert spack.config.get("config:verify_ssl") is False
         assert spack.config.get("config:dirty") is False
-        assert spack.config.get("packages:libelf:compiler") == ["gcc@4.5.3"]
+        assert spack.config.get("packages:all:compiler") == ["gcc@4.5.3"]
 
         # from the lower config scopes
         assert spack.config.get("config:checksum") is True
@@ -965,7 +965,7 @@ def test_single_file_scope_section_override(tmpdir, config):
     config:
         verify_ssl: False
     packages::
-        libelf:
+        all:
             compiler: [ 'gcc@4.5.3' ]
     repos:
         - /x/y/z
@@ -977,7 +977,7 @@ def test_single_file_scope_section_override(tmpdir, config):
     with spack.config.override(scope):
         # from the single-file config
         assert spack.config.get("config:verify_ssl") is False
-        assert spack.config.get("packages:libelf:compiler") == ["gcc@4.5.3"]
+        assert spack.config.get("packages:all:compiler") == ["gcc@4.5.3"]
 
         # from the lower config scopes
         assert spack.config.get("config:checksum") is True