Repository: spack/spack

Issue:
Clingo concretization picks wrong iconv provider
Hello Spack team,

I am facing an odd concretization behavior with the new concretizer. When I try to install `netcdf-c ~mpi` Spack is picking `libc` as the iconv provider. But the default variant for libc is `~iconv` so it does not provide the iconv libraries. As a result, the build for `netcdf-c~mpi` and a host of other packages fail.

In my packages.yaml, I have ONLY libiconv as the iconv provider:
```
# packages.yaml ....
      golang: [gcc]
      iconv: [libiconv]
      ...
```
But if I ask: `spack providers iconv`, I get:
```
iconv:
libc  libiconv
```
Oddly, the new concretizer picks libiconv as the provider when installing `netcdf-c+mpi`, but picks `libc~iconv` as the provider when installing `netcdf-c~mpi`.

**Expected Behavior:**
Clingo picks `libiconv` as the `iconv` provider in both cases.



Discussion:
Here are the full specs to help you identify the issue:
```
$ spack spec netcdf-c@4.7.4 +mpi ^hdf5@1.10.7+mpi ^openmpi %gcc@8.3.1
Input spec
--------------------------------
netcdf-c@4.7.4+mpi
    ^hdf5@1.10.7+mpi
    ^openmpi%gcc@8.3.1

Concretized
--------------------------------
netcdf-c@4.7.4%gcc@8.3.1~dap~fsync+hdf4~jna+mpi~parallel-netcdf+pic+shared arch=linux-centos8-zen
    ^hdf@4.2.15%gcc@8.3.1+external-xdr+fortran~java~netcdf+pic~shared+szip arch=linux-centos8-zen
        ^bison@3.7.6%gcc@8.3.1 arch=linux-centos8-zen
            ^diffutils@3.7%gcc@8.3.1 arch=linux-centos8-zen
                ^libiconv@1.16%gcc@8.3.1 arch=linux-centos8-zen
            ^m4@1.4.19%gcc@8.3.1+sigsegv arch=linux-centos8-zen
                ^libsigsegv@2.13%gcc@8.3.1 arch=linux-centos8-zen
            ^perl@5.32.0%gcc@8.3.1+cpanm+shared+threads arch=linux-centos8-zen
                ^berkeley-db@18.1.40%gcc@8.3.1+cxx~docs+stl patches=b231fcc4d5cff05e5c3a4814f6a5af0e9a966428dc2176540d2c05aff41de522 arch=linux-centos8-zen
                ^gdbm@1.19%gcc@8.3.1 arch=linux-centos8-zen
                    ^readline@8.1%gcc@8.3.1 arch=linux-centos8-zen
                        ^ncurses@6.2%gcc@8.3.1~symlinks+termlib abi=none arch=linux-centos8-zen
                            ^pkgconf@1.7.4%gcc@8.3.1 arch=linux-centos8-zen
        ^flex@2.6.3%gcc@8.3.1+lex~nls arch=linux-centos8-zen
            ^findutils@4.8.0%gcc@8.3.1 arch=linux-centos8-zen
        ^libjpeg-turbo@2.0.6%gcc@8.3.1 arch=linux-centos8-zen
            ^cmake@3.20.0%gcc@8.3.1~doc+ncurses+openssl+ownlibs~qt build_type=Release arch=linux-centos8-zen
                ^openssl@1.1.1g%gcc@8.3.1~docs+systemcerts arch=linux-centos8-zen
            ^nasm@2.15.05%gcc@8.3.1 arch=linux-centos8-zen
        ^libszip@2.1.1%gcc@8.3.1 arch=linux-centos8-zen
        ^libtirpc@1.2.6%gcc@8.3.1 arch=linux-centos8-zen
            ^krb5@1.18.2%gcc@8.3.1+shared patches=3d75052730690579484e65a5bf0f92f6c3b20d9c43a413862d087774f431d9e9 arch=linux-centos8-zen
        ^zlib@1.2.11%gcc@8.3.1+optimize+pic+shared arch=linux-centos8-zen
    ^hdf5@1.10.7%gcc@8.3.1+cxx~debug+fortran+hl~java+mpi+pic+shared+szip+threadsafe api=none arch=linux-centos8-zen
        ^numactl@2.0.14%gcc@8.3.1 patches=4e1d78cbbb85de625bad28705e748856033eaafab92a66dffd383a3d7e00cc94,62fc8a8bf7665a60e8f4c93ebbd535647cebf74198f7afafec4c085a8825c006 arch=linux-centos8-zen
            ^autoconf@2.69%gcc@8.3.1 arch=linux-centos8-zen
            ^automake@1.16.3%gcc@8.3.1 arch=linux-centos8-zen
            ^libtool@2.4.6%gcc@8.3.1 arch=linux-centos8-zen
        ^openmpi@4.0.5%gcc@8.3.1~atomics~cuda+cxx+cxx_exceptions+gpfs~internal-hwloc~java+legacylaunchers~lustre~memchecker+pmi~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=auto patches=60ce20bc14d98c572ef7883b9fcd254c3f232c2f3a13377480f96466169ac4c8 schedulers=slurm arch=linux-centos8-zen
            ^hwloc@2.4.1%gcc@8.3.1~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-centos8-zen
                ^libpciaccess@0.16%gcc@8.3.1 arch=linux-centos8-zen
                    ^util-macros@1.19.1%gcc@8.3.1 arch=linux-centos8-zen
                ^libxml2@2.9.10%gcc@8.3.1~python arch=linux-centos8-zen
                    ^xz@5.2.5%gcc@8.3.1~pic libs=shared,static arch=linux-centos8-zen
            ^libevent@2.1.12%gcc@8.3.1+openssl arch=linux-centos8-zen
            ^openssh@8.5p1%gcc@8.3.1 arch=linux-centos8-zen
                ^libedit@3.1-20210216%gcc@8.3.1 arch=linux-centos8-zen
            ^slurm@20-11-0-4%gcc@8.3.1~gtk~hdf5~hwloc~mariadb~pmix+readline~restd sysconfdir=PREFIX/etc arch=linux-centos8-zen

```
and the other one:
```
$ spack spec netcdf-c@4.7.4 ~mpi ^hdf5@1.10.7~mpi %gcc@8.3.1
Input spec
--------------------------------
netcdf-c@4.7.4~mpi
    ^hdf5@1.10.7%gcc@8.3.1~mpi

Concretized
--------------------------------
netcdf-c@4.7.4%gcc@8.3.1~dap~fsync+hdf4~jna~mpi~parallel-netcdf+pic+shared arch=linux-centos8-zen
    ^hdf@4.2.15%gcc@8.3.1+external-xdr+fortran~java~netcdf+pic~shared+szip arch=linux-centos8-zen
        ^bison@3.7.6%gcc@8.3.1 arch=linux-centos8-zen
            ^diffutils@3.7%gcc@8.3.1 arch=linux-centos8-zen
                ^libc@1.0%gcc@8.3.1~iconv~rpc arch=linux-centos8-zen
            ^m4@1.4.19%gcc@8.3.1+sigsegv arch=linux-centos8-zen
                ^libsigsegv@2.13%gcc@8.3.1 arch=linux-centos8-zen
            ^perl@5.32.0%gcc@8.3.1+cpanm+shared+threads arch=linux-centos8-zen
                ^berkeley-db@18.1.40%gcc@8.3.1+cxx~docs+stl patches=b231fcc4d5cff05e5c3a4814f6a5af0e9a966428dc2176540d2c05aff41de522 arch=linux-centos8-zen
                ^gdbm@1.19%gcc@8.3.1 arch=linux-centos8-zen
                    ^readline@8.1%gcc@8.3.1 arch=linux-centos8-zen
                        ^ncurses@6.2%gcc@8.3.1~symlinks+termlib abi=none arch=linux-centos8-zen
                            ^pkgconf@1.7.4%gcc@8.3.1 arch=linux-centos8-zen
        ^flex@2.6.4%gcc@8.3.1+lex~nls patches=09c22e5c6fef327d3e48eb23f0d610dcd3a35ab9207f12e0f875701c677978d3 arch=linux-centos8-zen
            ^autoconf@2.69%gcc@8.3.1 arch=linux-centos8-zen
            ^automake@1.16.3%gcc@8.3.1 arch=linux-centos8-zen
            ^findutils@4.8.0%gcc@8.3.1 arch=linux-centos8-zen
            ^gettext@0.21%gcc@8.3.1+bzip2+curses+git~libunistring+libxml2+tar+xz arch=linux-centos8-zen
                ^bzip2@1.0.8%gcc@8.3.1~debug~pic+shared arch=linux-centos8-zen
                ^libxml2@2.9.10%gcc@8.3.1~python arch=linux-centos8-zen
                    ^xz@5.2.5%gcc@8.3.1~pic libs=shared,static arch=linux-centos8-zen
                    ^zlib@1.2.11%gcc@8.3.1+optimize+pic+shared arch=linux-centos8-zen
                ^tar@1.34%gcc@8.3.1 arch=linux-centos8-zen
            ^help2man@1.47.16%gcc@8.3.1 arch=linux-centos8-zen
            ^libtool@2.4.6%gcc@8.3.1 arch=linux-centos8-zen
        ^libjpeg@9d%gcc@8.3.1 arch=linux-centos8-zen
        ^libszip@2.1.1%gcc@8.3.1 arch=linux-centos8-zen
    ^hdf5@1.10.7%gcc@8.3.1+cxx~debug+fortran+hl~java~mpi+pic+shared+szip+threadsafe api=none arch=linux-centos8-zen

```
I can't reproduce apparently. With latest `develop` I get:
```console
$ spack solve netcdf-c@4.7.4 ~mpi ^hdf5@1.10.7~mpi 
==> Best of 8 considered solutions.
==> Optimization Criteria:
  Priority  Criterion                                 Value
  1         deprecated versions used                      0
  2         version weight                                1
  3         number of non-default variants (roots)        0
  4         multi-valued variants                         0
  5         preferred providers for roots                 0
  6         number of non-default variants (non-roots)    0
  7         preferred providers (non-roots)               0
  8         compiler mismatches                           0
  9         version badness                               1
  10        count of non-root multi-valued variants       0
  11        non-preferred compilers                       0
  12        target mismatches                             0
  13        non-preferred targets                         0

netcdf-c@4.7.4%gcc@11.1.0~dap~fsync~hdf4~jna~mpi~parallel-netcdf+pic+shared arch=linux-ubuntu18.04-broadwell
    ^hdf5@1.10.7%gcc@11.1.0~cxx~debug~fortran+hl~java~mpi+pic+shared~szip~threadsafe api=none arch=linux-ubuntu18.04-broadwell
        ^zlib@1.2.11%gcc@11.1.0+optimize+pic+shared arch=linux-ubuntu18.04-broadwell
    ^m4@1.4.19%gcc@11.1.0+sigsegv arch=linux-ubuntu18.04-broadwell
        ^libsigsegv@2.13%gcc@11.1.0 arch=linux-ubuntu18.04-broadwell
```
@alalazo can you try this?
```
netcdf-c@4.7.4%gcc@8.3.1~dap~fsync+hdf4~jna~mpi~parallel-netcdf+pic+shared arch=linux-centos8-zen
    ^hdf@4.2.15%gcc@8.3.1+external-xdr+fortran~java~netcdf+pic~shared+szip arch=linux-centos8-zen
    ^hdf5@1.10.7%gcc@8.3.1+cxx~debug+fortran+hl~java~mpi+pic+shared+szip+threadsafe api=none arch=linux-centos8-zen
```
I have several variant preferences in packages.yaml.
i got this from `spack solve`
```
$ spack solve netcdf-c@4.7.4 ~mpi ^hdf5@1.10.7~mpi %gcc@8.3.1 
==> Best of 9 considered solutions.
==> Optimization Criteria:
  Priority  Criterion                                  Value
  1         deprecated versions used                       0
  2         version weight                                -2
  3         number of non-default variants (roots)         0
  4         multi-valued variants                          0
  5         preferred providers for roots                  0
  6         number of non-default variants (non-roots)     0
  7         number of non-default providers (non-roots)  141
  8         count of non-root multi-valued variants       -2
  9         compiler mismatches                            0
  10        version badness                               -5
  11        non-preferred compilers                       56
  12        target mismatches                              0
  13        non-preferred targets                         28

netcdf-c@4.7.4%gcc@8.3.1~dap~fsync+hdf4~jna~mpi~parallel-netcdf+pic+shared arch=linux-centos8-zen
    ^hdf@4.2.15%gcc@8.3.1+external-xdr+fortran~java~netcdf+pic~shared+szip arch=linux-centos8-zen
        ^bison@3.7.6%gcc@8.3.1 arch=linux-centos8-zen
            ^diffutils@3.7%gcc@8.3.1 arch=linux-centos8-zen
                ^libc@1.0%gcc@8.3.1~iconv~rpc arch=linux-centos8-zen
            ^m4@1.4.19%gcc@8.3.1+sigsegv arch=linux-centos8-zen
                ^libsigsegv@2.13%gcc@8.3.1 arch=linux-centos8-zen
            ^perl@5.32.0%gcc@8.3.1+cpanm+shared+threads arch=linux-centos8-zen
                ^berkeley-db@18.1.40%gcc@8.3.1+cxx~docs+stl patches=b231fcc4d5cff05e5c3a4814f6a5af0e9a966428dc2176540d2c05aff41de522 arch=linux-centos8-zen
                ^gdbm@1.19%gcc@8.3.1 arch=linux-centos8-zen
                    ^readline@8.1%gcc@8.3.1 arch=linux-centos8-zen
                        ^ncurses@6.2%gcc@8.3.1~symlinks+termlib abi=none arch=linux-centos8-zen
                            ^pkgconf@1.7.4%gcc@8.3.1 arch=linux-centos8-zen
        ^flex@2.6.4%gcc@8.3.1+lex~nls patches=09c22e5c6fef327d3e48eb23f0d610dcd3a35ab9207f12e0f875701c677978d3 arch=linux-centos8-zen
            ^autoconf@2.69%gcc@8.3.1 arch=linux-centos8-zen
            ^automake@1.16.3%gcc@8.3.1 arch=linux-centos8-zen
            ^findutils@4.8.0%gcc@8.3.1 arch=linux-centos8-zen
            ^gettext@0.21%gcc@8.3.1+bzip2+curses+git~libunistring+libxml2+tar+xz arch=linux-centos8-zen
                ^bzip2@1.0.8%gcc@8.3.1~debug~pic+shared arch=linux-centos8-zen
                ^libxml2@2.9.10%gcc@8.3.1~python arch=linux-centos8-zen
                    ^xz@5.2.5%gcc@8.3.1~pic libs=shared,static arch=linux-centos8-zen
                    ^zlib@1.2.11%gcc@8.3.1+optimize+pic+shared arch=linux-centos8-zen
                ^tar@1.34%gcc@8.3.1 arch=linux-centos8-zen
            ^help2man@1.47.16%gcc@8.3.1 arch=linux-centos8-zen
            ^libtool@2.4.6%gcc@8.3.1 arch=linux-centos8-zen
        ^libjpeg@9d%gcc@8.3.1 arch=linux-centos8-zen
        ^libszip@2.1.1%gcc@8.3.1 arch=linux-centos8-zen
    ^hdf5@1.10.7%gcc@8.3.1+cxx~debug+fortran+hl~java~mpi+pic+shared+szip+threadsafe api=none arch=linux-centos8-zen
```
Can you post the output of:
```console
$ spack debug report
```
?
```
$ spack debug report
* **Spack:** 0.16.2-3033-986bcef160
* **Python:** 3.6.8
* **Platform:** linux-centos8-zen2
* **Concretizer:** clingo
```
The commit you are using is from Sunday apparently. Can you retry with the latest `develop` commit? There might be chances this has been solved today. 
Hi @alalazo 
I did a git pull. Now both `netcdf-c+mpi` and `netcdf-c~mpi` are picking `libc~iconv` as the iconv provider.

Seems like it is ignoring `packages.yaml`?

Here is a test with a smaller spec. Notice the bison spec changed with hdf
```
[amaji@login00 20210608]$ spack spec bison %gcc@8.3.1 
Input spec
--------------------------------
bison%gcc@8.3.1

Concretized
--------------------------------
bison@3.7.6%gcc@8.3.1 arch=linux-centos8-zen
    ^diffutils@3.7%gcc@8.3.1 arch=linux-centos8-zen
        ^libiconv@1.16%gcc@8.3.1 arch=linux-centos8-zen
    ^m4@1.4.19%gcc@8.3.1+sigsegv arch=linux-centos8-zen
        ^libsigsegv@2.13%gcc@8.3.1 arch=linux-centos8-zen
    ^perl@5.32.0%gcc@8.3.1+cpanm+shared+threads arch=linux-centos8-zen
        ^berkeley-db@18.1.40%gcc@8.3.1+cxx~docs+stl patches=b231fcc4d5cff05e5c3a4814f6a5af0e9a966428dc2176540d2c05aff41de522 arch=linux-centos8-zen
        ^gdbm@1.19%gcc@8.3.1 arch=linux-centos8-zen
            ^readline@8.1%gcc@8.3.1 arch=linux-centos8-zen
                ^ncurses@6.2%gcc@8.3.1~symlinks+termlib abi=none arch=linux-centos8-zen
                    ^pkgconf@1.7.4%gcc@8.3.1 arch=linux-centos8-zen

[amaji@login00 20210608]$ spack spec hdf %gcc@8.3.1 
Input spec
--------------------------------
hdf%gcc@8.3.1

Concretized
--------------------------------
hdf@4.2.15%gcc@8.3.1+external-xdr+fortran~java~netcdf+pic~shared+szip arch=linux-centos8-zen
    ^bison@3.7.6%gcc@8.3.1 arch=linux-centos8-zen
        ^diffutils@3.7%gcc@8.3.1 arch=linux-centos8-zen
            ^libc@1.0%gcc@8.3.1~iconv~rpc arch=linux-centos8-zen
        ^m4@1.4.19%gcc@8.3.1+sigsegv arch=linux-centos8-zen
            ^libsigsegv@2.13%gcc@8.3.1 arch=linux-centos8-zen
        ^perl@5.32.0%gcc@8.3.1+cpanm+shared+threads arch=linux-centos8-zen
            ^berkeley-db@18.1.40%gcc@8.3.1+cxx~docs+stl patches=b231fcc4d5cff05e5c3a4814f6a5af0e9a966428dc2176540d2c05aff41de522 arch=linux-centos8-zen
            ^gdbm@1.19%gcc@8.3.1 arch=linux-centos8-zen
                ^readline@8.1%gcc@8.3.1 arch=linux-centos8-zen
                    ^ncurses@6.2%gcc@8.3.1~symlinks+termlib abi=none arch=linux-centos8-zen
                        ^pkgconf@1.7.4%gcc@8.3.1 arch=linux-centos8-zen
    ^flex@2.6.3%gcc@8.3.1+lex~nls arch=linux-centos8-zen
        ^findutils@4.8.0%gcc@8.3.1 arch=linux-centos8-zen
    ^libjpeg@9d%gcc@8.3.1 arch=linux-centos8-zen
    ^libszip@2.1.1%gcc@8.3.1 arch=linux-centos8-zen
    ^zlib@1.2.11%gcc@8.3.1+optimize+pic+shared arch=linux-centos8-zen

```
We need to narrow down what triggers the issue, I still get the correct behavior on my platform:
```console
$ spack solve hdf
==> Best of 10 considered solutions.
==> Optimization Criteria:
  Priority  Criterion                                 Value
  1         deprecated versions used                      0
  2         version weight                                0
  3         number of non-default variants (roots)        0
  4         multi-valued variants                         0
  5         preferred providers for roots                 0
  6         number of non-default variants (non-roots)    0
  7         preferred providers (non-roots)              50
  8         compiler mismatches                           0
  9         version badness                               0
  10        count of non-root multi-valued variants       0
  11        non-preferred compilers                       0
  12        target mismatches                             0
  13        non-preferred targets                         0

hdf@4.2.15%gcc@11.1.0+external-xdr~fortran~java~netcdf+pic~shared~szip arch=linux-ubuntu18.04-broadwell
    ^bison@3.7.6%gcc@11.1.0 arch=linux-ubuntu18.04-broadwell
        ^diffutils@3.7%gcc@11.1.0 arch=linux-ubuntu18.04-broadwell
            ^libiconv@1.16%gcc@11.1.0 arch=linux-ubuntu18.04-broadwell
[ ... ]
```
I just have default values in `packages.yaml`.
@alalazo Think I found a way to reproduce. My packages.yaml
```
#
# Per-spack-instance settings (overrides defaults):
#   $SPACK_ROOT/etc/spack/packages.yaml
#
# Per-user settings (overrides default and site settings):
#   ~/.spack/packages.yaml
# -------------------------------------------------------------------------
packages:
  openssl:
    externals:
    - spec: openssl@1.1.1g
      prefix: /usr
    buildable: false
  all:
    compiler: [gcc, intel, pgi, clang, xl, nag, fj, aocc]
    providers:
      awk: [gawk]
      blas: [openblas, amdblis]
      D: [ldc]
      daal: [intel-daal]
      elf: [elfutils]
      fftw-api: [fftw, amdfftw]
      flame: [libflame, amdlibflame]
      fuse: [libfuse]
      gl: [mesa+opengl, mesa18, opengl]
      glu: [mesa-glu, openglu]
      glx: [mesa+glx, mesa18+glx, opengl]
      golang: [gcc]
      iconv: [libiconv]
      ipp: [intel-ipp]
      java: [openjdk, jdk, ibm-java]
      jpeg: [libjpeg-turbo, libjpeg]
      lapack: [openblas, amdlibflame]
      mariadb-client: [mariadb-c-client, mariadb]
      mkl: [intel-mkl]
      mpe: [mpe2]
      mpi: [openmpi, mpich]
      mysql-client: [mysql, mariadb-c-client]
      opencl: [pocl]
      osmesa: [mesa+osmesa, mesa18+osmesa]
      pil: [py-pillow]
      pkgconfig: [pkgconf, pkg-config]
      rpc: [libtirpc]
      scalapack: [netlib-scalapack, amdscalapack]
      sycl: [hipsycl]
      szip: [libszip, libaec]
      tbb: [intel-tbb]
      unwind: [libunwind]
      uuid: [util-linux-uuid, libuuid]
      xxd: [xxd-standalone, vim]
      yacc: [bison, byacc]
      ziglang: [zig]
    permissions:
      read: world
      write: user
```

Now try: `spack solve hdf@4.2.15%gcc@8.3.1`. Please let me know if you can reproduce.
This is what I got on a fresh clone of Spack with that packages.yaml
```
[amaji@login01 spack]$ spack solve diffutils%gcc@8.3.1
==> Best of 5 considered solutions.
==> Optimization Criteria:
  Priority  Criterion                                 Value
  1         deprecated versions used                      0
  2         version weight                                0
  3         number of non-default variants (roots)        0
  4         multi-valued variants                         0
  5         preferred providers for roots                 0
  6         number of non-default variants (non-roots)    0
  7         preferred providers (non-roots)              10
  8         compiler mismatches                           0
  9         version badness                               0
  10        count of non-root multi-valued variants       0
  11        non-preferred compilers                       0
  12        target mismatches                             0
  13        non-preferred targets                         0

diffutils@3.7%gcc@8.3.1 arch=linux-centos8-zen
    ^libiconv@1.16%gcc@8.3.1 arch=linux-centos8-zen
[amaji@login01 spack]$ spack solve hdf@4.2.15%gcc@8.3.1
==> Best of 7 considered solutions.
==> Optimization Criteria:
  Priority  Criterion                                 Value
  1         deprecated versions used                      0
  2         version weight                                0
  3         number of non-default variants (roots)        0
  4         multi-valued variants                         0
  5         preferred providers for roots                 0
  6         number of non-default variants (non-roots)    0
  7         preferred providers (non-roots)             131
  8         compiler mismatches                           0
  9         version badness                               0
  10        count of non-root multi-valued variants       0
  11        non-preferred compilers                       0
  12        target mismatches                             0
  13        non-preferred targets                         0

hdf@4.2.15%gcc@8.3.1+external-xdr~fortran~java~netcdf+pic~shared~szip arch=linux-centos8-zen
    ^bison@3.7.6%gcc@8.3.1 arch=linux-centos8-zen
        ^diffutils@3.7%gcc@8.3.1 arch=linux-centos8-zen
            ^libc@1.0%gcc@8.3.1~iconv~rpc arch=linux-centos8-zen
        ^m4@1.4.19%gcc@8.3.1+sigsegv arch=linux-centos8-zen
            ^libsigsegv@2.13%gcc@8.3.1 arch=linux-centos8-zen
        ^perl@5.34.0%gcc@8.3.1+cpanm+shared+threads arch=linux-centos8-zen
            ^berkeley-db@18.1.40%gcc@8.3.1+cxx~docs+stl patches=b231fcc4d5cff05e5c3a4814f6a5af0e9a966428dc2176540d2c05aff41de522 arch=linux-centos8-zen
            ^gdbm@1.19%gcc@8.3.1 arch=linux-centos8-zen
                ^readline@8.1%gcc@8.3.1 arch=linux-centos8-zen
                    ^ncurses@6.2%gcc@8.3.1~symlinks+termlib abi=none arch=linux-centos8-zen
                        ^pkgconf@1.7.4%gcc@8.3.1 arch=linux-centos8-zen
    ^flex@2.6.3%gcc@8.3.1+lex~nls arch=linux-centos8-zen
        ^findutils@4.8.0%gcc@8.3.1 arch=linux-centos8-zen
    ^libjpeg@9d%gcc@8.3.1 arch=linux-centos8-zen
    ^zlib@1.2.11%gcc@8.3.1+optimize+pic+shared arch=linux-centos8-zen
```
**Stranger Things 2:** Removing `buildable: False` from packages.yaml seems to give correct concretization.
@amaji I think I got what's happening and, if I am correct, it's "by design". I don't know if there's any easy way to detect these kind of things and prompt a message to users. Anyhow, the longer diagnosis:
1. The ASP-based solver tries as much as possible to avoid choosing deprecated versions. That's the highest priority when searching for an optimal solution.
2. The version of `openssl` you have among externals is marked as deprecated AND not buildable. This means that if `openssl` has to be used in this spec, it should be a deprecated version.
3. The solver then tries not to use `openssl` at all in the spec which rules out `libtirps` as a provider for `rpc` needed by `hdf`. The only other possible provider for `rpc` is `libc`.
4. Since `libc` also provides `iconv` it is then preferable for the solver to use it for both virtual dependencies

Once you say that `openssl` is buildable, then the solver is allowed to build a new, non-deprecated, version of `openssl` and things get back to what you expect. Let me know if this makes sense to you.
Maybe a good way to warn users would be to add a check for deprecated versions in external packages in #23053 
Hi @alalazo,
Thank you for explaining so nicely. I think I understand the root cause now. 
But I have two concerns about this concretization heuristics.

1. Sometimes, packages are in externals not by choice but by necessity. For example, I have openssl, slurm, rdma-core, cuda in externals and these are not updated often. So, if I don't use `buildable: false`, the ASP solver will automatically build a new version (non-deprecated) which may not work on the system. The openssl in question is provided by CentOS8 and the reason for choosing this version is here: https://spack.readthedocs.io/en/latest/getting_started.html#openssl
I think a better solution may be to add back priorities to deprecated versions, if they are in externals.
2. The alternative chosen here is not a provider. The default spec for `libc` is `~iconv~rpc`. So spack should be more restrictive about providers by looking at the variants?
For point 1 I need to think about it. I don't like that we treat deprecated versions differently for externals, at least if we keep everything else as is. A workaround would be to declare a custom version in your `packages.yaml` that doesn't match any of the deprecated versions in `openssl/package.py`. For instance `openssl@centos8` does the trick.

For point 2 I need to look more closely at reproducing your exact behavior, since it seems I overlooked the 2 variant values on `libc`. It may be that we have some bug lurking there (the only dependency of `diffutils` is on `iconv` so there's something strange happening here).

`openssl@centos8` does seem to concretize. Thanks for the workaround! I'll check if it actually installs. There is no checksum for `openssl@centos8` in the package.py, so `spack install` may just error out.

Maybe in future we can have a config flag like `ignore_deprecated_versions` which would suppress this behavior by the concretizer?

I think I'll just edit `openssl/package.py` to remove the deprecated tag for that version in my packages for now.

Big thanks to @alalazo for being so prompt and responsive! :) 
Another example of bug 2. The concretizer thinks `intel-parallel-studio~mpi` is the mpi provider.
```
$ spack spec netcdf-c@4.7.4 +mpi ^hdf5@1.10.7+mpi
Input spec
--------------------------------
netcdf-c@4.7.4+mpi
    ^hdf5@1.10.7+mpi

Concretized
--------------------------------
netcdf-c@4.7.4%intel@19.0.5~dap~fsync+hdf4~jna+mpi~parallel-netcdf+pic+shared arch=linux-centos8-zen2
    ^hdf@4.2.15%intel@19.0.5+external-xdr+fortran~java~netcdf+pic~shared+szip arch=linux-centos8-zen2
        ^bison@3.7.6%intel@19.0.5 arch=linux-centos8-zen2
            ^diffutils@3.7%intel@19.0.5 arch=linux-centos8-zen2
                ^libc@1.0%intel@19.0.5~iconv~rpc arch=linux-centos8-zen2
            ^m4@1.4.19%intel@19.0.5+sigsegv arch=linux-centos8-zen2
                ^libsigsegv@2.13%intel@19.0.5 arch=linux-centos8-zen2
            ^perl@5.32.0%intel@19.0.5+cpanm+shared+threads arch=linux-centos8-zen2
                ^berkeley-db@18.1.40%intel@19.0.5+cxx~docs+stl patches=b231fcc4d5cff05e5c3a4814f6a5af0e9a966428dc2176540d2c05aff41de522 arch=linux-centos8-zen2
                ^gdbm@1.19%intel@19.0.5 arch=linux-centos8-zen2
                    ^readline@8.1%intel@19.0.5 arch=linux-centos8-zen2
                        ^ncurses@6.2%intel@19.0.5~symlinks+termlib abi=none arch=linux-centos8-zen2
                            ^pkgconf@1.7.4%intel@19.0.5 arch=linux-centos8-zen2
        ^flex@2.6.3%intel@19.0.5+lex~nls arch=linux-centos8-zen2
            ^findutils@4.8.0%intel@19.0.5 arch=linux-centos8-zen2
        ^libjpeg@9d%intel@19.0.5 arch=linux-centos8-zen2
        ^libszip@2.1.1%intel@19.0.5 arch=linux-centos8-zen2
        ^zlib@1.2.11%intel@19.0.5+optimize+pic+shared arch=linux-centos8-zen2
    ^hdf5@1.10.7%intel@19.0.5+cxx~debug+fortran+hl~java+mpi+pic+shared+szip+threadsafe api=none arch=linux-centos8-zen2
        ^intel-parallel-studio@cluster.2019.5%intel@19.0.5+advisor+clck+daal+gdb~ilp64+inspector+ipp+itac+mkl~mpi~newdtags+rpath+shared+tbb+vtune auto_dispatch=none threads=openmp arch=linux-centos8-zen2
        ^numactl@2.0.14%intel@19.0.5 patches=4e1d78cbbb85de625bad28705e748856033eaafab92a66dffd383a3d7e00cc94,62fc8a8bf7665a60e8f4c93ebbd535647cebf74198f7afafec4c085a8825c006 arch=linux-centos8-zen2
            ^autoconf@2.69%intel@19.0.5 arch=linux-centos8-zen2
            ^automake@1.16.3%intel@19.0.5 arch=linux-centos8-zen2
            ^libtool@2.4.6%intel@19.0.5 arch=linux-centos8-zen2
```

Hi @alalazo,

I need your advice with another issue. I am trying to install `qt+gtk+opengl ^python@3.9.5`, but somehow this is choosing `mesa18` as the glx provider (instead of `mesa`). Since `mesa18` is old, it does not build with `python@3.9.5`. 

I am wondering why is the ASP solver choosing mesa18 ahead of mesa. This does not change even if I remove `mesa18` as a glx provider in my `packages.yaml`.

1. Are provider preferences in packages.yaml ignored by the ASP solver? I am noticing many instances where editing packages.yaml has no impact on providers.
2. How can I force the solver to use mesa as my glx provider? I am looking for a more elegant solution than explicitly specifying `^mesa`.
3. Is there a way for specifying "don't use this as a provider"?
Shortest example:
```
$ spack solve libepoxy ^python@3.9.5
==> Best of 8 considered solutions.
==> Optimization Criteria:
  Priority  Criterion                                 Value
  1         deprecated versions used                      0
  2         version weight                                0
  3         number of non-default variants (roots)        0
  4         multi-valued variants                         0
  5         preferred providers for roots                 0
  6         number of non-default variants (non-roots)    0
  7         preferred providers (non-roots)              51
  8         compiler mismatches                           0
  9         version badness                               5
  10        count of non-root multi-valued variants      -6
  11        non-preferred compilers                       0
  12        target mismatches                             0
  13        non-preferred targets                         0

libepoxy@1.4.3%gcc@8.3.1+glx arch=linux-centos8-zen
    ^libx11@1.7.0%gcc@8.3.1 arch=linux-centos8-zen
        ^inputproto@2.3.2%gcc@8.3.1 arch=linux-centos8-zen
            ^pkgconf@1.7.4%gcc@8.3.1 arch=linux-centos8-zen
            ^util-macros@1.19.1%gcc@8.3.1 arch=linux-centos8-zen
        ^kbproto@1.0.7%gcc@8.3.1 arch=linux-centos8-zen
        ^libxcb@1.14%gcc@8.3.1 arch=linux-centos8-zen
            ^libpthread-stubs@0.4%gcc@8.3.1 arch=linux-centos8-zen
            ^libxau@1.0.8%gcc@8.3.1 arch=linux-centos8-zen
                ^xproto@7.0.31%gcc@8.3.1 arch=linux-centos8-zen
            ^libxdmcp@1.1.2%gcc@8.3.1 arch=linux-centos8-zen
                ^libbsd@0.11.3%gcc@8.3.1 arch=linux-centos8-zen
                    ^libmd@1.0.3%gcc@8.3.1 arch=linux-centos8-zen
            ^xcb-proto@1.14.1%gcc@8.3.1 arch=linux-centos8-zen
        ^perl@5.34.0%gcc@8.3.1+cpanm+shared+threads arch=linux-centos8-zen
            ^berkeley-db@18.1.40%gcc@8.3.1+cxx~docs+stl patches=b231fcc4d5cff05e5c3a4814f6a5af0e9a966428dc2176540d2c05aff41de522 arch=linux-centos8-zen
            ^gdbm@1.19%gcc@8.3.1 arch=linux-centos8-zen
                ^readline@8.1%gcc@8.3.1 arch=linux-centos8-zen
                    ^ncurses@6.2%gcc@8.3.1~symlinks+termlib abi=none arch=linux-centos8-zen
        ^xextproto@7.3.0%gcc@8.3.1 arch=linux-centos8-zen
        ^xtrans@1.3.5%gcc@8.3.1 arch=linux-centos8-zen
    ^mesa18@18.3.6%gcc@8.3.1+glx+llvm~opengles+osmesa patches=55a5611ca9fcbe8324c4d68a07b338134954ff12c5b122dc78ff376f012a1414 swr=none arch=linux-centos8-zen
        ^autoconf@2.69%gcc@8.3.1 arch=linux-centos8-zen
            ^m4@1.4.19%gcc@8.3.1+sigsegv arch=linux-centos8-zen
                ^libsigsegv@2.13%gcc@8.3.1 arch=linux-centos8-zen
        ^automake@1.16.3%gcc@8.3.1 arch=linux-centos8-zen
        ^binutils@2.36.1%gcc@8.3.1~gas+gold~headers~interwork+ld~libiberty~lto+nls+plugins libs=shared,static arch=linux-centos8-zen
            ^diffutils@3.7%gcc@8.3.1 arch=linux-centos8-zen
                ^libiconv@1.16%gcc@8.3.1 arch=linux-centos8-zen
            ^gettext@0.21%gcc@8.3.1+bzip2+curses+git~libunistring+libxml2+tar+xz arch=linux-centos8-zen
                ^bzip2@1.0.8%gcc@8.3.1~debug~pic+shared arch=linux-centos8-zen
                ^libxml2@2.9.10%gcc@8.3.1~python arch=linux-centos8-zen
                    ^xz@5.2.5%gcc@8.3.1~pic libs=shared,static arch=linux-centos8-zen
                    ^zlib@1.2.11%gcc@8.3.1+optimize+pic+shared arch=linux-centos8-zen
                ^tar@1.34%gcc@8.3.1 arch=linux-centos8-zen
        ^bison@3.7.6%gcc@8.3.1 arch=linux-centos8-zen
        ^expat@2.3.0%gcc@8.3.1+libbsd arch=linux-centos8-zen
        ^flex@2.6.3%gcc@8.3.1+lex~nls arch=linux-centos8-zen
            ^findutils@4.8.0%gcc@8.3.1 arch=linux-centos8-zen
        ^glproto@1.4.17%gcc@8.3.1 arch=linux-centos8-zen
        ^libtool@2.4.6%gcc@8.3.1 arch=linux-centos8-zen
        ^libxext@1.3.3%gcc@8.3.1 arch=linux-centos8-zen
        ^llvm@10.0.1%gcc@8.3.1~all_targets+clang~code_signing+compiler-rt~cuda~flang+gold+internal_unwind~ipo+libcxx+lld+lldb~llvm_dylib~mlir~omp_debug~omp_tsan+polly~python~shared_libs~split_dwarf build_type=Release cuda_arch=none patches=47284669559b98dd068b193b72a48add09112fcb90a9bfe5c609b79628a20d2b arch=linux-centos8-zen
            ^cmake@3.20.3%gcc@8.3.1~doc+ncurses+openssl+ownlibs~qt build_type=Release arch=linux-centos8-zen
                ^openssl@1.1.1k%gcc@8.3.1~docs+systemcerts arch=linux-centos8-zen
            ^hwloc@2.4.1%gcc@8.3.1~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-centos8-zen
                ^libpciaccess@0.16%gcc@8.3.1 arch=linux-centos8-zen
            ^libedit@3.1-20210216%gcc@8.3.1 arch=linux-centos8-zen
            ^perl-data-dumper@2.173%gcc@8.3.1 arch=linux-centos8-zen
            ^python@3.9.5%gcc@8.3.1+bz2+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib patches=0d98e93189bc278fbc37a50ed7f183bd8aaf249a8e1670a465f0db6bb4f8cf87 arch=linux-centos8-zen
                ^libffi@3.3%gcc@8.3.1 patches=26f26c6f29a7ce9bf370ad3ab2610f99365b4bdd7b82e7c31df41a3370d685c0 arch=linux-centos8-zen
                ^sqlite@3.35.5%gcc@8.3.1+column_metadata+fts~functions~rtree arch=linux-centos8-zen
                ^util-linux-uuid@2.36.2%gcc@8.3.1 arch=linux-centos8-zen
            ^swig@4.0.2%gcc@8.3.1 arch=linux-centos8-zen
                ^pcre@8.44%gcc@8.3.1~jit+multibyte+utf arch=linux-centos8-zen
            ^z3@4.8.9%gcc@8.3.1+python arch=linux-centos8-zen
                ^py-setuptools@50.3.2%gcc@8.3.1 arch=linux-centos8-zen
        ^py-mako@1.1.4%gcc@8.3.1 arch=linux-centos8-zen
            ^py-markupsafe@1.1.1%gcc@8.3.1 arch=linux-centos8-zen
```
I am using default `packages.yaml`.
@amaji I'll try to have a look at these two issues going to Monday.
For what is worth, I'm looking into a subtle bug here. This diff seems to resolve this issue:
```diff
diff --git a/lib/spack/spack/solver/concretize.lp b/lib/spack/spack/solver/concretize.lp
index c7d4c1ae44..ca828862a5 100644
--- a/lib/spack/spack/solver/concretize.lp
+++ b/lib/spack/spack/solver/concretize.lp
@@ -145,7 +145,7 @@ path(Parent, Descendant) :- path(Parent, A), depends_on(A, Descendant).
 % provider for that virtual then it depends on the provider
 depends_on(Package, Provider, Type)
   :- dependency_holds(Package, Virtual, Type),
-     provides_virtual(Provider, Virtual),
+     provider(Provider, Virtual),
      not external(Package).
 
 % dependencies on virtuals also imply that the virtual is a virtual node
@@ -154,13 +154,12 @@ virtual_node(Virtual)
      virtual(Virtual), not external(Package).
 
 % if there's a virtual node, we must select one provider
-1 { provides_virtual(Package, Virtual) : possible_provider(Package, Virtual) } 1
+1 { provider(Package, Virtual) : possible_provider(Package, Virtual) } 1
  :- virtual_node(Virtual).
 
 % virtual roots imply virtual nodes, and that one provider is a root
 virtual_node(Virtual) :- virtual_root(Virtual).
-1 { root(Package) : provides_virtual(Package, Virtual) } 1
-  :- virtual_root(Virtual).
+root(Package) :- virtual_root(Virtual), provider(Package, Virtual).
 
 % The provider provides the virtual if some provider condition holds.
 provides_virtual(Provider, Virtual) :-
@@ -168,12 +167,10 @@ provides_virtual(Provider, Virtual) :-
    condition_holds(ID),
    virtual(Virtual).
 
-% a node that provides a virtual is a provider
-provider(Package, Virtual)
-  :- node(Package), provides_virtual(Package, Virtual).
+:- provider(Package, Virtual), not provides_virtual(Package, Virtual).
 
-% for any virtual, there can be at most one provider in the DAG
-0 { node(Package) : provides_virtual(Package, Virtual) } 1 :- virtual(Virtual).
+:- provides_virtual(Package, V1), provides_virtual(Package, V2), V1 != V2,
+   provider(Package, V1), not provider(Package, V2), virtual_node(V1), virtual_node(V2).
 
 #defined possible_provider/2.
```
but unit tests are still failing on an edge case. Will continue working on it tomorrow.

Resolving patch:
diff --git a/var/spack/repos/builtin.mock/packages/unsat-provider/package.py b/var/spack/repos/builtin.mock/packages/unsat-provider/package.py
new file mode 100644
--- /dev/null
+++ b/var/spack/repos/builtin.mock/packages/unsat-provider/package.py
@@ -0,0 +1,15 @@
+# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+class UnsatProvider(Package):
+    """This package has a dependency on a virtual that cannot be provided"""
+    homepage = "http://www.example.com"
+    url = "http://www.example.com/v1.0.tgz"
+
+    version('1.0', sha256='foobarbaz')
+
+    variant('foo', default=True, description='')
+
+    provides('unsatvdep', when='+foo')
+    conflicts('+foo')
diff --git a/var/spack/repos/builtin.mock/packages/unsat-virtual-dependency/package.py b/var/spack/repos/builtin.mock/packages/unsat-virtual-dependency/package.py
new file mode 100644
--- /dev/null
+++ b/var/spack/repos/builtin.mock/packages/unsat-virtual-dependency/package.py
@@ -0,0 +1,12 @@
+# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+class UnsatVirtualDependency(Package):
+    """This package has a dependency on a virtual that cannot be provided"""
+    homepage = "http://www.example.com"
+    url = "http://www.example.com/v1.0.tgz"
+
+    version('1.0', sha256='foobarbaz')
+
+    depends_on('unsatvdep')


Test patch:
diff --git a/lib/spack/spack/test/concretize.py b/lib/spack/spack/test/concretize.py
--- a/lib/spack/spack/test/concretize.py
+++ b/lib/spack/spack/test/concretize.py
@@ -1253,3 +1253,11 @@ def test_version_badness_more_important_than_default_mv_variants(self):
         # criteria.
         s = spack.spec.Spec('root').concretized()
         assert s['gmt'].satisfies('@2.0')
+
+    @pytest.mark.regression('24205')
+    def test_provider_must_meet_requirements(self):
+        # A package can be a provider of a virtual only if the underlying
+        # requirements are met.
+        s = spack.spec.Spec('unsat-virtual-dependency')
+        with pytest.raises((RuntimeError, spack.error.UnsatisfiableSpecError)):
+            s.concretize()