Repository: spack/spack

Issue:
Error: Invalid Spec DAG
<!-- Explain, in a clear and concise way, the command you ran and the result you were trying to achieve.
Example: "I ran `spack find` to list all the installed packages and ..." -->

I ran 
```console
$ spack spec -I lammps ~asphere~body~class2~colloid+compress~coreshell+cuda~cuda_mps~dipole~exceptions~ffmpeg~granular~ipo~jpeg~kokkos+kspace~latte+lib+manybody+mc~meam+misc+molecule+mpi~mpiio~opencl+openmp+opt~peri~png~poems~python~qeq+replica+rigid~shock~snap~spin+srd+user-atc~user-h5md~user-lb~user-meamc+user-misc~user-netcdf+user-omp~user-reaxc~voronoi~kim target=zen2 ^cmake /ondqobr ^cuda /vtzojeg ^fftw /fj2ea2u ^openblas /g5utlbh ^openmpi /c2bl75v
```
to prepare an installation of lammps with cuda support. If I don't define the existing packages, it will concretize.

I am mainly reporting this, because I could not find any issue that has precisely this error. I recently updated spack with `git pull` in case that could be a cause of the problem. I don't belive this is an issue with lammps, but rather with libxml2 in some way.

### Steps to reproduce the issue

Run the above command.

### Error Message

<!-- If Spack reported an error, provide the error message. If it did not report an error but the output appears incorrect, provide the incorrect output. If there was no error message and no output but the result is incorrect, describe how it does not match what you expect. -->

Output:
```console
$ spack spec -I lammps ~asphere~body~class2~colloid+compress~coreshell+cuda~cuda_mps~dipole~exceptions~ffmpeg~granular~ipo~jpeg~kokkos+kspace~latte+lib+manybody+mc~meam+misc+molecule+mpi~mpiio~opencl+openmp+opt~peri~png~poems~python~qeq+replica+rigid~shock~snap~spin+srd+user-atc~user-h5md~user-lb~user-meamc+user-misc~user-netcdf+user-omp~user-reaxc~voronoi~kim target=zen2 ^cmake /ondqobr ^cuda /vtzojeg ^fftw /fj2ea2u ^openblas /g5utlbh ^openmpi /c2bl75v
Input spec
--------------------------------
 -   lammps~asphere~body~class2~colloid+compress~coreshell+cuda~cuda_mps~dipole~exceptions~ffmpeg~granular~ipo~jpeg~kim~kokkos+kspace~latte+lib+manybody+mc~meam+misc+molecule+mpi~mpiio~opencl+openmp+opt~peri~png~poems~python~qeq+replica+rigid~shock~snap~spin+srd+user-atc~user-h5md~user-lb~user-meamc+user-misc~user-netcdf+user-omp~user-reaxc~voronoi arch=linux-None-zen2
[+]      ^cmake@3.19.2%gcc@10.2.0~doc+ncurses+openssl+ownlibs~qt arch=linux-fedora33-x86_64
[+]          ^ncurses@6.2%gcc@10.2.0~symlinks+termlib arch=linux-fedora33-x86_64
[+]          ^openssl@1.1.1i%gcc@10.2.0+systemcerts arch=linux-fedora33-x86_64
[+]      ^cuda@11.2.2%gcc@10.2.0~dev arch=linux-fedora33-x86_64
[+]          ^libxml2@2.9.10%gcc@10.2.0~python arch=linux-fedora33-x86_64
[+]              ^libiconv@1.16%gcc@10.2.0 arch=linux-fedora33-x86_64
[+]              ^xz@5.2.5%gcc@10.2.0~pic arch=linux-fedora33-x86_64
[+]              ^zlib@1.2.11%gcc@10.2.0+optimize+pic+shared arch=linux-fedora33-x86_64
[+]      ^fftw@3.3.9%gcc@10.2.0~mpi~openmp~pfft_patches precision=double,float arch=linux-fedora33-x86_64
[+]      ^openblas@0.3.13%gcc@10.2.0~consistent_fpcsr~ilp64+pic+shared threads=none arch=linux-fedora33-x86_64
[+]      ^openmpi@4.0.5%gcc@10.2.0+atomics~cuda+cxx+cxx_exceptions+gpfs~java+legacylaunchers~lustre~memchecker+pmi~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=none schedulers=slurm arch=linux-fedora33-x86_64
[+]          ^hwloc@2.4.0%gcc@10.2.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-fedora33-x86_64
[+]              ^libpciaccess@0.16%gcc@10.2.0 arch=linux-fedora33-x86_64
[+]              ^libxml2@2.9.10%gcc@10.2.0~python arch=linux-fedora33-x86_64
[+]                  ^libiconv@1.16%gcc@10.2.0 arch=linux-fedora33-x86_64
[+]                  ^xz@5.2.5%gcc@10.2.0~pic arch=linux-fedora33-x86_64
[+]                  ^zlib@1.2.11%gcc@10.2.0+optimize+pic+shared arch=linux-fedora33-x86_64
[+]          ^libevent@2.1.12%gcc@10.2.0+openssl arch=linux-fedora33-x86_64
[+]              ^openssl@1.1.1i%gcc@10.2.0+systemcerts arch=linux-fedora33-x86_64
[+]          ^numactl@2.0.14%gcc@10.2.0 patches=4e1d78cbbb85de625bad28705e748856033eaafab92a66dffd383a3d7e00cc94 arch=linux-fedora33-x86_64
[+]          ^slurm@20.11.2%gcc@10.2.0~gtk~hdf5~hwloc~mariadb~pmix+readline~restd sysconfdir=PREFIX/etc arch=linux-fedora33-x86_64

Concretized
--------------------------------
==> Error: Invalid Spec DAG: libxml2@2.9.10%gcc@10.2.0~python arch=linux-fedora33-x86_64 does not satisfy libxml2@2.9.10%gcc@10.2.0~python arch=linux-fedora33-x86_64 ^libiconv@1.16%gcc@10.2.0 arch=linux-fedora33-x86_64 ^xz@5.2.5%gcc@10.2.0~pic arch=linux-fedora33-x86_64 ^zlib@1.2.11%gcc@10.2.0+optimize+pic+shared arch=linux-fedora33-x86_64
```

```console
$ spack --debug --stacktrace  spec -I lammps ~asphere~body~class2~colloid+compress~coreshell+cuda~cuda_mps~dipole~exceptions~ffmpeg~granular~ipo~jpeg~kokkos+kspace~latte+lib+manybody+mc~meam+misc+molecule+mpi~mpiio~opencl+openmp+opt~peri~png~poems~python~qeq+replica+rigid~shock~snap~spin+srd+user-atc~user-h5md~user-lb~user-meamc+user-misc~user-netcdf+user-omp~user-reaxc~voronoi~kim target=zen2 ^cmake /ondqobr ^cuda /vtzojeg ^fftw /fj2ea2u ^openblas /g5utlbh ^openmpi /c2bl75v
lib/spack/spack/cmd/__init__.py:122 ==> [2021-04-21-14:35:53.774341] Imported spec from built-in commands
lib/spack/spack/cmd/__init__.py:122 ==> [2021-04-21-14:35:53.775590] Imported spec from built-in commands
lib/spack/spack/config.py:981 ==> [2021-04-21-14:35:53.790076] Reading config file /shared/spack/etc/spack/defaults/config.yaml
lib/spack/spack/config.py:981 ==> [2021-04-21-14:35:53.816802] Reading config file /shared/spack/etc/spack/config.yaml
lib/spack/spack/database.py:363 ==> [2021-04-21-14:35:53.819672] DATABASE LOCK TIMEOUT: 3s
lib/spack/spack/database.py:367 ==> [2021-04-21-14:35:53.819888] PACKAGE LOCK TIMEOUT: No timeout
lib/spack/spack/config.py:981 ==> [2021-04-21-14:35:53.863117] Reading config file /shared/spack/etc/spack/defaults/repos.yaml
Input spec
--------------------------------
 -   lammps~asphere~body~class2~colloid+compress~coreshell+cuda~cuda_mps~dipole~exceptions~ffmpeg~granular~ipo~jpeg~kim~kokkos+kspace~latte+lib+manybody+mc~meam+misc+molecule+mpi~mpiio~opencl+openmp+opt~peri~png~poems~python~qeq+replica+rigid~shock~snap~spin+srd+user-atc~user-h5md~user-lb~user-meamc+user-misc~user-netcdf+user-omp~user-reaxc~voronoi arch=linux-None-zen2
[+]      ^cmake@3.19.2%gcc@10.2.0~doc+ncurses+openssl+ownlibs~qt arch=linux-fedora33-x86_64
[+]          ^ncurses@6.2%gcc@10.2.0~symlinks+termlib arch=linux-fedora33-x86_64
[+]          ^openssl@1.1.1i%gcc@10.2.0+systemcerts arch=linux-fedora33-x86_64
[+]      ^cuda@11.2.2%gcc@10.2.0~dev arch=linux-fedora33-x86_64
[+]          ^libxml2@2.9.10%gcc@10.2.0~python arch=linux-fedora33-x86_64
[+]              ^libiconv@1.16%gcc@10.2.0 arch=linux-fedora33-x86_64
[+]              ^xz@5.2.5%gcc@10.2.0~pic arch=linux-fedora33-x86_64
[+]              ^zlib@1.2.11%gcc@10.2.0+optimize+pic+shared arch=linux-fedora33-x86_64
[+]      ^fftw@3.3.9%gcc@10.2.0~mpi~openmp~pfft_patches precision=double,float arch=linux-fedora33-x86_64
[+]      ^openblas@0.3.13%gcc@10.2.0~consistent_fpcsr~ilp64+pic+shared threads=none arch=linux-fedora33-x86_64
[+]      ^openmpi@4.0.5%gcc@10.2.0+atomics~cuda+cxx+cxx_exceptions+gpfs~java+legacylaunchers~lustre~memchecker+pmi~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=none schedulers=slurm arch=linux-fedora33-x86_64
[+]          ^hwloc@2.4.0%gcc@10.2.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-fedora33-x86_64
[+]              ^libpciaccess@0.16%gcc@10.2.0 arch=linux-fedora33-x86_64
[+]              ^libxml2@2.9.10%gcc@10.2.0~python arch=linux-fedora33-x86_64
[+]                  ^libiconv@1.16%gcc@10.2.0 arch=linux-fedora33-x86_64
[+]                  ^xz@5.2.5%gcc@10.2.0~pic arch=linux-fedora33-x86_64
[+]                  ^zlib@1.2.11%gcc@10.2.0+optimize+pic+shared arch=linux-fedora33-x86_64
[+]          ^libevent@2.1.12%gcc@10.2.0+openssl arch=linux-fedora33-x86_64
[+]              ^openssl@1.1.1i%gcc@10.2.0+systemcerts arch=linux-fedora33-x86_64
[+]          ^numactl@2.0.14%gcc@10.2.0 patches=4e1d78cbbb85de625bad28705e748856033eaafab92a66dffd383a3d7e00cc94 arch=linux-fedora33-x86_64
[+]          ^slurm@20.11.2%gcc@10.2.0~gtk~hdf5~hwloc~mariadb~pmix+readline~restd sysconfdir=PREFIX/etc arch=linux-fedora33-x86_64

Concretized
--------------------------------
lib/spack/spack/spec.py:2816 ==> [2021-04-21-14:35:54.063791] lammps applying constraint cuda
lib/spack/spack/spec.py:2816 ==> [2021-04-21-14:35:54.066424] lammps applying constraint cmake
lib/spack/spack/spec.py:2816 ==> [2021-04-21-14:35:54.072950] lammps applying constraint openmpi@4.0.5%gcc@10.2.0+atomics~cuda+cxx+cxx_exceptions+gpfs~java+legacylaunchers~lustre~memchecker+pmi~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=none schedulers=slurm arch=linux-fedora33-x86_64 ^hwloc@2.4.0%gcc@10.2.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-fedora33-x86_64 ^libevent@2.1.12%gcc@10.2.0+openssl arch=linux-fedora33-x86_64 ^libiconv@1.16%gcc@10.2.0 arch=linux-fedora33-x86_64 ^libpciaccess@0.16%gcc@10.2.0 arch=linux-fedora33-x86_64 ^libxml2@2.9.10%gcc@10.2.0~python arch=linux-fedora33-x86_64 ^numactl@2.0.14%gcc@10.2.0 patches=4e1d78cbbb85de625bad28705e748856033eaafab92a66dffd383a3d7e00cc94 arch=linux-fedora33-x86_64 ^openssl@1.1.1i%gcc@10.2.0+systemcerts arch=linux-fedora33-x86_64 ^slurm@20.11.2%gcc@10.2.0~gtk~hdf5~hwloc~mariadb~pmix+readline~restd sysconfdir=PREFIX/etc arch=linux-fedora33-x86_64 ^xz@5.2.5%gcc@10.2.0~pic arch=linux-fedora33-x86_64 ^zlib@1.2.11%gcc@10.2.0+optimize+pic+shared arch=linux-fedora33-x86_64
lib/spack/spack/spec.py:2816 ==> [2021-04-21-14:35:54.075495] lammps applying constraint fftw@3.3.9%gcc@10.2.0~mpi~openmp~pfft_patches precision=double,float arch=linux-fedora33-x86_64
lib/spack/spack/spec.py:2816 ==> [2021-04-21-14:35:54.081538] lammps applying constraint openblas@0.3.13%gcc@10.2.0~consistent_fpcsr~ilp64+pic+shared threads=none arch=linux-fedora33-x86_64
lib/spack/spack/spec.py:2816 ==> [2021-04-21-14:35:54.084147] lammps applying constraint openblas@0.3.13%gcc@10.2.0~consistent_fpcsr~ilp64+pic+shared threads=none arch=linux-fedora33-x86_64
lib/spack/spack/config.py:981 ==> [2021-04-21-14:35:54.108342] Reading config file /shared/spack/etc/spack/defaults/packages.yaml
lib/spack/spack/config.py:981 ==> [2021-04-21-14:35:54.137378] Reading config file /shared/spack/etc/spack/packages.yaml
lib/spack/spack/config.py:981 ==> [2021-04-21-14:35:54.149051] Reading config file /shared/spack/etc/spack/compilers.yaml
lib/spack/spack/spec.py:2816 ==> [2021-04-21-14:35:54.214926] lammps applying constraint cuda
lib/spack/spack/spec.py:2816 ==> [2021-04-21-14:35:54.218575] lammps applying constraint cmake
lib/spack/spack/spec.py:2816 ==> [2021-04-21-14:35:54.225871] lammps applying constraint openmpi@4.0.5%gcc@10.2.0+atomics~cuda+cxx+cxx_exceptions+gpfs~java+legacylaunchers~lustre~memchecker+pmi~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=none schedulers=slurm arch=linux-fedora33-x86_64 ^hwloc@2.4.0%gcc@10.2.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-fedora33-x86_64 ^libevent@2.1.12%gcc@10.2.0+openssl arch=linux-fedora33-x86_64 ^libiconv@1.16%gcc@10.2.0 arch=linux-fedora33-x86_64 ^libpciaccess@0.16%gcc@10.2.0 arch=linux-fedora33-x86_64 ^libxml2@2.9.10%gcc@10.2.0~python arch=linux-fedora33-x86_64 ^numactl@2.0.14%gcc@10.2.0 patches=4e1d78cbbb85de625bad28705e748856033eaafab92a66dffd383a3d7e00cc94 arch=linux-fedora33-x86_64 ^openssl@1.1.1i%gcc@10.2.0+systemcerts arch=linux-fedora33-x86_64 ^slurm@20.11.2%gcc@10.2.0~gtk~hdf5~hwloc~mariadb~pmix+readline~restd sysconfdir=PREFIX/etc arch=linux-fedora33-x86_64 ^xz@5.2.5%gcc@10.2.0~pic arch=linux-fedora33-x86_64 ^zlib@1.2.11%gcc@10.2.0+optimize+pic+shared arch=linux-fedora33-x86_64
lib/spack/spack/spec.py:2816 ==> [2021-04-21-14:35:54.229462] lammps applying constraint fftw@3.3.9%gcc@10.2.0~mpi~openmp~pfft_patches precision=double,float arch=linux-fedora33-x86_64
lib/spack/spack/spec.py:2816 ==> [2021-04-21-14:35:54.238541] lammps applying constraint openblas@0.3.13%gcc@10.2.0~consistent_fpcsr~ilp64+pic+shared threads=none arch=linux-fedora33-x86_64
lib/spack/spack/spec.py:2816 ==> [2021-04-21-14:35:54.242155] lammps applying constraint openblas@0.3.13%gcc@10.2.0~consistent_fpcsr~ilp64+pic+shared threads=none arch=linux-fedora33-x86_64
lib/spack/spack/main.py:775 ==> [2021-04-21-14:35:54.327519] InconsistentSpecError: Invalid Spec DAG: libxml2@2.9.10%gcc@10.2.0~python arch=linux-fedora33-x86_64 does not satisfy libxml2@2.9.10%gcc@10.2.0~python arch=linux-fedora33-x86_64 ^libiconv@1.16%gcc@10.2.0 arch=linux-fedora33-x86_64 ^xz@5.2.5%gcc@10.2.0~pic arch=linux-fedora33-x86_64 ^zlib@1.2.11%gcc@10.2.0+optimize+pic+shared arch=linux-fedora33-x86_64
lib/spack/spack/error.py:55 ==> [2021-04-21-14:35:54.327780] Error: Invalid Spec DAG: libxml2@2.9.10%gcc@10.2.0~python arch=linux-fedora33-x86_64 does not satisfy libxml2@2.9.10%gcc@10.2.0~python arch=linux-fedora33-x86_64 ^libiconv@1.16%gcc@10.2.0 arch=linux-fedora33-x86_64 ^xz@5.2.5%gcc@10.2.0~pic arch=linux-fedora33-x86_64 ^zlib@1.2.11%gcc@10.2.0+optimize+pic+shared arch=linux-fedora33-x86_64
Traceback (most recent call last):
  File "/shared/spack/lib/spack/spack/spec.py", line 2644, in flat_dependencies
    flat_deps[spec.name].constrain(spec)
  File "/shared/spack/lib/spack/spack/spec.py", line 3051, in constrain
    raise spack.error.UnsatisfiableSpecError(
spack.error.UnsatisfiableSpecError: libxml2@2.9.10%gcc@10.2.0~python arch=linux-fedora33-x86_64 does not satisfy libxml2@2.9.10%gcc@10.2.0~python arch=linux-fedora33-x86_64 ^libiconv@1.16%gcc@10.2.0 arch=linux-fedora33-x86_64 ^xz@5.2.5%gcc@10.2.0~pic arch=linux-fedora33-x86_64 ^zlib@1.2.11%gcc@10.2.0+optimize+pic+shared arch=linux-fedora33-x86_64

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/shared/spack/lib/spack/spack/main.py", line 772, in main
    return _invoke_command(command, parser, args, unknown)
  File "/shared/spack/lib/spack/spack/main.py", line 496, in _invoke_command
    return_val = command(parser, args)
  File "/shared/spack/lib/spack/spack/cmd/spec.py", line 110, in spec
    print(spec.tree(**kwargs))
  File "/shared/spack/lib/spack/spack/spec.py", line 4203, in tree
    status = status_fn(node)
  File "/shared/spack/lib/spack/spack/spec.py", line 4158, in install_status
    record = spack.store.db.get_record(self)
  File "/shared/spack/lib/spack/spack/database.py", line 122, in converter
    return function(self, spec_like, *args, **kwargs)
  File "/shared/spack/lib/spack/spack/database.py", line 1152, in get_record
    key = self._get_matching_spec_key(spec, **kwargs)
  File "/shared/spack/lib/spack/spack/database.py", line 1147, in _get_matching_spec_key
    raise KeyError("No such spec in database! %s" % spec)
  File "/shared/spack/lib/spack/spack/spec.py", line 4150, in __str__
    ret = self.format() + self.dep_string()
  File "/shared/spack/lib/spack/spack/spec.py", line 4147, in dep_string
    return ''.join(" ^" + dep.format() for dep in self.sorted_deps())
  File "/shared/spack/lib/spack/spack/spec.py", line 3578, in sorted_deps
    deps = self.flat_dependencies()
  File "/shared/spack/lib/spack/spack/spec.py", line 2660, in flat_dependencies
    raise InconsistentSpecError("Invalid Spec DAG: %s" % e.message)
spack.spec.InconsistentSpecError: Invalid Spec DAG: libxml2@2.9.10%gcc@10.2.0~python arch=linux-fedora33-x86_64 does not satisfy libxml2@2.9.10%gcc@10.2.0~python arch=linux-fedora33-x86_64 ^libiconv@1.16%gcc@10.2.0 arch=linux-fedora33-x86_64 ^xz@5.2.5%gcc@10.2.0~pic arch=linux-fedora33-x86_64 ^zlib@1.2.11%gcc@10.2.0+optimize+pic+shared arch=linux-fedora33-x86_64
```

### Information on your system

<!-- Please include the output of `spack debug report` -->
```console
$ spack debug report
* **Spack:** 0.16.0-2351-9e620c732b
* **Python:** 3.9.2
* **Platform:** linux-fedora33-sandybridge
* **Concretizer:** original
```

<!-- If you have any relevant configuration detail (custom `packages.yaml` or `modules.yaml`, etc.) you can add that here as well. -->
```console
$ cat /shared/spack/etc/spack/packages.yaml
packages:
  all:
    compiler: [gcc@10.2.0]
    target: [x86_64]
  openssl:
    buildable: false
    externals:
    - spec: openssl@1.1.1i
      prefix: /usr
  slurm:
    buildable: false
    externals:
    - spec: slurm@20.11.2
      prefix: /usr
```


### Additional information

<!-- These boxes can be checked by replacing [ ] with [x] or by clicking them after submitting the issue. -->
- [x] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [x] I have searched the issues of this repo and believe this is not a duplicate
- [x] I have run the failing commands in debug mode and reported the output

<!-- We encourage you to try, as much as possible, to reduce your problem to the minimal example that still reproduces the issue. That would help us a lot in fixing it quickly and effectively!

If you want to ask a question about the tool (how to use it, what it can currently do, etc.), try the `#general` channel on our Slack first. We have a welcoming community and chances are you'll get your reply faster and without opening an issue.

Other than that, thanks for taking the time to contribute to Spack! -->



Discussion:
Now this is weird: if I don't specify the cuda version it works, but uses the same cuda version
```console
$ spack spec -I lammps ~asphere~body~class2~colloid+compress~coreshell+cuda~cuda_mps~dipole~exceptions~ffmpeg~granular~ipo~jpeg~kokkos+kspace~latte+lib+manybody+mc~meam+misc+molecule+mpi~mpiio~opencl+openmp+opt~peri~png~poems~python~qeq+replica+rigid~shock~snap~spin+srd+user-atc~user-h5md~user-lb~user-meamc+user-misc~user-netcdf+user-omp~user-reaxc~voronoi~kim  ^cmake /ondqobr ^fftw /fj2ea2u ^openblas /g5utlbh ^openmpi /c2bl75v 
Input spec
--------------------------------
 -   lammps~asphere~body~class2~colloid+compress~coreshell+cuda~cuda_mps~dipole~exceptions~ffmpeg~granular~ipo~jpeg~kim~kokkos+kspace~latte+lib+manybody+mc~meam+misc+molecule+mpi~mpiio~opencl+openmp+opt~peri~png~poems~python~qeq+replica+rigid~shock~snap~spin+srd+user-atc~user-h5md~user-lb~user-meamc+user-misc~user-netcdf+user-omp~user-reaxc~voronoi
[+]      ^cmake@3.19.2%gcc@10.2.0~doc+ncurses+openssl+ownlibs~qt arch=linux-fedora33-x86_64
[+]          ^ncurses@6.2%gcc@10.2.0~symlinks+termlib arch=linux-fedora33-x86_64
[+]          ^openssl@1.1.1i%gcc@10.2.0+systemcerts arch=linux-fedora33-x86_64
[+]      ^fftw@3.3.9%gcc@10.2.0~mpi~openmp~pfft_patches precision=double,float arch=linux-fedora33-x86_64
[+]      ^openblas@0.3.13%gcc@10.2.0~consistent_fpcsr~ilp64+pic+shared threads=none arch=linux-fedora33-x86_64
[+]      ^openmpi@4.0.5%gcc@10.2.0+atomics~cuda+cxx+cxx_exceptions+gpfs~java+legacylaunchers~lustre~memchecker+pmi~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=none schedulers=slurm arch=linux-fedora33-x86_64
[+]          ^hwloc@2.4.0%gcc@10.2.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-fedora33-x86_64
[+]              ^libpciaccess@0.16%gcc@10.2.0 arch=linux-fedora33-x86_64
[+]              ^libxml2@2.9.10%gcc@10.2.0~python arch=linux-fedora33-x86_64
[+]                  ^libiconv@1.16%gcc@10.2.0 arch=linux-fedora33-x86_64
[+]                  ^xz@5.2.5%gcc@10.2.0~pic arch=linux-fedora33-x86_64
[+]                  ^zlib@1.2.11%gcc@10.2.0+optimize+pic+shared arch=linux-fedora33-x86_64
[+]          ^libevent@2.1.12%gcc@10.2.0+openssl arch=linux-fedora33-x86_64
[+]              ^openssl@1.1.1i%gcc@10.2.0+systemcerts arch=linux-fedora33-x86_64
[+]          ^numactl@2.0.14%gcc@10.2.0 patches=4e1d78cbbb85de625bad28705e748856033eaafab92a66dffd383a3d7e00cc94 arch=linux-fedora33-x86_64
[+]          ^slurm@20.11.2%gcc@10.2.0~gtk~hdf5~hwloc~mariadb~pmix+readline~restd sysconfdir=PREFIX/etc arch=linux-fedora33-x86_64

Concretized
--------------------------------
 -   lammps@20210310%gcc@10.2.0~asphere~body~class2~colloid+compress~coreshell+cuda~cuda_mps~dipole~exceptions~ffmpeg~granular~ipo~jpeg~kim~kokkos+kspace~latte+lib+manybody+mc~meam+misc~mliap+molecule+mpi~mpiio~opencl+openmp+opt~peri~png~poems~python~qeq+replica+rigid~shock~snap~spin+srd~user-adios+user-atc~user-awpmd~user-bocs~user-cgsdk~user-colvars~user-diffraction~user-dpd~user-drude~user-eff~user-fep~user-h5md~user-lb~user-manifold~user-meamc~user-mesodpd~user-mesont~user-mgpt+user-misc~user-mofff~user-netcdf+user-omp~user-phonon~user-plumed~user-ptm~user-qtb~user-reaction~user-reaxc~user-sdpd~user-smd~user-smtbq~user-sph~user-tally~user-uef~user-yaff~voronoi build_type=RelWithDebInfo cuda_arch=none arch=linux-fedora33-x86_64
[+]      ^cmake@3.19.2%gcc@10.2.0~doc+ncurses+openssl+ownlibs~qt arch=linux-fedora33-x86_64
[+]          ^ncurses@6.2%gcc@10.2.0~symlinks+termlib arch=linux-fedora33-x86_64
[+]          ^openssl@1.1.1i%gcc@10.2.0+systemcerts arch=linux-fedora33-x86_64
[+]      ^cuda@11.2.2%gcc@10.2.0~dev arch=linux-fedora33-x86_64
[+]          ^libxml2@2.9.10%gcc@10.2.0~python arch=linux-fedora33-x86_64
[+]              ^libiconv@1.16%gcc@10.2.0 arch=linux-fedora33-x86_64
[+]              ^xz@5.2.5%gcc@10.2.0~pic arch=linux-fedora33-x86_64
[+]              ^zlib@1.2.11%gcc@10.2.0+optimize+pic+shared arch=linux-fedora33-x86_64
[+]      ^fftw@3.3.9%gcc@10.2.0~mpi~openmp~pfft_patches precision=double,float arch=linux-fedora33-x86_64
[+]      ^openblas@0.3.13%gcc@10.2.0~consistent_fpcsr~ilp64+pic+shared threads=none arch=linux-fedora33-x86_64
[+]      ^openmpi@4.0.5%gcc@10.2.0+atomics~cuda+cxx+cxx_exceptions+gpfs~java+legacylaunchers~lustre~memchecker+pmi~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=none schedulers=slurm arch=linux-fedora33-x86_64
[+]          ^hwloc@2.4.0%gcc@10.2.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-fedora33-x86_64
[+]              ^libpciaccess@0.16%gcc@10.2.0 arch=linux-fedora33-x86_64
[+]          ^libevent@2.1.12%gcc@10.2.0+openssl arch=linux-fedora33-x86_64
[+]              ^openssl@1.1.1i%gcc@10.2.0+systemcerts arch=linux-fedora33-x86_64
[+]          ^numactl@2.0.14%gcc@10.2.0 patches=4e1d78cbbb85de625bad28705e748856033eaafab92a66dffd383a3d7e00cc94 arch=linux-fedora33-x86_64
[+]          ^slurm@20.11.2%gcc@10.2.0~gtk~hdf5~hwloc~mariadb~pmix+readline~restd sysconfdir=PREFIX/etc arch=linux-fedora33-x86_64
```
Can you please double check what would be the behavior of the new concretizer?
so I added `concretizer: clingo` to config.yml.

Now it suddenly starts building packages!? Is this the expected behavior?
```console
$ spack spec -I lammps ~asphere~body~class2~colloid+compress~coreshell+cuda~cuda_mps~dipole~exceptions~ffmpeg~granular~ipo~jpeg~kokkos+kspace~latte+lib+manybody+mc~meam+misc+molecule+mpi~mpiio~opencl+openmp+opt~peri~png~poems~python~qeq+replica+rigid~shock~snap~spin+srd+user-atc~user-h5md~user-lb~user-meamc+user-misc~user-netcdf+user-omp~user-reaxc~voronoi~kim target=zen2 ^cmake /ondqobr ^cuda /vtzojeg ^fftw /fj2ea2u ^openblas /g5utlbh ^openmpi /c2bl75v
Input spec
--------------------------------
 -   lammps~asphere~body~class2~colloid+compress~coreshell+cuda~cuda_mps~dipole~exceptions~ffmpeg~granular~ipo~jpeg~kim~kokkos+kspace~latte+lib+manybody+mc~meam+misc+molecule+mpi~mpiio~opencl+openmp+opt~peri~png~poems~python~qeq+replica+rigid~shock~snap~spin+srd+user-atc~user-h5md~user-lb~user-meamc+user-misc~user-netcdf+user-omp~user-reaxc~voronoi arch=linux-None-zen2
[+]      ^cmake@3.19.2%gcc@10.2.0~doc+ncurses+openssl+ownlibs~qt arch=linux-fedora33-x86_64
[+]          ^ncurses@6.2%gcc@10.2.0~symlinks+termlib arch=linux-fedora33-x86_64
[+]          ^openssl@1.1.1i%gcc@10.2.0+systemcerts arch=linux-fedora33-x86_64
[+]      ^cuda@11.2.2%gcc@10.2.0~dev arch=linux-fedora33-x86_64
[+]          ^libxml2@2.9.10%gcc@10.2.0~python arch=linux-fedora33-x86_64
[+]              ^libiconv@1.16%gcc@10.2.0 arch=linux-fedora33-x86_64
[+]              ^xz@5.2.5%gcc@10.2.0~pic arch=linux-fedora33-x86_64
[+]              ^zlib@1.2.11%gcc@10.2.0+optimize+pic+shared arch=linux-fedora33-x86_64
[+]      ^fftw@3.3.9%gcc@10.2.0~mpi~openmp~pfft_patches precision=double,float arch=linux-fedora33-x86_64
[+]      ^openblas@0.3.13%gcc@10.2.0~consistent_fpcsr~ilp64+pic+shared threads=none arch=linux-fedora33-x86_64
[+]      ^openmpi@4.0.5%gcc@10.2.0+atomics~cuda+cxx+cxx_exceptions+gpfs~java+legacylaunchers~lustre~memchecker+pmi~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=none schedulers=slurm arch=linux-fedora33-x86_64
[+]          ^hwloc@2.4.0%gcc@10.2.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-fedora33-x86_64
[+]              ^libpciaccess@0.16%gcc@10.2.0 arch=linux-fedora33-x86_64
[+]              ^libxml2@2.9.10%gcc@10.2.0~python arch=linux-fedora33-x86_64
[+]                  ^libiconv@1.16%gcc@10.2.0 arch=linux-fedora33-x86_64
[+]                  ^xz@5.2.5%gcc@10.2.0~pic arch=linux-fedora33-x86_64
[+]                  ^zlib@1.2.11%gcc@10.2.0+optimize+pic+shared arch=linux-fedora33-x86_64
[+]          ^libevent@2.1.12%gcc@10.2.0+openssl arch=linux-fedora33-x86_64
[+]              ^openssl@1.1.1i%gcc@10.2.0+systemcerts arch=linux-fedora33-x86_64
[+]          ^numactl@2.0.14%gcc@10.2.0 patches=4e1d78cbbb85de625bad28705e748856033eaafab92a66dffd383a3d7e00cc94 arch=linux-fedora33-x86_64
[+]          ^slurm@20.11.2%gcc@10.2.0~gtk~hdf5~hwloc~mariadb~pmix+readline~restd sysconfdir=PREFIX/etc arch=linux-fedora33-x86_64

Concretized
--------------------------------
==> Warning: Missing a source id for python@3.9
==> Installing libiconv-1.16-vrzcj5hwzrs3vzp3kxezuil54iirjvwc
==> No binary for libiconv-1.16-vrzcj5hwzrs3vzp3kxezuil54iirjvwc found: installing from source
==> Using cached archive: /shared/spack/var/spack/cache/_source-cache/archive/e6/e6a1b1b589654277ee790cce3734f07876ac4ccfaecbee8afa0b649cf529cc04.tar.gz
==> No patches needed for libiconv
==> libiconv: Executing phase: 'autoreconf'
==> libiconv: Executing phase: 'configure'
==> libiconv: Executing phase: 'build'
==> libiconv: Executing phase: 'install'
==> Warning: Module file already exists : skipping creation
file : /shared/spack/share/spack/modules/linux-fedora33-x86_64/libiconv-1.16-gcc-10.2.0-vrzcj5h
spec : libiconv@1.16%gcc@10.2.0 arch=linux-fedora33-x86_64
==> libiconv: Successfully installed libiconv-1.16-vrzcj5hwzrs3vzp3kxezuil54iirjvwc
  Fetch: 0.05s.  Build: 25.32s.  Total: 25.37s.
[+] /home/mbernhardt/.spack/bootstrap/store/linux-fedora33-x86_64/gcc-10.2.0/libiconv-1.16-vrzcj5hwzrs3vzp3kxezuil54iirjvwc
==> Installing pkgconf-1.7.4-tayfpqdxbvcfcsmxmknpxmhcg2syrdsn
==> No binary for pkgconf-1.7.4-tayfpqdxbvcfcsmxmknpxmhcg2syrdsn found: installing from source
==> Using cached archive: /shared/spack/var/spack/cache/_source-cache/archive/d7/d73f32c248a4591139a6b17777c80d4deab6b414ec2b3d21d0a24be348c476ab.tar.xz
==> No patches needed for pkgconf
==> pkgconf: Executing phase: 'autoreconf'
==> pkgconf: Executing phase: 'configure'
==> Error: Failed to install pkgconf due to KeyboardInterrupt: 
==> Error: Exception occurred in writer daemon!
Traceback (most recent call last):
  File "/shared/spack/lib/spack/llnl/util/tty/log.py", line 775, in _writer_daemon
    line = _retry(in_pipe.readline)()
  File "/shared/spack/lib/spack/llnl/util/tty/log.py", line 837, in wrapped
    return function(*args, **kwargs)
KeyboardInterrupt

==> Error: Keyboard interrupt.
```
>  Is this the expected behavior?

@marvinbernhardt If you don't have the `clingo` module importable in Python yes. 

What happened is that Spack started bootstrapping `clingo` from sources to use it during concretization. We'll soon be able to bootstrap from binaries which would be much faster. Also, since a couple of days another alternative if you're using a Python interpreter which is at version 3.6 or later is to:
```
pip install clingo
```
I see, thanks for the clarification.

Now when I use it I get `==> Error: variant "adios" not found in package "lammps" [required from package "lammps"]`.
Super weird, as there is only a variant user-adios.

<details>
  <summary>debug log</summary>

```console
$ spack --debug --stacktrace spec -I lammps ~asphere~body~class2~colloid+compress~coreshell+cuda~cuda_mps~dipole~exceptions~ffmpeg~granular~ipo~jpeg~kokkos+kspace~latte+lib+manybody+mc~meam+misc+molecule+mpi~mpiio~opencl+openmp+opt~peri~png~poems~python~qeq+replica+rigid~shock~snap~spin+srd+user-atc~user-h5md~user-lb~user-meamc+user-misc~user-netcdf+user-omp~user-reaxc~voronoi~kim target=zen2 ^cmake /ondqobr ^cuda /vtzojeg ^fftw /fj2ea2u ^openblas /g5utlbh ^openmpi /c2bl75v
lib/spack/spack/cmd/__init__.py:122 ==> [2021-04-23-17:16:01.518604] Imported spec from built-in commands
lib/spack/spack/cmd/__init__.py:122 ==> [2021-04-23-17:16:01.519679] Imported spec from built-in commands
lib/spack/spack/config.py:981 ==> [2021-04-23-17:16:01.521584] Reading config file /shared/spack/etc/spack/defaults/config.yaml
lib/spack/spack/config.py:981 ==> [2021-04-23-17:16:01.544098] Reading config file /shared/spack/etc/spack/config.yaml
lib/spack/spack/database.py:364 ==> [2021-04-23-17:16:01.546873] DATABASE LOCK TIMEOUT: 3s
lib/spack/spack/database.py:368 ==> [2021-04-23-17:16:01.547062] PACKAGE LOCK TIMEOUT: No timeout
lib/spack/spack/config.py:981 ==> [2021-04-23-17:16:01.583423] Reading config file /shared/spack/etc/spack/defaults/repos.yaml
Input spec
--------------------------------
 -   lammps~asphere~body~class2~colloid+compress~coreshell+cuda~cuda_mps~dipole~exceptions~ffmpeg~granular~ipo~jpeg~kim~kokkos+kspace~latte+lib+manybody+mc~meam+misc+molecule+mpi~mpiio~opencl+openmp+opt~peri~png~poems~python~qeq+replica+rigid~shock~snap~spin+srd+user-atc~user-h5md~user-lb~user-meamc+user-misc~user-netcdf+user-omp~user-reaxc~voronoi arch=linux-None-zen2
[+]      ^cmake@3.19.2%gcc@10.2.0~doc+ncurses+openssl+ownlibs~qt arch=linux-fedora33-x86_64
[+]          ^ncurses@6.2%gcc@10.2.0~symlinks+termlib arch=linux-fedora33-x86_64
[+]          ^openssl@1.1.1i%gcc@10.2.0+systemcerts arch=linux-fedora33-x86_64
[+]      ^cuda@11.2.2%gcc@10.2.0~dev arch=linux-fedora33-x86_64
[+]          ^libxml2@2.9.10%gcc@10.2.0~python arch=linux-fedora33-x86_64
[+]              ^libiconv@1.16%gcc@10.2.0 arch=linux-fedora33-x86_64
[+]              ^xz@5.2.5%gcc@10.2.0~pic arch=linux-fedora33-x86_64
[+]              ^zlib@1.2.11%gcc@10.2.0+optimize+pic+shared arch=linux-fedora33-x86_64
[+]      ^fftw@3.3.9%gcc@10.2.0~mpi~openmp~pfft_patches precision=double,float arch=linux-fedora33-x86_64
[+]      ^openblas@0.3.13%gcc@10.2.0~consistent_fpcsr~ilp64+pic+shared threads=none arch=linux-fedora33-x86_64
[+]      ^openmpi@4.0.5%gcc@10.2.0+atomics~cuda+cxx+cxx_exceptions+gpfs~java+legacylaunchers~lustre~memchecker+pmi~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=none schedulers=slurm arch=linux-fedora33-x86_64
[+]          ^hwloc@2.4.0%gcc@10.2.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-fedora33-x86_64
[+]              ^libpciaccess@0.16%gcc@10.2.0 arch=linux-fedora33-x86_64
[+]              ^libxml2@2.9.10%gcc@10.2.0~python arch=linux-fedora33-x86_64
[+]                  ^libiconv@1.16%gcc@10.2.0 arch=linux-fedora33-x86_64
[+]                  ^xz@5.2.5%gcc@10.2.0~pic arch=linux-fedora33-x86_64
[+]                  ^zlib@1.2.11%gcc@10.2.0+optimize+pic+shared arch=linux-fedora33-x86_64
[+]          ^libevent@2.1.12%gcc@10.2.0+openssl arch=linux-fedora33-x86_64
[+]              ^openssl@1.1.1i%gcc@10.2.0+systemcerts arch=linux-fedora33-x86_64
[+]          ^numactl@2.0.14%gcc@10.2.0 patches=4e1d78cbbb85de625bad28705e748856033eaafab92a66dffd383a3d7e00cc94 arch=linux-fedora33-x86_64
[+]          ^slurm@20.11.2%gcc@10.2.0~gtk~hdf5~hwloc~mariadb~pmix+readline~restd sysconfdir=PREFIX/etc arch=linux-fedora33-x86_64

Concretized
--------------------------------
lib/spack/spack/config.py:981 ==> [2021-04-23-17:16:03.041598] Reading config file /shared/spack/etc/spack/compilers.yaml
lib/spack/spack/config.py:981 ==> [2021-04-23-17:16:03.117511] Reading config file /shared/spack/etc/spack/defaults/packages.yaml
lib/spack/spack/config.py:981 ==> [2021-04-23-17:16:03.148997] Reading config file /shared/spack/etc/spack/packages.yaml
Traceback (most recent call last):
  File "/shared/spack/lib/spack/spack/solver/asp.py", line 971, in spec_clauses
    variant_def = spec.package.variants[vname]
KeyError: 'adios'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/shared/spack/lib/spack/spack/solver/asp.py", line 892, in checked_spec_clauses
    clauses = self.spec_clauses(*args, **kwargs)
  File "/shared/spack/lib/spack/spack/solver/asp.py", line 974, in spec_clauses
    raise RuntimeError(msg.format(vname, spec.name))
RuntimeError: variant "adios" not found in package "lammps"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/shared/spack/bin/spack", line 76, in <module>
    sys.exit(spack.main.main())
  File "/shared/spack/lib/spack/spack/main.py", line 772, in main
    return _invoke_command(command, parser, args, unknown)
  File "/shared/spack/lib/spack/spack/main.py", line 496, in _invoke_command
    return_val = command(parser, args)
  File "/shared/spack/lib/spack/spack/cmd/spec.py", line 109, in spec
    spec.concretize()
  File "/shared/spack/lib/spack/spack/spec.py", line 2577, in concretize
    self._new_concretize(tests)
  File "/shared/spack/lib/spack/spack/spec.py", line 2547, in _new_concretize
    result = spack.solver.asp.solve([self], tests=tests)
  File "/shared/spack/lib/spack/spack/solver/asp.py", line 1652, in solve
    return driver.solve(setup, specs, dump, models, timers, stats, tests)
  File "/shared/spack/lib/spack/spack/solver/asp.py", line 315, in solve
    solver_setup.setup(self, specs, tests=tests)
  File "/shared/spack/lib/spack/spack/solver/asp.py", line 1371, in setup
    self.pkg_rules(pkg, tests=tests)
  File "/shared/spack/lib/spack/spack/solver/asp.py", line 639, in pkg_rules
    self.conflict_rules(pkg)
  File "/shared/spack/lib/spack/spack/solver/asp.py", line 510, in conflict_rules
    trigger_id = self.condition(spack.spec.Spec(trigger), name=pkg.name)
  File "/shared/spack/lib/spack/spack/solver/asp.py", line 680, in condition
    named_cond, body=True, required_from=name)
  File "/shared/spack/lib/spack/spack/solver/asp.py", line 897, in checked_spec_clauses
    raise RuntimeError(msg)
RuntimeError: variant "adios" not found in package "lammps" [required from package "lammps"]
```
</details>
Ah ok, those seem to be faulty checks in lammps package.py around line 152.
When I fix package.py, I get the same error again

<details>
  <summary>debug log</summary>
  
  ```console
$ spack --debug --stacktrace spec -I lammps ~asphere~body~class2~colloid+compress~coreshell+cuda~cuda_mps~dipole~exceptions~ffmpeg~granular~ipo~jpeg~kokkos+kspace~latte+lib+manybody+mc~meam+misc+molecule+mpi~mpiio~opencl+openmp+opt~peri~png~poems~python~qeq+replica+rigid~shock~snap~spin+srd+user-atc~user-h5md~user-lb~user-meamc+user-misc~user-netcdf+user-omp~user-reaxc~voronoi~kim target=zen2 ^cmake /ondqobr ^cuda /vtzojeg ^fftw /fj2ea2u ^openblas /g5utlbh ^openmpi /c2bl75v
lib/spack/spack/cmd/__init__.py:122 ==> [2021-04-23-17:23:42.443010] Imported spec from built-in commands
lib/spack/spack/cmd/__init__.py:122 ==> [2021-04-23-17:23:42.444464] Imported spec from built-in commands
lib/spack/spack/config.py:981 ==> [2021-04-23-17:23:42.447025] Reading config file /shared/spack/etc/spack/defaults/config.yaml
lib/spack/spack/config.py:981 ==> [2021-04-23-17:23:42.475793] Reading config file /shared/spack/etc/spack/config.yaml
lib/spack/spack/database.py:364 ==> [2021-04-23-17:23:42.479282] DATABASE LOCK TIMEOUT: 3s
lib/spack/spack/database.py:368 ==> [2021-04-23-17:23:42.479499] PACKAGE LOCK TIMEOUT: No timeout
lib/spack/spack/config.py:981 ==> [2021-04-23-17:23:42.527875] Reading config file /shared/spack/etc/spack/defaults/repos.yaml
Input spec
--------------------------------
 -   lammps~asphere~body~class2~colloid+compress~coreshell+cuda~cuda_mps~dipole~exceptions~ffmpeg~granular~ipo~jpeg~kim~kokkos+kspace~latte+lib+manybody+mc~meam+misc+molecule+mpi~mpiio~opencl+openmp+opt~peri~png~poems~python~qeq+replica+rigid~shock~snap~spin+srd+user-atc~user-h5md~user-lb~user-meamc+user-misc~user-netcdf+user-omp~user-reaxc~voronoi arch=linux-None-zen2
[+]      ^cmake@3.19.2%gcc@10.2.0~doc+ncurses+openssl+ownlibs~qt arch=linux-fedora33-x86_64
[+]          ^ncurses@6.2%gcc@10.2.0~symlinks+termlib arch=linux-fedora33-x86_64
[+]          ^openssl@1.1.1i%gcc@10.2.0+systemcerts arch=linux-fedora33-x86_64
[+]      ^cuda@11.2.2%gcc@10.2.0~dev arch=linux-fedora33-x86_64
[+]          ^libxml2@2.9.10%gcc@10.2.0~python arch=linux-fedora33-x86_64
[+]              ^libiconv@1.16%gcc@10.2.0 arch=linux-fedora33-x86_64
[+]              ^xz@5.2.5%gcc@10.2.0~pic arch=linux-fedora33-x86_64
[+]              ^zlib@1.2.11%gcc@10.2.0+optimize+pic+shared arch=linux-fedora33-x86_64
[+]      ^fftw@3.3.9%gcc@10.2.0~mpi~openmp~pfft_patches precision=double,float arch=linux-fedora33-x86_64
[+]      ^openblas@0.3.13%gcc@10.2.0~consistent_fpcsr~ilp64+pic+shared threads=none arch=linux-fedora33-x86_64
[+]      ^openmpi@4.0.5%gcc@10.2.0+atomics~cuda+cxx+cxx_exceptions+gpfs~java+legacylaunchers~lustre~memchecker+pmi~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=none schedulers=slurm arch=linux-fedora33-x86_64
[+]          ^hwloc@2.4.0%gcc@10.2.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-fedora33-x86_64
[+]              ^libpciaccess@0.16%gcc@10.2.0 arch=linux-fedora33-x86_64
[+]              ^libxml2@2.9.10%gcc@10.2.0~python arch=linux-fedora33-x86_64
[+]                  ^libiconv@1.16%gcc@10.2.0 arch=linux-fedora33-x86_64
[+]                  ^xz@5.2.5%gcc@10.2.0~pic arch=linux-fedora33-x86_64
[+]                  ^zlib@1.2.11%gcc@10.2.0+optimize+pic+shared arch=linux-fedora33-x86_64
[+]          ^libevent@2.1.12%gcc@10.2.0+openssl arch=linux-fedora33-x86_64
[+]              ^openssl@1.1.1i%gcc@10.2.0+systemcerts arch=linux-fedora33-x86_64
[+]          ^numactl@2.0.14%gcc@10.2.0 patches=4e1d78cbbb85de625bad28705e748856033eaafab92a66dffd383a3d7e00cc94 arch=linux-fedora33-x86_64
[+]          ^slurm@20.11.2%gcc@10.2.0~gtk~hdf5~hwloc~mariadb~pmix+readline~restd sysconfdir=PREFIX/etc arch=linux-fedora33-x86_64

Concretized
--------------------------------
lib/spack/spack/config.py:981 ==> [2021-04-23-17:23:44.033280] Reading config file /shared/spack/etc/spack/compilers.yaml
lib/spack/spack/config.py:981 ==> [2021-04-23-17:23:44.110276] Reading config file /shared/spack/etc/spack/defaults/packages.yaml
lib/spack/spack/config.py:981 ==> [2021-04-23-17:23:44.142768] Reading config file /shared/spack/etc/spack/packages.yaml
lib/spack/spack/main.py:775 ==> [2021-04-23-17:23:47.831412] InconsistentSpecError: Invalid Spec DAG: libxml2@2.9.10%gcc@10.2.0~python arch=linux-fedora33-x86_64 does not satisfy libxml2@2.9.10%gcc@10.2.0~python arch=linux-fedora33-x86_64 ^libiconv@1.16%gcc@10.2.0 arch=linux-fedora33-x86_64 ^xz@5.2.5%gcc@10.2.0~pic arch=linux-fedora33-x86_64 ^zlib@1.2.11%gcc@10.2.0+optimize+pic+shared arch=linux-fedora33-x86_64
lib/spack/spack/error.py:55 ==> [2021-04-23-17:23:47.831767] Error: Invalid Spec DAG: libxml2@2.9.10%gcc@10.2.0~python arch=linux-fedora33-x86_64 does not satisfy libxml2@2.9.10%gcc@10.2.0~python arch=linux-fedora33-x86_64 ^libiconv@1.16%gcc@10.2.0 arch=linux-fedora33-x86_64 ^xz@5.2.5%gcc@10.2.0~pic arch=linux-fedora33-x86_64 ^zlib@1.2.11%gcc@10.2.0+optimize+pic+shared arch=linux-fedora33-x86_64
Traceback (most recent call last):
  File "/shared/spack/lib/spack/spack/spec.py", line 2644, in flat_dependencies
    flat_deps[spec.name].constrain(spec)
  File "/shared/spack/lib/spack/spack/spec.py", line 3052, in constrain
    self, other, 'constrain a concrete spec'
spack.error.UnsatisfiableSpecError: libxml2@2.9.10%gcc@10.2.0~python arch=linux-fedora33-x86_64 does not satisfy libxml2@2.9.10%gcc@10.2.0~python arch=linux-fedora33-x86_64 ^libiconv@1.16%gcc@10.2.0 arch=linux-fedora33-x86_64 ^xz@5.2.5%gcc@10.2.0~pic arch=linux-fedora33-x86_64 ^zlib@1.2.11%gcc@10.2.0+optimize+pic+shared arch=linux-fedora33-x86_64

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/shared/spack/lib/spack/spack/main.py", line 772, in main
    return _invoke_command(command, parser, args, unknown)
  File "/shared/spack/lib/spack/spack/main.py", line 496, in _invoke_command
    return_val = command(parser, args)
  File "/shared/spack/lib/spack/spack/cmd/spec.py", line 109, in spec
    spec.concretize()
  File "/shared/spack/lib/spack/spack/spec.py", line 2577, in concretize
    self._new_concretize(tests)
  File "/shared/spack/lib/spack/spack/spec.py", line 2547, in _new_concretize
    result = spack.solver.asp.solve([self], tests=tests)
  File "/shared/spack/lib/spack/spack/solver/asp.py", line 1652, in solve
    return driver.solve(setup, specs, dump, models, timers, stats, tests)
  File "/shared/spack/lib/spack/spack/solver/asp.py", line 315, in solve
    solver_setup.setup(self, specs, tests=tests)
  File "/shared/spack/lib/spack/spack/solver/asp.py", line 1386, in setup
    self.gen.h2('Spec: %s' % str(spec))
  File "/shared/spack/lib/spack/spack/spec.py", line 4150, in __str__
    ret = self.format() + self.dep_string()
  File "/shared/spack/lib/spack/spack/spec.py", line 4147, in dep_string
    return ''.join(" ^" + dep.format() for dep in self.sorted_deps())
  File "/shared/spack/lib/spack/spack/spec.py", line 3578, in sorted_deps
    deps = self.flat_dependencies()
  File "/shared/spack/lib/spack/spack/spec.py", line 2660, in flat_dependencies
    raise InconsistentSpecError("Invalid Spec DAG: %s" % e.message)
spack.spec.InconsistentSpecError: Invalid Spec DAG: libxml2@2.9.10%gcc@10.2.0~python arch=linux-fedora33-x86_64 does not satisfy libxml2@2.9.10%gcc@10.2.0~python arch=linux-fedora33-x86_64 ^libiconv@1.16%gcc@10.2.0 arch=linux-fedora33-x86_64 ^xz@5.2.5%gcc@10.2.0~pic arch=linux-fedora33-x86_64 ^zlib@1.2.11%gcc@10.2.0+optimize+pic+shared arch=linux-fedora33-x86_64

  ```
</details>
Any tip on how I could resolve this? I would really like to install lammps+cuda with spack.
It gets much weirder when I add `^hwloc`

Tried that with the latest develop branch. Looks like there is some Python bug there, right?

```console
$ spack --debug --stacktrace spec -I lammps ~asphere~body~class2~colloid+compress~coreshell+cuda~cuda_mps~dipole~exceptions~ffmpeg~granular~ipo~jpeg~kokkos+kspace~latte+lib+manybody+mc~meam+misc+molecule+mpi~mpiio~opencl+openmp+opt~peri~png~poems~python~qeq+replica+rigid~shock~snap~spin+srd+user-atc~user-h5md~user-lb~user-meamc+user-misc~user-netcdf+user-omp~user-reaxc~voronoi~kim target=zen2 ^cmake /ondqobr ^cuda /vtzojeg ^fftw /fj2ea2u ^openblas /g5utlbh ^openmpi /c2bl75v ^hwloc
lib/spack/spack/cmd/__init__.py:122 ==> [2021-04-30-14:18:59.388528] Imported spec from built-in commands
lib/spack/spack/cmd/__init__.py:122 ==> [2021-04-30-14:18:59.389491] Imported spec from built-in commands
lib/spack/spack/config.py:981 ==> [2021-04-30-14:18:59.391319] Reading config file /shared/spack/etc/spack/defaults/config.yaml
lib/spack/spack/config.py:981 ==> [2021-04-30-14:18:59.412153] Reading config file /shared/spack/etc/spack/config.yaml
lib/spack/spack/database.py:363 ==> [2021-04-30-14:18:59.414357] DATABASE LOCK TIMEOUT: 3s
lib/spack/spack/database.py:367 ==> [2021-04-30-14:18:59.414523] PACKAGE LOCK TIMEOUT: No timeout
lib/spack/spack/config.py:981 ==> [2021-04-30-14:18:59.447804] Reading config file /shared/spack/etc/spack/defaults/repos.yaml
Input spec
--------------------------------
 -   lammps~asphere~body~class2~colloid+compress~coreshell+cuda~cuda_mps~dipole~exceptions~ffmpeg~granular~ipo~jpeg~kim~kokkos+kspace~latte+lib+manybody+mc~meam+misc+molecule+mpi~mpiio~opencl+openmp+opt~peri~png~poems~python~qeq+replica+rigid~shock~snap~spin+srd+user-atc~user-h5md~user-lb~user-meamc+user-misc~user-netcdf+user-omp~user-reaxc~voronoi arch=linux-None-zen2
[+]      ^cmake@3.19.2%gcc@10.2.0~doc+ncurses+openssl+ownlibs~qt arch=linux-fedora33-x86_64
[+]          ^ncurses@6.2%gcc@10.2.0~symlinks+termlib arch=linux-fedora33-x86_64
[+]          ^openssl@1.1.1i%gcc@10.2.0+systemcerts arch=linux-fedora33-x86_64
[+]      ^cuda@11.2.2%gcc@10.2.0~dev arch=linux-fedora33-x86_64
[+]          ^libxml2@2.9.10%gcc@10.2.0~python arch=linux-fedora33-x86_64
[+]              ^libiconv@1.16%gcc@10.2.0 arch=linux-fedora33-x86_64
[+]              ^xz@5.2.5%gcc@10.2.0~pic arch=linux-fedora33-x86_64
[+]              ^zlib@1.2.11%gcc@10.2.0+optimize+pic+shared arch=linux-fedora33-x86_64
[+]      ^fftw@3.3.9%gcc@10.2.0~mpi~openmp~pfft_patches precision=double,float arch=linux-fedora33-x86_64
 -       ^hwloc
[+]      ^openblas@0.3.13%gcc@10.2.0~consistent_fpcsr~ilp64+pic+shared threads=none arch=linux-fedora33-x86_64
[+]      ^openmpi@4.0.5%gcc@10.2.0+atomics~cuda+cxx+cxx_exceptions+gpfs~java+legacylaunchers~lustre~memchecker+pmi~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=none schedulers=slurm arch=linux-fedora33-x86_64
[+]          ^hwloc@2.4.0%gcc@10.2.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-fedora33-x86_64
[+]              ^libpciaccess@0.16%gcc@10.2.0 arch=linux-fedora33-x86_64
[+]              ^libxml2@2.9.10%gcc@10.2.0~python arch=linux-fedora33-x86_64
[+]                  ^libiconv@1.16%gcc@10.2.0 arch=linux-fedora33-x86_64
[+]                  ^xz@5.2.5%gcc@10.2.0~pic arch=linux-fedora33-x86_64
[+]                  ^zlib@1.2.11%gcc@10.2.0+optimize+pic+shared arch=linux-fedora33-x86_64
[+]          ^libevent@2.1.12%gcc@10.2.0+openssl arch=linux-fedora33-x86_64
[+]              ^openssl@1.1.1i%gcc@10.2.0+systemcerts arch=linux-fedora33-x86_64
[+]          ^numactl@2.0.14%gcc@10.2.0 patches=4e1d78cbbb85de625bad28705e748856033eaafab92a66dffd383a3d7e00cc94 arch=linux-fedora33-x86_64
[+]          ^slurm@20.11.2%gcc@10.2.0~gtk~hdf5~hwloc~mariadb~pmix+readline~restd sysconfdir=PREFIX/etc arch=linux-fedora33-x86_64

Concretized
--------------------------------
lib/spack/spack/main.py:776 ==> [2021-04-30-14:18:59.640842] InvalidDependencyError: Package hwloc does not depend on l, i, b, i, c, o, n, or v
lib/spack/spack/error.py:55 ==> [2021-04-30-14:18:59.641070] Error: Package hwloc does not depend on l, i, b, i, c, o, n, or v
Traceback (most recent call last):
  File "/shared/spack/lib/spack/spack/main.py", line 773, in main
    return _invoke_command(command, parser, args, unknown)
  File "/shared/spack/lib/spack/spack/main.py", line 497, in _invoke_command
    return_val = command(parser, args)
  File "/shared/spack/lib/spack/spack/cmd/spec.py", line 109, in spec
    spec.concretize()
  File "/shared/spack/lib/spack/spack/spec.py", line 2579, in concretize
    self._old_concretize(tests)
  File "/shared/spack/lib/spack/spack/spec.py", line 2373, in _old_concretize
    user_spec_deps = self.flat_dependencies(copy=False)
  File "/shared/spack/lib/spack/spack/spec.py", line 2644, in flat_dependencies
    flat_deps[spec.name].constrain(spec)
  File "/shared/spack/lib/spack/spack/spec.py", line 3115, in constrain
    changed |= self._constrain_dependencies(other)
  File "/shared/spack/lib/spack/spack/spec.py", line 3145, in _constrain_dependencies
    dep_spec_copy = other.get_dependency(name)
  File "/shared/spack/lib/spack/spack/spec.py", line 1143, in get_dependency
    raise InvalidDependencyError(self.name, name)
spack.spec.InvalidDependencyError: Package hwloc does not depend on l, i, b, i, c, o, n, or v
```
Bringing @becker33 into this since he's been dealing with some similar issues—I think in particular #22991 might help? I recall that the `l, i, b, i, c, o, n, or v` issue was solved for me with that.

Regarding the `InconsistentSpecError` I've actually debugged that separately, since I'd been encountering it as well—it's due to a frankly terrible implementation of `sorted_deps` (used in `__str__`) that constructs a deep copy of `self` for some strange reason and causes issues when that fails. It should really just sort the traversal of the deps...
So you mean this one could be implemented better?
https://github.com/spack/spack/blob/7f91c1a5109e4c7f92c22594d0bcfc1b94bf230b/lib/spack/spack/spec.py#L3576-L3579
@marvinbernhardt Can you create a smaller reproducer that can be used to debug on another platform?

Resolving patch:
diff --git a/lib/spack/spack/spec.py b/lib/spack/spack/spec.py
--- a/lib/spack/spack/spec.py
+++ b/lib/spack/spack/spec.py
@@ -3573,11 +3573,6 @@ def __contains__(self, spec):
         else:
             return any(s.satisfies(spec) for s in self.traverse(root=False))
 
-    def sorted_deps(self):
-        """Return a list of all dependencies sorted by name."""
-        deps = self.flat_dependencies()
-        return tuple(deps[name] for name in sorted(deps))
-
     def eq_dag(self, other, deptypes=True, vs=None, vo=None):
         """True if the full dependency DAGs of specs are equal."""
         if vs is None:
@@ -3887,7 +3882,9 @@ def write_attribute(spec, attribute, color):
                 'Format string terminated while reading attribute.'
                 'Missing terminating }.'
             )
-        return out.getvalue()
+
+        formatted_spec = out.getvalue()
+        return formatted_spec.strip()
 
     def old_format(self, format_string='$_$@$%@+$+$=', **kwargs):
         """
@@ -4143,12 +4140,12 @@ def cformat(self, *args, **kwargs):
         kwargs.setdefault('color', None)
         return self.format(*args, **kwargs)
 
-    def dep_string(self):
-        return ''.join(" ^" + dep.format() for dep in self.sorted_deps())
-
     def __str__(self):
-        ret = self.format() + self.dep_string()
-        return ret.strip()
+        sorted_nodes = [self] + sorted(
+            self.traverse(root=False), key=lambda x: x.name
+        )
+        spec_str = " ^".join(d.format() for d in sorted_nodes)
+        return spec_str.strip()
 
     def install_status(self):
         """Helper for tree to print DB install status."""


Test patch:
diff --git a/lib/spack/spack/test/spec_semantics.py b/lib/spack/spack/test/spec_semantics.py
--- a/lib/spack/spack/test/spec_semantics.py
+++ b/lib/spack/spack/test/spec_semantics.py
@@ -776,7 +776,7 @@ def test_spec_formatting(self):
         sigil_package_segments = [("{@VERSIONS}", '@' + str(spec.version)),
                                   ("{%compiler}", '%' + str(spec.compiler)),
                                   ("{arch=architecture}",
-                                   ' arch=' + str(spec.architecture))]
+                                   'arch=' + str(spec.architecture))]
 
         compiler_segments = [("{compiler.name}", "name"),
                              ("{compiler.version}", "versions")]
@@ -798,7 +798,7 @@ def test_spec_formatting(self):
         for named_str, prop in package_segments:
             expected = getattr(spec, prop, "")
             actual = spec.format(named_str)
-            assert str(expected) == actual
+            assert str(expected).strip() == actual
 
         for named_str, expected in sigil_package_segments:
             actual = spec.format(named_str)