Repository: spack/spack

Issue:
Concretization performance is really bad after #29926
### Steps to reproduce

I know that #29926 acknowledged that there were performance issues but it seems to be much worse than indicated there.
### Concretization of texlive prior to change
```console
$ time spack spec texlive
real    0m10.752s
user    0m10.104s
sys     0m0.192s
```
### Concretization of texlive after the change
```console
$ time spack spec texlive
real    11m21.115s
user    11m18.896s
sys     0m1.580s
```

### Error message

```console
$spack --debug spec texlive
==> [2022-06-04-20:36:29.467464] Imported spec from built-in commands
==> [2022-06-04-20:36:29.468258] Imported spec from built-in commands
==> [2022-06-04-20:36:29.469563] Reading config file /home/gpjohnsn/spack/etc/spack/defaults/config.yaml
==> [2022-06-04-20:36:29.481497] Reading config file /home/gpjohnsn/spack/etc/spack/defaults/bootstrap.yaml
==> [2022-06-04-20:36:29.489377] Reading config file /home/gpjohnsn/spack/etc/spack/defaults/config.yaml
==> [2022-06-04-20:36:29.500719] DATABASE LOCK TIMEOUT: 3s
==> [2022-06-04-20:36:29.500809] PACKAGE LOCK TIMEOUT: No timeout
==> [2022-06-04-20:36:29.501192] Reading config file /home/gpjohnsn/spack/etc/spack/defaults/bootstrap.yaml
==> [2022-06-04-20:36:29.505200] DATABASE LOCK TIMEOUT: 3s
==> [2022-06-04-20:36:29.505269] PACKAGE LOCK TIMEOUT: No timeout
==> [2022-06-04-20:36:29.505334] [BOOTSTRAP CONFIG SCOPE] name=_builtin
==> [2022-06-04-20:36:29.505626] [BOOTSTRAP CONFIG SCOPE] name=defaults, path=/home/gpjohnsn/spack/etc/spack/defaults
==> [2022-06-04-20:36:29.505674] [BOOTSTRAP CONFIG SCOPE] name=defaults/linux, path=/home/gpjohnsn/spack/etc/spack/defaults/linux
==> [2022-06-04-20:36:29.505723] [BOOTSTRAP CONFIG SCOPE] name=bootstrap, path=/home/gpjohnsn/.spack/bootstrap/config
==> [2022-06-04-20:36:29.505762] [BOOTSTRAP CONFIG SCOPE] name=bootstrap/linux, path=/home/gpjohnsn/.spack/bootstrap/config/linux
==> [2022-06-04-20:36:29.506001] Reading config file /home/gpjohnsn/.spack/bootstrap/config/linux/compilers.yaml
==> [2022-06-04-20:36:29.522482] [BOOTSTRAP ROOT SPEC] clingo-bootstrap@spack+python %gcc target=x86_64
==> [2022-06-04-20:36:29.522571] [BOOTSTRAP MODULE clingo] Try importing from Python
==> [2022-06-04-20:36:29.522774] Reading config file /home/gpjohnsn/spack/etc/spack/defaults/bootstrap.yaml
==> [2022-06-04-20:36:29.644403] Reading config file /home/gpjohnsn/spack/etc/spack/defaults/config.yaml
==> [2022-06-04-20:36:50.942029] [BOOTSTRAP MODULE clingo] The installed spec "clingo-bootstrap@spack+python %gcc target=x86_64 ^python@3.8 /shqedxgvjnhiwdcdrvjhbd73jaevv7wt" provides the "clingo" Python module
==> [2022-06-04-20:36:50.942341] Reading config file /home/gpjohnsn/spack/etc/spack/defaults/concretizer.yaml
==> [2022-06-04-20:36:50.945789] Reading config file /home/gpjohnsn/spack/etc/spack/defaults/repos.yaml
==> [2022-06-04-20:36:50.948944] Reading config file /home/gpjohnsn/spack/etc/spack/defaults/config.yaml
==> [2022-06-04-20:36:50.960601] Reading config file /home/gpjohnsn/spack/etc/spack/defaults/mirrors.yaml
==> [2022-06-04-20:36:51.541328] Reading config file /home/gpjohnsn/.spack/linux/compilers.yaml
==> [2022-06-04-20:36:51.545425] Reading config file /home/gpjohnsn/spack/etc/spack/defaults/packages.yaml
        ==> [2022-06-04-20:47:45.559882] build(texlive)
==> [2022-06-04-20:47:45.560375] build(cairo)
==> [2022-06-04-20:47:45.560405] build(freetype)
==> [2022-06-04-20:47:45.560421] build(ghostscript)
==> [2022-06-04-20:47:45.560434] build(gmp)
==> [2022-06-04-20:47:45.560457] build(harfbuzz)
==> [2022-06-04-20:47:45.560477] build(icu4c)
==> [2022-06-04-20:47:45.560492] build(libgd)
==> [2022-06-04-20:47:45.560504] build(libpaper)
==> [2022-06-04-20:47:45.560537] build(libpng)
==> [2022-06-04-20:47:45.560589] build(libxaw)
==> [2022-06-04-20:47:45.560613] build(libxt)
==> [2022-06-04-20:47:45.560629] build(mpfr)
==> [2022-06-04-20:47:45.560643] build(perl)
==> [2022-06-04-20:47:45.560657] build(pixman)
==> [2022-06-04-20:47:45.560671] build(poppler)
==> [2022-06-04-20:47:45.560684] build(teckit)
==> [2022-06-04-20:47:45.560723] build(zlib)
==> [2022-06-04-20:47:45.560762] build(zziplib)
==> [2022-06-04-20:47:45.560786] build(pkgconf)
==> [2022-06-04-20:47:45.560802] build(coreutils)
==> [2022-06-04-20:47:45.560817] build(expat)
==> [2022-06-04-20:47:45.560831] build(cmake)
==> [2022-06-04-20:47:45.560862] build(poppler-data)
==> [2022-06-04-20:47:45.560903] build(berkeley-db)
==> [2022-06-04-20:47:45.560945] build(gdbm)
==> [2022-06-04-20:47:45.560994] build(autoconf-archive)
==> [2022-06-04-20:47:45.561020] build(texinfo)
==> [2022-06-04-20:47:45.561036] build(kbproto)
==> [2022-06-04-20:47:45.561050] build(libice)
==> [2022-06-04-20:47:45.561082] build(libsm)
==> [2022-06-04-20:47:45.561125] build(libxmu)
==> [2022-06-04-20:47:45.561185] build(libxpm)
==> [2022-06-04-20:47:45.561230] build(util-macros)
==> [2022-06-04-20:47:45.561278] build(xextproto)
==> [2022-06-04-20:47:45.561306] build(xproto)
==> [2022-06-04-20:47:45.561321] build(gettext)
==> [2022-06-04-20:47:45.561352] build(gobject-introspection)
==> [2022-06-04-20:47:45.561393] build(graphite2)
==> [2022-06-04-20:47:45.561436] build(meson)
==> [2022-06-04-20:47:45.561497] build(ninja)
==> [2022-06-04-20:47:45.561521] build(gtkplus)
==> [2022-06-04-20:47:45.561537] build(krb5)
==> [2022-06-04-20:47:45.561566] build(lcms)
==> [2022-06-04-20:47:45.561579] build(libtiff)
==> [2022-06-04-20:47:45.561608] build(bzip2)
==> [2022-06-04-20:47:45.561632] build(autoconf)
==> [2022-06-04-20:47:45.561665] build(automake)
==> [2022-06-04-20:47:45.561689] build(fontconfig)
==> [2022-06-04-20:47:45.561731] build(glib)
==> [2022-06-04-20:47:45.561786] build(librsvg)
==> [2022-06-04-20:47:45.561826] build(libtool)
==> [2022-06-04-20:47:45.561850] build(libx11)
==> [2022-06-04-20:47:45.561866] build(libxcb)
==> [2022-06-04-20:47:45.561881] build(libxext)
==> [2022-06-04-20:47:45.561899] build(libxrender)
==> [2022-06-04-20:47:45.561913] build(m4)
==> [2022-06-04-20:47:45.561942] build(python)
==> [2022-06-04-20:47:45.561968] build(libiconv)
==> [2022-06-04-20:47:45.562001] build(libjpeg-turbo)
==> [2022-06-04-20:47:45.562040] build(libsigsegv)
==> [2022-06-04-20:47:45.562080] build(renderproto)
==> [2022-06-04-20:47:45.562129] build(libpthread-stubs)
==> [2022-06-04-20:47:45.562176] build(libxau)
==> [2022-06-04-20:47:45.562203] build(libxdmcp)
==> [2022-06-04-20:47:45.562217] build(xcb-proto)
==> [2022-06-04-20:47:45.562230] build(libcroco)
==> [2022-06-04-20:47:45.562256] build(rust)
==> [2022-06-04-20:47:45.562303] build(font-util)
==> [2022-06-04-20:47:45.562343] build(diffutils)
==> [2022-06-04-20:47:45.562383] build(at-spi2-atk)
==> [2022-06-04-20:47:45.562423] build(atk)
==> [2022-06-04-20:47:45.562464] build(fixesproto)
==> [2022-06-04-20:47:45.562504] build(fribidi)
==> [2022-06-04-20:47:45.562528] build(gdk-pixbuf)
==> [2022-06-04-20:47:45.562544] build(libepoxy)
==> [2022-06-04-20:47:45.562559] build(pango)
==> [2022-06-04-20:47:45.562589] build(shared-mime-info)
==> [2022-06-04-20:47:45.562629] build(xrandr)
==> [2022-06-04-20:47:45.562654] build(py-pip)
==> [2022-06-04-20:47:45.562687] build(py-setuptools)
==> [2022-06-04-20:47:45.562728] build(py-wheel)
==> [2022-06-04-20:47:45.562768] build(libffi)
==> [2022-06-04-20:47:45.562808] build(tar)
==> [2022-06-04-20:47:45.562847] build(xz)
==> [2022-06-04-20:47:45.562871] build(xtrans)
==> [2022-06-04-20:47:45.562887] build(readline)
==> [2022-06-04-20:47:45.562901] build(bison)
==> [2022-06-04-20:47:45.562915] build(dbus)
==> [2022-06-04-20:47:45.562945] build(flex)
==> [2022-06-04-20:47:45.562984] build(gperf)
==> [2022-06-04-20:47:45.563024] build(inputproto)
==> [2022-06-04-20:47:45.563071] build(libxi)
==> [2022-06-04-20:47:45.563116] build(libxkbcommon)
==> [2022-06-04-20:47:45.563140] build(libxml2)
==> [2022-06-04-20:47:45.563156] build(libxrandr)
==> [2022-06-04-20:47:45.563187] build(libxtst)
==> [2022-06-04-20:47:45.563228] build(llvm)
==> [2022-06-04-20:47:45.563272] build(pcre)
==> [2022-06-04-20:47:45.563317] build(sqlite)
==> [2022-06-04-20:47:45.563359] build(zstd)
==> [2022-06-04-20:47:45.563403] build(libssh2)
==> [2022-06-04-20:47:45.563448] build(ncurses)
==> [2022-06-04-20:47:45.563491] build(openssl)
==> [2022-06-04-20:47:45.563532] build(libbsd)
==> [2022-06-04-20:47:45.563572] build(nasm)
==> [2022-06-04-20:47:45.563612] build(util-linux-uuid)
==> [2022-06-04-20:47:45.563636] build(mesa)
==> [2022-06-04-20:47:45.563654] build(libmd)
==> [2022-06-04-20:47:45.563671] build(libedit)
==> [2022-06-04-20:47:45.563685] build(perl-data-dumper)
==> [2022-06-04-20:47:45.563720] build(swig)
==> [2022-06-04-20:47:45.563774] build(recordproto)
==> [2022-06-04-20:47:45.563799] build(xkbdata)
==> [2022-06-04-20:47:45.563831] build(libxfixes)
==> [2022-06-04-20:47:45.563873] build(pigz)
==> [2022-06-04-20:47:45.563923] build(randrproto)
==> [2022-06-04-20:47:45.563948] build(intltool)
==> [2022-06-04-20:47:45.563964] build(json-glib)
==> [2022-06-04-20:47:45.563995] build(at-spi2-core)
==> [2022-06-04-20:47:45.564035] build(bdftopcf)
==> [2022-06-04-20:47:45.564059] build(mkfontdir)
==> [2022-06-04-20:47:45.564075] build(mkfontscale)
==> [2022-06-04-20:47:45.564090] build(gmake)
==> [2022-06-04-20:47:45.564122] build(libgit2)
==> [2022-06-04-20:47:45.564166] build(docbook-xml)
==> [2022-06-04-20:47:45.564207] build(docbook-xsl)
==> [2022-06-04-20:47:45.564248] build(libxft)
==> [2022-06-04-20:47:45.564289] build(findutils)
==> [2022-06-04-20:47:45.564329] build(hwloc)
==> [2022-06-04-20:47:45.564370] build(libpciaccess)
==> [2022-06-04-20:47:45.564410] build(binutils)
==> [2022-06-04-20:47:45.564450] build(glproto)
==> [2022-06-04-20:47:45.564489] build(py-mako)
==> [2022-06-04-20:47:45.564529] build(libunwind)
==> [2022-06-04-20:47:45.564569] build(libfontenc)
==> [2022-06-04-20:47:45.564609] build(fontsproto)
==> [2022-06-04-20:47:45.564649] build(libxfont)
==> [2022-06-04-20:47:45.564689] build(perl-xml-parser)
==> [2022-06-04-20:47:45.564730] build(xkbcomp)
==> [2022-06-04-20:47:45.564771] build(py-markupsafe)
==> [2022-06-04-20:47:45.564811] build(libxkbfile)
==> [2022-06-04-20:47:45.564851] build(perl-libwww-perl)
==> [2022-06-04-20:47:45.564892] build(perl-module-build)
==> [2022-06-04-20:47:45.564932] build(perl-encode-locale)
==> [2022-06-04-20:47:45.564973] build(perl-file-listing)
==> [2022-06-04-20:47:45.565013] build(perl-html-parser)
==> [2022-06-04-20:47:45.565066] build(perl-http-cookies)
==> [2022-06-04-20:47:45.565107] build(perl-http-daemon)
==> [2022-06-04-20:47:45.565147] build(perl-http-date)
==> [2022-06-04-20:47:45.565187] build(perl-http-message)
==> [2022-06-04-20:47:45.565227] build(perl-http-negotiate)
==> [2022-06-04-20:47:45.565267] build(perl-lwp-mediatypes)
==> [2022-06-04-20:47:45.565310] build(perl-net-http)
==> [2022-06-04-20:47:45.565350] build(perl-try-tiny)
==> [2022-06-04-20:47:45.565390] build(perl-uri)
==> [2022-06-04-20:47:45.565431] build(perl-www-robotrules)
==> [2022-06-04-20:47:45.565472] build(perl-test-needs)
==> [2022-06-04-20:47:45.565522] build(perl-io-html)
==> [2022-06-04-20:47:45.565571] build(perl-module-build-tiny)
==> [2022-06-04-20:47:45.565620] build(perl-html-tagset)
==> [2022-06-04-20:47:45.565669] build(perl-extutils-config)
==> [2022-06-04-20:47:45.565730] build(perl-extutils-helpers)
==> [2022-06-04-20:47:45.565783] build(perl-extutils-installpaths)
==> [2022-06-04-20:47:45.616489] Ordered hashes [cairo]: cairo/401/7c4da77767fe9feb03f8051def0832f0c67f99162913275cfa127a88df19cf51
==> [2022-06-04-20:47:45.616716] Ordered hashes [autoconf]: autoconf/607/7793209b33013dc0f81208718c68440c5aae80e7a1c4b8d336e382525af791a7, autoconf/608/35c449281546376449766f92d49fc121ca50e330e60fefcfc9be2af3253082c2, autoconf/609/a49dd5bac3b62daa0ff688ab4d508d71dbd2f4f8d7e2a02321926346161bf3ee
==> [2022-06-04-20:47:45.616783] Ordered hashes [m4]: m4/974/bfdffa7c2eb01021d5849b36972c069693654ad826c1a20b53534009a4ec7a89, m4/975/9dc5fbd0d5cb1037ab1e6d0ecc74a30df218d0a94bdd5a02759a97f62daca573
==> [2022-06-04-20:47:45.616854] Ordered hashes [berkeley-db]: berkeley-db/1133/b231fcc4d5cff05e5c3a4814f6a5af0e9a966428dc2176540d2c05aff41de522
==> [2022-06-04-20:47:45.616995] Ordered hashes [zlib]: zlib/1430/0d38234384870bfd34dfcb738a9083952656f0c766a0f5990b1893076b084b76
==> [2022-06-04-20:47:45.617305] Ordered hashes [python]: python/7/0d98e93189bc278fbc37a50ed7f183bd8aaf249a8e1670a465f0db6bb4f8cf87, python/8/f2fd060afc4b4618fe8104c4c5d771f36dc55b1db5a4623785a4ea707ec72fb4, python/11/4c2457325f2b608b1b6a2c63087df8c26e07db3e3d493caf36a56f0ecf6fb768
==> [2022-06-04-20:47:45.617523] Ordered hashes [glib]: glib/352/b3fd45063a19c871048aa1f28692293ab8971a871bdcbe65f06f17fdd79db9e2
==> [2022-06-04-20:47:45.617608] Ordered hashes [meson]: meson/1816/aa6c50d5a2aeb1a487d16f6712be4357fefb923aae37ab830699b07338388287
==> [2022-06-04-20:47:45.618048] Ordered hashes [docbook-xsl]: docbook-xsl/207/a92c39715c54949ba9369add1809527b8f155b7e2a2b2e30cb4b39ee715f2e30
==> [2022-06-04-20:47:45.618273] Ordered hashes [findutils]: findutils/1326/440b9543365b4692a2e6e0b5674809659846658d34d1dfc542c4397c8d668b92
==> [2022-06-04-20:47:45.618651] Ordered hashes [intltool]: intltool/797/ca9d6562f29f06c64150f50369a24402b7aa01a3a0dc73dce55106f3224330a1
==> [2022-06-04-20:47:45.619025] Ordered hashes [mesa]: mesa/1089/ada85be0f30bdd61a7c4eb24afec741da81bc411f7227e2d9014c862eef58b64
==> [2022-06-04-20:47:45.619188] Ordered hashes [llvm]: llvm/257/f920173d0258c934f246fcee65864bcbe9dcb1be32d4298f9b6323c364919b99
==> [2022-06-04-20:47:45.619608] Ordered hashes [graphite2]: graphite2/390/39e58d629b090bee4a319bfc581e59c4f4b0453f8b13ec77693a52be8d13a90d
==> [2022-06-04-20:47:45.619778] Ordered hashes [texinfo]: texinfo/1331/1732115f651cff98989cb0215d8f64da5e0f7911ebf0c13b064920f088f2ffe1, texinfo/1334/12f6edb0c6b270b8c8dba2ce17998c580db01182d871ee32b7b6e4129bd1d23a
==> [2022-06-04-20:47:45.620248] Ordered hashes [autoconf-archive]: autoconf-archive/1455/130cd480d3d0157da25ec831723da1c9fb212d20d8906bde603b18b424b2ef1c
...
```

### Information on your system

```console
* **Spack:** 0.19.0.dev0 (03c54aebdd914ada16925db7ca455132c45b8a45)
* **Python:** 3.8.10
* **Platform:** linux-ubuntu20.04-icelake
* **Concretizer:** clingo
```

### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have searched the issues of this repo and believe this is not a duplicate
- [X] I have run the failing commands in debug mode and reported the output


Discussion:
I'm using:

* **Spack:** 0.19.0.dev0 (8c58c14c3d77420c86b0ae852bc4a680c2aca472)
* **Python:** 3.8.10
* **Platform:** linux-ubuntu20.04-icelake
* **Concretizer:** clingo

and in my case:
```console
$ spack solve --timers texlive
Time:
    setup:         5.3355
    load:          0.0200
    ground:        2.0644
    solve:         2.6992
Total: 11.0180
```
so nowhere near the 10 minutes. Do you still experience this performance regression?
Well, this is interesting. I first noticed the problem on my CentOS-7 system, which is my main system, used for my HPC stack. I used that to `git bisect` this. I then verified it on a Ubuntu WSL2 system, with a fresh checkout of spack. The problem was the same, and the "fix" was to back out 19087c9d35abd849975a8a5c0451ed47e958c23d.

Now, however, I am not seeing the problem on my CentOS system, but am still seeing it on my Ubuntu WSL2 system. 
```console
$ spack solve --timers texlive
Time:
    setup:         4.8320
    load:          0.0188
    ground:        1.6895
    solve:         667.0861
Total: 674.6858

==> Best of 5 considered solutions.
==> Optimization Criteria:
  Priority  Criterion                                            Installed  ToBuild
  1         number of input specs not concretized                        -        0
  2         number of packages to build (vs. reuse)                      -        0
  3         deprecated versions used                                     0        0
  4         version weight                                               0        0
  5         number of non-default variants (roots)                       0        0
  6         preferred providers for roots                                0        0
  7         default values of variants not being used (roots)            0        0
  8         number of non-default variants (non-roots)                   0        0
  9         preferred providers (non-roots)                              0        0
  10        compiler mismatches                                          0        0
  11        OS mismatches                                                0        0
  12        non-preferred OS's                                           0        0
  13        version badness                                             20        0
  14        default values of variants not being used (non-roots)       13        0
  15        non-preferred compilers                                      0        0
  16        target mismatches                                            0        0
  17        non-preferred targets                                     4920        0
```
Reverting 19087c9d35abd849975a8a5c0451ed47e958c23d:
```console
$ spack solve --timers texlive
Time:
    setup:         4.6419
    load:          0.0191
    ground:        1.7244
    solve:         2.3350
Total: 9.7520

==> Best of 5 considered solutions.
==> Optimization Criteria:
  Priority  Criterion                                            Installed  ToBuild
  1         number of input specs not concretized                        -        0
  2         number of packages to build (vs. reuse)                      -        0
  3         deprecated versions used                                     0        0
  4         version weight                                               0        0
  5         number of non-default variants (roots)                       0        0
  6         preferred providers for roots                                0        0
  7         default values of variants not being used (roots)            0        0
  8         number of non-default variants (non-roots)                   0        0
  9         preferred providers (non-roots)                              0        0
  10        compiler mismatches                                          0        0
  11        OS mismatches                                                0        0
  12        non-preferred OS's                                           0        0
  13        version badness                                             20        0
  14        default values of variants not being used (non-roots)       13        0
  15        non-preferred compilers                                      0        0
  16        target mismatches                                            0        0
  17        non-preferred targets                                        0        0
```

I had updated the spack repository on my CentOS system but even reverting back to the commit that it was on when I filed this issue, I get normal times now. Perhaps I am the only one seeing this but it feels like something is lurking there. Let me know if there is anyway to dig into the long solve time on the Ubuntu WSL2 system. 
Hmm, I also redid my bootstrap on my CentOS-7 system. Could there be something with that?

### CentOS-7
```console
$ spack find --bootstrap
==> Warning: `spack find --bootstrap` is deprecated and will be removed in v0.19.
  Use `spack --bootstrap find` instead.
==> Showing internal bootstrap store at "/Users/gpjohnsn/.spack/bootstrap/store"
==> 3 installed packages
-- linux-centos7-x86_64 / gcc@4.8.5 -----------------------------
patchelf@0.13.1

-- linux-rhel5-x86_64 / gcc@9.3.0 -------------------------------
clingo-bootstrap@spack  python@3.6
```
### Ubuntu WSL2
```console
$ spack find --bootstrap
==> Warning: `spack find --bootstrap` is deprecated and will be removed in v0.19.
  Use `spack --bootstrap find` instead.
==> Showing internal bootstrap store at "/home/gpjohnsn/.spack/bootstrap/store"
==> 3 installed packages
-- linux-centos7-x86_64 / gcc@10.2.1 ----------------------------
clingo-bootstrap@spack  python@3.8

-- linux-ubuntu20.04-x86_64 / gcc@9.4.0 -------------------------
patchelf@0.13.1
```
So, on CentOS 7 you are using the first set of binaries we generated to bootstrap Spack (notice the rhel5, i.e. manylinux1), while on WSL2 you're using the latest binaries (CentOS 7 a.k.a. manylinux2014). That said I have never seen different performance due to using one and not the other.
I'd definitely try to get to the bottom of this, since as you say it would be good to rule out having a huge performance regression waiting to come out. I assume when you get the 10 minutes you get them consistently. In case, can you dump:
```console
$ spack solve --show=asp texlive &> texlive.lp
```
and post it here? If I can reproduce the 10 minutes solve with just `clingo` then we'll look into what causes that and potentially which configuration entry you customized that triggered this.
@glennpj Just a guess. Could it be that in the meanwhile you uninstalled a few specs? Or removed a binary mirror from your configuration? With `--reuse` by default we are stateful, so it might be that the spec being reused are a more likely cause of the slowdown you see, rather than the binary used for clingo
I found one problem where texlive takes a lot. Posting the ASP (all in one file) here.
[problem.lp.txt](https://github.com/spack/spack/files/8929373/problem.lp.txt)

Thanks for reporting this @glennpj ! Nailed a reproducer down. It is effectively related to #29926 

Start from a version of Spack with nothing set in `packages.yaml` in terms of target preferences:
```console
$ time spack spec texlive
[ ... ]
real	7m46,639s
user	7m45,783s
sys	0m0,414s
```
Then add target preferences to `packages.yaml`:
```console
$ spack config add packages:all:target:[x86_64]
$ time spack spec texlive
[ ... ]
real	0m13,829s
user	0m12,734s
sys	0m0,291s
```
Looking at the way the slow problem above is getting solved, it seems that to decide on the `non-preferred-targets` criterion `clingo` has to explore a lot of the search space. For the slow case you posted above, where the optimal value is 4920, I think it has to prove that no value between 0 and 4920 can give a better solution.
A solution to this issue might be what I propose in #31199 (which of course needs some discussion to find an agreement on). I'll try to see if I can find some workaround in the meanwhile.
> So, on CentOS 7 you are using the first set of binaries we generated to bootstrap Spack (notice the rhel5, i.e. manylinux1), while on WSL2 you're using the latest binaries (CentOS 7 a.k.a. manylinux2014). That said I have never seen different performance due to using one and not the other.

I am using whatever Spack bootstraps to, I did not select anything. I am not sure what heuristics are at play.
Sorry for the delay, I have fallen behind on Spack again. It seems like this issue might also be related to the oddities I reported in #30996.
> Sorry for the delay, I have fallen behind on Spack again. 

No worries, Spack is moving faster than usual lately - so it's easy to fall behind. I'm looking into this, but it might take a few days before I post a proper solution.
> Start from a version of Spack with nothing set in `packages.yaml` in terms of target preferences:

Yes, that is how this showed up on my Ububtu WSL2 system. I simply cloned spack and ran a concretization without any additional setup. What is puzzling to me is why the problem cleared up on my production system. 
I spent some time doing a small performance study on these specs [radiuss.txt](https://github.com/spack/spack/files/8949322/radiuss.txt) There are indeed a few outliers which require around 10 min. instead of around 10 sec. So the issue is not specific to `texlive`. 

![total_time_vs_deps](https://user-images.githubusercontent.com/4199709/174812662-82528cdb-00af-4a06-bb6e-8959a57a87d5.png)


I partially implemented the idea in #31199, by still maintaining a per-package soft-preference - but avoiding the 30 weight offset. Here are the timings, which are much better:
![total_time_vs_deps](https://user-images.githubusercontent.com/4199709/174977442-a1e31fc4-6b86-4423-8a8b-4675e95e2ab4.png)


Resolving patch:
diff --git a/lib/spack/spack/solver/asp.py b/lib/spack/spack/solver/asp.py
--- a/lib/spack/spack/solver/asp.py
+++ b/lib/spack/spack/solver/asp.py
@@ -717,7 +717,7 @@ def __init__(self, tests=False):
         self.variant_values_from_specs = set()
         self.version_constraints = set()
         self.target_constraints = set()
-        self.default_targets = {}
+        self.default_targets = []
         self.compiler_version_constraints = set()
         self.post_facts = []
 
@@ -1178,29 +1178,19 @@ def target_preferences(self, pkg_name):
         if not self.target_specs_cache:
             self.target_specs_cache = [
                 spack.spec.Spec('target={0}'.format(target_name))
-                for target_name in archspec.cpu.TARGETS
+                for _, target_name in self.default_targets
             ]
 
-        target_specs = self.target_specs_cache
-        preferred_targets = [x for x in target_specs if key_fn(x) < 0]
+        package_targets = self.target_specs_cache[:]
+        package_targets.sort(key=key_fn)
 
-        for i, preferred in enumerate(preferred_targets):
-            self.gen.fact(fn.package_target_weight(
-                str(preferred.architecture.target), pkg_name, i
-            ))
-
-        # generate weights for non-preferred targets on a per-package basis
-        default_targets = {
-            name: weight for
-            name, weight in self.default_targets.items()
-            if not any(preferred.architecture.target.name == name
-                       for preferred in preferred_targets)
-        }
-
-        num_preferred = len(preferred_targets)
-        for name, weight in default_targets.items():
-            self.gen.fact(fn.default_target_weight(
-                name, pkg_name, weight + num_preferred + 30
+        offset = 0
+        best_default = self.default_targets[0][1]
+        for i, preferred in enumerate(package_targets):
+            if str(preferred.architecture.target) == best_default and i != 0:
+                offset = 100
+            self.gen.fact(fn.target_weight(
+                pkg_name, str(preferred.architecture.target), i + offset
             ))
 
     def flag_defaults(self):
@@ -1604,11 +1594,12 @@ def target_defaults(self, specs):
             # these are stored to be generated as facts later offset by the
             # number of preferred targets
             if target.name in best_targets:
-                self.default_targets[target.name] = i
+                self.default_targets.append((i, target.name))
                 i += 1
             else:
-                self.default_targets[target.name] = 100
+                self.default_targets.append((100, target.name))
 
+            self.default_targets = list(sorted(set(self.default_targets)))
             self.gen.newline()
 
     def virtual_providers(self):


Test patch: