Repository: spack/spack

Issue:
Conretizer uses old dependency version
### Steps to reproduce

When building a new package which depends on `py-numba` spack concretizes to use the oldest available version`py-numba@0.35.0` instead the newest one `py-numba@0.55.1` without any reason. Therefore also downgrading python to version 3.6.15

Reproduce:
- fresh spack clone
- no config (removed `~/.spack`)
- create a new package with a `py-numba` dependency

I used `py-fracridge` (the package I wanted to create when encountering this problem) with the `package.py`
```python
from spack import *


class PyFracridge(PythonPackage):
    """Fractional Ridge Regression."""

    homepage = "https://nrdg.github.io/fracridge"
    pypi     = "fracridge/fracridge-1.4.3.tar.gz"

    version('1.4.3', sha256='0446d486f00fea02110567fd9df14b8b2a7b155dc72700af9201873ea11c27cc')

    depends_on('py-numba', type=('build', 'run'))
```

ping @adamjstewart @alalazo 

### Error message

```console
$ spack solve py-fracridge
==> Best of 6 considered solutions.
==> Optimization Criteria:
  Priority  Criterion                                            Installed  ToBuild
  1         number of packages to build (vs. reuse)                      -       30
  2         deprecated versions used                                     0        0
  3         version weight                                               0        0
  4         number of non-default variants (roots)                       0        0
  5         preferred providers for roots                                0        0
  6         default values of variants not being used (roots)            0        0
  7         number of non-default variants (non-roots)                   0        0
  8         preferred providers (non-roots)                              0        0
  9         compiler mismatches                                          0        0
  10        OS mismatches                                                0        0
  11        non-preferred OS's                                           0        0
  12        version badness                                              0       71
  13        default values of variants not being used (non-roots)        0        1
  14        non-preferred compilers                                      0        0
  15        target mismatches                                            0        0
  16        non-preferred targets                                        0        0

py-fracridge@1.4.3%gcc@11.3.1 arch=linux-fedora35-skylake
    ^py-numba@0.35.0%gcc@11.3.1 arch=linux-fedora35-skylake
        ^py-numpy@1.19.5%gcc@11.3.1+blas+lapack patches=8a9d5d1 arch=linux-fedora35-skylake
            ^openblas@0.3.20%gcc@11.3.1~bignuma~consistent_fpcsr~ilp64+locking+pic+shared symbol_suffix=none threads=none arch=linux-fedora35-skylake
                ^perl@5.34.1%gcc@11.3.1+cpanm+shared+threads arch=linux-fedora35-skylake
                    ^berkeley-db@18.1.40%gcc@11.3.1+cxx~docs+stl patches=b231fcc arch=linux-fedora35-skylake
                    ^bzip2@1.0.8%gcc@11.3.1~debug~pic+shared arch=linux-fedora35-skylake
                        ^diffutils@3.8%gcc@11.3.1 arch=linux-fedora35-skylake
                            ^libiconv@1.16%gcc@11.3.1 libs=shared,static arch=linux-fedora35-skylake
                    ^gdbm@1.19%gcc@11.3.1 arch=linux-fedora35-skylake
                        ^readline@8.1%gcc@11.3.1 arch=linux-fedora35-skylake
                            ^ncurses@6.2%gcc@11.3.1~symlinks+termlib abi=none arch=linux-fedora35-skylake
                                ^pkgconf@1.8.0%gcc@11.3.1 arch=linux-fedora35-skylake
                    ^zlib@1.2.12%gcc@11.3.1+optimize+pic+shared patches=0d38234 arch=linux-fedora35-skylake
            ^py-cython@0.29.24%gcc@11.3.1 arch=linux-fedora35-skylake
                ^py-pip@21.3.1%gcc@11.3.1 arch=linux-fedora35-skylake
                    ^python@3.6.15%gcc@11.3.1+bz2+ctypes+dbm~debug+ensurepip+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib arch=linux-fedora35-skylake
                        ^expat@2.4.8%gcc@11.3.1+libbsd arch=linux-fedora35-skylake
                            ^libbsd@0.11.5%gcc@11.3.1 arch=linux-fedora35-skylake
                                ^libmd@1.0.4%gcc@11.3.1 arch=linux-fedora35-skylake
                        ^gettext@0.21%gcc@11.3.1+bzip2+curses+git~libunistring+libxml2+tar+xz arch=linux-fedora35-skylake
                            ^libxml2@2.9.13%gcc@11.3.1~python arch=linux-fedora35-skylake
                                ^xz@5.2.5%gcc@11.3.1~pic libs=shared,static arch=linux-fedora35-skylake
                            ^tar@1.34%gcc@11.3.1 arch=linux-fedora35-skylake
                        ^libffi@3.4.2%gcc@11.3.1 arch=linux-fedora35-skylake
                        ^openssl@1.1.1o%gcc@11.3.1~docs~shared certs=system arch=linux-fedora35-skylake
                        ^sqlite@3.38.5%gcc@11.3.1+column_metadata+dynamic_extensions+fts~functions+rtree arch=linux-fedora35-skylake
                        ^util-linux-uuid@2.37.4%gcc@11.3.1 arch=linux-fedora35-skylake
                ^py-setuptools@59.4.0%gcc@11.3.1 arch=linux-fedora35-skylake
                    ^py-wheel@0.37.0%gcc@11.3.1 arch=linux-fedora35-skylake
```


### Information on your system

* **Spack:** 0.18.0.dev0 (d7d0c892d8e33ed0157140a28ef3e14c5bfe6577)
* **Python:** 3.10.4
* **Platform:** linux-fedora35-skylake
* **Concretizer:** clingo


### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have searched the issues of this repo and believe this is not a duplicate
- [X] I have run the failing commands in debug mode and reported the output


Discussion:
What happens if you try:
```
$ spack solve py-fracridge ^py-numba@0.55.1
```
?
Then it concretizes using this version without error

```
$ spack solve py-fracridge ^py-numba@0.55.1
==> Best of 5 considered solutions.
==> Optimization Criteria:
  Priority  Criterion                                            Installed  ToBuild
  1         number of packages to build (vs. reuse)                      -       45
  2         deprecated versions used                                     0        0
  3         version weight                                               0        0
  4         number of non-default variants (roots)                       0        0
  5         preferred providers for roots                                0        0
  6         default values of variants not being used (roots)            0        0
  7         number of non-default variants (non-roots)                   0        1
  8         preferred providers (non-roots)                              0        0
  9         compiler mismatches                                          0        0
  10        OS mismatches                                                0        0
  11        non-preferred OS's                                           0        0
  12        version badness                                              0       15
  13        default values of variants not being used (non-roots)        0        5
  14        non-preferred compilers                                      0        0
  15        target mismatches                                            0        0
  16        non-preferred targets                                        0        0

py-fracridge@1.4.3%gcc@11.3.1 arch=linux-fedora35-skylake
    ^py-numba@0.55.1%gcc@11.3.1 arch=linux-fedora35-skylake
        ^py-llvmlite@0.38.0%gcc@11.3.1 arch=linux-fedora35-skylake
            ^binutils@2.38%gcc@11.3.1~gas+gold~headers~interwork+ld~libiberty~lto+nls+plugins libs=shared,static arch=linux-fedora35-skylake
                ^diffutils@3.8%gcc@11.3.1 arch=linux-fedora35-skylake
                    ^libiconv@1.16%gcc@11.3.1 libs=shared,static arch=linux-fedora35-skylake
                ^gettext@0.21%gcc@11.3.1+bzip2+curses+git~libunistring+libxml2+tar+xz arch=linux-fedora35-skylake
                    ^bzip2@1.0.8%gcc@11.3.1~debug~pic+shared arch=linux-fedora35-skylake
                    ^libxml2@2.9.13%gcc@11.3.1~python arch=linux-fedora35-skylake
                        ^pkgconf@1.8.0%gcc@11.3.1 arch=linux-fedora35-skylake
                        ^xz@5.2.5%gcc@11.3.1~pic libs=shared,static arch=linux-fedora35-skylake
                        ^zlib@1.2.12%gcc@11.3.1+optimize+pic+shared patches=0d38234 arch=linux-fedora35-skylake
                    ^ncurses@6.2%gcc@11.3.1~symlinks+termlib abi=none arch=linux-fedora35-skylake
                    ^tar@1.34%gcc@11.3.1 arch=linux-fedora35-skylake
            ^llvm@11.1.0%gcc@11.3.1+clang~code_signing+compiler-rt~cuda~flang+gold+internal_unwind~ipo+libcxx~link_llvm_dylib+lld+lldb~llvm_dylib~mlir~omp_as_runtime~omp_debug~omp_tsan+polly~python~shared_libs~split_dwarf~z3 build_type=Release patches=1b1f41a,6811ad4,f920173 targets=none version_suffix=none arch=linux-fedora35-skylake
                ^cmake@3.23.1%gcc@11.3.1~doc+ncurses+ownlibs~qt build_type=Release arch=linux-fedora35-skylake
                    ^openssl@1.1.1o%gcc@11.3.1~docs~shared certs=system arch=linux-fedora35-skylake
                        ^perl@5.34.1%gcc@11.3.1+cpanm+shared+threads arch=linux-fedora35-skylake
                            ^berkeley-db@18.1.40%gcc@11.3.1+cxx~docs+stl patches=b231fcc arch=linux-fedora35-skylake
                            ^gdbm@1.19%gcc@11.3.1 arch=linux-fedora35-skylake
                                ^readline@8.1%gcc@11.3.1 arch=linux-fedora35-skylake
                ^hwloc@2.7.1%gcc@11.3.1~cairo~cuda~gl~libudev+libxml2~netloc~nvml~opencl+pci~rocm+shared arch=linux-fedora35-skylake
                    ^libpciaccess@0.16%gcc@11.3.1 arch=linux-fedora35-skylake
                        ^libtool@2.4.7%gcc@11.3.1 arch=linux-fedora35-skylake
                            ^m4@1.4.19%gcc@11.3.1+sigsegv patches=9dc5fbd,bfdffa7 arch=linux-fedora35-skylake
                                ^libsigsegv@2.13%gcc@11.3.1 arch=linux-fedora35-skylake
                        ^util-macros@1.19.3%gcc@11.3.1 arch=linux-fedora35-skylake
                ^libedit@3.1-20210216%gcc@11.3.1 arch=linux-fedora35-skylake
                ^ninja@1.10.2%gcc@11.3.1 arch=linux-fedora35-skylake
                    ^python@3.9.12%gcc@11.3.1+bz2+ctypes+dbm~debug+ensurepip+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib patches=0d98e93,4c24573,f2fd060 arch=linux-fedora35-skylake
                        ^expat@2.4.8%gcc@11.3.1+libbsd arch=linux-fedora35-skylake
                            ^libbsd@0.11.5%gcc@11.3.1 arch=linux-fedora35-skylake
                                ^libmd@1.0.4%gcc@11.3.1 arch=linux-fedora35-skylake
                        ^libffi@3.4.2%gcc@11.3.1 arch=linux-fedora35-skylake
                        ^sqlite@3.38.5%gcc@11.3.1+column_metadata+dynamic_extensions+fts~functions+rtree arch=linux-fedora35-skylake
                        ^util-linux-uuid@2.37.4%gcc@11.3.1 arch=linux-fedora35-skylake
                ^perl-data-dumper@2.173%gcc@11.3.1 arch=linux-fedora35-skylake
                ^swig@4.0.2%gcc@11.3.1 arch=linux-fedora35-skylake
                    ^pcre@8.45%gcc@11.3.1~jit+multibyte+utf arch=linux-fedora35-skylake
            ^py-pip@21.3.1%gcc@11.3.1 arch=linux-fedora35-skylake
            ^py-setuptools@59.4.0%gcc@11.3.1 arch=linux-fedora35-skylake
                ^py-wheel@0.37.0%gcc@11.3.1 arch=linux-fedora35-skylake
        ^py-numpy@1.21.6%gcc@11.3.1+blas+lapack patches=802970a,873745d arch=linux-fedora35-skylake
            ^openblas@0.3.20%gcc@11.3.1~bignuma~consistent_fpcsr~ilp64+locking+pic+shared symbol_suffix=none threads=none arch=linux-fedora35-skylake
            ^py-cython@0.29.24%gcc@11.3.1 arch=linux-fedora35-skylake
```
@tgamblin @becker33 "Interesting" use case here. This happens because when we have "reuse" as a default the latest version of this dependency forces a lot more packages to be built. Since minimizing the rebuilds takes precedence over having more recent versions the result is that we use an old version of the dependency.
After setting reuse to false, the result does not change for me and it still concretizes to use the old `py-numba` version.
@manuelakuhn can you attach the tarball created with this command?

```
spack debug create-db-tarball
```
Old version with reuse is the intended behavior — but `—fresh` and `concretizer:reuse:false` should exhibit the old behavior.

We are working on more flexible ways to control what is reused and what’s built fresh — e.g., @adamjstewart wants an option for *just* the roots of an env to be updated, but everything else optimized for reuse.

I’m curious to see @manuelakuhn’s installs because I think here there are weird choices being made among various installs that could be reused.
@tgamblin I cannot create a db tarball because the command fails:
```
$ spack debug create-db-tarball
/usr/bin/tar: spack/.spack-db/index.json: Cannot stat: No such file or directory
/usr/bin/tar: Exiting with failure status due to previous errors
==> Error: Command exited with status 2:
'/usr/bin/tar' '-czf' '/tmp/spack/spack-db.develop.c775c322ec.2022-05-18-184559.tar.gz' '--transform' 's/^spack/spack-db.develop.c775c322ec.2022-05-18-184559.tar.gz/' 'spack/.spack-db/index.json'
```

Ugh that's unfortunate.  Any chance you're in an environment that moves the store to a new (empty?) location?  Can you try:

```
spack -E debug create-db-tarball
```

If not can you find your install tree and just grab the `.spack-db/index.json` there?
The commands fail because it is a completely new spack clone with nothing installed yet.
Ok I dug into this a bit and I know *why* it's preferring what you see, but we'll need to dig a bit more to figure out how best to fix it.

One way to understand these concretizer preferences is to look at the version of `py-numba` that you *want* and try to force it, then compare to the weights you get with `spack solve` to what you get *without* forcing it.

So in this case:

```
> spack solve py-fracridge ^py-numba@0.55.1
==> Best of 8 considered solutions.
==> Optimization Criteria:
  Priority  Criterion                                            Installed  ToBuild
  1         number of packages to build (vs. reuse)                      -       65
  2         deprecated versions used                                     0        0
  3         version weight                                               0        0
  4         number of non-default variants (roots)                       0        0
  5         preferred providers for roots                                0        0
  6         default values of variants not being used (roots)            0        0
  7         number of non-default variants (non-roots)                   1        1    <-- this is why
  8         preferred providers (non-roots)                              0        0
  9         compiler mismatches                                          0        0
  10        OS mismatches                                                0        0
  11        non-preferred OS's                                           0        0
  12        version badness                                              0       24
  13        default values of variants not being used (non-roots)        0        4
  14        non-preferred compilers                                      0        0
  15        target mismatches                                            0        0
  16        non-preferred targets                                        0        0
```

The (bad) configuration you're seeing has:

```
  7         number of non-default variants (non-roots)                   0        0
...
  12        version badness                                              0       71
```

"version weight" and "version badness" should really be "version badness (roots)" and "version badness (non-roots)" -- we should rename those the output for clarity.

So to the concretizer (at least right now), it's "better" because it has no non-default variants in non-root nodes.  It *does* have a ton of bad versions: 71 of them vs. 24 for the configuration I tried to force, but the optimizer optimizes highest priority criteria first.  It turns out that in this case, newer `py-numba` versions depend on `py-llvmlite`, which depends on `llvm`, and somehow `llvm` is somehow getting forced to turn off `omp_as_runtime`, which gives us that one non-default variant.

This doesn't happen if you just `spack spec py-numba` because we prefer the newest version for the roots more strongly than for non-roots.  But when `py-numba` is a dependency of the root this pops up because we lump all non-roots together.  We do that because we have not yet figured out a cheap way to impose a total order over preferences across nodes in the DAG -- when we tried it in the solver it was very expensive.  We probably need to revisit that.

Ideally, any parent node's preferences should take precedence over those of its children. If we did that here, we'd strongly prefer for the new number version and other attributes of numba to be right over anything about transitive dependencies.  I think we need to start digging into this problem so that we can impose a stricter order on these preferences, as right now really subtle differences due to requirements on transitive dependencies will compete with preferences on more important nodes in the graph, and we really don't want that.

We could look at doing a quick fix where we swap variant and version weight for non-roots, but I believe @alalazo has them in that order b/c preferring newest versions could have some really weird consequences w.r.t. variants.  It's another case where we need to prefer higher-up nodes as opposed to aggregate categories over non-roots.  The roots vs. non-roots thing is a (blunt) stopgap until we have that.




@alalazo @becker33 FYI
> somehow llvm is somehow getting forced to turn off omp_as_runtime, which gives us that one non-default variant.

This should not happen.  It really shouldn’t even be a variant, it should be on 100% of the time in versions where it’s supported.  I know this issue is about more than that, but there must be a package doing something strange for that to happen.
It's because `py-llvmlite` depends on either `llvm@10` or `llvm@11`, both of which are too old for it to be on by default.
If that's the case, this specific issue can be solved by turning:
```python
variant(
    "omp_as_runtime",
    default=True,
    description="Build OpenMP runtime via ENABLE_RUNTIME by just-built Clang",
)
conflicts("+omp_as_runtime", when="~clang", msg="omp_as_runtime requires clang being built.")
conflicts("+omp_as_runtime", when="@:11.1", msg="omp_as_runtime works since LLVM 12.")
```
into:
```python
variant(
    "omp_as_runtime",
    default=True,
    when='+clang @12:',
    description="Build OpenMP runtime via ENABLE_RUNTIME by just-built Clang",
)

```
I'll try this and submit a PR if it works.

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/llvm/package.py b/var/spack/repos/builtin/packages/llvm/package.py
--- a/var/spack/repos/builtin/packages/llvm/package.py
+++ b/var/spack/repos/builtin/packages/llvm/package.py
@@ -165,6 +165,7 @@ class Llvm(CMakePackage, CudaPackage):
     variant(
         "omp_as_runtime",
         default=True,
+        when='+clang @12:',
         description="Build OpenMP runtime via ENABLE_RUNTIME by just-built Clang",
     )
     variant('code_signing', default=False,
@@ -268,10 +269,6 @@ class Llvm(CMakePackage, CudaPackage):
     # OMP TSAN exists in > 5.x
     conflicts("+omp_tsan", when="@:5")
 
-    # OpenMP via ENABLE_RUNTIME restrictions
-    conflicts("+omp_as_runtime", when="~clang", msg="omp_as_runtime requires clang being built.")
-    conflicts("+omp_as_runtime", when="@:11.1", msg="omp_as_runtime works since LLVM 12.")
-
     # cuda_arch value must be specified
     conflicts("cuda_arch=none", when="+cuda", msg="A value for cuda_arch must be specified.")
 


Test patch: