Repository: spack/spack

Issue:
ASP solver unable to sort variant_values_from_specs when variant can have both bool and str values
### Steps to reproduce

Attempting to re-concretize an environment with the v0.17.1 clingo solver with

```console
spack concretize --reuse
```

fails at [`spack.solver.asp.SpackSolverSetup.define_variant_values`](https://github.com/spack/spack/blob/b23bd81cd9dbccb536778a9e7bd8a05954ce3300/lib/spack/spack/solver/asp.py#L1603) due to a type comparison error while sorting the [`variant_values_from_specs`](https://github.com/spack/spack/blob/b23bd81cd9dbccb536778a9e7bd8a05954ce3300/lib/spack/spack/solver/asp.py#L657) if an installed reuse candidate and new spec candidates have values that cannot be natively compared for the sort (eg, a `bool` and `str` value). Specifically, the [kokkos-kernels `eti` variant](https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/kokkos-kernels/package.py#L50-L61) seems to have both an `"auto"` (string type) value and apparently a `bool` value option (under some circumstances not clearly known to me) which cannot be directly compared natively with `obj.__lt__`

As a work-around, I've changed [`spack.solver.asp.SpackSolverSetup.define_variant_values`](https://github.com/spack/spack/blob/b23bd81cd9dbccb536778a9e7bd8a05954ce3300/lib/spack/spack/solver/asp.py#L1603) to sort only on the first element of the tuples in the set:

```diff
diff --git a/lib/spack/spack/solver/asp.py b/lib/spack/spack/solver/asp.py
index 957f363960..d8e5c7e338 100644
--- a/lib/spack/spack/solver/asp.py
+++ b/lib/spack/spack/solver/asp.py
@@ -1600,7 +1600,7 @@ def define_variant_values(self):
         """
         # Tell the concretizer about possible values from specs we saw in
         # spec_clauses()
-        for pkg, variant, value in sorted(self.variant_values_from_specs):
+        for pkg, variant, value in sorted(self.variant_values_from_specs, key=lambda x: x[0]):
             self.gen.fact(fn.variant_possible_value(pkg, variant, value))
 
     def _facts_from_concrete_spec(self, spec, possible):
```

though I'm not sure if this is safe or appropriate. What is the purpose of using `sorted()` here and is it necessary for this set to be sorted in a specific way beyond the first two keys in each member tuple of `variant_values_from_specs`?

### Error message

```console
spack -vd --stacktrace concretize --reuse -f
```

```console
[PID=3602254] Traceback (most recent call last):
  File "/autofs/nccs-svm1_home2/belhorn/development/olcf-spack-environments/spack/lib/spack/spack/util/parallel.py", line 53, in __call__
    value = self.func(*args, **kwargs)
  File "/autofs/nccs-svm1_home2/belhorn/development/olcf-spack-environments/spack/lib/spack/spack/environment/environment.py", line 2057, in _concretize_task
    return _concretize_from_constraints(spec_constraints, tests, reuse)
  File "/autofs/nccs-svm1_home2/belhorn/development/olcf-spack-environments/spack/lib/spack/spack/environment/environment.py", line 2035, in _concretize_from_constraints
    return s.concretized(tests=tests, reuse=reuse)
  File "/autofs/nccs-svm1_home2/belhorn/development/olcf-spack-environments/spack/lib/spack/spack/spec.py", line 2692, in concretized
    clone.concretize(tests=tests, reuse=reuse)
  File "/autofs/nccs-svm1_home2/belhorn/development/olcf-spack-environments/spack/lib/spack/spack/spec.py", line 2648, in concretize
    self._new_concretize(tests, reuse=reuse)
  File "/autofs/nccs-svm1_home2/belhorn/development/olcf-spack-environments/spack/lib/spack/spack/spec.py", line 2619, in _new_concretize
    result = spack.solver.asp.solve([self], tests=tests, reuse=reuse)
  File "/autofs/nccs-svm1_home2/belhorn/development/olcf-spack-environments/spack/lib/spack/spack/solver/asp.py", line 2054, in solve
    setup, specs, dump, models, timers, stats, tests, reuse 
  File "/autofs/nccs-svm1_home2/belhorn/development/olcf-spack-environments/spack/lib/spack/spack/solver/asp.py", line 552, in solve
    solver_setup.setup(self, specs, tests=tests, reuse=reuse)
  File "/autofs/nccs-svm1_home2/belhorn/development/olcf-spack-environments/spack/lib/spack/spack/solver/asp.py", line 1752, in setup
    self.define_variant_values()
  File "/autofs/nccs-svm1_home2/belhorn/development/olcf-spack-environments/spack/lib/spack/spack/solver/asp.py", line 1603, in define_variant_values
    for pkg, variant, value in sorted(self.variant_values_from_specs):
TypeError: '<' not supported between instances of 'str' and 'bool'
```


### Information on your system

$ spack debug report
* **Spack:** 0.17.1
* **Python:** 3.6.8
* **Platform:** linux-rhel8-zen2
* **Concretizer:** clingo

### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have searched the issues of this repo and believe this is not a duplicate
- [X] I have run the failing commands in debug mode and reported the output


Discussion:
I think both this issue and #28259 require us to add a "namespace" dimension to our solves. Currently we don't have that:

https://github.com/spack/spack/blob/c791ddc742474226bed94fcfadd5965148957918/lib/spack/spack/solver/asp.py#L1980-L1984

Adding that dimension is probably best done after #27964 is merged, or at least assuming #27964 is merged first, to avoid conflicts in `concretize.lp` during the refactoring 
@becker33  fyi
Maybe we could associate an ID with each `package.py` and turn facts like:
```
version_declared("zlib","1.2.11",0,"package_py").
version_declared("zlib","1.2.8",1,"package_py").
version_declared("zlib","1.2.3",2,"package_py").
```
into:
```
package_name(id, "zlib")
package_namespace(id, "builtin")
version_declared(id,"1.2.11",0,"package_py").
version_declared(id,"1.2.8",1,"package_py").
version_declared(id,"1.2.3",2,"package_py").
```
?
> What is the purpose of using sorted() here and is it necessary for this set to be sorted in a specific way beyond the first two keys in each member tuple of variant_values_from_specs?

I'm having a closer look at this issue. Sorting is just for readability when debugging clingo programs and can be omitted entirely. That part of the code emits input facts for the solver, so the order is not important as long as the set of facts stays the same.
I can reproduce this with:
```console
$ spack install --fake kokkos-kernels
$ spack install --reuse kokkos-kernels
==> Error: '<' not supported between instances of 'str' and 'bool'
```

Resolving patch:
diff --git a/lib/spack/spack/solver/asp.py b/lib/spack/spack/solver/asp.py
--- a/lib/spack/spack/solver/asp.py
+++ b/lib/spack/spack/solver/asp.py
@@ -1589,8 +1589,9 @@ def define_variant_values(self):
 
         """
         # Tell the concretizer about possible values from specs we saw in
-        # spec_clauses()
-        for pkg, variant, value in sorted(self.variant_values_from_specs):
+        # spec_clauses(). We might want to order these facts by pkg and name
+        # if we are debugging.
+        for pkg, variant, value in self.variant_values_from_specs:
             self.gen.fact(fn.variant_possible_value(pkg, variant, value))
 
     def _facts_from_concrete_spec(self, spec, possible):


Test patch: