Repository: spack/spack

Issue:
Stacks: matrix expansion failing with spack.spec.DuplicateCompilerSpecError
### Steps to reproduce

This error has been originally reported on Slack by @g-mathias (thanks!)

The following environment:
```yaml
spack:
  # add package specs to the `specs` list
  specs:
    # 2020 plumed mpi
    - matrix:
      - [gromacs]
      - ['@2020.1:2020.4'] 
      - ['%intel'] #compiler in extra matrix element
      - ['+plumed ^plumed@2.6.4%intel']

  view: false
```
fails to parse when expanding the matrix. To trigger the error:
```console
$ spack -e . concretize
```

### Error message

The error with a stacktrace is:

<details>
<summary>$ spack -d -e . concretize</summary>

```
==> [2022-02-03-13:13:40.299700] Reading config file /home/culpo/PycharmProjects/spack/etc/spack/defaults/config.yaml
==> [2022-02-03-13:13:40.321068] Reading config file /tmp/matrix/spack.yaml
==> [2022-02-03-13:13:40.323254] Using environment '/tmp/matrix'
==> [2022-02-03-13:13:40.324073] Imported concretize from built-in commands
==> [2022-02-03-13:13:40.324521] Imported concretize from built-in commands
==> [2022-02-03-13:13:40.327576] DuplicateCompilerSpecError: Spec for 'plumed' cannot have two compilers.
==> [2022-02-03-13:13:40.327616] Error: Spec for 'plumed' cannot have two compilers.
Traceback (most recent call last):
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/main.py", line 900, in main
    return _main(argv)
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/main.py", line 883, in _main
    return _invoke_command(command, parser, args, unknown)
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/main.py", line 553, in _invoke_command
    return_val = command(parser, args)
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/cmd/concretize.py", line 39, in concretize
    concretized_specs = env.concretize(
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/environment/environment.py", line 1118, in concretize
    return self._concretize_separately(tests=tests, reuse=reuse)
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/environment/environment.py", line 1184, in _concretize_separately
    self.user_specs, self.user_specs.specs_as_constraints
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/spec_list.py", line 63, in specs_as_constraints
    constraints.extend(_expand_matrix_constraints(item))
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/spec_list.py", line 203, in _expand_matrix_constraints
    test_spec = Spec(' '.join(ordered_combo))
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/spec.py", line 1115, in __init__
    spec_list = SpecParser(self).parse(spec_like)
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/parse.py", line 153, in parse
    return self.do_parse()
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/spec.py", line 4747, in do_parse
    dep = self.spec(dep_name)
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/spec.py", line 4884, in spec
    spec._set_compiler(self.compiler())
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/spec.py", line 1241, in _set_compiler
    raise DuplicateCompilerSpecError(
spack.spec.DuplicateCompilerSpecError: Spec for 'plumed' cannot have two compilers.
```

</details>

As a further data point, the following environment which should be equivalent:
```yaml
spack:
  # add package specs to the `specs` list
  specs:
    # 2020 plumed mpi
    - matrix:
      - [gromacs]
      - ['@2020.1:2020.4%intel']
      - [+plumed ^plumed@2.6.4%intel]
  view: false
```
works correctly.

### Information on your system

* **Spack:** 0.17.1-1061-18b83c3833
* **Python:** 3.8.10
* **Platform:** linux-ubuntu20.04-icelake
* **Concretizer:** clingo

See also #20340 for another issue with best-effort matrix expansion. Adding it here since they can be probably solved at the same time with a refactor PR.


### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have searched the issues of this repo and believe this is not a duplicate
- [X] I have run the failing commands in debug mode and reported the output


Discussion:
Looking a bit into it it seems that the issue is caused by:

https://github.com/spack/spack/blob/5208fcf1ea135b4500c38fbc7dba44e8558994bd/lib/spack/spack/spec_list.py#L14-L24

which is used during matrix expansion here:

https://github.com/spack/spack/blob/5208fcf1ea135b4500c38fbc7dba44e8558994bd/lib/spack/spack/spec_list.py#L201

I think the current implementation overlooked the possibility to have composite requirements like:
```
+plumed ^plumed@2.6.4
```
that impose a constraint on the root spec and on a dependency simultaneously. We should probably use "spec merge" semantics instead of string concatenation:

https://github.com/spack/spack/blob/5208fcf1ea135b4500c38fbc7dba44e8558994bd/lib/spack/spack/spec_list.py#L203

 in this function.


I think we could also "intersect" constraints here

```
$ spack spec zlib %gcc %gcc
==> Error: Spec for 'zlib' cannot have two compilers.
```

the second %gcc is not constraining the solution space, so it doesn't need to error. And this

```
$ spack spec zlib %gcc %gcc@8: %gcc@:10
```
can be equivalent to `%gcc@8:10`
This error is more fundamental, since the strings get reordered to:
```
gromacs@2020.1:2020.4 +plumed ^plumed@2.6.4%intel %intel
```

I stumbled across this bug because going from spack 0.16.3 to 0.17.1 the concretizer returned the combination `gromacs%intel ^plumed%gcc ^cmake%gcc` from `gromacs%intel ^plumed ^cmake%gcc` (without `--reuse`). I could not figure out, where the `plumed%gcc` was coming from (giving me trouble), so I tried to force to use the intel compiler on `plumed`. 

Resolving patch:
diff --git a/lib/spack/spack/spec.py b/lib/spack/spack/spec.py
--- a/lib/spack/spack/spec.py
+++ b/lib/spack/spack/spec.py
@@ -3170,6 +3170,15 @@ def constrain(self, other, deps=True):
                     raise UnsatisfiableArchitectureSpecError(sarch, oarch)
 
         changed = False
+
+        if not self.name and other.name:
+            self.name = other.name
+            changed = True
+
+        if not self.namespace and other.namespace:
+            self.namespace = other.namespace
+            changed = True
+
         if self.compiler is not None and other.compiler is not None:
             changed |= self.compiler.constrain(other.compiler)
         elif self.compiler is None:
diff --git a/lib/spack/spack/spec_list.py b/lib/spack/spack/spec_list.py
--- a/lib/spack/spack/spec_list.py
+++ b/lib/spack/spack/spec_list.py
@@ -11,19 +11,6 @@
 from spack.spec import Spec
 
 
-def spec_ordering_key(s):
-    if s.startswith('^'):
-        return 5
-    elif s.startswith('/'):
-        return 4
-    elif s.startswith('%'):
-        return 3
-    elif any(s.startswith(c) for c in '~-+@') or '=' in s:
-        return 2
-    else:
-        return 1
-
-
 class SpecList(object):
 
     def __init__(self, name='specs', yaml_list=None, reference=None):
@@ -177,30 +164,36 @@ def __getitem__(self, key):
         return self.specs[key]
 
 
-def _expand_matrix_constraints(object, specify=True):
-    # recurse so we can handle nexted matrices
+def _expand_matrix_constraints(matrix_config):
+    # recurse so we can handle nested matrices
     expanded_rows = []
-    for row in object['matrix']:
+    for row in matrix_config['matrix']:
         new_row = []
         for r in row:
             if isinstance(r, dict):
+                # Flatten the nested matrix into a single row of constraints
                 new_row.extend(
-                    [[' '.join(c)]
-                     for c in _expand_matrix_constraints(r, specify=False)])
+                    [[' '.join([str(c) for c in expanded_constraint_list])]
+                     for expanded_constraint_list in _expand_matrix_constraints(r)]
+                )
             else:
                 new_row.append([r])
         expanded_rows.append(new_row)
 
-    excludes = object.get('exclude', [])  # only compute once
-    sigil = object.get('sigil', '')
+    excludes = matrix_config.get('exclude', [])  # only compute once
+    sigil = matrix_config.get('sigil', '')
 
     results = []
     for combo in itertools.product(*expanded_rows):
         # Construct a combined spec to test against excludes
-        flat_combo = [constraint for list in combo for constraint in list]
-        ordered_combo = sorted(flat_combo, key=spec_ordering_key)
+        flat_combo = [constraint for constraint_list in combo
+                      for constraint in constraint_list]
+        flat_combo = [Spec(x) for x in flat_combo]
+
+        test_spec = flat_combo[0].copy()
+        for constraint in flat_combo[1:]:
+            test_spec.constrain(constraint)
 
-        test_spec = Spec(' '.join(ordered_combo))
         # Abstract variants don't have normal satisfaction semantics
         # Convert all variants to concrete types.
         # This method is best effort, so all existing variants will be
@@ -214,14 +207,12 @@ def _expand_matrix_constraints(object, specify=True):
         if any(test_spec.satisfies(x) for x in excludes):
             continue
 
-        if sigil:  # add sigil if necessary
-            ordered_combo[0] = sigil + ordered_combo[0]
+        if sigil:
+            flat_combo[0] = Spec(sigil + str(flat_combo[0]))
 
         # Add to list of constraints
-        if specify:
-            results.append([Spec(x) for x in ordered_combo])
-        else:
-            results.append(ordered_combo)
+        results.append(flat_combo)
+
     return results
 
 


Test patch:
diff --git a/lib/spack/spack/test/spec_list.py b/lib/spack/spack/test/spec_list.py
--- a/lib/spack/spack/test/spec_list.py
+++ b/lib/spack/spack/test/spec_list.py
@@ -45,24 +45,34 @@ def test_spec_list_expansions(self):
         assert speclist.specs_as_constraints == self.default_constraints
         assert speclist.specs == self.default_specs
 
-    def test_spec_list_constraint_ordering(self):
-        specs = [{'matrix': [
+    @pytest.mark.regression('28749')
+    @pytest.mark.parametrize('specs,expected', [
+        # Constraints are ordered randomly
+        ([{'matrix': [
             ['^zmpi'],
             ['%gcc@4.5.0'],
             ['hypre', 'libelf'],
             ['~shared'],
             ['cflags=-O3', 'cflags="-g -O0"'],
             ['^foo']
-        ]}]
-
+        ]}], [
+            'hypre cflags=-O3 ~shared %gcc@4.5.0 ^foo ^zmpi',
+            'hypre cflags="-g -O0" ~shared %gcc@4.5.0 ^foo ^zmpi',
+            'libelf cflags=-O3 ~shared %gcc@4.5.0 ^foo ^zmpi',
+            'libelf cflags="-g -O0" ~shared %gcc@4.5.0 ^foo ^zmpi',
+        ]),
+        # A constraint affects both the root and a dependency
+        ([{'matrix': [
+            ['gromacs'],
+            ['%gcc'],
+            ['+plumed ^plumed%gcc']
+        ]}], [
+            'gromacs+plumed%gcc ^plumed%gcc'
+        ])
+    ])
+    def test_spec_list_constraint_ordering(self, specs, expected):
         speclist = SpecList('specs', specs)
-
-        expected_specs = [
-            Spec('hypre cflags=-O3 ~shared %gcc@4.5.0 ^foo ^zmpi'),
-            Spec('hypre cflags="-g -O0" ~shared %gcc@4.5.0 ^foo ^zmpi'),
-            Spec('libelf cflags=-O3 ~shared %gcc@4.5.0 ^foo ^zmpi'),
-            Spec('libelf cflags="-g -O0" ~shared %gcc@4.5.0 ^foo ^zmpi'),
-        ]
+        expected_specs = [Spec(x) for x in expected]
         assert speclist.specs == expected_specs
 
     def test_spec_list_add(self):
diff --git a/lib/spack/spack/test/spec_semantics.py b/lib/spack/spack/test/spec_semantics.py
--- a/lib/spack/spack/test/spec_semantics.py
+++ b/lib/spack/spack/test/spec_semantics.py
@@ -1228,3 +1228,15 @@ def test_merge_abstract_anonymous_specs(specs, expected):
     specs = [Spec(x) for x in specs]
     result = spack.spec.merge_abstract_anonymous_specs(*specs)
     assert result == Spec(expected)
+
+
+@pytest.mark.parametrize('anonymous,named,expected', [
+    ('+plumed', 'gromacs', 'gromacs+plumed'),
+    ('+plumed ^plumed%gcc', 'gromacs', 'gromacs+plumed ^plumed%gcc'),
+    ('+plumed', 'builtin.gromacs', 'builtin.gromacs+plumed')
+])
+def test_merge_anonymous_spec_with_named_spec(anonymous, named, expected):
+    s = Spec(anonymous)
+    changed = s.constrain(named)
+    assert changed
+    assert s == Spec(expected)