Repository: spack/spack

Issue:
Setting `cxxflags="stdlib=libc++"` results in `Error: {'stdlib=libc++'} does not equal set()` after #34218
### Steps to reproduce

The `cxxflags` variant doesn't seem to accept certain types of flags:
```console
> spack spec cmake cxxflags="-stdlib=libc++"
==> Error: {'-stdlib=libc++'} does not equal set()
```
Curiously, e.g.
```console
> spack spec cmake cxxflags="-stdlib=libc+"
```
or

```console
> spack spec cmake cxxflags="-stdlib=libc+++"
```
or
```console
> spack spec cmake cxxflags="-O3"
```
do concretize (is `-stdlib=libc++` being filtered out at some point?).

### Error message

```console
> spack -d spec -lI cmake cxxflags="-stdlib=libc++"
==> [2022-12-07-13:30:07.174827] Imported spec from built-in commands
==> [2022-12-07-13:30:07.175544] Imported spec from built-in commands
==> [2022-12-07-13:30:07.183557] Reading config from file /apps/daint/SSL/pika/spack/etc/spack/defaults/config.yaml
==> [2022-12-07-13:30:07.210096] Reading config from file /apps/daint/SSL/pika/spack/etc/spack/config.yaml
==> [2022-12-07-13:30:07.215239] DATABASE LOCK TIMEOUT: 3s
==> [2022-12-07-13:30:07.215334] PACKAGE LOCK TIMEOUT: No timeout
==> [2022-12-07-13:30:07.216063] Reading config from file /apps/daint/SSL/pika/spack/etc/spack/defaults/bootstrap.yaml
==> [2022-12-07-13:30:07.228291] Reading config from file /apps/daint/SSL/pika/spack/etc/spack/defaults/repos.yaml
==> [2022-12-07-13:30:07.233727] Reading config from file /apps/daint/SSL/pika/spack/etc/spack/defaults/config.yaml
==> [2022-12-07-13:30:07.256862] Reading config from file /apps/daint/SSL/pika/spack/etc/spack/config.yaml
==> [2022-12-07-13:30:07.258273] DATABASE LOCK TIMEOUT: 3s
==> [2022-12-07-13:30:07.258328] PACKAGE LOCK TIMEOUT: No timeout
==> [2022-12-07-13:30:07.258419] [BOOTSTRAP CONFIG SCOPE] name=_builtin
==> [2022-12-07-13:30:07.258863] Reading config from file /apps/daint/SSL/pika/spack/etc/spack/defaults/bootstrap.yaml
==> [2022-12-07-13:30:07.266142] [BOOTSTRAP CONFIG SCOPE] name=defaults, path=/apps/daint/SSL/pika/spack/etc/spack/defaults
==> [2022-12-07-13:30:07.266192] [BOOTSTRAP CONFIG SCOPE] name=defaults/linux, path=/apps/daint/SSL/pika/spack/etc/spack/defaults/linux
==> [2022-12-07-13:30:07.266250] [BOOTSTRAP CONFIG SCOPE] name=bootstrap, path=/users/simbergm/.spack/cache/bootstrap/config
==> [2022-12-07-13:30:07.266288] [BOOTSTRAP CONFIG SCOPE] name=bootstrap/linux, path=/users/simbergm/.spack/cache/bootstrap/config/linux
==> [2022-12-07-13:30:07.266713] Reading config from file /users/simbergm/.spack/cache/bootstrap/config/linux/compilers.yaml
==> [2022-12-07-13:30:07.297600] [BOOTSTRAP ROOT SPEC] patchelf@0.13.1: %gcc target=x86_64
==> [2022-12-07-13:30:07.298538] Reading config from file /apps/daint/SSL/pika/spack/etc/spack/defaults/bootstrap.yaml
==> [2022-12-07-13:30:07.311453] [BOOTSTRAP EXECUTABLES patchelf] Try installed specs with query 'patchelf@0.13.1: %gcc target=x86_64'
==> [2022-12-07-13:30:07.493116] Reading config from file /apps/daint/SSL/pika/spack/etc/spack/defaults/config.yaml
==> [2022-12-07-13:30:07.574441] Reading config from file /apps/daint/SSL/pika/spack/etc/spack/defaults/modules.yaml
==> [2022-12-07-13:30:07.589838] Reading config from file /apps/daint/SSL/pika/spack/etc/spack/defaults/linux/modules.yaml
==> [2022-12-07-13:30:07.730110] Reading config from file /apps/daint/SSL/pika/spack/etc/spack/defaults/packages.yaml
==> [2022-12-07-13:30:07.768774] [BOOTSTRAP ROOT SPEC] clingo-bootstrap@spack+python %gcc target=x86_64
==> [2022-12-07-13:30:07.768877] [BOOTSTRAP MODULE clingo] Try importing from Python
==> [2022-12-07-13:30:07.875007] [BOOTSTRAP MODULE clingo] The installed spec "clingo-bootstrap@spack+python %gcc target=x86_64 ^python@3.9/gqcctd2ejbgvyvyt4umqpetfoogfycwu" provides the "clingo" Python module
==> [2022-12-07-13:30:07.878519] [BOOTSTRAP ROOT SPEC] gnupg@2.3: %gcc target=x86_64
==> [2022-12-07-13:30:07.879786] Reading config from file /apps/daint/SSL/pika/spack/etc/spack/defaults/concretizer.yaml
==> [2022-12-07-13:30:07.885561] Reading config from file /apps/daint/SSL/pika/spack/../spack-user-config/concretizer.yaml
==> [2022-12-07-13:30:07.887251] Reading config from file /apps/daint/SSL/pika/spack/etc/spack/defaults/repos.yaml
==> [2022-12-07-13:30:08.009689] '/bin/bash' '--norc' '--noprofile' '-lc' 'echo $CRAY_CPU_TARGET'
==> [2022-12-07-13:30:08.015746] using front-end architecture or available ancestor
==> [2022-12-07-13:30:08.072519] Reading config from file /apps/daint/SSL/pika/spack/etc/spack/defaults/packages.yaml
==> [2022-12-07-13:30:08.105827] Reading config from file /apps/daint/UES/jenkins/7.0.UP03/21.09/daint-gpu/tools/software/spack-config/1.7/cdt-cuda-21.09/packages.yaml
==> [2022-12-07-13:30:08.131242] Reading config from file /apps/daint/SSL/pika/spack/../spack-user-config/packages.yaml
==> [2022-12-07-13:30:09.055865] Reading config from file /apps/daint/SSL/pika/spack/etc/spack/defaults/config.yaml
==> [2022-12-07-13:30:09.078436] Reading config from file /apps/daint/SSL/pika/spack/etc/spack/config.yaml
==> [2022-12-07-13:30:09.081298] Reading config from file /apps/daint/SSL/pika/spack/etc/spack/defaults/mirrors.yaml
==> [2022-12-07-13:30:09.083429] Checking existence of https://mirror.spack.io/build_cache/index.json
==> [2022-12-07-13:30:09.217385] Failure reading URL: Download failed: HTTP Error 404: Not Found
==> [2022-12-07-13:30:11.218379] Reading config from file /apps/daint/UES/jenkins/7.0.UP03/21.09/daint-gpu/tools/software/spack-config/1.7/cdt-cuda-21.09/compilers.yaml
==> [2022-12-07-13:30:11.255550] Reading config from file /apps/daint/SSL/pika/spack/../spack-user-config/cray/compilers.yaml
Traceback (most recent call last):
  File "/apps/daint/SSL/pika/spack/bin/spack", line 54, in <module>
    sys.exit(main())
  File "/apps/daint/SSL/pika/spack/lib/spack/spack_installable/main.py", line 36, in main
    sys.exit(spack.main.main(argv))
  File "/apps/daint/SSL/pika/spack/lib/spack/spack/main.py", line 1000, in main
    return _main(argv)
  File "/apps/daint/SSL/pika/spack/lib/spack/spack/main.py", line 955, in _main
    return finish_parse_and_run(parser, cmd_name, env_format_error)
  File "/apps/daint/SSL/pika/spack/lib/spack/spack/main.py", line 983, in finish_parse_and_run
    return _invoke_command(command, parser, args, unknown)
  File "/apps/daint/SSL/pika/spack/lib/spack/spack/main.py", line 632, in _invoke_command
    return_val = command(parser, args)
  File "/apps/daint/SSL/pika/spack/lib/spack/spack/cmd/spec.py", line 103, in spec
    concretized_specs = spack.cmd.parse_specs(args.specs, concretize=True)
  File "/apps/daint/SSL/pika/spack/lib/spack/spack/cmd/__init__.py", line 223, in parse_specs
    spec.concretize(tests=tests)  # implies normalize
  File "/apps/daint/SSL/pika/spack/lib/spack/spack/spec.py", line 2978, in concretize
    self._new_concretize(tests)
  File "/apps/daint/SSL/pika/spack/lib/spack/spack/spec.py", line 2953, in _new_concretize
    result = solver.solve([self], tests=tests)
  File "/apps/daint/SSL/pika/spack/lib/spack/spack/solver/asp.py", line 2518, in solve
    result, _, _ = self.driver.solve(setup, specs, reuse=reusable_specs, output=output)
  File "/apps/daint/SSL/pika/spack/lib/spack/spack/solver/asp.py", line 765, in solve
    answers = builder.build_specs(spec_attrs)
  File "/apps/daint/SSL/pika/spack/lib/spack/spack/solver/asp.py", line 2376, in build_specs
    self.reorder_flags()
  File "/apps/daint/SSL/pika/spack/lib/spack/spack/solver/asp.py", line 2304, in reorder_flags
    assert set(compiler_flags) == set(ordered_compiler_flags), msg
AssertionError: {'-stdlib=libc++'} does not equal set()
```

### Information on your system

The assertion points to https://github.com/spack/spack/blob/194f9a9ca97063c302a6fb0e4456b653a876315a/lib/spack/spack/solver/asp.py#L2304 which was introduced in #34218. Concretization works before that was merged.

```console
> spack debug report
* **Spack:** 0.20.0.dev0 (194f9a9ca97063c302a6fb0e4456b653a876315a)
* **Python:** 3.9.4
* **Platform:** cray-sles15-haswell
* **Concretizer:** clingo
```

### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have searched the issues of this repo and believe this is not a duplicate
- [X] I have run the failing commands in debug mode and reported the output


Discussion:
I'm hitting what appears to be the same problem if I have a root spec in an environment that specifies custom cflags, like in this example:

```
 - json-c cflags="-fsanitize=address -fno-omit-frame-pointer -g -Wall" ldflags="-fsanitize=address"
 ```

I can create the environment but when I try to install it produces a `==> Error: {'-Wall', '-fsanitize=address', '-fno-omit-frame-pointer', '-g'} does not equal set()` error.

Actually it looks like I get that error even after removing the cflags from my spack.yaml root spec, apparently because I had packages already installed with those flags.  I cleaned out all of my packages (this is a dev environment) and started over with no cflags to get past it.
FWIW, I have mostly seen this `does not equal set()` error in the past on specs that are trying to set multiple cflags, but for some reason today I'm even getting it when the root specs just have `cflags="-g"`.  I think at this point I can't reliably  set any cflags at all in my development environment.

Resolving patch:
diff --git a/lib/spack/spack/compilers/__init__.py b/lib/spack/spack/compilers/__init__.py
--- a/lib/spack/spack/compilers/__init__.py
+++ b/lib/spack/spack/compilers/__init__.py
@@ -84,7 +84,7 @@ def _to_dict(compiler):
     d = {}
     d["spec"] = str(compiler.spec)
     d["paths"] = dict((attr, getattr(compiler, attr, None)) for attr in _path_instance_vars)
-    d["flags"] = dict((fname, fvals) for fname, fvals in compiler.flags)
+    d["flags"] = dict((fname, " ".join(fvals)) for fname, fvals in compiler.flags.items())
     d["flags"].update(
         dict(
             (attr, getattr(compiler, attr, None))
diff --git a/lib/spack/spack/solver/asp.py b/lib/spack/spack/solver/asp.py
--- a/lib/spack/spack/solver/asp.py
+++ b/lib/spack/spack/solver/asp.py
@@ -1402,7 +1402,12 @@ def flag_defaults(self):
 
         # flags from compilers.yaml
         compilers = all_compilers_in_config()
+        seen = set()
         for compiler in compilers:
+            # if there are multiple with the same spec, only use the first
+            if compiler.spec in seen:
+                continue
+            seen.add(compiler.spec)
             for name, flags in compiler.flags.items():
                 for flag in flags:
                     self.gen.fact(
@@ -2287,7 +2292,8 @@ def reorder_flags(self):
         The solver determines wihch flags are on nodes; this routine
         imposes order afterwards.
         """
-        compilers = dict((c.spec, c) for c in all_compilers_in_config())
+        # reverse compilers so we get highest priority compilers that share a spec
+        compilers = dict((c.spec, c) for c in reversed(all_compilers_in_config()))
         cmd_specs = dict((s.name, s) for spec in self._command_line_specs for s in spec.traverse())
 
         for spec in self._specs.values():


Test patch:
diff --git a/lib/spack/spack/test/concretize.py b/lib/spack/spack/test/concretize.py
--- a/lib/spack/spack/test/concretize.py
+++ b/lib/spack/spack/test/concretize.py
@@ -2,6 +2,7 @@
 # Spack Project Developers. See the top-level COPYRIGHT file for details.
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
+import copy
 import os
 import sys
 
@@ -332,6 +333,24 @@ def test_compiler_flags_from_compiler_and_dependent(self):
         for spec in [client, cmake]:
             assert spec.compiler_flags["cflags"] == ["-O3", "-g"]
 
+    def test_compiler_flags_differ_identical_compilers(self):
+        # Correct arch to use test compiler that has flags
+        spec = Spec("a %clang@12.2.0 platform=test os=fe target=fe")
+
+        # Get the compiler that matches the spec (
+        compiler = spack.compilers.compiler_for_spec("clang@12.2.0", spec.architecture)
+        # Clear cache for compiler config since it has its own cache mechanism outside of config
+        spack.compilers._cache_config_file = []
+
+        # Configure spack to have two identical compilers with different flags
+        default_dict = spack.compilers._to_dict(compiler)
+        different_dict = copy.deepcopy(default_dict)
+        different_dict["compiler"]["flags"] = {"cflags": "-O2"}
+
+        with spack.config.override("compilers", [different_dict]):
+            spec.concretize()
+            assert spec.satisfies("cflags=-O2")
+
     def test_concretize_compiler_flag_propagate(self):
         spec = Spec("hypre cflags=='-g' ^openblas")
         spec.concretize()