Repository: spack/spack

Issue:
AssertionError is seen if user update compiler.yaml file with compiler flags section and then pass 'c' and 'cxx' flags to spack install command.
### Steps to reproduce

```
$spack config edit compilers
update cflags, cxxflags and fflags with any flag value say '-O3 -fPIC'

--snip--
- compiler:
    spec: gcc@11.2.0
    paths:
      cc: /home/amd/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-9.3.0/gcc-11.2.0-36ugcqpr5laqaukcht5ldyi6eh4e4ddu/bin/gcc
      cxx: /home/amd/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-9.3.0/gcc-11.2.0-36ugcqpr5laqaukcht5ldyi6eh4e4ddu/bin/g++
      f77: /home/amd/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-9.3.0/gcc-11.2.0-36ugcqpr5laqaukcht5ldyi6eh4e4ddu/bin/gfortran
      fc: /home/amd/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-9.3.0/gcc-11.2.0-36ugcqpr5laqaukcht5ldyi6eh4e4ddu/bin/gfortran
    flags:
      cflags: -O3 -fPIC
      cxxflags: -O3 -fPIC
      fflags: -O3 -fPIC
    operating_system: ubuntu20.04
    target: x86_64
    modules: []
    environment: {}
    extra_rpaths: []
--snip--

Pass 'c' and 'cxx' flags to spack install command for any spack product
say:
 spack install sqlite cflags="-w" cxxflags="-w" fflags="-w"
```


### Error message

```

$ spack --debug --stacktrace install sqlite cflags="-w" cxxflags="-w" fflags="-w"
lib/spack/spack/cmd/__init__.py:123 ==> [2021-12-01-11:36:05.147719] Imported install from built-in commands
lib/spack/spack/config.py:1007 ==> [2021-12-01-11:36:05.149329] Reading config file /home/amd/spack/etc/spack/defaults/config.yaml
lib/spack/spack/cmd/__init__.py:123 ==> [2021-12-01-11:36:05.162727] Imported install from built-in commands
lib/spack/spack/config.py:1007 ==> [2021-12-01-11:36:05.171718] Reading config file /home/amd/spack/etc/spack/defaults/bootstrap.yaml
lib/spack/spack/config.py:1007 ==> [2021-12-01-11:36:05.177781] Reading config file /home/amd/spack/etc/spack/defaults/config.yaml
lib/spack/spack/database.py:373 ==> [2021-12-01-11:36:05.190519] DATABASE LOCK TIMEOUT: 3s
lib/spack/spack/database.py:377 ==> [2021-12-01-11:36:05.190631] PACKAGE LOCK TIMEOUT: No timeout
lib/spack/spack/config.py:1007 ==> [2021-12-01-11:36:05.190958] Reading config file /home/amd/spack/etc/spack/defaults/bootstrap.yaml
lib/spack/spack/database.py:373 ==> [2021-12-01-11:36:05.195618] DATABASE LOCK TIMEOUT: 3s
lib/spack/spack/database.py:377 ==> [2021-12-01-11:36:05.195732] PACKAGE LOCK TIMEOUT: No timeout
lib/spack/spack/bootstrap.py:636 ==> [2021-12-01-11:36:05.195845] [BOOTSTRAP CONFIG SCOPE] name=_builtin
lib/spack/spack/bootstrap.py:652 ==> [2021-12-01-11:36:05.196129] [BOOTSTRAP CONFIG SCOPE] name=defaults, path=/home/amd/spack/etc/spack/defaults
lib/spack/spack/bootstrap.py:653 ==> [2021-12-01-11:36:05.196220] [BOOTSTRAP CONFIG SCOPE] name=defaults/linux, path=/home/amd/spack/etc/spack/defaults/linux
lib/spack/spack/bootstrap.py:652 ==> [2021-12-01-11:36:05.196309] [BOOTSTRAP CONFIG SCOPE] name=bootstrap, path=/home/amd/.spack/bootstrap/config
lib/spack/spack/bootstrap.py:653 ==> [2021-12-01-11:36:05.196391] [BOOTSTRAP CONFIG SCOPE] name=bootstrap/linux, path=/home/amd/.spack/bootstrap/config/linux
lib/spack/spack/config.py:1007 ==> [2021-12-01-11:36:05.196743] Reading config file /home/amd/.spack/bootstrap/config/linux/compilers.yaml
lib/spack/spack/bootstrap.py:763 ==> [2021-12-01-11:36:05.217430] [BOOTSTRAP ROOT SPEC] clingo-bootstrap@spack+python %gcc target=x86_64
lib/spack/spack/bootstrap.py:482 ==> [2021-12-01-11:36:05.217533] [BOOTSTRAP MODULE clingo] Try importing from Python
lib/spack/spack/config.py:1007 ==> [2021-12-01-11:36:05.217741] Reading config file /home/amd/spack/etc/spack/defaults/bootstrap.yaml
lib/spack/spack/config.py:1007 ==> [2021-12-01-11:36:05.292413] Reading config file /home/amd/spack/etc/spack/defaults/config.yaml
lib/spack/spack/bootstrap.py:98 ==> [2021-12-01-11:36:21.105665] [BOOTSTRAP MODULE clingo] The installed spec "clingo-bootstrap@spack+python %gcc target=x86_64 ^python@3.8 /kt74l7kjzrlp3cgtj2576o33mhsrgyrw" provides the "clingo" Python module
lib/spack/spack/config.py:1007 ==> [2021-12-01-11:36:21.106005] Reading config file /home/amd/spack/etc/spack/defaults/repos.yaml
lib/spack/spack/config.py:1007 ==> [2021-12-01-11:36:21.134718] Reading config file /home/amd/.spack/linux/compilers.yaml
lib/spack/spack/config.py:1007 ==> [2021-12-01-11:36:21.147125] Reading config file /home/amd/spack/etc/spack/defaults/config.yaml
lib/spack/spack/config.py:1007 ==> [2021-12-01-11:36:21.158502] Reading config file /home/amd/spack/etc/spack/defaults/packages.yaml
lib/spack/spack/solver/asp.py:1932 ==> [2021-12-01-11:36:21.248433] build(sqlite)
lib/spack/spack/solver/asp.py:1932 ==> [2021-12-01-11:36:21.248533] build(readline)
lib/spack/spack/solver/asp.py:1932 ==> [2021-12-01-11:36:21.248616] build(zlib)
lib/spack/spack/solver/asp.py:1932 ==> [2021-12-01-11:36:21.248695] build(ncurses)
lib/spack/spack/solver/asp.py:1932 ==> [2021-12-01-11:36:21.248770] build(pkgconf)
Traceback (most recent call last):
  File "/home/amd/spack/bin/spack", line 100, in <module>
    sys.exit(spack.main.main())
  File "/home/amd/spack/lib/spack/spack/main.py", line 881, in main
    return _main(argv)
  File "/home/amd/spack/lib/spack/spack/main.py", line 864, in _main
    return _invoke_command(command, parser, args, unknown)
  File "/home/amd/spack/lib/spack/spack/main.py", line 534, in _invoke_command
    return_val = command(parser, args)
  File "/home/amd/spack/lib/spack/spack/cmd/install.py", line 394, in install
    specs = spack.cmd.parse_specs(
  File "/home/amd/spack/lib/spack/spack/cmd/__init__.py", line 166, in parse_specs
    spec.concretize(tests=tests, reuse=reuse)  # implies normalize
  File "/home/amd/spack/lib/spack/spack/spec.py", line 2639, in concretize
    self._new_concretize(tests, reuse=reuse)
  File "/home/amd/spack/lib/spack/spack/spec.py", line 2610, in _new_concretize
    result = spack.solver.asp.solve([self], tests=tests, reuse=reuse)
  File "/home/amd/spack/lib/spack/spack/solver/asp.py", line 2024, in solve
    return driver.solve(
  File "/home/amd/spack/lib/spack/spack/solver/asp.py", line 592, in solve
    answers = builder.build_specs(tuples)
  File "/home/amd/spack/lib/spack/spack/solver/asp.py", line 1958, in build_specs
    self.reorder_flags()
  File "/home/amd/spack/lib/spack/spack/solver/asp.py", line 1875, in reorder_flags
    check_same_flags(spec.compiler_flags, compiler_flags)
  File "/home/amd/spack/lib/spack/spack/solver/asp.py", line 262, in check_same_flags
    assert values1 == values2
AssertionError

```


### Information on your system

$ spack debug report
* **Spack:** 0.17.0-1-f92a2d688d
* **Python:** 3.8.10
* **Platform:** linux-ubuntu20.04-zen3
* **Concretizer:** clingo


### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have searched the issues of this repo and believe this is not a duplicate
- [X] I have run the failing commands in debug mode and reported the output


Discussion:
Maybe I can give a little more info on this issue. I see the same problem from my spack environment, here's a simple reproducer:
```
spack:
  config:
    concretizer: clingo
  compilers:
  - compiler:
      spec: gcc@11.2.0
      paths:
        cc: cc
        cxx: CC
        f77: ftn
        fc: ftn
      flags:
        cflags: -O3
      operating_system: sles15
      target: x86_64
      modules:
      - PrgEnv-gnu
      - libfabric
      - gcc/11.2.0
      environment: {}
      extra_rpaths: []
  specs:
  - vampirtrace%gcc@11.2.0 +mpi cflags='-DMPICH_SUPPRESS_PROTOTYPES'
 ```

This occurs with `clingo` only, the `original` concretizer works.
```
(2) lukebr@login2:/ccs/proj/ven114/lukebr/spackBug spack concretize -f
==> Starting concretization
==> Error: 
(2) lukebr@login2:/ccs/proj/ven114/lukebr/spackBug vi spack.yaml 
(2) lukebr@login2:/ccs/proj/ven114/lukebr/spackBug spack concretize -f
==> Starting concretization
==> Warning: the original concretizer is currently being used.
	Upgrade to "clingo" at your earliest convenience. The original concretizer will be removed from Spack starting at v0.18.0
==> Environment concretized in 6.06 seconds.
==> Concretized vampirtrace%gcc@11.2.0 cflags="-DMPICH_SUPPRESS_PROTOTYPES" +mpi
 -   ddbfdvw  vampirtrace@5.14.4%gcc@11.2.0 cflags="-O3 -DMPICH_SUPPRESS_PROTOTYPES" +mpi patches=6b96a81 arch=cray-sles15-zen3
 -   fp27bg6      ^openmpi@4.1.2%gcc@11.2.0 cflags="-O3 -DMPICH_SUPPRESS_PROTOTYPES" ~atomics~cuda~cxx~cxx_exceptions~gpfs~internal-hwloc~java~legacylaunchers~lustre~memchecker~pmi~pmix+romio~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=none schedulers=none arch=cray-sles15-zen3
 -   o46qxen          ^hwloc@2.7.0%gcc@11.2.0 cflags="-O3 -DMPICH_SUPPRESS_PROTOTYPES" ~cairo~cuda~gl~libudev+libxml2~netloc~nvml~opencl+pci~rocm+shared arch=cray-sles15-zen3
 -   a7wliph              ^libpciaccess@0.16%gcc@11.2.0 cflags="-O3 -DMPICH_SUPPRESS_PROTOTYPES"  arch=cray-sles15-zen3
 -   dcn5ano                  ^libtool@2.4.6%gcc@11.2.0 cflags="-O3 -DMPICH_SUPPRESS_PROTOTYPES"  arch=cray-sles15-zen3
 -   7pxsx5l                      ^m4@1.4.19%gcc@11.2.0 cflags="-O3 -DMPICH_SUPPRESS_PROTOTYPES" +sigsegv patches=9dc5fbd,bfdffa7 arch=cray-sles15-zen3
 -   m7r62rz                          ^libsigsegv@2.13%gcc@11.2.0 cflags="-O3 -DMPICH_SUPPRESS_PROTOTYPES"  arch=cray-sles15-zen3
 -   ce5tvec                  ^pkgconf@1.8.0%gcc@11.2.0 cflags="-O3 -DMPICH_SUPPRESS_PROTOTYPES"  arch=cray-sles15-zen3
 -   6stt4bn                  ^util-macros@1.19.3%gcc@11.2.0 cflags="-O3 -DMPICH_SUPPRESS_PROTOTYPES"  arch=cray-sles15-zen3
 -   kkqxjh6              ^libxml2@2.9.12%gcc@11.2.0 cflags="-O3 -DMPICH_SUPPRESS_PROTOTYPES" ~python arch=cray-sles15-zen3
 -   ktd2rtp                  ^libiconv@1.16%gcc@11.2.0 cflags="-O3 -DMPICH_SUPPRESS_PROTOTYPES"  libs=shared,static arch=cray-sles15-zen3
 -   qdkb5iq                  ^xz@5.2.5%gcc@11.2.0 cflags="-O3 -DMPICH_SUPPRESS_PROTOTYPES" ~pic libs=shared,static arch=cray-sles15-zen3
 -   2bnlcpl                  ^zlib@1.2.11%gcc@11.2.0 cflags="-O3 -DMPICH_SUPPRESS_PROTOTYPES" +optimize+pic+shared arch=cray-sles15-zen3
 -   vaqxzoi              ^ncurses@6.2%gcc@11.2.0 cflags="-O3 -DMPICH_SUPPRESS_PROTOTYPES" ~symlinks+termlib abi=none arch=cray-sles15-zen3
 -   fpwyohg          ^libevent@2.1.12%gcc@11.2.0 cflags="-O3 -DMPICH_SUPPRESS_PROTOTYPES" +openssl arch=cray-sles15-zen3
 -   hjxxrow              ^openssl@1.1.1n%gcc@11.2.0 cflags="-O3 -DMPICH_SUPPRESS_PROTOTYPES" ~docs~shared certs=system arch=cray-sles15-zen3
 -   qrd57wl                  ^perl@5.34.0%gcc@11.2.0 cflags="-O3 -DMPICH_SUPPRESS_PROTOTYPES" +cpanm+shared+threads arch=cray-sles15-zen3
 -   jqmpsja                      ^berkeley-db@18.1.40%gcc@11.2.0 cflags="-O3 -DMPICH_SUPPRESS_PROTOTYPES" +cxx~docs+stl patches=b231fcc arch=cray-sles15-zen3
 -   r2yxqo3                      ^bzip2@1.0.8%gcc@11.2.0 cflags="-O3 -DMPICH_SUPPRESS_PROTOTYPES" ~debug~pic+shared arch=cray-sles15-zen3
 -   loagpfa                          ^diffutils@3.8%gcc@11.2.0 cflags="-O3 -DMPICH_SUPPRESS_PROTOTYPES"  arch=cray-sles15-zen3
 -   upa6edt                      ^gdbm@1.19%gcc@11.2.0 cflags="-O3 -DMPICH_SUPPRESS_PROTOTYPES"  arch=cray-sles15-zen3
 -   dmxvc3a                          ^readline@8.1%gcc@11.2.0 cflags="-O3 -DMPICH_SUPPRESS_PROTOTYPES"  arch=cray-sles15-zen3
 -   ibt4rse          ^numactl@2.0.14%gcc@11.2.0 cflags="-O3 -DMPICH_SUPPRESS_PROTOTYPES"  patches=4e1d78c,62fc8a8,ff37630 arch=cray-sles15-zen3
 -   vadrpwo              ^autoconf@2.69%gcc@11.2.0 cflags="-O3 -DMPICH_SUPPRESS_PROTOTYPES"  patches=35c4492,7793209,a49dd5b arch=cray-sles15-zen3
 -   73qu7q3              ^automake@1.16.5%gcc@11.2.0 cflags="-O3 -DMPICH_SUPPRESS_PROTOTYPES"  arch=cray-sles15-zen3
 -   zmzxtwv          ^openssh@8.9p1%gcc@11.2.0 cflags="-O3 -DMPICH_SUPPRESS_PROTOTYPES"  arch=cray-sles15-zen3
 -   yqqxfjc              ^libedit@3.1-20210216%gcc@11.2.0 cflags="-O3 -DMPICH_SUPPRESS_PROTOTYPES"  arch=cray-sles15-zen3
 -   n26wveh      ^otf@1.12.5salmon%gcc@11.2.0 cflags="-O3 -DMPICH_SUPPRESS_PROTOTYPES"  arch=cray-sles15-zen3
 -   u2fodns      ^papi@6.0.0.1%gcc@11.2.0 cflags="-O3 -DMPICH_SUPPRESS_PROTOTYPES" ~cuda+example~infiniband~lmsensors~nvml~powercap~rapl~rocm~rocm_smi~sde+shared~static_tools arch=cray-sles15-zen3

```

We expect this to be fixed by https://github.com/spack/spack/pull/29761 and are going to prioritize that PR

Resolving patch:
diff --git a/lib/spack/spack/solver/asp.py b/lib/spack/spack/solver/asp.py
--- a/lib/spack/spack/solver/asp.py
+++ b/lib/spack/spack/solver/asp.py
@@ -2176,48 +2176,41 @@ def reorder_flags(self):
         The solver determines wihch flags are on nodes; this routine
         imposes order afterwards.
         """
-        # nodes with no flags get flag order from compiler
         compilers = dict((c.spec, c) for c in all_compilers_in_config())
-        for pkg in self._flag_compiler_defaults:
-            spec = self._specs[pkg]
-            compiler_flags = compilers[spec.compiler].flags
-            for key in spec.compiler_flags:
-                spec_compiler_flags_set = set(spec.compiler_flags.get(key, []))
-                compiler_flags_set = set(compiler_flags.get(key, []))
-
-                assert spec_compiler_flags_set == compiler_flags_set, "%s does not equal %s" % (
-                    spec_compiler_flags_set,
-                    compiler_flags_set,
-                )
-
-                spec.compiler_flags[key] = compiler_flags.get(key, [])
-        # index of all specs (and deps) from the command line by name
         cmd_specs = dict((s.name, s) for spec in self._command_line_specs for s in spec.traverse())
 
-        # iterate through specs with specified flags
-        for key, sources in self._flag_sources.items():
-            pkg, flag_type = key
-            spec = self._specs[pkg]
-            compiler_flags = spec.compiler_flags.get(flag_type, [])
+        for spec in self._specs.values():
+            # if bootstrapping, compiler is not in config and has no flags
+            flagmap_from_compiler = {}
+            if spec.compiler in compilers:
+                flagmap_from_compiler = compilers[spec.compiler].flags
+
+            for flag_type in spec.compiler_flags.valid_compiler_flags():
+                from_compiler = flagmap_from_compiler.get(flag_type, [])
+                from_sources = []
+
+                # order is determined by the  DAG. A spec's flags come after any of its ancestors
+                # on the compile line
+                source_key = (spec.name, flag_type)
+                if source_key in self._flag_sources:
+                    order = [s.name for s in spec.traverse(order="post", direction="parents")]
+                    sorted_sources = sorted(
+                        self._flag_sources[source_key], key=lambda s: order.index(s)
+                    )
 
-            # order is determined by the DAG.  A spec's flags come after
-            # any from its ancestors on the compile line.
-            order = [s.name for s in spec.traverse(order="post", direction="parents")]
+                    # add flags from each source, lowest to highest precedence
+                    for source_name in sorted_sources:
+                        source = cmd_specs[source_name]
+                        extend_flag_list(from_sources, source.compiler_flags.get(flag_type, []))
 
-            # sort the sources in our DAG order
-            sorted_sources = sorted(sources, key=lambda s: order.index(s))
+                # compiler flags from compilers config are lowest precedence
+                ordered_compiler_flags = from_compiler + from_sources
+                compiler_flags = spec.compiler_flags.get(flag_type, [])
 
-            # add flags from each source, lowest to highest precedence
-            flags = []
-            for source_name in sorted_sources:
-                source = cmd_specs[source_name]
-                extend_flag_list(flags, source.compiler_flags.get(flag_type, []))
+                msg = "%s does not equal %s" % (set(compiler_flags), set(ordered_compiler_flags))
+                assert set(compiler_flags) == set(ordered_compiler_flags), msg
 
-            assert set(compiler_flags) == set(flags), "%s does not equal %s" % (
-                set(compiler_flags),
-                set(flags),
-            )
-            spec.compiler_flags.update({flag_type: source.compiler_flags[flag_type]})
+                spec.compiler_flags.update({flag_type: ordered_compiler_flags})
 
     def deprecated(self, pkg, version):
         msg = 'using "{0}@{1}" which is a deprecated version'


Test patch:
diff --git a/lib/spack/spack/test/concretize.py b/lib/spack/spack/test/concretize.py
--- a/lib/spack/spack/test/concretize.py
+++ b/lib/spack/spack/test/concretize.py
@@ -325,6 +325,13 @@ def test_different_compilers_get_different_flags(self):
         assert set(client.compiler_flags["fflags"]) == set(["-O0", "-g"])
         assert not set(cmake.compiler_flags["fflags"])
 
+    def test_compiler_flags_from_compiler_and_dependent(self):
+        client = Spec("cmake-client %clang@12.2.0 platform=test os=fe target=fe cflags==-g")
+        client.concretize()
+        cmake = client["cmake"]
+        for spec in [client, cmake]:
+            assert spec.compiler_flags["cflags"] == ["-O3", "-g"]
+
     def test_concretize_compiler_flag_propagate(self):
         spec = Spec("hypre cflags=='-g' ^openblas")
         spec.concretize()