Repository: spack/spack

Issue:
Compiler flags have a random order
When using python 3, flags randomised at build time resulting in multiple installs based on all possible combinations of multiple flags - issue arose as Ubuntu 18.04 doesn't include python2 in a base image.

### Steps to reproduce the issue
1. Not necessary on ubuntu 18.04 - but necessary on SLES 12 SP3 
vim \`which spack\`
change:
#!/usr/bin/env python
to
#!/usr/bin/env python3

2. modify compilers.yaml
spack config --scope defaults edit compilers
#### modify gcc/default entry with e.g.
    flags:
       cppflags: -fPIC -mtune=native
       cflags: -fPIC -mtune=native
       cxxflags: -fPIC -mtune=native
       fflags: -fPIC -mtune=native
       ldflags: -fPIC -Wl,--as-needed

#### install a package multiple times - e.g. more than 4 times in the above
for i in \`seq 1 10\` ; do spack install zlib; done

#### expected results - 1 install, 9 already installed messages
#### actual results - 4 installs of packages with different concretizations based on order of flags (easiest show by uninstall I think):

### Error Message

spack uninstall --all zlib
==> The following packages will be uninstalled:

    -- linux-ubuntu18.04-x86_64 / gcc@7.3.0 -------------------------
    fj6ajxq zlib@1.2.11%gcc cflags="-fPIC -mtune=native" cppflags="-fPIC -mtune=native" cxxflags="-fPIC -mtune=native" fflags="-fPIC -mtune=native" ldflags="-Wl,--as-needed -fPIC" +optimize+pic+shared
    h227bca zlib@1.2.11%gcc cflags="-fPIC -mtune=native" cppflags="-fPIC -mtune=native" cxxflags="-fPIC -mtune=native" fflags="-fPIC -mtune=native" ldflags="-fPIC -Wl,--as-needed" +optimize+pic+shared
    gbbmeio zlib@1.2.11%gcc cflags="-mtune=native -fPIC" cppflags="-mtune=native -fPIC" cxxflags="-mtune=native -fPIC" fflags="-mtune=native -fPIC" ldflags="-Wl,--as-needed -fPIC" +optimize+pic+shared
    qn3j54v zlib@1.2.11%gcc cflags="-mtune=native -fPIC" cppflags="-mtune=native -fPIC" cxxflags="-mtune=native -fPIC" fflags="-mtune=native -fPIC" ldflags="-fPIC -Wl,--as-needed" +optimize+pic+shared
==> Do you want to proceed? [y/N] 

### Information on your system

tested on:
spack 0.11.2 and 0.12.0
Ubuntu 18.04 - python 3.6.7
SLES 12 SP3 - python/3.5.1 python/3.6.1
Also SLES 12 SP3 - python/2.7.13 where everything works as expected.



Discussion:
The issue can't be reproduced injecting the flags from the command line:
```console
$ for ii in $(seq 1 10) ; do spack spec -l zlib cppflags="-fPIC -mtune=native" ; done
Input spec
--------------------------------
zlib cppflags="-fPIC -mtune=native" 

Concretized
--------------------------------
6lcnhlc  zlib@1.2.11%gcc@8.0.1 cppflags="-fPIC -mtune=native" +optimize+pic+shared arch=linux-ubuntu18.04-x86_64 

...

Input spec
--------------------------------
zlib cppflags="-fPIC -mtune=native" 

Concretized
--------------------------------
6lcnhlc  zlib@1.2.11%gcc@8.0.1 cppflags="-fPIC -mtune=native" +optimize+pic+shared arch=linux-ubuntu18.04-x86_64 
```
while it can be reproduced using flags from `compilers.yaml` (as reported above).
I think I found the root of the problem:

https://github.com/spack/spack/blob/06dcf72f142cf2637a3044aae7c5bff429a9e7b4/lib/spack/spack/concretize.py#L404-L409

In the lines above during concretization we reset the flags for a `spec` after manipulating `set` objects, which are inherently unordered. This happens only if we have flags attached to the compiler (i.e. if we have flags in `compilers.yaml`) and that explains why we don't see the issue when injecting from the command line.
Apparently we have the same issue here:

https://github.com/spack/spack/blob/06dcf72f142cf2637a3044aae7c5bff429a9e7b4/lib/spack/spack/concretize.py#L379-L390

where it is noted in a comment that the set operation might reorder flags.
duplicate of #6179

Turns out that @baip already posted a similar issue
Duplicate of #6179
@tgamblin @scheibelp @adamjstewart @becker33

I think the current logic for compiler flags is broken, in the sense that:
1. It relies on the fact that a random permutation of the same set of flags will produce the same binary
2. It produces, under the same initial conditions (same config, same command, etc.), different concrete specs starting from the same abstract spec

To solve just point 2. we could probably sort the list of flags:
```python
spec.compiler_flags[flag] = sorted(x for x in nearest_flags | flags)
```
but that won't handle the false assumption at point 1. 

A simple example to support this point is that `-O3 -O1` is different from `-O1 -O3`. As silly as it might seem, this case models well a real use-case where e.g. `compilers.yaml` sets flags to `-O3` on the compiler and the spec injects `-O1` because of known problems of that particular package with a specific compiler.

Do we have docs other than [this](https://spack.readthedocs.io/en/latest/tutorial_configuration.html#compiler-flags) where we specify exactly the semantics of the various way in which we can inject compiler flags? I couldn't find it and currently I am under the impression that to fix this issue properly we should:
1. Decide a clear precedence order for the various methods ("command line" over "spec" over "compilers.yaml"?)
2. Keep all the flags in the same order as they were given in input
3. (Bonus) Have a way not to propagate down the DAG the flags that are passed on the command line (if I say `foo%gcc cflags='-funroll-loops'` I probably don't mean to compile all its dependencies with the same flag)

I can take care of implementing a solution to this as soon as we reach an agreement on what to do. Thoughts?
> Do we have docs other than [this](https://spack.readthedocs.io/en/latest/tutorial_configuration.html#compiler-flags) where we specify exactly the semantics of the various way in which we can inject compiler flags?

Not that I know of.

> to fix this issue properly we should:

Overall I think 2 can be fixed independently from 1 and 3. I agree that all three are worth addressing but 1 and 3 will take more effort (and potentially some consensus on conventions).

> 1. Decide a clear precedence order for the various methods ("command line" over "spec" over "compilers.yaml"?)

Enforcing a precedence other than what comes about "naturally" (if you were to append all flags rather than doing set combinations) would likely be difficult. It's probably easiest to determine what the precedence order *is* for now, record it, and then change that later if it ought to be different.

Note that in addition to the sources of compiler flags you mentioned, packages can also add compiler flags by overriding the `flag_handler` function. See: https://spack.readthedocs.io/en/latest/packaging_guide.html#compiler-flags

> 2. Keep all the flags in the same order as they were given in input

This is the most important thing to fix the bug as it is reported here. I think the required changes would also be smaller and simpler compared to 1 and 3.

> 3. (Bonus) Have a way not to propagate down the DAG the flags that are passed on the command line (if I say foo%gcc cflags='-funroll-loops' I probably don't mean to compile all its dependencies with the same flag)

Perhaps we could eliminate all compiler flag propagation, and the only universal compiler flags would come from `compilers.yaml`. IMO controlling whether flags are propagated from the command line would be complicated vs. just having the user specify what flags to use in `packages.yaml`.

https://github.com/spack/spack/issues/6154 provides a (potentially more-motivating) example of flag reordering leading to different semantics

Resolving patch:
diff --git a/lib/spack/spack/concretize.py b/lib/spack/spack/concretize.py
--- a/lib/spack/spack/concretize.py
+++ b/lib/spack/spack/concretize.py
@@ -380,13 +380,12 @@ def concretize_compiler_flags(self, spec):
                                if (compiler_match(p) and
                                    (p is not spec) and
                                    flag in p.compiler_flags))
-                nearest_flags = set(nearest.compiler_flags.get(flag, []))
-                flags = set(spec.compiler_flags.get(flag, []))
-                if (nearest_flags - flags):
-                    # TODO: these set operations may reorder the flags, which
-                    # for some orders of flags can be invalid. See:
-                    # https://github.com/spack/spack/issues/6154#issuecomment-342365573
-                    spec.compiler_flags[flag] = list(nearest_flags | flags)
+                nearest_flags = nearest.compiler_flags.get(flag, [])
+                flags = spec.compiler_flags.get(flag, [])
+                if set(nearest_flags) - set(flags):
+                    spec.compiler_flags[flag] = list(
+                        llnl.util.lang.dedupe(nearest_flags + flags)
+                    )
                     ret = True
             except StopIteration:
                 pass
@@ -402,10 +401,12 @@ def concretize_compiler_flags(self, spec):
                 raise
             return ret
         for flag in compiler.flags:
-            config_flags = set(compiler.flags.get(flag, []))
-            flags = set(spec.compiler_flags.get(flag, []))
-            spec.compiler_flags[flag] = list(config_flags | flags)
-            if (config_flags - flags):
+            config_flags = compiler.flags.get(flag, [])
+            flags = spec.compiler_flags.get(flag, [])
+            spec.compiler_flags[flag] = list(
+                llnl.util.lang.dedupe(config_flags + flags)
+            )
+            if set(config_flags) - set(flags):
                 ret = True
 
         return ret


Test patch:
diff --git a/lib/spack/spack/test/concretize.py b/lib/spack/spack/test/concretize.py
--- a/lib/spack/spack/test/concretize.py
+++ b/lib/spack/spack/test/concretize.py
@@ -179,9 +179,9 @@ def test_different_compilers_get_different_flags(self):
                       ' ^cmake %clang@3.5 platform=test os=fe target=fe')
         client.concretize()
         cmake = client['cmake']
-        assert set(client.compiler_flags['cflags']) == set(['-O0'])
+        assert set(client.compiler_flags['cflags']) == set(['-O0', '-g'])
         assert set(cmake.compiler_flags['cflags']) == set(['-O3'])
-        assert set(client.compiler_flags['fflags']) == set(['-O0'])
+        assert set(client.compiler_flags['fflags']) == set(['-O0', '-g'])
         assert not set(cmake.compiler_flags['fflags'])
 
     def test_architecture_inheritance(self):
diff --git a/lib/spack/spack/test/data/compilers.yaml b/lib/spack/spack/test/data/compilers.yaml
--- a/lib/spack/spack/test/data/compilers.yaml
+++ b/lib/spack/spack/test/data/compilers.yaml
@@ -98,9 +98,9 @@ compilers:
       f77: /path/to/gfortran472
       fc: /path/to/gfortran472
     flags:
-      cflags: -O0
-      cxxflags: -O0
-      fflags: -O0
+      cflags: -O0 -g
+      cxxflags: -O0 -g
+      fflags: -O0 -g
     modules: 'None'
 - compiler:
     spec: clang@3.5
diff --git a/lib/spack/spack/test/spec_semantics.py b/lib/spack/spack/test/spec_semantics.py
--- a/lib/spack/spack/test/spec_semantics.py
+++ b/lib/spack/spack/test/spec_semantics.py
@@ -743,3 +743,16 @@ def test_spec_formatting(self):
             expected = getattr(arch, prop, "")
             actual = spec.format(named_str)
             assert str(expected) == actual
+
+    @pytest.mark.regression('9908')
+    def test_spec_flags_maintain_order(self):
+        # Spack was assembling flags in a manner that could result in
+        # different orderings for repeated concretizations of the same
+        # spec and config
+        spec_str = 'libelf %gcc@4.7.2 os=redhat6'
+        for _ in range(25):
+            s = Spec(spec_str).concretized()
+            assert all(
+                s.compiler_flags[x] == ['-O0', '-g']
+                for x in ('cflags', 'cxxflags', 'fflags')
+            )