Repository: spack/spack

Issue:
spack choose older compiler by default
don't know when this started, but at the moment Spack choose older compiler by default, i.e. `gcc@4.7` instead of `gcc@5.4.0`. I see it on a fresh copy on Ubuntu 16.04 with 715ac8b7e67b094c094ec290729c5257cd565a0c.


Discussion:
Hmm, I'm not seeing this behavior. Can you post your `compilers.yaml` and `packages.yaml`?
I also noticed some weird behavior with compilers lately:
```console
$ spack compilers
==> Available compilers
-- gcc ubuntu14-x86_64 ------------------------------------------
gcc@6.3.0  gcc@4.8

$ spack spec -I netlib-scalapack ^mpich ^openblas
Input spec
--------------------------------
     netlib-scalapack
         ^mpich
         ^openblas

Normalized
--------------------------------
     netlib-scalapack
         ^cmake
         ^mpich
         ^openblas

Concretized
--------------------------------
     netlib-scalapack@2.0.2%gcc@4.8~fpic+shared arch=linux-ubuntu14-x86_64 
         ^cmake@3.7.2%gcc@4.8~doc+ncurses+openssl+ownlibs~qt arch=linux-ubuntu14-x86_64 
[+]          ^ncurses@6.0%gcc@4.8~symlinks arch=linux-ubuntu14-x86_64 
[+]              ^mawk@1.3.4%gcc@4.8 arch=linux-ubuntu14-x86_64 
[+]              ^pkg-config@0.29.2%gcc@4.8+internal_glib arch=linux-ubuntu14-x86_64 
[+]          ^openssl@1.0.2k%gcc@4.8 arch=linux-ubuntu14-x86_64 
[+]              ^perl@5.24.1%gcc@4.8+cpanm arch=linux-ubuntu14-x86_64 
[+]              ^zlib@1.2.11%gcc@4.8+pic+shared arch=linux-ubuntu14-x86_64 
         ^mpich@3.2%gcc@4.8+hydra+pmi+romio~verbs arch=linux-ubuntu14-x86_64 
         ^openblas@0.2.19%gcc@4.8~openmp+pic+shared arch=linux-ubuntu14-x86_64 
```

Are any of those dependencies externally installed packages? If `buildable: False` then Spack will try to use the same compiler as the one you specified for that package. See #3166.
I have no external packages on Ubuntu, I will post the configuration files on Monday. 
It seems my problem stems from the configuration hierarchy in some way. I always added compilers using:
```console
$ spack compiler add --scope=site
```
on a particular checkout of `spack`.  This created a `configuration.yaml` that looks like:
```yaml
compilers:
- compiler:
    environment: {}
    extra_rpaths: []
    flags: {}
    modules: []
    operating_system: ubuntu14
    paths:
      cc: /home/mculpo/hpcac-2017/spack-modulefiles/opt/spack/linux-ubuntu14-x86_64/gcc-4.8/gcc-6.3.0-24puqverktq45jg5jwtcusii3o77kwos/bin/gcc
      cxx: /home/mculpo/hpcac-2017/spack-modulefiles/opt/spack/linux-ubuntu14-x86_64/gcc-4.8/gcc-6.3.0-24puqverktq45jg5jwtcusii3o77kwos/bin/g++
      f77: /home/mculpo/hpcac-2017/spack-modulefiles/opt/spack/linux-ubuntu14-x86_64/gcc-4.8/gcc-6.3.0-24puqverktq45jg5jwtcusii3o77kwos/bin/gfortran
      fc: /home/mculpo/hpcac-2017/spack-modulefiles/opt/spack/linux-ubuntu14-x86_64/gcc-4.8/gcc-6.3.0-24puqverktq45jg5jwtcusii3o77kwos/bin/gfortran
    spec: gcc@6.3.0
    target: x86_64
- compiler:
    environment: {}
    extra_rpaths: []
    flags: {}
    modules: []
    operating_system: ubuntu14
    paths:
      cc: /usr/bin/gcc-4.8
      cxx: /usr/bin/g++-4.8
      f77: /usr/bin/gfortran-4.8
      fc: /usr/bin/gfortran-4.8
    spec: gcc@4.8
    target: x86_64
```
Then in another part of the filesystem I started working on a new clone of spack. This clone added an entry in `~/.spack/compilers.yaml`:
```yaml
compilers:
- compiler:
    environment: {}
    extra_rpaths: []
    flags: {}
    modules: []
    operating_system: ubuntu14
    paths:
      cc: /usr/bin/gcc-4.8
      cxx: /usr/bin/g++-4.8
      f77: /usr/bin/gfortran-4.8
      fc: /usr/bin/gfortran-4.8
    spec: gcc@4.8
    target: x86_64
```
From this point on the first version of Spack concretizes as you see above:
```console
$ spack compilers
==> Available compilers
-- gcc ubuntu14-x86_64 ------------------------------------------
gcc@6.3.0  gcc@4.8

$ spack spec -I netlib-scalapack ^mpich ^openblas
Input spec
--------------------------------
     netlib-scalapack
         ^mpich
         ^openblas

Normalized
--------------------------------
     netlib-scalapack
         ^cmake
         ^mpich
         ^openblas

Concretized
--------------------------------
     netlib-scalapack@2.0.2%gcc@4.8~fpic+shared arch=linux-ubuntu14-x86_64 
         ^cmake@3.7.2%gcc@4.8~doc+ncurses+openssl+ownlibs~qt arch=linux-ubuntu14-x86_64 
[+]          ^ncurses@6.0%gcc@4.8~symlinks arch=linux-ubuntu14-x86_64 
[+]              ^mawk@1.3.4%gcc@4.8 arch=linux-ubuntu14-x86_64 
[+]              ^pkg-config@0.29.2%gcc@4.8+internal_glib arch=linux-ubuntu14-x86_64 
[+]          ^openssl@1.0.2k%gcc@4.8 arch=linux-ubuntu14-x86_64 
[+]              ^perl@5.24.1%gcc@4.8+cpanm arch=linux-ubuntu14-x86_64 
[+]              ^zlib@1.2.11%gcc@4.8+pic+shared arch=linux-ubuntu14-x86_64 
         ^mpich@3.2%gcc@4.8+hydra+pmi+romio~verbs arch=linux-ubuntu14-x86_64 
         ^openblas@0.2.19%gcc@4.8~openmp+pic+shared arch=linux-ubuntu14-x86_64 
```
If I remove `~/.spack/compilers.yaml` things go back to what they used to be:
```console
$ mv ~/.spack/linux/compilers.yaml ~/.spack/linux/compilers.yaml.bkp

$ spack compilers
==> Available compilers
-- gcc ubuntu14-x86_64 ------------------------------------------
gcc@6.3.0  gcc@4.8

$ spack spec -I netlib-scalapack ^mpich ^openblas
Input spec
--------------------------------
     netlib-scalapack
         ^mpich
         ^openblas

Normalized
--------------------------------
     netlib-scalapack
         ^cmake
         ^mpich
         ^openblas

Concretized
--------------------------------
[+]  netlib-scalapack@2.0.2%gcc@6.3.0~fpic+shared arch=linux-ubuntu14-x86_64 
[+]      ^cmake@3.7.2%gcc@6.3.0~doc+ncurses+openssl+ownlibs~qt arch=linux-ubuntu14-x86_64 
[+]          ^ncurses@6.0%gcc@6.3.0~symlinks arch=linux-ubuntu14-x86_64 
[+]              ^mawk@1.3.4%gcc@6.3.0 arch=linux-ubuntu14-x86_64 
[+]              ^pkg-config@0.29.2%gcc@6.3.0+internal_glib arch=linux-ubuntu14-x86_64 
[+]          ^openssl@1.0.2k%gcc@6.3.0 arch=linux-ubuntu14-x86_64 
[+]              ^perl@5.24.1%gcc@6.3.0+cpanm arch=linux-ubuntu14-x86_64 
[+]              ^zlib@1.2.11%gcc@6.3.0+pic+shared arch=linux-ubuntu14-x86_64 
[+]      ^mpich@3.2%gcc@6.3.0+hydra+pmi+romio~verbs arch=linux-ubuntu14-x86_64 
[+]      ^openblas@0.2.19%gcc@6.3.0~openmp+pic+shared arch=linux-ubuntu14-x86_64 

```
@davydden Is this your same problem?
Just to be clear:

1. I think we should track down this bug, and see why a change in the configuration hierarchy affects concretization
1. After using Spack for a while on clusters and workstations I propose we make `site` the default `scope`. The `user` scope is potentially a shared state among many installations that can make the behavior of Spack change inadvertently.

What do you think @davydden @tgamblin @adamjstewart ?
Why not set the default configuration scope in `config.yaml`? Then everyone will be happy. Seriously, we don't use that thing enough. We still have a half dozen issues open asking for the ability to set the default parallelism.
@adamjstewart I agree on all the line. In fact in #3183 I added a list of directories in `config.yaml` where to find the templates. We can use the same mechanism for configurations files like `compiler.yaml` or `packages.yaml` and there make `site` the default configuration scope. Something like:
```yaml
config:
  ...
  configuration_scopes:
    site: ${spack}/etc/spack
    user: ${home}/.spack
    .... 
```
If there's agreement I'll start working on a PR later next week.
@citibeth was interested in project-scope configuration and might find it nice to be able to configure this in `config.yaml`.
@alalazo I had some questions about your example:

* ~~does ```spack compiler find``` locate gcc 6.3 or is it outside of ```/usr/bin/```?~~ (whoops sorry just looked at your example again)
* are the versions of the two spack instances the same?
* how recent is the version of each spack instance?
* when ```~/.spack/compilers.yaml``` is present, what happens when you explicitly specify gcc 6.3 or 4.8? For example ```spack spec -I netlib-scalapack%gcc@4.8 ^mpich ^openblas```

To me there are two possibilities:

* ```spack compilers``` is finding different compilers vs. ```spack spec```: ~~I think this is more likely. ```spack compilers``` ends up calling compilers.all_compilers with init_config=True, so in that case Spack will search for compilers~~ EDIT: didn't finish that thought; for now I'll simply leave the possibility as is and exclude speculation
* ```spack spec``` is finding gcc 6.3 and 4.8 but for some reason 4.8 gets prioritized: this wouldn't be a config file problem IMO: the prioritization logic should be able to order the compilers regardless of where they come from

@scheibelp Thanks for looking into this. In case, this week I won't have time to debug the issue, but from next week I should be able to work on it.

The two versions I used above were not the same, but they were pretty recent clones of develop:

1. bd9213a55fecfefd2b67de0383bfb568a58438f0
1. 1d21a11ff4b646666ae8027e7594476f045833d1

> when ~/.spack/compilers.yaml is present, what happens when you explicitly specify gcc 6.3 or 4.8? 

Here's the behavior:
```console
$ spack compilers
==> Available compilers
-- gcc ubuntu14-x86_64 ------------------------------------------
gcc@6.3.0  gcc@4.8

$ spack spec -I netlib-scalapack ^mpich ^openblas
Input spec
--------------------------------
     netlib-scalapack
         ^mpich
         ^openblas

Normalized
--------------------------------
     netlib-scalapack
         ^cmake
         ^mpich
         ^openblas

Concretized
--------------------------------
     netlib-scalapack@2.0.2%gcc@4.8~fpic+shared arch=linux-ubuntu14-x86_64 
         ^cmake@3.7.2%gcc@4.8~doc+ncurses+openssl+ownlibs~qt arch=linux-ubuntu14-x86_64 
[+]          ^ncurses@6.0%gcc@4.8~symlinks arch=linux-ubuntu14-x86_64 
[+]              ^mawk@1.3.4%gcc@4.8 arch=linux-ubuntu14-x86_64 
[+]              ^pkg-config@0.29.2%gcc@4.8+internal_glib arch=linux-ubuntu14-x86_64 
[+]          ^openssl@1.0.2k%gcc@4.8 arch=linux-ubuntu14-x86_64 
[+]              ^perl@5.24.1%gcc@4.8+cpanm arch=linux-ubuntu14-x86_64 
[+]              ^zlib@1.2.11%gcc@4.8+pic+shared arch=linux-ubuntu14-x86_64 
         ^mpich@3.2%gcc@4.8+hydra+pmi+romio~verbs arch=linux-ubuntu14-x86_64 
         ^openblas@0.2.19%gcc@4.8~openmp+pic+shared arch=linux-ubuntu14-x86_64 

$ spack spec -I netlib-scalapack ^mpich ^openblas %gcc@6.3.0
Input spec
--------------------------------
     netlib-scalapack
         ^mpich
         ^openblas%gcc@6.3.0

Normalized
--------------------------------
     netlib-scalapack
         ^cmake
         ^mpich
         ^openblas%gcc@6.3.0

Concretized
--------------------------------
[+]  netlib-scalapack@2.0.2%gcc@6.3.0~fpic+shared arch=linux-ubuntu14-x86_64 
[+]      ^cmake@3.7.2%gcc@6.3.0~doc+ncurses+openssl+ownlibs~qt arch=linux-ubuntu14-x86_64 
[+]          ^ncurses@6.0%gcc@6.3.0~symlinks arch=linux-ubuntu14-x86_64 
[+]              ^mawk@1.3.4%gcc@6.3.0 arch=linux-ubuntu14-x86_64 
[+]              ^pkg-config@0.29.2%gcc@6.3.0+internal_glib arch=linux-ubuntu14-x86_64 
[+]          ^openssl@1.0.2k%gcc@6.3.0 arch=linux-ubuntu14-x86_64 
[+]              ^perl@5.24.1%gcc@6.3.0+cpanm arch=linux-ubuntu14-x86_64 
[+]              ^zlib@1.2.11%gcc@6.3.0+pic+shared arch=linux-ubuntu14-x86_64 
[+]      ^mpich@3.2%gcc@6.3.0+hydra+pmi+romio~verbs arch=linux-ubuntu14-x86_64 
[+]      ^openblas@0.2.19%gcc@6.3.0~openmp+pic+shared arch=linux-ubuntu14-x86_64 
```

sorry for delay in posting, here are my `linux/compilers.yaml`:
```
compilers:
- compiler:
    environment: {}
    extra_rpaths: []
    flags: {}
    modules: []
    operating_system: ubuntu16
    paths:
      cc: /usr/bin/gcc-4.7
      cxx: /usr/bin/g++-4.7
      f77: /usr/bin/gfortran-4.7
      fc: /usr/bin/gfortran-4.7
    spec: gcc@4.7
    target: x86_64
- compiler:
    environment: {}
    extra_rpaths: []
    flags: {}
    modules: []
    operating_system: ubuntu16
    paths:
      cc: /usr/bin/gcc
      cxx: /usr/bin/g++
      f77: /usr/bin/gfortran
      fc: /usr/bin/gfortran
    spec: gcc@5.4.0
    target: x86_64
- compiler:
    environment: {}
    extra_rpaths: []
    flags: {}
    modules: []
    operating_system: ubuntu16
    paths:
      cc: /usr/bin/gcc-4.8
      cxx: /usr/bin/g++-4.8
      f77: /usr/bin/gfortran-4.8
      fc: /usr/bin/gfortran-4.8
    spec: gcc@4.8
    target: x86_64
```
config.yaml:
```
config:
  build_stage:
    - $spack/var/spack/stage
```
and packages.yaml
```
packages:
  all:
    compiler: [gcc]
  #mkl:
  #  version: [11.3.3.210]
  #  paths:
  #    mkl@11.3.3.210: /home/davydden/intel/mkl
  #  buildable: False
```
And I have those issues even on the most simple packages like `libsigsegv`:
```
$ spack spec libsigsegv
Input spec
--------------------------------
libsigsegv

Normalized
--------------------------------
libsigsegv

Concretized
--------------------------------
libsigsegv@2.11%gcc@4.7 arch=linux-ubuntu16-x86_64 
```

@davydden @alalazo @scheibelp I finally figured out how to reproduce this!

With the following `compilers.yaml`:
```yaml
compilers:
- compiler:
    environment: {}
    extra_rpaths: []
    flags: {}
    modules: []
    operating_system: centos7
    paths:
      cc: /blues/gpfs/home/ajstewart/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-7.1.0-4bgguypumyezlt6vfhha3ukntrlrx7fq/bin/gcc
      cxx: /blues/gpfs/home/ajstewart/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-7.1.0-4bgguypumyezlt6vfhha3ukntrlrx7fq/bin/g++
      f77: /blues/gpfs/home/ajstewart/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-7.1.0-4bgguypumyezlt6vfhha3ukntrlrx7fq/bin/gfortran
      fc: /blues/gpfs/home/ajstewart/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-7.1.0-4bgguypumyezlt6vfhha3ukntrlrx7fq/bin/gfortran
    spec: gcc@7.1.0
    target: x86_64
- compiler:
    environment: {}
    extra_rpaths: []
    flags: {}
    modules: []
    operating_system: centos7
    paths:
      cc: /usr/bin/gcc
      cxx: /usr/bin/g++
      f77: /usr/bin/gfortran
      fc: /usr/bin/gfortran
    spec: gcc@4.8.5
    target: x86_64
```
it uses the newer compiler like I would expect:
```
$ spack spec zlib
...
zlib@1.2.11%gcc@7.1.0+pic+shared arch=linux-centos7-x86_64 
```
If I change the order of the compilers:
```yaml
compilers:
- compiler:
    environment: {}
    extra_rpaths: []
    flags: {}
    modules: []
    operating_system: centos7
    paths:
      cc: /usr/bin/gcc
      cxx: /usr/bin/g++
      f77: /usr/bin/gfortran
      fc: /usr/bin/gfortran
    spec: gcc@4.8.5
    target: x86_64
- compiler:
    environment: {}
    extra_rpaths: []
    flags: {}
    modules: []
    operating_system: centos7
    paths:
      cc: /blues/gpfs/home/ajstewart/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-7.1.0-4bgguypumyezlt6vfhha3ukntrlrx7fq/bin/gcc
      cxx: /blues/gpfs/home/ajstewart/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-7.1.0-4bgguypumyezlt6vfhha3ukntrlrx7fq/bin/g++
      f77: /blues/gpfs/home/ajstewart/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-7.1.0-4bgguypumyezlt6vfhha3ukntrlrx7fq/bin/gfortran
      fc: /blues/gpfs/home/ajstewart/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-7.1.0-4bgguypumyezlt6vfhha3ukntrlrx7fq/bin/gfortran
    spec: gcc@7.1.0
    target: x86_64
```
I can reproduce the bug:
```
$ spack spec zlib
...
zlib@1.2.11%gcc@4.8.5+pic+shared arch=linux-centos7-x86_64 
```
So it seems to be choosing the first compiler it sees, not the newest version.

Does anyone know where the default compiler is chosen? It doesn't seem like this should be hard to fix but I don't even know where to start looking.
> Does anyone know where the default compiler is chosen? It doesn't seem like this should be hard to fix but I don't even know where to start looking.

@scheibelp @tgamblin @becker33 ping. I also would like to get this fixed. So if you point us in the right direction, this should be doable. Currently compilers are chosen based on their order in `compilers.yaml` as opposed to their version.
https://github.com/LLNL/spack/blob/462de9847c2271dbd7de149ae775e785c5d31daf/lib/spack/spack/concretize.py#L321

@davydden I believe this is the relevant line. That's where the candidate compilers are sorted during concretization.
@becker33 thanks! it looks like one should get the right order in `spack.compilers.all_compiler_specs()`, then the rest should follow...

EDIT: unfortunately, it's beyond my knowledge of Spack and Python, but i would say that this is the part where sorting should be added https://github.com/LLNL/spack/blob/462de9847c2271dbd7de149ae775e785c5d31daf/lib/spack/spack/compilers/__init__.py#L178-L181
The key in line 321 comes from the previous line, which is the hook that sorts by package preferences. The way to do it is to change the comparison key that comes from the package preferences to sort by version when all else is equal.

Resolving patch:
diff --git a/lib/spack/spack/concretize.py b/lib/spack/spack/concretize.py
--- a/lib/spack/spack/concretize.py
+++ b/lib/spack/spack/concretize.py
@@ -317,6 +317,9 @@ def _proper_compiler_style(cspec, aspec):
             # No compiler with a satisfactory spec was found
             raise UnavailableCompilerVersionError(other_compiler)
 
+        # By default, prefer later versions of compilers
+        compiler_list = sorted(
+            compiler_list, key=lambda x: (x.name, x.version), reverse=True)
         ppk = PackagePrefs(other_spec.name, 'compiler')
         matches = sorted(compiler_list, key=ppk)
 


Test patch:
diff --git a/lib/spack/spack/test/conftest.py b/lib/spack/spack/test/conftest.py
--- a/lib/spack/spack/test/conftest.py
+++ b/lib/spack/spack/test/conftest.py
@@ -259,7 +259,10 @@ def _refresh():
 
     with spack.store.db.write_transaction():
         for spec in spack.store.db.query():
-            t.uninstall(spec)
+            if spec.package.installed:
+                t.uninstall(spec)
+            else:
+                spack.store.db.remove(spec)
 
     install_path.remove(rec=1)
     spack.store.root = str(spack_install_path)
diff --git a/lib/spack/spack/test/spec_syntax.py b/lib/spack/spack/test/spec_syntax.py
--- a/lib/spack/spack/test/spec_syntax.py
+++ b/lib/spack/spack/test/spec_syntax.py
@@ -25,6 +25,7 @@
 import pytest
 import shlex
 
+import spack
 import spack.spec as sp
 from spack.parse import Token
 from spack.spec import *
@@ -314,33 +315,20 @@ def test_multiple_specs_with_hash(self, database):
         assert len(specs) == 2
 
     def test_ambiguous_hash(self, database):
-        dbspecs = database.mock.db.query()
-
-        def find_ambiguous(specs, keyfun):
-            """Return the first set of specs that's ambiguous under a
-               particular key function."""
-            key_to_spec = {}
-            for spec in specs:
-                key = keyfun(spec)
-                speclist = key_to_spec.setdefault(key, [])
-                speclist.append(spec)
-                if len(speclist) > 1:
-                    return (key, speclist)
-
-            # If we fail here, we may need to guarantee that there are
-            # some ambiguos specs by adding more specs to the test DB
-            # until this succeeds.
-            raise RuntimeError("no ambiguous specs found for keyfun!")
+        x1 = Spec('a')
+        x1._hash = 'xy'
+        x1._concrete = True
+        x2 = Spec('a')
+        x2._hash = 'xx'
+        x2._concrete = True
+        database.mock.db.add(x1, spack.store.layout)
+        database.mock.db.add(x2, spack.store.layout)
 
         # ambiguity in first hash character
-        char, specs = find_ambiguous(dbspecs, lambda s: s.dag_hash()[0])
-        self._check_raises(AmbiguousHashError, ['/' + char])
+        self._check_raises(AmbiguousHashError, ['/x'])
 
         # ambiguity in first hash character AND spec name
-        t, specs = find_ambiguous(dbspecs,
-                                  lambda s: (s.name, s.dag_hash()[0]))
-        name, char = t
-        self._check_raises(AmbiguousHashError, [name + '/' + char])
+        self._check_raises(AmbiguousHashError, ['a/x'])
 
     def test_invalid_hash(self, database):
         mpileaks_zmpi = database.mock.db.query_one('mpileaks ^zmpi')