Repository: spack/spack

Issue:
Cryptic error message when configuring a compiler for a specific target
I have clang correctly configured in my `compilers.yaml` file with `target: ivybridge`.

Upon attempting to install any arbitrary package with `%clang`, spack fails:

```console
$ spack install bash%clang@8.0.0
==> Error: No compilers with spec clang@8.0.0 found for operating system ubuntu16.04 and target ivybridge.
Run 'spack compiler find' to add compilers.
```
Changing `target` to `linux-ubuntu16.04-ivybridge` does not fix the error. Changing to `x86_64` _does_ fix the error, but reporting this as a bug because `ivybridge` is an appropriate target and the compiler has been appropriately configured for it.

### Steps to reproduce the issue

Edit `compilers.yaml` and change the target of a compiler to the output of `spack arch -t`. Then attempt to install any package with the affected compiler.

### Error Message

```console
$ spack -d --stacktrace install bash%clang@8.0.0
lib/spack/spack/config.py:706 ==> [2019-11-14-15:33:01.315799] Reading config file /home/hdiags/src/spack/etc/spack/defaults/modules.yaml
lib/spack/spack/config.py:706 ==> [2019-11-14-15:33:01.328014] Reading config file /home/hdiags/src/spack/etc/spack/defaults/linux/modules.yaml
lib/spack/spack/config.py:706 ==> [2019-11-14-15:33:01.330991] Reading config file /home/hdiags/src/spack/etc/spack/defaults/config.yaml
lib/spack/spack/cmd/__init__.py:101 ==> [2019-11-14-15:33:01.347916] Imported install from built-in commands
lib/spack/spack/cmd/__init__.py:101 ==> [2019-11-14-15:33:01.349093] Imported install from built-in commands
lib/spack/spack/config.py:706 ==> [2019-11-14-15:33:01.367195] Reading config file /home/hdiags/src/spack/etc/spack/defaults/repos.yaml
lib/spack/spack/config.py:706 ==> [2019-11-14-15:33:01.368930] Reading config file /home/hdiags/src/spack/etc/spack/repos.yaml
lib/spack/spack/config.py:706 ==> [2019-11-14-15:33:01.369852] Reading config file /home/hdiags/.spack/linux/repos.yaml
lib/spack/spack/config.py:706 ==> [2019-11-14-15:33:01.422301] Reading config file /home/hdiags/src/spack/etc/spack/defaults/packages.yaml
lib/spack/spack/config.py:706 ==> [2019-11-14-15:33:01.435570] Reading config file /home/hdiags/src/spack/etc/spack/packages.yaml
lib/spack/llnl/util/lock.py:334 ==> [2019-11-14-15:33:01.443640] READ LOCK: /home/hdiags/.spack/cache/providers/.builtin-index.json.lock[0:0] [Acquiring]
lib/spack/llnl/util/lock.py:334 ==> [2019-11-14-15:33:01.444069] READ LOCK: /home/hdiags/.spack/cache/providers/.builtin-index.json.lock[0:0] [Acquired]
lib/spack/llnl/util/lock.py:334 ==> [2019-11-14-15:33:01.465539] READ LOCK: /home/hdiags/.spack/cache/providers/.builtin-index.json.lock[0:0] [Released]
lib/spack/llnl/util/lock.py:334 ==> [2019-11-14-15:33:01.468488] READ LOCK: /home/hdiags/.spack/cache/patches/.builtin-index.json.lock[0:0] [Acquiring]
lib/spack/llnl/util/lock.py:334 ==> [2019-11-14-15:33:01.468827] READ LOCK: /home/hdiags/.spack/cache/patches/.builtin-index.json.lock[0:0] [Acquired]
lib/spack/llnl/util/lock.py:334 ==> [2019-11-14-15:33:01.476680] READ LOCK: /home/hdiags/.spack/cache/patches/.builtin-index.json.lock[0:0] [Released]
lib/spack/llnl/util/lock.py:334 ==> [2019-11-14-15:33:01.478625] READ LOCK: /home/hdiags/.spack/cache/tags/.builtin-index.json.lock[0:0] [Acquiring]
lib/spack/llnl/util/lock.py:334 ==> [2019-11-14-15:33:01.478946] READ LOCK: /home/hdiags/.spack/cache/tags/.builtin-index.json.lock[0:0] [Acquired]
lib/spack/llnl/util/lock.py:334 ==> [2019-11-14-15:33:01.479452] READ LOCK: /home/hdiags/.spack/cache/tags/.builtin-index.json.lock[0:0] [Released]
lib/spack/llnl/util/lock.py:334 ==> [2019-11-14-15:33:01.487132] READ LOCK: /home/hdiags/.spack/cache/providers/.hdiags-index.json.lock[0:0] [Acquiring]
lib/spack/llnl/util/lock.py:334 ==> [2019-11-14-15:33:01.487653] READ LOCK: /home/hdiags/.spack/cache/providers/.hdiags-index.json.lock[0:0] [Acquired]
lib/spack/llnl/util/lock.py:334 ==> [2019-11-14-15:33:01.488146] READ LOCK: /home/hdiags/.spack/cache/providers/.hdiags-index.json.lock[0:0] [Released]
lib/spack/llnl/util/lock.py:334 ==> [2019-11-14-15:33:01.488749] READ LOCK: /home/hdiags/.spack/cache/patches/.hdiags-index.json.lock[0:0] [Acquiring]
lib/spack/llnl/util/lock.py:334 ==> [2019-11-14-15:33:01.489359] READ LOCK: /home/hdiags/.spack/cache/patches/.hdiags-index.json.lock[0:0] [Acquired]
lib/spack/llnl/util/lock.py:334 ==> [2019-11-14-15:33:01.489827] READ LOCK: /home/hdiags/.spack/cache/patches/.hdiags-index.json.lock[0:0] [Released]
lib/spack/llnl/util/lock.py:334 ==> [2019-11-14-15:33:01.490399] READ LOCK: /home/hdiags/.spack/cache/tags/.hdiags-index.json.lock[0:0] [Acquiring]
lib/spack/llnl/util/lock.py:334 ==> [2019-11-14-15:33:01.490866] READ LOCK: /home/hdiags/.spack/cache/tags/.hdiags-index.json.lock[0:0] [Acquired]
lib/spack/llnl/util/lock.py:334 ==> [2019-11-14-15:33:01.491371] READ LOCK: /home/hdiags/.spack/cache/tags/.hdiags-index.json.lock[0:0] [Released]
lib/spack/spack/config.py:706 ==> [2019-11-14-15:33:01.502424] Reading config file /home/hdiags/.spack/linux/compilers.yaml
lib/spack/spack/cmd/install.py:266 ==> [2019-11-14-15:33:01.518251] UnavailableCompilerVersionError: No compilers with spec clang@8.0.0 found for operating system ubuntu16.04 and target ivybridge.
    Run 'spack compiler find' to add compilers.
lib/spack/spack/main.py:718 ==> [2019-11-14-15:33:01.518412] UnavailableCompilerVersionError: No compilers with spec clang@8.0.0 found for operating system ubuntu16.04 and target ivybridge.
    Run 'spack compiler find' to add compilers.
lib/spack/spack/error.py:55 ==> [2019-11-14-15:33:01.518641] Error: No compilers with spec clang@8.0.0 found for operating system ubuntu16.04 and target ivybridge.
Run 'spack compiler find' to add compilers.
Traceback (most recent call last):
  File "/home/hdiags/src/spack/lib/spack/spack/main.py", line 715, in main
    return _invoke_command(command, parser, args, unknown)
  File "/home/hdiags/src/spack/lib/spack/spack/main.py", line 457, in _invoke_command
    return_val = command(parser, args)
  File "/home/hdiags/src/spack/lib/spack/spack/cmd/install.py", line 264, in install
    args.package, concretize=True, tests=tests)
  File "/home/hdiags/src/spack/lib/spack/spack/cmd/__init__.py", line 141, in parse_specs
    spec.concretize(tests=tests)  # implies normalize
  File "/home/hdiags/src/spack/lib/spack/spack/spec.py", line 2151, in concretize
    self._concretize_helper(concretizer))
  File "/home/hdiags/src/spack/lib/spack/spack/spec.py", line 1979, in _concretize_helper
    concretizer, presets, visited
  File "/home/hdiags/src/spack/lib/spack/spack/spec.py", line 1991, in _concretize_helper
    concretizer.concretize_compiler(self),
  File "/home/hdiags/src/spack/lib/spack/spack/concretize.py", line 376, in concretize_compiler
    other_compiler, spec.architecture)
  File "/home/hdiags/src/spack/lib/spack/spack/concretize.py", line 611, in _compiler_concretization_failure
    raise UnavailableCompilerVersionError(compiler_spec, arch)
spack.concretize.UnavailableCompilerVersionError: No compilers with spec clang@8.0.0 found for operating system ubuntu16.04 and target ivybridge.
    Run 'spack compiler find' to add compilers.

```

### Information on your system

Affects me on both `master` and `develop` (`HEAD` at 3f861e18b).

```console
$ spack arch
linux-ubuntu16.04-ivybridge
```
Affects me for both clang and gcc:

```console
$ cat ~/.spack/linux/compilers.yaml
compilers:
- compiler:
    modules: []
    target: ivybridge
    operating_system: ubuntu16.04
    paths:
      cxx: /usr/bin/g++-5
      fc: /usr/bin/gfortran-5
      f77: /usr/bin/gfortran-5
      cc: /usr/bin/gcc-5
    extra_rpaths: []
    environment: {}
    spec: gcc@5.4.0
    flags: {}
- compiler:
    environment: {}
    extra_rpaths: [/home/hdiags/src/spack/opt/spack/linux-ubuntu16.04-ivybridge/gcc-5.4.0/llvm-8.0.0-cie23zscq3gpdqpvb5gutx3exqldhdgo/lib]
    flags: {}
    modules: [llvm-8.0.0-gcc-5.4.0-cie23zs]
    operating_system: ubuntu16.04
    paths:
      cc: /home/hdiags/src/spack/opt/spack/linux-ubuntu16.04-ivybridge/gcc-5.4.0/llvm-8.0.0-cie23zscq3gpdqpvb5gutx3exqldhdgo/bin/clang
      cxx: /home/hdiags/src/spack/opt/spack/linux-ubuntu16.04-ivybridge/gcc-5.4.0/llvm-8.0.0-cie23zscq3gpdqpvb5gutx3exqldhdgo/bin/clang++
      f77: /usr/bin/gfortran
      fc: /usr/bin/gfortran
    spec: clang@8.0.0
    target: ivybridge
```



Discussion:
Does the output of `spack config blame compilers` look significantly different?
Whoops I just read more carefully:

> Changing to x86_64 does fix the error, but reporting this as a bug because ivybridge is an appropriate target and the compiler has been appropriately configured for it.

Based on that, `spack config blame` will probably look the same
It works as designed, so I wouldn't consider this a bug. To maintain backward compatibility and not have a needless proliferation of compilers the `target` in `compilers.yaml` is the architecture family. 
Oh, I am assuming that this entry in `compilers.yaml` was manually edited, let me know if this was not the case.
@alalazo hi, yes, you're correct--it was manually added, although in an automated fashion (using the output of `spack arch -t`).

I recognise the motivation in avoiding unnecessary specialisation in `compilers.yaml`, but would argue that it's nevertheless a bug for it to refuse to recognise a compiler configured for the very target that spack was specifically seeking to compile for:

`Error: No compilers with spec clang@8.0.0 found for operating system ubuntu16.04 and target ivybridge.`

The workaround (specifying the generic `x86_64` target) is by no means difficult, so I recognise it may not be a high priority, but it caused some amount of confusion for me at least.
@scheibelp yep, for removal of doubt, `spack config blame compilers` confirms spack's using the intended `compilers.yaml` file.
@cdfh In my opinion what's missing here is:
1. A better error message
2. Documentation on the meaning of the `target` field in `compilers.yaml`

If you think about it, when you install a compiler with some architecture support then it's able to generate binaries for all the microarchitectures it knows about belonging to that "family" (e.g. if you install `x86_64` you can optimize for `skylake` as well as `broadwell` etc.).

For the error message I was thinking of something like:
```console
==> Error: the "target" field in compilers.yaml accepts only target families [substitute "ivybridge" with "x86_64"]
```
If that's fine with you I'll change the title of this issue to "Cryptic error message when configuring a compiler for a specific target". Waiting for your reply to do that.
@alalazo Yep, I'm fine with that, thanks!

Resolving patch:
diff --git a/lib/spack/spack/compilers/__init__.py b/lib/spack/spack/compilers/__init__.py
--- a/lib/spack/spack/compilers/__init__.py
+++ b/lib/spack/spack/compilers/__init__.py
@@ -413,6 +413,14 @@ def get_compilers(config, cspec=None, arch_spec=None):
             assert arch_spec is None
 
         if arch_spec and target and (target != family and target != 'any'):
+            # If the family of the target is the family we are seeking,
+            # there's an error in the underlying configuration
+            if llnl.util.cpu.targets[target].family == family:
+                msg = ('the "target" field in compilers.yaml accepts only '
+                       'target families [replace "{0}" with "{1}"'
+                       ' in "{2}" specification]')
+                msg = msg.format(str(target), family, items.get('spec', '??'))
+                raise ValueError(msg)
             continue
 
         compilers.append(_compiler_from_config_entry(items))


Test patch:
diff --git a/lib/spack/spack/test/compilers.py b/lib/spack/spack/test/compilers.py
--- a/lib/spack/spack/test/compilers.py
+++ b/lib/spack/spack/test/compilers.py
@@ -485,3 +485,28 @@ def test_fj_version_detection(version_str, expected_version):
 def test_detecting_mixed_toolchains(compiler_spec, expected_result, config):
     compiler = spack.compilers.compilers_for_spec(compiler_spec).pop()
     assert spack.compilers.is_mixed_toolchain(compiler) is expected_result
+
+
+@pytest.mark.regression('14798,13733')
+def test_raising_if_compiler_target_is_over_specific(config):
+    # Compiler entry with an overly specific target
+    compilers = [{'compiler': {
+        'spec': 'gcc@9.0.1',
+        'paths': {
+            'cc': '/usr/bin/gcc-9',
+            'cxx': '/usr/bin/g++-9',
+            'f77': '/usr/bin/gfortran-9',
+            'fc': '/usr/bin/gfortran-9'
+        },
+        'flags': {},
+        'operating_system': 'ubuntu18.04',
+        'target': 'haswell',
+        'modules': [],
+        'environment': {},
+        'extra_rpaths': []
+    }}]
+    arch_spec = spack.spec.ArchSpec(('linux', 'ubuntu18.04', 'haswell'))
+    with spack.config.override('compilers', compilers):
+        cfg = spack.compilers.get_compiler_config()
+        with pytest.raises(ValueError):
+            spack.compilers.get_compilers(cfg, 'gcc@9.0.1', arch_spec)