Repository: spack/spack

Issue:
"Installation issue: openblas: default blas/lapack 'openblas' can't build on RHEL6 with gcc@7.2.0 built by Spack
**snl-atdm-issue**

### Steps to reproduce the issue

```console
$ ./atdm-spack/spack/bin/spack install openblas %gcc@7.2.0
```

with build errors starting with:

```
==> Installing openblas
==> Searching for binary cache of openblas
==> Finding buildcaches in /ascldap/users/rabartl/Spack.base3/atdm-spack/atdm-spack-tarball-mirror/build_cache
==> No binary for openblas found: installing from source
==> Using cached archive: /home/rabartl/Spack.base3/atdm-spack/spack/var/spack/cache/openblas/openblas-0.3.5.tar.gz
==> Staging archive: /home/rabartl/Spack.base3/atdm-spack/spack/var/spack/stage/openblas-0.3.5-ase7rloe2w5cvr7egf4fglyx5kw2egrl/v0.3.5.tar.gz
==> Created stage in /home/rabartl/Spack.base3/atdm-spack/spack/var/spack/stage/openblas-0.3.5-ase7rloe2w5cvr7egf4fglyx5kw2egrl
==> No patches needed for openblas
==> Building openblas [MakefilePackage]
==> Executing phase: 'edit'
==> Executing phase: 'build'
==> Error: ProcessError: Command exited with status 2:
    'make' '-j32' 'CC=/home/rabartl/Spack.base3/atdm-spack/spack/lib/spack/env/gcc/gcc' 'FC=/home/rabartl/Spack.base3/atdm-spack/spack/lib/spack/env/gcc/gfortran' 'MAKE_NB_JOBS=0' 'USE_OPENMP=0' 'USE_THREAD=0' 'libs' 'netlib' 'shared'

577 errors found in build log:
     1348    /home/rabartl/Spack.base3/atdm-spack/spack/lib/spack/env/gcc/gcc -O2 -DMAX_STACK_ALLOC=2048 -Wall -m64 -DF_INTERFACE_GFORT -fPIC -DNO_AVX512 -DNO_WARMUP -DMAX_CPU_NUMBER=32 -DMAX_PARALLEL_NUMBER=1 
             -DVERSION=\"0.3.5\" -mavx2 -DASMNAME=dtrmm_kernel_LN -DASMFNAME=dtrmm_kernel_LN_ -DNAME=dtrmm_kernel_LN_ -DCNAME=dtrmm_kernel_LN -DCHAR_NAME=\"dtrmm_kernel_LN_\" -DCHAR_CNAME=\"dtrmm_kernel_LN\" -D
             NO_AFFINITY -I.. -DDOUBLE  -UCOMPLEX -c -DTRMMKERNEL -DDOUBLE -UCOMPLEX -DLEFT -UTRANSA ../kernel/x86_64/dtrmm_kernel_4x8_haswell.c -o dtrmm_kernel_LN.o
     1349    /home/rabartl/Spack.base3/atdm-spack/spack/lib/spack/env/gcc/gcc -O2 -DMAX_STACK_ALLOC=2048 -Wall -m64 -DF_INTERFACE_GFORT -fPIC -DNO_AVX512 -DNO_WARMUP -DMAX_CPU_NUMBER=32 -DMAX_PARALLEL_NUMBER=1 
             -DVERSION=\"0.3.5\" -mavx2 -DASMNAME=dtrmm_kernel_LT -DASMFNAME=dtrmm_kernel_LT_ -DNAME=dtrmm_kernel_LT_ -DCNAME=dtrmm_kernel_LT -DCHAR_NAME=\"dtrmm_kernel_LT_\" -DCHAR_CNAME=\"dtrmm_kernel_LT\" -D
             NO_AFFINITY -I.. -DDOUBLE  -UCOMPLEX -c -DTRMMKERNEL -DDOUBLE -UCOMPLEX -DLEFT -DTRANSA ../kernel/x86_64/dtrmm_kernel_4x8_haswell.c -o dtrmm_kernel_LT.o
     1350    /home/rabartl/Spack.base3/atdm-spack/spack/lib/spack/env/gcc/gcc -O2 -DMAX_STACK_ALLOC=2048 -Wall -m64 -DF_INTERFACE_GFORT -fPIC -DNO_AVX512 -DNO_WARMUP -DMAX_CPU_NUMBER=32 -DMAX_PARALLEL_NUMBER=1 
             -DVERSION=\"0.3.5\" -mavx2 -DASMNAME=dtrmm_kernel_RN -DASMFNAME=dtrmm_kernel_RN_ -DNAME=dtrmm_kernel_RN_ -DCNAME=dtrmm_kernel_RN -DCHAR_NAME=\"dtrmm_kernel_RN_\" -DCHAR_CNAME=\"dtrmm_kernel_RN\" -D
             NO_AFFINITY -I.. -DDOUBLE  -UCOMPLEX -c -DTRMMKERNEL -DDOUBLE -UCOMPLEX -ULEFT -UTRANSA ../kernel/x86_64/dtrmm_kernel_4x8_haswell.c -o dtrmm_kernel_RN.o
     1351    /home/rabartl/Spack.base3/atdm-spack/spack/lib/spack/env/gcc/gcc -O2 -DMAX_STACK_ALLOC=2048 -Wall -m64 -DF_INTERFACE_GFORT -fPIC -DNO_AVX512 -DNO_WARMUP -DMAX_CPU_NUMBER=32 -DMAX_PARALLEL_NUMBER=1 
             -DVERSION=\"0.3.5\" -mavx2 -DASMNAME=dtrmm_kernel_RT -DASMFNAME=dtrmm_kernel_RT_ -DNAME=dtrmm_kernel_RT_ -DCNAME=dtrmm_kernel_RT -DCHAR_NAME=\"dtrmm_kernel_RT_\" -DCHAR_CNAME=\"dtrmm_kernel_RT\" -D
             NO_AFFINITY -I.. -DDOUBLE  -UCOMPLEX -c -DTRMMKERNEL -DDOUBLE -UCOMPLEX -ULEFT -DTRANSA ../kernel/x86_64/dtrmm_kernel_4x8_haswell.c -o dtrmm_kernel_RT.o
     1352    /home/rabartl/Spack.base3/atdm-spack/spack/lib/spack/env/gcc/gcc -c -O2 -DMAX_STACK_ALLOC=2048 -Wall -m64 -DF_INTERFACE_GFORT -fPIC -DNO_AVX512 -DNO_WARMUP -DMAX_CPU_NUMBER=32 -DMAX_PARALLEL_NUMBER
             =1 -DVERSION=\"0.3.5\" -mavx2 -DASMNAME=dtrsm_kernel_LN -DASMFNAME=dtrsm_kernel_LN_ -DNAME=dtrsm_kernel_LN_ -DCNAME=dtrsm_kernel_LN -DCHAR_NAME=\"dtrsm_kernel_LN_\" -DCHAR_CNAME=\"dtrsm_kernel_LN\"
              -DNO_AFFINITY -I.. -DDOUBLE  -UCOMPLEX -DTRSMKERNEL -UCOMPLEX -DDOUBLE -DUPPER -DLN -UCONJ ../kernel/x86_64/../generic/trsm_kernel_LN.c -o dtrsm_kernel_LN.o
     1353    ../kernel/x86_64/dgemm_kernel_4x8_haswell.S: Assembler messages:
  >> 1354    ../kernel/x86_64/dgemm_kernel_4x8_haswell.S:1759: Error: no such instruction: `vpermpd $ 0xb1,%ymm0,%ymm0'
  >> 1355    ../kernel/x86_64/dgemm_kernel_4x8_haswell.S:1759: Error: no such instruction: `vpermpd $ 0x1b,%ymm0,%ymm0'
```

The STDOUT from spack is given in:

* [spack.install.openblas.gcc-7.2.0.rhel6..out.txt](https://github.com/spack/spack/files/3458812/spack.install.openblas.gcc-7.2.0.rhel6.out.txt)

The the build log file:

```
atdm-spack/spack/var/spack/stage/openblas-0.3.5-ase7rloe2w5cvr7egf4fglyx5kw2egrl/OpenBLAS-0.3.5/spack-build.out
```

is:

* [OpenBLAS-0.3.5-spack-build.out.txt](https://github.com/spack/spack/files/3458814/OpenBLAS-0.3.5-spack-build.out.txt)

### Platform and user environment

Please report your OS here:
```commandline
 $ uname -a
Linux crf450.srn.sandia.gov 2.6.32-696.30.1.el6.x86_64 #1 SMP Fri May 18 11:50:44 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux

$ lsb_release -d
Description:    Red Hat Enterprise Linux Workstation release 6.9 (Santiago)
``` 

Here are the compilers (where gcc@7.2.0 was built by spack itself with gcc@4.4.7 on the system).

* [compilers.yaml.txt](https://github.com/spack/spack/files/3458813/compilers.yaml.txt)

### Additional information

Sometimes the issue benefits from additional details. In these cases there are
a few things we can suggest doing. First of all, you can post the full output of:

```console
./atdm-spack/spack/bin/spack spec --install-status openblas %gcc@7.2.0
Input spec
--------------------------------
 -   openblas%gcc@7.2.0

Concretized
--------------------------------
==> Error: String contains no specs: 
```

to show people whether Spack installed a faulty software or if it was not able to
build it at all. 



Discussion:
I have added the bug tag back because of 

```
./atdm-spack/spack/bin/spack spec --install-status openblas %gcc@7.2.0
Input spec
--------------------------------
 -   openblas%gcc@7.2.0

Concretized
--------------------------------
==> Error: String contains no specs: 
```

It should be possible to get more info with `spack spec` (and it implies a bug in Spack if it isn't returning any info when that command is run).
@bartlettroscoe can you provide the output of 

```
./atdm-spack/spack/bin/spack -d --stacktrace spec --install-status openblas %gcc@7.2.0
```

? (i.e. rerun that command with `-d --stacktrace`)
@bartlettroscoe can you also provide the processor make of the node you ran on? I.E. skylake/haswell/sandybridge?
@scheibelp asked:

> can you provide the output of
>
> `./atdm-spack/spack/bin/spack -d --stacktrace spec --install-status openblas %gcc@7.2.0`
>
> ?

It is attached in [spack.d.stacktrace.install.openblas.out.txt](https://github.com/spack/spack/files/3468690/spack.d.stacktrace.install.openblas.out.txt).





@becker33 asked:

> @bartlettroscoe can you also provide the processor make of the node you ran on? I.E. skylake/haswell/sandybridge?

```
$ lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                32
On-line CPU(s) list:   0-31
Thread(s) per core:    2
Core(s) per socket:    8
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 79
Model name:            Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz
Stepping:              1
CPU MHz:               3192.902
BogoMIPS:              6384.74
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              25600K
NUMA node0 CPU(s):     0-7,16-23
NUMA node1 CPU(s):     8-15,24-31
```

@bartlettroscoe have you tried with other compilers and/or other versions of openblas?

The reason I ask is that's a weird error to be getting on a Broadwell chip with `-mavx2` flags (as shown in your output. The `A` version of the `vpermpd` operation is part of the avx2 standard, and I'm not quite sure why a broadwell chip would be rejecting it. So my first thought is if there's some bad combination of compiler/version that manifests a bug.
Is there anyone with ECP funding that can get on the SNL SRN system, log into one of these RHEL6 machines and then try to reproduce this?  It seems like we should not have to fight with installing the default BLAS/LAPACK on a RHEL6 machine.
I don't think I have access to any RHEL6 machines at the moment. I'm trying to build gcc@7.2.0 to see if I can reproduce on RHEL7, but building gcc takes a while
@becker33 said:

>  I'm trying to build gcc@7.2.0 to see if I can reproduce on RHEL7, but building gcc takes a while

Thanks @becker33!  I think @fryeguy52 reported that he got the same build error on his RHEL7 machine (but I can't find documentation for that).
Hi,
Redhat 6 has binutils 2.20 which doesn't support avx2 (or at least this instruction)
If you install a recent binutils and add it to the module section in your compiler.yaml it should work

on CentOs 6.10 I have:
as --version
GNU assembler version 2.20.51.0.2-5.48.el6_10.1 20100205
with asm-vpermd.s : 
       .text
        vpermpd $0xb1,%ymm0,%ymm0
as asm-vpermd.s
asm-vpermd.s: Assembler messages:
asm-vpermd.s:2: Error: no such instruction: `vpermpd $0xb1,%ymm0,%ymm0'

It works on CentOS7 with binutils 2.27


@laubry, Spack is already building binutils@2.31.1.  Why does the 'openblas' package not have dependence on 'binutils'?

@bartlettroscoe From what I understand, gcc is looking for gas on the PATH, unless you compile it explicitly with binutils.

It's possible the build will succeed if you add a depends_on("binutils") in openblas.
Openblas shouldn't depend on binutils, I bet it will compile fine with intel compiler or llvm.

The real problem is that "gcc -mavx2" will tell the build system that you  can compile avx2 instructions, but it doesn't check that the assembler it uses will support the instruction set.

The correct fix, should be for spack to provide an "as" wrapper as part of the "compiler" when providing a gcc toolchain.
My workaround is to load the binutils package as a module from the compiler definition.
@bartlettroscoe Since you built binutils, does adding the module in your compilers.yaml fix the build ?
ie:
```
compilers:
- compiler:
    environment: {}
    extra_rpaths: []
    flags: {}
    modules: ["binutils-2.31.1-gcc-x.y.z-wwwwww"]
    operating_system: rhel6
    paths:
      cc: /home/rabartl/Spack.base3/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-
[...]
```
@laubry, thanks for the suggestion.  The problem is that we are building 'openblas' in the same `spack install` command as we are building `binutils`.  So do you think it is worth trying hacking `openblas/package.py` to add a call to `depends_on('binutils')`?
@bartlettroscoe It should work.

Make sure you do depends_on('binutils', type='build'), or you'll drag a runtime dependency.
@laubry, what is the difference between a 'build' dependency and a regular dependency?  There is no mention of the `type` argument at all for `depends_on()` in:

* https://spack.readthedocs.io/en/latest/tutorial_packaging.html

And what would a runtime dependency even mean for a library like 'openblas'?  It has no executbles (or at least none that we would care about).

@bartlettroscoe It is explained in the packaging guide : 
[https://spack.readthedocs.io/en/latest/packaging_guide.html#depends-on](https://spack.readthedocs.io/en/latest/packaging_guide.html#depends-on)

As you say, openblas doesn't need binutils at runtime, but I think that by default, the type for depends_on is type=("build", "link", "run")

Also, I'm not sure (maybe a dev can confirm), I think the hash for the package include link and run dependency, but not the build dep. Which means, if you drop the change at some point (because the problem is fixed) you won't trigger a rebuild of openblas.


@bartlettroscoe I should have read the link I gave you all the way... It's all explained : 

the default type is ('build', 'link'), but you should really use build only.
@laubry, the last thing I am worried about at this point is having to rebuild packages if dependencies change.  We have rebuilt these packages from scratch at least 100 times in the last 6 months trying to get all of this to work.
Wasn't actually closed by #14380 due to reverted code. I hit this again with spack-built gcc 8.3 using openblas.

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/gcc/package.py b/var/spack/repos/builtin/packages/gcc/package.py
--- a/var/spack/repos/builtin/packages/gcc/package.py
+++ b/var/spack/repos/builtin/packages/gcc/package.py
@@ -188,6 +188,11 @@ class Gcc(AutotoolsPackage, GNUMirrorPackage):
     conflicts('languages=objc', when='+nvptx')
     conflicts('languages=obj-c++', when='+nvptx')
 
+    # Newer binutils than RHEL's is required to run `as` on some instructions
+    # generated by new GCC (see https://github.com/spack/spack/issues/12235)
+    conflicts('~binutils', when='@7: os=rhel6',
+              msg='New GCC cannot use system assembler on RHEL6')
+
     if sys.platform == 'darwin':
         # Fix parallel build on APFS filesystem
         # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81797


Test patch: