Repository: spack/spack

Issue:
Boost fails to build on ANL's Theta
I'm trying to install boost with spack on ANL's Theta machine, using the GNU compilers. I tried version 1.68.0 and 1.67.0. In both cases the build fails when calling `b2 --clean`. When I go in the the build directory and call `./b2 -- clean` myself, I get `Illegal instruction (core dumped)`.

The same happens with Intel compiler instead of GNU.

If I try to build boost manually (i.e. downloading the archive, running `./bootstrap.sh` with the libraries I want, then `./b2` and `./b2 install`) it works fine.

### Steps to reproduce the issue

```console
$ spack install boost
==> diffutils is already installed in /gpfs/mira-home/mdorier/spack/opt/spack/cray-cnl6-mic_knl/gcc-7.3.0/diffutils-3.6-hdksfzqzgtfwvwbvhu7s75u6f7tkudvz
==> bzip2 is already installed in /gpfs/mira-home/mdorier/spack/opt/spack/cray-cnl6-mic_knl/gcc-7.3.0/bzip2-1.0.6-m22cmmtaf36fnfmvwy36dssvltqfw73q
==> zlib is already installed in /gpfs/mira-home/mdorier/spack/opt/spack/cray-cnl6-mic_knl/gcc-7.3.0/zlib-1.2.11-yuj7ryis72kd6iwpq3qmvp627umsbymc
==> Installing boost
==> Fetching https://dl.bintray.com/boostorg/release/1.67.0/source/boost_1_67_0.tar.bz2
######################################################################## 100.0%
==> Staging archive: /gpfs/mira-home/mdorier/spack/var/spack/stage/boost-1.67.0-a5gxl76fce2fobsekfnhvrhsl5j6j6cw/boost_1_67_0.tar.bz2
==> Created stage in /gpfs/mira-home/mdorier/spack/var/spack/stage/boost-1.67.0-a5gxl76fce2fobsekfnhvrhsl5j6j6cw
==> Applied patch python_jam.patch
==> Building boost [Package]
==> Executing phase: 'install'
==> Error: ProcessError: Command exited with status -4:
    './b2' '--clean'
```

### Platform and user environment

```commandline
$ uname -a 
Linux thetalogin6 4.4.103-6.38-default #1 SMP Mon Dec 25 20:44:33 UTC 2017 (e4b9067) x86_64 x86_64 x86_64 GNU/Linux
$ lsb_release -d
Description:	SUSE Linux Enterprise Server 12 SP3
``` 

- `compilers.yaml`

```
compilers:
- compiler:
    environment: {}
    extra_rpaths: []
    flags: {}
    modules:
    - PrgEnv-cray
    - cce/8.7.3
    operating_system: cnl6
    paths:
      cc: cc
      cxx: CC
      f77: ftn
      fc: ftn
    spec: cce@8.7.3
    target: any
- compiler:
    environment: {}
    extra_rpaths: []
    flags: {}
    modules:
    - PrgEnv-gnu
    - gcc/7.3.0
    operating_system: cnl6
    paths:
      cc: cc
      cxx: CC
      f77: ftn
      fc: ftn
    spec: gcc@7.3.0
    target: any
- compiler:
    environment: {}
    extra_rpaths: []
    flags: {}
    modules: []
    operating_system: cnl6
    paths:
      cc: /opt/cray/pe/craype/2.5.15/bin/cc
      cxx: /opt/cray/pe/craype/2.5.15/bin/CC
      f77: /opt/cray/pe/craype/2.5.15/bin/ftn
      fc: /opt/cray/pe/craype/2.5.15/bin/ftn
    spec: intel@18.0.2.199
    target: any
```
boost fails to build on Cray (Theta) with "illegal instruction"
'spack install boost' fails trying to run 'b2 --clean' :  b2 was built with back-end instructions and so fails with 

```
 ./b2 --clean   
[1]    57337 illegal hardware instruction (core dumped)  ./b2 --clean
```
my default environment uses the intel compilers, though 'spack spec' suggests things are built with gcc-6.3.

I do not see a way to ask spack to tell boost to use "toolset=cray".  (was trying to follow advice found in https://cug.org/proceedings/cug2016_proceedings/includes/files/pap127.pdf , which did seem to work for a non-spack build)


Discussion:
I get a similar problem when trying to build `cmake`, which builds fine manual, but not with spack. I think what happens is both cmake and boost are first compiling an executable that is later called during the build process, but on Theta compilers are actually cross-compiling for the compute nodes (cray-cnl6-mic_knl architecture), so some instructions don't exist on the login node where spack is used.
@dorier I just ran into this (boost failure) too!
Thanks for posting.
@jgalarowicz For now the only way to solve this is to build manually and have you packages.yaml file point to the manually installed boost. I think the Spack developers are working on a way to deal with such cross-compiling issues.
@dorier Thanks for the information!
This bug goes back at least two years to #3209.

Actually, you can build boost now by usetting CRAYPE_VERSION and
building for the front-end.  This is the same result you'd get by
building boost natively, outside of spack, using FE compilers.

But this would build for arch=linux-sles12-x86_64 and I don't know if
that would fit into a regular, back-end build for Cray with
arch=cray-cnl6-mic_knl.

----------

To recap the problem, boost builds b2/bjam, a C program and then runs
it on the FE as its own version of make.  The spack wrappers override
the C compiler (cc) and turn it into a BE compiler.  The binary dies
on SIGILL, in this case vinserti64x4.

```
Program terminated with signal SIGILL, Illegal instruction.
#0  0x000000000042c2ad in ?? ()

  42c2ad:    62 e3 f5 48 3a cb 01   vinserti64x4 $0x1,%ymm3,%zmm1,%zmm17
  42c2b4:    c4 61 f9 6e c3         vmovq  %rbx,%xmm8
  42c2b9:    c4 e1 f9 6e c9         vmovq  %rcx,%xmm1
```

Btw, if you run `spack install --keep-stage`, you can see that bjam
was built for BE because ldd reports that it links with `librca.so.0`
and `libAtpSigHandler.so.0`.  Building it for FE does not.  Also,
`spack -d install` shows the spack wrappers used for the bjam source
files.

To fix the problem, you need to run the bootstrap step without the
spack wrappers and build bjam with an unmodified FE compiler.

I don't know a general purpose mechanism for doing this.  I think you
have to fix it package by package.  Perhaps spack could write a method
for running a process as part of the build and disable the spack
wrappers for that step.  That would cover some common uses.  Maybe
something like:

```
@run_without_compiler_wrappers
bootstrap(*bootstrap_options)
```

@tgamblin  Is this sort of thing possible in spack?

Btw, libdwarf is also technically broken for the same reason.  It also
builds gennames as a C program and runs it as part of the build.
Again, the spack wrappers build it for the BE.

However, the libdwarf build doesn't fall over because gennames is a
much simpler binary and doesn't generate any vinsert instructions.
So, even though it was built for the BE, it runs on the FE and the
build succeeds.

Cray and Blue Gene are simplified versions of the more general problem
here.  A lot of FE code runs on the BE, so it's easy to miss the real
problem.  (I call this a 'cross-link' as opposed to a true cross-
compile.)

If the BE was k1om or ARM or something that was a true cross compile,
then the real problem would be more obvious.  Fundamentally, for boost
and libdwarf (probably others), the compiler for bjam or gennames must
be a different compiler than the one to build libboost or libdwarf.

----------

Anyway, I think there's a simple way to fix the problem, at least for
this case with Cray.  It's only 10 lines of shell code to patch
bootstrap.sh to remove the spack wrappers from PATH and build bjam the
native boost way.  This is more or less Todd's suggestion from #3209,
but I guess no one ever followed up.

This only affects building bjam.  The boost libraries are still built
the normal spack way with the spack wrappers, as you can see from
`spack -d install`.

I'll submit a PR to fix this, unless someone has a better way.

@roblatham00: Spack knows what the default front-end compiler is, and we could expose that within the build -- we just don't currently do that.

We should probably patch the boost build so that `b2` is always built with the frontend compiler, and the other stuff is built with whatever the compiler is using.

This is related to @scheibelp's #2548, but it's a) different because `b2` isn't an explicit dependency -- it's a build tool created by a single package during its own x-compile build, and b) simpler, in that we just need to export a couple more variables to tell packages about the host compiler.

@scheibelp: do you know where in the boost build `b2` gets built and whether it would be easy to patch it as I'm describing?
> do you know where in the boost build b2 gets built and whether it would be easy to patch it as I'm describing?

b2 is built with bootstrap.sh. I suspect the solution is to invoke ```bootstrap --with-toolset=<frontend compiler toolset>``` and then write out user-config.jam with the backend compiler (and specifically to do this when the boost target does not match the frontend target). Assuming I'm right about that I think it would be fairly straightforward and I could try to get a PR out by the end of Monday 2/27.
> my default environment uses the intel compilers, though 'spack spec' suggests things are built with gcc-6.3

The best way to ensure building with intel is to include it in the spec: ```spack spec boost%intel``` or ```spack install boost%intel```

@tgamblin this suggests that this particular error message is not an issue with using the backend compiler to build b2 (although that problem may come up when spack tries to build boost with cce). In fact as of now there isn't a toolset entry for the cce compiler for boost's package.py. Once that is in place the problem you describe may come up.
@scheibelp: I think it may still be a backend issue -- gcc can still build for the backend or frontend OS on the cray... right @mamelara and @becker33?

~~@roblatham00 @wscullin: is the compute node arch on Theta different from the architecture on the login nodes?~~

nevermind I read the Theta description.  It almost certainly is, as Theta is all Xeon Phi.  I was scanning the docs for "KNL" and read "Xeon Phi" as "Xeon" 😄 .

@scheibelp: that PR would be really useful.
> I think it may still be a backend issue -- gcc can still build for the backend or frontend OS on the cray

If that's true, then I presume it is some sort of compiler flag? In which case would a user creates two entries the same compiler paths but with different flags and targets?

> I do not see a way to ask spack to tell boost to use "toolset=cray"

Apologies as I didn't answer this one: at the moment there isn't a way. In many cases Spack picks a boost toolset based on the compiler (and there is a 1-to-1 mapping of compilers to boost toolsets). This may be a case which doesn't fit that.
@roblatham00: if you want to programmatically set `toolset=Cray`, even for intel compiler builds, you can check he platform like this:

```python
if 'platform=cray' in spec:
    # set boost args 
```

I'm still not sure boost will properly cross compile b2 if you do that, though, as on most Crays you can get away with compiling front end tools with the backend compiler, so it may not be build into their build.  But you can try that while @scheibelp is working on this...
I've requested access to a theta system so I should be able to resolve the issue. I don't think my 2/27 estimate was accurate though. I'll aim to provide an update by the end of the week (3/3). 
I am not sure if this is relevant for boost scenario here but if there are certain component needs to be built for front-end (in two-stage-installation), I have set cflags/cxxflags as: 

```
        if 'cray' in self.spec.architecture:
            flags = '-target-cpu=x86_64 -target-network=none'
            options.extend(['CFLAGS=%s' % flags,
                                        'CXXFLAGS=%s' % flags])
```

With this cray wrappers (`cc`, `CC`) will compile code for fronted-arch ([cray-docs](http://docs.cray.com/books/S-2801-1608/S-2801-1608.pdf)).

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/boost/package.py b/var/spack/repos/builtin/packages/boost/package.py
--- a/var/spack/repos/builtin/packages/boost/package.py
+++ b/var/spack/repos/builtin/packages/boost/package.py
@@ -156,6 +156,9 @@ class Boost(Package):
     patch('boost_1.63.0_pgi.patch', when='@1.63.0%pgi')
     patch('boost_1.63.0_pgi_17.4_workaround.patch', when='@1.63.0%pgi@17.4')
 
+    # Fix the bootstrap/bjam build for Cray
+    patch('bootstrap-path.patch', when='@1.39.0: platform=cray')
+
     def url_for_version(self, version):
         if version >= Version('1.63.0'):
             url = "https://dl.bintray.com/boostorg/release/{0}/source/boost_{1}.tar.bz2"


Test patch: