Repository: spack/spack

Issue:
Cray: how do I use MPI compile wrappers?
I'm trying to follow the [current documentation for Crays](http://spack.readthedocs.io/en/latest/basic_usage.html#spack-on-cray) to build parmetis on trinitite (Cray machine at LANL), but spack is trying to use `mpicc` instead of the Cray compiler wrapper `cc`. Hopefully, you can point out my configuration or install command syntax error.

Configuration files:
[compilers.yaml](https://github.com/LLNL/spack/files/487944/compilers.yaml.txt) - automatically generated from `spack compilers`
[packages.yaml](https://github.com/LLNL/spack/files/487945/packages.yaml.txt) - generated based on online documentation.

The documentation uses `mpi:` instead of `mpich:` in `packages.yaml`.  I'm not sure if that is a bug, but I had to change it to `mpich` in my configuration files to allow spack to resolve the mpi-flavor.

Here's the spec:

```
 % spack spec parmetis target=haswell os=CNL ^mpich
Input spec
------------------------------
  parmetis
      ^mpich@7.4.2 arch=cray-CNL-haswell

Normalized
------------------------------
  parmetis
      ^cmake@2.8:
      ^metis@5:
      ^mpich

Concretized
------------------------------
  parmetis@4.0.3%intel@16.0.3~debug~gdb+shared arch=cray-CNL-haswell
      ^cmake@3.6.2%intel@16.0.3~doc+ncurses+openssl~ownlibs~qt arch=cray-CNL-haswell
      ^metis@5.1.0%intel@16.0.3~debug~gdb~idx64~real64+shared arch=cray-CNL-haswell
      ^mpich@7.4.2%intel@16.0.3+hydra+pmi~verbs arch=cray-CNL-haswell
```

That looks good, so try to install:

```
% spack install --dirty  parmetis target=haswell os=CNL
==> Installing parmetis
==> mpich is externally installed in /opt/cray/pe/mpt/7.4.2/gni/mpich-intel/15.0
==> metis is already installed in /usr/projects/draco/vendors/spack.tt.20160920/opt/spack/cray-CNL-haswell/intel-16.0.3/metis-5.1.0-sd6v7to2csrd5oun2hkzufo2rs7sxquh
==> cmake is externally installed in /usr/projects/hpcsoft/cle6.0/common/cmake
==> Already downloaded /usr/projects/draco/vendors/spack.tt.20160920/var/spack/stage/parmetis-4.0.3-vbjpbrq7dxi4q5qei442i6aeij6hwkoa/parmetis-4.0.3.tar.gz
==> Already staged parmetis-4.0.3-vbjpbrq7dxi4q5qei442i6aeij6hwkoa in /usr/projects/draco/vendors/spack.tt.20160920/var/spack/stage/parmetis-4.0.3-vbjpbrq7dxi4q5qei442i6aeij6hwkoa
==> Already patched parmetis
==> Building parmetis
==> Error: Command exited with status 1:
'cmake' '/usr/projects/draco/vendors/spack.tt.20160920/var/spack/stage/parmetis-4.0.3-vbjpbrq7dxi4q5qei442i6aeij6hwkoa/parmetis-4.0.3' '-DCMAKE_INSTALL_PREFIX=/usr/projects/draco/vendors/spack.tt.20160920/opt/spack/cray-CNL-haswell/intel-16.0.3/parmetis-4.0.3-vbjpbrq7dxi4q5qei442i6aeij6hwkoa' '-DCMAKE_BUILD_TYPE=RelWithDebInfo' '-DCMAKE_INSTALL_RPATH_USE_LINK_PATH=FALSE' '-DCMAKE_INSTALL_RPATH=/usr/projects/draco/vendors/spack.tt.20160920/opt/spack/cray-CNL-haswell/intel-16.0.3/parmetis-4.0.3-vbjpbrq7dxi4q5qei442i6aeij6hwkoa/lib:/usr/projects/draco/vendors/spack.tt.20160920/opt/spack/cray-CNL-haswell/intel-16.0.3/parmetis-4.0.3-vbjpbrq7dxi4q5qei442i6aeij6hwkoa/lib64:/opt/cray/pe/mpt/7.4.2/gni/mpich-intel/15.0/lib:/usr/projects/draco/vendors/spack.tt.20160920/opt/spack/cray-CNL-haswell/intel-16.0.3/metis-5.1.0-sd6v7to2csrd5oun2hkzufo2rs7sxquh/lib:/opt/gcc/5.3.0/snos' '-DGKLIB_PATH:PATH=/usr/projects/draco/vendors/spack.tt.20160920/opt/spack/cray-CNL-haswell/intel-16.0.3/metis-5.1.0-sd6v7to2csrd5oun2hkzufo2rs7sxquh/include/GKlib' '-DMETIS_PATH:PATH=/usr/projects/draco/vendors/spack.tt.20160920/opt/spack/cray-CNL-haswell/intel-16.0.3/metis-5.1.0-sd6v7to2csrd5oun2hkzufo2rs7sxquh' '-DCMAKE_C_COMPILER:STRING=/opt/cray/pe/mpt/7.4.2/gni/mpich-intel/15.0/bin/mpicc' '-DCMAKE_CXX_COMPILER:STRING=/opt/cray/pe/mpt/7.4.2/gni/mpich-intel/15.0/bin/mpic++' '-DSHARED:BOOL=ON'

See build log for details:
  /tmp/kellyt/spack-stage/spack-stage-IdePiO/parmetis-4.0.3/spack-build.out
```

Here's the spack-build.log:
[parmetis-spack-build.out.txt](https://github.com/LLNL/spack/files/487961/parmetis-spack-build.out.txt). 

Spack has set `CMAKE_C_COMPILER:STRING=/opt/cray/pe/mpt/7.4.2/gni/mpich-intel/15.0/bin/mpicc` instead of using the Cray compile wrapper `cc` 

Should I have something different in `packages.yaml` or should the install command have different options since we want to use the cray compile wrappers?  



Discussion:
when using `mpi` normally you need to use `mpi` wrappers as otherwise you would need to manually link hell lot of `mpi` libs. Thereby `'-DCMAKE_C_COMPILER:STRING=%s' % spec['mpi'].mpicc,` is correct from my perspective. I don't think it has anything to do with your settings.

What is the error you are getting if you try using `mpi` wrappers?

ps. can you build any other package which uses `mpi` wrappers for compilers? If not, maybe your `mpi` build is broken?

@davydden  On Cray machines, the Cray compile wrappers (CC, cc, ftn) are MPI compile wrappers.  You can change the compiler or MPI modules, but you always use the same wrapper script (for me it is /opt/cray/pe/craype/2.5.6/bin/CC).  These scripts know which compiler and mpi to use or link against based on the loaded modules (or programming environment modules).  That is, spack should always use /opt/cray/pe/craype/2.5.6/bin/CC, /opt/cray/pe/craype/2.5.6/bin/cc, or /opt/cray/pe/craype/2.5.6/bin/ftn.

> On Cray machines, the Cray compile wrappers (CC, cc, ftn) are MPI compile wrappers. You can change the compiler or MPI modules, but you always use the same wrapper script (for me it is /opt/cray/pe/craype/2.5.6/bin/CC)

Tricky. Then i guess one needs a way to customize `spec['mpi'].mpicc` and alike from `packages.yaml`, AFAIK you can't do it now.

Hopefully others who have experience with Cray can suggest a solution.

The package could probably check whether it is on a cray and swap the MPI compilers out.  Better, the MPI packages should just set `mpicc` to BE the regular compiler if on a cray.

@mamelara: this is probably something NERSC folks would want too.  Thoughts?

> Better, the MPI packages should just set mpicc to BE the regular compiler if on a cray.

:+1: Otherwise one would need to add `if cray:` to each packages with `mpi`.

@KineticTheory: if you want to try this, edit the MPI package you're using (I'm assuming you probably have an external set up for cray-mpich, so you could edit `mpich`).  Then in `setup_dependent_package`, make it set `self.spec.mpicc = spack_cc` when on Cray, instead of the path to the wrapper.  And likewise for the other `self.spec.mpi*` attributes.

@tgamblin I missed your most recent comment on this issue until someone on the mailing list pointed it out to me.  I just tried your suggestion:

`$ git diff mpich/package.py`

``` diff
     def setup_dependent_package(self, module, dep_spec):
-        self.spec.mpicc = join_path(self.prefix.bin, 'mpicc')
-        self.spec.mpicxx = join_path(self.prefix.bin, 'mpic++')
-        self.spec.mpifc = join_path(self.prefix.bin, 'mpif90')
-        self.spec.mpif77 = join_path(self.prefix.bin, 'mpif77')
+        self.spec.mpicc = spack_cc
+        self.spec.mpicxx = spack_cxx
+        self.spec.mpifc = spack_f77
+        self.spec.mpif77 = spack_fc
         self.spec.mpicxx_shared_libs = [
             join_path(self.prefix.lib, 'libmpicxx.{0}'.format(dso_suffix)),
             join_path(self.prefix.lib, 'libmpi.{0}'.format(dso_suffix))
```

and this appears to fix the issue on Cray.

```
% spack install --dirty parmetis%intel@16.0.3 ^mpich@7.4.2 os=CNL
==> Installing parmetis
...
==> Building parmetis
==> Successfully installed parmetis
  Fetch: 0.03s.  Build: 21.07s.  Total: 21.10s.
[+] /usr/projects/draco/vendors/spack.tt.20161005/opt/spack/cray-CNL-haswell/intel-16.0.3/parmetis-4.0.3-vbjpbrq7dxi4q5qei442i6aeij6hwkoa
```

> until someone on the mailing list pointed it out to me.

that was me, actually :smile:

> this appears to fix the issue on Cray.

Could you please create a PR which would choose between the two cases using

```
import platform


if platform.system() == "Cray"
```

or something like this?

@davydden I'll do that once I get finish building some tools. At a minimum, I'll also need to update the hdf5 recipe to use `spec['mpi'].mpicc`.

Using `platform.system()` will not work because this just reports `linux`.  I'm testing my changes with `os.environ.get('CRAYPE_VERSION')`, but I know that isn't sufficient for detecting all Cray environments.  I'm hoping someone who helps develop and test spack on Crays can provide advice on how to ask spack if the current machine is a Cray. 

@KineticTheory : did you also add mpich as external package in ~/.spack/packages.yaml ? as a path or module ?

@pramodk Yes, I added an external package definition:

```
packages:
  mpich:
    modules:
      mpich@7.4.2%intel@16.0.3 arch=cray-CNL-haswell: cray-mpich/7.4.2
    buildable: False
  all:
    compiler: [intel@16.0.3]
    providers:
      mpi: [mpich]
```


Resolving patch:
diff --git a/var/spack/repos/builtin/packages/mpich/package.py b/var/spack/repos/builtin/packages/mpich/package.py
--- a/var/spack/repos/builtin/packages/mpich/package.py
+++ b/var/spack/repos/builtin/packages/mpich/package.py
@@ -62,10 +62,18 @@ def setup_dependent_environment(self, spack_env, run_env, dependent_spec):
         spack_env.set('MPICH_FC', spack_fc)
 
     def setup_dependent_package(self, module, dep_spec):
-        self.spec.mpicc = join_path(self.prefix.bin, 'mpicc')
-        self.spec.mpicxx = join_path(self.prefix.bin, 'mpic++')
-        self.spec.mpifc = join_path(self.prefix.bin, 'mpif90')
-        self.spec.mpif77 = join_path(self.prefix.bin, 'mpif77')
+        # Is this a Cray machine? (TODO: We need a better test than this.)
+        if os.environ.get('CRAYPE_VERSION'):
+            self.spec.mpicc = spack_cc
+            self.spec.mpicxx = spack_cxx
+            self.spec.mpifc = spack_fc
+            self.spec.mpif77 = spack_f77
+        else:
+            self.spec.mpicc = join_path(self.prefix.bin, 'mpicc')
+            self.spec.mpicxx = join_path(self.prefix.bin, 'mpic++')
+            self.spec.mpifc = join_path(self.prefix.bin, 'mpif90')
+            self.spec.mpif77 = join_path(self.prefix.bin, 'mpif77')
+
         self.spec.mpicxx_shared_libs = [
             join_path(self.prefix.lib, 'libmpicxx.{0}'.format(dso_suffix)),
             join_path(self.prefix.lib, 'libmpi.{0}'.format(dso_suffix))


Test patch: