Repository: spack/spack

Issue:
Spack dependency resolution and normalization broken
I'm on the latest develop branch and having problems specifying an installation. I want to build parallel NetCDF and NetCDF-Fortran with PGI. OpenSSL and libpciaccess can't be built with PGI, so I'm linking to GCC versions. When I run:

```
$ spack spec netcdf-fortran %pgi ^netcdf+mpi ^openssl%gcc
Input spec
------------------------------
  netcdf-fortran%pgi
      ^netcdf+mpi
      ^openssl%gcc

Normalized
------------------------------
  netcdf-fortran%pgi
      ^netcdf+mpi
          ^curl
              ^openssl%gcc
                  ^zlib
          ^m4

Concretized
------------------------------
  netcdf-fortran@4.4.4%pgi@16.5-0 arch=linux-centos6-x86_64
      ^netcdf@4.4.1%pgi@16.5-0+hdf4+mpi arch=linux-centos6-x86_64
          ^curl@7.50.1%pgi@16.5-0 arch=linux-centos6-x86_64
              ^openssl@1.0.2h%gcc@6.1.0 arch=linux-centos6-x86_64
                  ^zlib@1.2.8%pgi@16.5-0 arch=linux-centos6-x86_64
          ^hdf@4.2.11%pgi@16.5-0+szip arch=linux-centos6-x86_64
              ^jpeg@9b%pgi@16.5-0 arch=linux-centos6-x86_64
              ^szip@2.1%pgi@16.5-0 arch=linux-centos6-x86_64
          ^hdf5@1.10.0-patch1%pgi@16.5-0+cxx~debug+fortran+mpi+shared+szip~threadsafe arch=linux-centos6-x86_64
              ^mvapich2@2.2b%pgi@16.5-0~debug~gforker+hydra~mrail~nemesis~nemesisib~nemesisibtcp+psm~remshell~slurm~sock arch=linux-centos6-x86_64
                  ^libpciaccess@0.13.4%pgi@16.5-0 arch=linux-centos6-x86_64
                      ^libtool@2.4.6%pgi@16.5-0 arch=linux-centos6-x86_64
                          ^m4@1.4.17%pgi@16.5-0+sigsegv arch=linux-centos6-x86_64
                              ^libsigsegv@2.10%pgi@16.5-0 arch=linux-centos6-x86_64
```

It works. But when I also specify the compiler to use for libpciaccess:

```
$ spack spec netcdf-fortran %pgi ^netcdf+mpi ^openssl%gcc ^libpciaccess%pgi
Input spec
------------------------------
  netcdf-fortran%pgi
      ^libpciaccess%pgi
      ^netcdf+mpi
      ^openssl%gcc

Normalized
------------------------------
==> Error: netcdf-fortran does not depend on libpciaccess
```



Discussion:
A simpler example. The following crashes:

```
$ spack spec netcdf ^hdf5@1.8.16
Input spec
------------------------------
  netcdf
      ^hdf5

Normalized
------------------------------
==> Error: netcdf does not depend on hdf5
```

But the following succeeds:

```
$ spack spec netcdf ^zlib
Input spec
------------------------------
  netcdf
      ^zlib

Normalized
------------------------------
  netcdf
      ^curl
          ^openssl
              ^zlib
      ^m4

Concretized
------------------------------
  netcdf@4.4.1%gcc@6.1.0~hdf4+mpi arch=linux-centos6-x86_64
      ^curl@7.50.1%gcc@6.1.0 arch=linux-centos6-x86_64
          ^openssl@1.0.2h%gcc@6.1.0 arch=linux-centos6-x86_64
              ^zlib@1.2.8%gcc@6.1.0 arch=linux-centos6-x86_64
      ^hdf5@1.10.0-patch1%gcc@6.1.0+cxx~debug+fortran+mpi+shared~szip~threadsafe arch=linux-centos6-x86_64
          ^openmpi@2.0.0%gcc@6.1.0~mxm~pmi~psm~psm2~slurm~sqlite3~thread_multiple~tm+verbs+vt arch=linux-centos6-x86_64
              ^hwloc@1.11.3%gcc@6.1.0 arch=linux-centos6-x86_64
                  ^libpciaccess@0.13.4%gcc@6.1.0 arch=linux-centos6-x86_64
                      ^libtool@2.4.6%gcc@6.1.0 arch=linux-centos6-x86_64
                          ^m4@1.4.17%gcc@6.1.0+sigsegv arch=linux-centos6-x86_64
                              ^libsigsegv@2.10%gcc@6.1.0 arch=linux-centos6-x86_64
```

The difference is that hdf5 does not appear in the normalized spec. This is because Spack doesn't think NetCDF always depends on HDF5:

```
    depends_on("zlib")                                                          
    depends_on('hdf5@:1.8+mpi', when='@:4.4.0+mpi')                             
    depends_on('hdf5+mpi', when='@4.4.1:+mpi')                                  
    depends_on('hdf5@:1.8~mpi', when='@:4.4.0~mpi')                             
    depends_on('hdf5~mpi', when='@4.4.1:~mpi')
```

This means that it's no longer possible to specify the MPI provider at all:

```
$ spack spec netcdf+mpi ^mvapich2
Input spec
------------------------------
  netcdf+mpi
      ^mvapich2

Normalized
------------------------------
==> Error: netcdf does not depend on mvapich2
```

I'm starting to wonder if this problem only applies to NetCDF's complicated HDF5 dependency. I'll try to find an example that doesn't involve either of these packages.

I think this may just be a problem with NetCDF's HDF5 dependency. I'll put out a PR shortly.


Resolving patch:
diff --git a/var/spack/repos/builtin/packages/hdf5/package.py b/var/spack/repos/builtin/packages/hdf5/package.py
--- a/var/spack/repos/builtin/packages/hdf5/package.py
+++ b/var/spack/repos/builtin/packages/hdf5/package.py
@@ -52,7 +52,7 @@ class Hdf5(Package):
     variant('cxx', default=True, description='Enable C++ support')
     variant('fortran', default=True, description='Enable Fortran support')
 
-    variant('mpi', default=False, description='Enable MPI support')
+    variant('mpi', default=True, description='Enable MPI support')
     variant('szip', default=False, description='Enable szip support')
     variant('threadsafe', default=False,
             description='Enable thread-safe capabilities')
diff --git a/var/spack/repos/builtin/packages/netcdf/package.py b/var/spack/repos/builtin/packages/netcdf/package.py
--- a/var/spack/repos/builtin/packages/netcdf/package.py
+++ b/var/spack/repos/builtin/packages/netcdf/package.py
@@ -50,12 +50,18 @@ class Netcdf(Package):
 
     # Required for NetCDF-4 support
     depends_on("zlib")
-    depends_on('hdf5@:1.8+mpi', when='@:4.4.0+mpi')
-    depends_on('hdf5+mpi', when='@4.4.1:+mpi')
-    depends_on('hdf5@:1.8~mpi', when='@:4.4.0~mpi')
-    depends_on('hdf5~mpi', when='@4.4.1:~mpi')
+    depends_on('hdf5')
+
+    # NetCDF 4.4.0 and prior have compatibility issues with HDF5 1.10 and later
+    # https://github.com/Unidata/netcdf-c/issues/250
+    depends_on('hdf5@:1.8', when='@:4.4.0')
 
     def install(self, spec, prefix):
+        # Workaround until variant forwarding works properly
+        if '+mpi' in spec and spec.satisfies('^hdf5~mpi'):
+            raise RuntimeError('Invalid spec. Package netcdf requires '
+                               'hdf5+mpi, but spec asked for hdf5~mpi.')
+
         # Environment variables
         CPPFLAGS = []
         LDFLAGS  = []


Test patch: