Repository: spack/spack

Issue:
Mumps is broken by PR 1023
Mumps is broken on OSX with clang+gfortran by https://github.com/LLNL/spack/pull/1023. It manifests in a segmentation fault when trying to use Trilinos + MUMPS direct solver as a [unit test during installation of  deal.ii@8.4.1](https://github.com/LLNL/spack/blob/develop/var/spack/repos/builtin/packages/dealii/package.py#L279) which worked before. 

To confirm it I took a [pre-PR version of MUMPS](https://github.com/LLNL/spack/blob/ddaba07d8297544f2587f0c31fcbd2dafa824eb3/var/spack/repos/builtin/packages/mumps/package.py), build it with current develop and my [usual patch](https://github.com/dealii/dealii/wiki/deal.II-in-Spack#mixing-gcc-and-clang-on-osx) to mix clang and fortran and it works good and does not produce a segmentation fault!

Most likely it is [related to this](https://github.com/LLNL/spack/commit/9b299cb2bbe6e1e20df38c1900d6a05483c81a3f#diff-2b1c47a5b3acd753325522d1c1c65956L137), but could also be related to the big shared patch introduced in https://github.com/LLNL/spack/pull/1023.
Alternatively, it could be related to the way [MUMPS is used in Trilinos](https://github.com/LLNL/spack/blob/develop/var/spack/repos/builtin/packages/trilinos/package.py#L221).

@nrichart would you mind having a second look at your PR and see what could cause the issue?



Discussion:
I do not have access to an OSX machine yet but I will try to see if I can already reproduce this on linux or if it might be linked on the specific lines for darwin.
Is it possible to reproduce the bug with a simple case that does not depend on Trilinos and deal.ii ?

i will test tomorrow with a clean install on linux, but i am afraid it is related to either (i) darwin or (ii) clang. As for a stand alone test, perhaps some small example program which use only Mumps would do. Unfortunately the current small test programs run ok, it seems.

Looking at [mumps in homebrew](https://github.com/Homebrew/homebrew-science/blob/master/mumps.rb), there must be `c_example.c` file, which could show the issue. Will try tomorrow...

If it is clang I might be able to reproduce it, I'm spack installing a clang right know.
You installed deal.ii@8.4.1 with everything default or with some specific variants ? so I install the same 

`dealii@8.4.1` (default) should show the issue.

@nrichart i checked my installation logs on Linux, it builds fine with gcc. The bug must be related to this line you changed https://github.com/LLNL/spack/commit/9b299cb2bbe6e1e20df38c1900d6a05483c81a3f#diff-2b1c47a5b3acd753325522d1c1c65956L140 which is related to reworking the makefile.

Today I couldn't work on it I will try to find a mac osx tomorrow to try to compile.

@nrichart this PR https://github.com/LLNL/spack/pull/1145 exposes the segmentation fault in MUMPS when running PETSc, which is way faster to compile compared to Trilinos.


Resolving patch:
diff --git a/var/spack/repos/builtin/packages/mumps/package.py b/var/spack/repos/builtin/packages/mumps/package.py
--- a/var/spack/repos/builtin/packages/mumps/package.py
+++ b/var/spack/repos/builtin/packages/mumps/package.py
@@ -23,10 +23,7 @@
 # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
 ##############################################################################
 from spack import *
-import os
-import sys
-import subprocess
-
+import os, sys, glob
 
 class Mumps(Package):
     """MUMPS: a MUltifrontal Massively Parallel sparse direct Solver"""
@@ -47,6 +44,7 @@ class Mumps(Package):
     variant('idx64', default=False, description='Use int64_t/integer*8 as default index type')
     variant('shared', default=True, description='Build shared libraries')
 
+
     depends_on('scotch + esmumps', when='~ptscotch+scotch')
     depends_on('scotch + esmumps + mpi', when='+ptscotch')
     depends_on('metis@5:', when='+metis')
@@ -56,64 +54,49 @@ class Mumps(Package):
     depends_on('scalapack', when='+mpi')
     depends_on('mpi', when='+mpi')
 
-    patch('mumps-shared.patch', when='+shared')
-
     # this function is not a patch function because in case scalapack
     # is needed it uses self.spec['scalapack'].fc_link set by the
     # setup_dependent_environment in scalapck. This happen after patch
     # end before install
     # def patch(self):
     def write_makefile_inc(self):
-        if (('+parmetis' in self.spec or
-             '+ptscotch' in self.spec)) and '+mpi' not in self.spec:
-            raise RuntimeError('You cannot use the variants parmetis or ptscotch without mpi')  # NOQA: E501
+        if ('+parmetis' in self.spec or '+ptscotch' in self.spec) and '+mpi' not in self.spec:
+            raise RuntimeError('You cannot use the variants parmetis or ptscotch without mpi')
 
-        makefile_conf = [
-            "LIBBLAS = -L%s -lblas" % self.spec['blas'].prefix.lib
-        ]
+        makefile_conf = ["LIBBLAS = -L%s -lblas" % self.spec['blas'].prefix.lib]
 
         orderings = ['-Dpord']
 
         if '+ptscotch' in self.spec or '+scotch' in self.spec:
             join_lib = ' -l%s' % ('pt' if '+ptscotch' in self.spec else '')
-            makefile_conf.extend([
-                "ISCOTCH = -I%s" % self.spec['scotch'].prefix.include,
-                "LSCOTCH = -L%s %s%s" % (self.spec['scotch'].prefix.lib,
-                                         join_lib,
-                                         join_lib.join(['esmumps',
-                                                        'scotch',
-                                                        'scotcherr']))
-            ])
+            makefile_conf.extend(
+                ["ISCOTCH = -I%s" % self.spec['scotch'].prefix.include,
+                 "LSCOTCH = -L%s %s%s" % (self.spec['scotch'].prefix.lib,
+                                          join_lib,
+                                          join_lib.join(['esmumps', 'scotch', 'scotcherr']))])
             orderings.append('-Dscotch')
             if '+ptscotch' in self.spec:
                 orderings.append('-Dptscotch')
 
         if '+parmetis' in self.spec and '+metis' in self.spec:
-            makefile_conf.extend([
-                "IMETIS = -I%s" % self.spec['parmetis'].prefix.include,
-                "LMETIS = -L%s -l%s -L%s -l%s" % (
-                    self.spec['parmetis'].prefix.lib, 'parmetis',
-                    self.spec['metis'].prefix.lib, 'metis')
-            ])
+            libname = 'parmetis' if '+parmetis' in self.spec else 'metis'
+            makefile_conf.extend(
+                ["IMETIS = -I%s" % self.spec['parmetis'].prefix.include,
+                 "LMETIS = -L%s -l%s -L%s -l%s" % (self.spec['parmetis'].prefix.lib, 'parmetis',self.spec['metis'].prefix.lib, 'metis')])
 
             orderings.append('-Dparmetis')
         elif '+metis' in self.spec:
-            makefile_conf.extend([
-                "IMETIS = -I%s" % self.spec['metis'].prefix.include,
-                "LMETIS = -L%s -l%s" % (self.spec['metis'].prefix.lib,
-                                        'metis')
-            ])
+            makefile_conf.extend(
+                ["IMETIS = -I%s" % self.spec['metis'].prefix.include,
+                 "LMETIS = -L%s -l%s" % (self.spec['metis'].prefix.lib, 'metis')])
 
             orderings.append('-Dmetis')
 
         makefile_conf.append("ORDERINGSF = %s" % (' '.join(orderings)))
 
-        # when building shared libs need -fPIC, otherwise /usr/bin/ld:
-        # graph.o: relocation R_X86_64_32 against `.rodata.str1.1' can
-        # not be used when making a shared object; recompile with
-        # -fPIC
+        # when building shared libs need -fPIC, otherwise
+        # /usr/bin/ld: graph.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
         fpic = '-fPIC' if '+shared' in self.spec else ''
-
         # TODO: test this part, it needs a full blas, scalapack and
         # partitionning environment with 64bit integers
         if '+idx64' in self.spec:
@@ -121,7 +104,7 @@ def write_makefile_inc(self):
                 # the fortran compilation flags most probably are
                 # working only for intel and gnu compilers this is
                 # perhaps something the compiler should provide
-                ['OPTF    = %s -O  -DALLOW_NON_INIT %s' % (fpic, '-fdefault-integer-8' if self.compiler.name == "gcc" else '-i8'),  # NOQA: E501
+                ['OPTF    = %s -O  -DALLOW_NON_INIT %s' % (fpic,'-fdefault-integer-8' if self.compiler.name == "gcc" else '-i8'),
                  'OPTL    = %s -O ' % fpic,
                  'OPTC    = %s -O -DINTSIZE64' % fpic])
         else:
@@ -130,47 +113,49 @@ def write_makefile_inc(self):
                  'OPTL    = %s -O ' % fpic,
                  'OPTC    = %s -O ' % fpic])
 
+
         if '+mpi' in self.spec:
             makefile_conf.extend(
-                ["CC = %s" % self.spec['mpi'].mpicc,
-                 "FC = %s" % self.spec['mpi'].mpifc,
+                ["CC = %s" % join_path(self.spec['mpi'].prefix.bin, 'mpicc'),
+                 "FC = %s" % join_path(self.spec['mpi'].prefix.bin, 'mpif90'),
+                 "FL = %s" % join_path(self.spec['mpi'].prefix.bin, 'mpif90'),
                  "SCALAP = %s" % self.spec['scalapack'].fc_link,
                  "MUMPS_TYPE = par"])
         else:
             makefile_conf.extend(
                 ["CC = cc",
                  "FC = fc",
+                 "FL = fc",
                  "MUMPS_TYPE = seq"])
 
         # TODO: change the value to the correct one according to the
         # compiler possible values are -DAdd_, -DAdd__ and/or -DUPPER
-        makefile_conf.extend([
-            'CDEFS   = -DAdd_',
-            'FL = $(FC)',
-        ])
+        makefile_conf.append("CDEFS   = -DAdd_")
 
         if '+shared' in self.spec:
-            makefile_conf.append('SHLIBEXT = .%s' % dso_suffix)
             if sys.platform == 'darwin':
-                makefile_conf.append(
-                    'LDFLAGS = -dynamiclib -Wl,-install_name -Wl,{0}/$(notdir $@) {1}{0} -undefined dynamic_lookup'.format(prefix.lib, self.compiler.fc_rpath_arg)  # NOQA: E501
-                )
+                # Building dylibs with mpif90 causes segfaults on 10.8 and 10.10. Use gfortran. (Homebrew)
+                makefile_conf.extend([
+                    'LIBEXT=.dylib',
+                    'AR=%s -dynamiclib -Wl,-install_name -Wl,%s/$(notdir $@) -undefined dynamic_lookup -o ' % (os.environ['FC'],prefix.lib),
+                    'RANLIB=echo'
+                ])
             else:
-                makefile_conf.append(
-                    'LDFLAGS = -shared {0}{1}'.format(
-                        self.compiler.fc_rpath_arg,
-                        prefix.lib)
-                )
-
-        makefile_conf.extend([
-            'LIBEXT  = .a',
-            'AR = ar vr ',
-            'RANLIB = ranlib'
-        ])
-
-        makefile_inc_template = \
-            join_path(os.path.dirname(self.module.__file__),
-                      'Makefile.inc')
+                makefile_conf.extend([
+                    'LIBEXT=.so',
+                    'AR=$(FL) -shared -Wl,-soname -Wl,%s/$(notdir $@) -o' % prefix.lib,
+                    'RANLIB=echo'
+                ])
+        else:
+            makefile_conf.extend([
+                'LIBEXT  = .a',
+                'AR = ar vr',
+                'RANLIB = ranlib'
+            ])
+
+
+        makefile_inc_template = join_path(os.path.dirname(self.module.__file__),
+                                          'Makefile.inc')
         with open(makefile_inc_template, "r") as fh:
             makefile_conf.extend(fh.read().split('\n'))
 
@@ -179,53 +164,46 @@ def write_makefile_inc(self):
                 makefile_inc = '\n'.join(makefile_conf)
                 fh.write(makefile_inc)
 
+
+
     def install(self, spec, prefix):
         make_libs = []
 
         # the choice to compile ?examples is to have kind of a sanity
         # check on the libraries generated.
         if '+float' in spec:
-            make_libs.append('s')
+            make_libs.append('sexamples')
             if '+complex' in spec:
-                make_libs.append('c')
+                make_libs.append('cexamples')
 
         if '+double' in spec:
-            make_libs.append('d')
+            make_libs.append('dexamples')
             if '+complex' in spec:
-                make_libs.append('z')
+                make_libs.append('zexamples')
 
         self.write_makefile_inc()
 
-        make('mumps_lib', parallel=False)
-        make(*make_libs)
+        # Build fails in parallel
+        make(*make_libs, parallel=False)
 
         install_tree('lib', prefix.lib)
         install_tree('include', prefix.include)
 
-        if '~mpi' in spec:
-            install('libseq/libmpiseq.a', prefix.lib)
-            if '+shared' in spec:
-                install('libseq/libmpiseq.{0}'.format(dso_suffix), prefix.lib)
-            install('libseq/mpi.h', prefix.include)
-            install('libseq/mpif.h', prefix.include)
-
-        # FIXME: extend the tests to mpirun -np 2 (or alike) when
-        # build with MPI
-        # FIXME: use something like numdiff to compare blessed output
-        # with the current
-        # TODO: test the installed mumps and not the one in stage
-        if '~mpi' in spec:
-            for t in make_libs:
-                make('{0}examples'.format(t))
-
-            with working_dir('examples'):
-                for t in make_libs:
-                    input_file = 'input_simpletest_{0}'.format(
-                        'real' if t in ['s', 'd'] else 'cmplx')
-                    with open(input_file) as input:
-                        test = './{0}simpletest'.format(t)
-                        ret = subprocess.call(test,
-                                              stdin=input)
-                        if ret is not 0:
-                            raise RuntimeError(
-                                'The test {0} did not pass'.format(test))
+        if '~mpi' in spec:            
+            lib_dsuffix = '.dylib' if sys.platform == 'darwin' else '.so'
+            lib_suffix = lib_dsuffix if '+shared' in spec else '.a'
+            install('libseq/libmpiseq%s' % lib_suffix, prefix.lib)
+            for f in glob.glob(join_path('libseq','*.h')):
+                install(f, prefix.include)
+
+        # FIXME: extend the tests to mpirun -np 2 (or alike) when build with MPI
+        # FIXME: use something like numdiff to compare blessed output with the current
+        with working_dir('examples'):
+            if '+float' in spec:
+                os.system('./ssimpletest < input_simpletest_real')
+                if '+complex' in spec:
+                    os.system('./csimpletest < input_simpletest_real')
+            if '+double' in spec:
+                os.system('./dsimpletest < input_simpletest_real')
+                if '+complex' in spec:
+                    os.system('./zsimpletest < input_simpletest_cmplx')


Test patch: