Repository: spack/spack

Issue:
trilinos recipe: spec['blas'].prefix.lib is wrong for mkl
I can no longer build `trilinos ^mkl` because `spec['blas'].prefix.lib` does not point to the correct directory for MKL.  At least for my installation, the MKL libraries are located at `$MKLROOT/lib/intel64` and not at  `$MKLROOT/lib`.  I'm not sure how to fix this because I don't know if there is significant variation in directory names for MKL installations. 

It is simple to modify the trilinos recipe to do something special for MKL.  Should I submit a PR with these changes?  Does anyone else have experience building trilinos against MKL? Are there other configurations that I should be aware of?



Discussion:
@alalazo @davydden This is an external package not built by Spack, so the directory layout doesn't match. We're going to have to figure out some way to handle this...

Specs are marked external so we have the information.  May need to record it better.

> I can no longer build trilinos ^mkl because spec['blas'].prefix.lib does not point to the correct directory for MKL.

you can't use external MKL, you should build it with Spack, then symlinks will be created. That way Trilinos is buildable with MKL.

p.s. tested on on Ubuntu@16.04+gcc@5.4.0 in this PR https://github.com/LLNL/spack/pull/1852

I think we need to make external MKL work. A lot of users will already have it installed. Can't we search for libraries recursively instead of asserting that they have to be in `prefix.lib`?

> I think we need to make external MKL work.

essentially that would amount to changing everywhere `spec['blas'].prefix.lib` to `spec['blas']. directories` (the latter is a list, AFAIK). Same for `lapack` and `scalapack`.
This would rely on `LibraryList` class, introduced in https://github.com/LLNL/spack/pull/1682  .

@KineticTheory 

> It is simple to modify the trilinos recipe to do something special for MKL. Should I submit a PR with these changes?

do you want to try this for `Trilinos` and submit a PR? If that works for your external MKL, the other packages can be gradually switched to this scheme.

@KineticTheory For the record, this is a modification that should be made in the `mkl` package, not in the `trilinos` package.

> @KineticTheory For the record, this is a modification that should be made in the mkl package, not in the trilinos package.

Not exactly. It should be done in both places. 
In `trilinos`:

```
spec['blas'].prefix.lib -> ";".join(blas.directories)
```

and in `MKL` actually you don't need to do anything as 

```
mkl_libs = find_libraries(
            mkl_integer + ['libmkl_core'] + mkl_threading,
            root=join_path(self.prefix.lib, 'intel64'),
            shared=shared
        )
```

already searches in native directory structure. One can then remove

```
# Unfortunately MKL libs are natively distrubted in prefix/lib/intel64.
        # To make MKL play nice with Spack, symlink all files to prefix/lib:
        mkl_lib_dir = os.path.join(prefix, "lib", "intel64")
        for f in os.listdir(mkl_lib_dir):
            os.symlink(os.path.join(mkl_lib_dir, f),
                       os.path.join(self.prefix, "lib", f))
```

from `MKL`.

p.s. symlinking was introduced prior to https://github.com/LLNL/spack/pull/1682 .

With the new LAPACK/BLAS functionality do we need symlinking?  I think depending on the native directory structure for externals is better.

>  With the new LAPACK/BLAS functionality do we need symlinking? I think depending on the native directory structure for externals is better.

we don't need to symlink. It is indeed good to avoid it everywhere.


Resolving patch:
diff --git a/var/spack/repos/builtin/packages/mkl/package.py b/var/spack/repos/builtin/packages/mkl/package.py
--- a/var/spack/repos/builtin/packages/mkl/package.py
+++ b/var/spack/repos/builtin/packages/mkl/package.py
@@ -66,13 +66,6 @@ def install(self, spec, prefix):
         for f in os.listdir(mkl_dir):
             os.symlink(os.path.join(mkl_dir, f), os.path.join(self.prefix, f))
 
-        # Unfortunately MKL libs are natively distrubted in prefix/lib/intel64.
-        # To make MKL play nice with Spack, symlink all files to prefix/lib:
-        mkl_lib_dir = os.path.join(prefix, "lib", "intel64")
-        for f in os.listdir(mkl_lib_dir):
-            os.symlink(os.path.join(mkl_lib_dir, f),
-                       os.path.join(self.prefix, "lib", f))
-
     def setup_dependent_environment(self, spack_env, run_env, dependent_spec):
         # set up MKLROOT for everyone using MKL package
         spack_env.set('MKLROOT', self.prefix)
diff --git a/var/spack/repos/builtin/packages/trilinos/package.py b/var/spack/repos/builtin/packages/trilinos/package.py
--- a/var/spack/repos/builtin/packages/trilinos/package.py
+++ b/var/spack/repos/builtin/packages/trilinos/package.py
@@ -149,10 +149,10 @@ def install(self, spec, prefix):
             '-DMPI_BASE_DIR:PATH=%s' % spec['mpi'].prefix,
             '-DTPL_ENABLE_BLAS=ON',
             '-DBLAS_LIBRARY_NAMES=%s' % ';'.join(blas.names),
-            '-DBLAS_LIBRARY_DIRS=%s' % spec['blas'].prefix.lib,
+            '-DBLAS_LIBRARY_DIRS=%s' % ';'.join(blas.directories),
             '-DTPL_ENABLE_LAPACK=ON',
             '-DLAPACK_LIBRARY_NAMES=%s' % ';'.join(lapack.names),
-            '-DLAPACK_LIBRARY_DIRS=%s' % spec['lapack'].prefix.lib,
+            '-DLAPACK_LIBRARY_DIRS=%s' % ';'.join(lapack.directories),
             '-DTrilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON',
             '-DTrilinos_ENABLE_CXX11:BOOL=ON',
             '-DTPL_ENABLE_Netcdf:BOOL=ON',


Test patch: