Repository: easybuilders/easybuild-framework

Issue:
Modules: Set CMAKE_PREFIX_PATH
Hi,

I am a user of several HPC systems that use Easybuild as a package manager (CSCS, TU Dresden). As a power-user and heavy developer, we use package software from modules on HPC systems to develop and build further software - outside of those package managers.

In the case of our software stack, we heavily rely on CMake and for such we would like to set the `CMAKE_PREFIX_PATH` when loading modules automatically to the prefix of installed software. The CMake build system heavily relies on those to find software of any kind (and prefer it over e.g. system paths).

How can one generally *configure* Easybuild to *always* prepend `CMAKE_PREFIX_PATH` in modules?
Should one introduce a more general templating mechanism for modules outside of [`--modules-footer`](https://easybuild.readthedocs.io/en/latest/version-specific/help.html)?

As a heavy [Spack](http://spack.io/) user myself, I am quite in love with the module configuration I can use there and therefore I am quite biased :) Maybe that's useful for Easybuild as well?
https://spack.readthedocs.io/en/latest/module_file_support.html#write-a-configuration-file


Discussion:
Hmm, I'd say people probably have some opinions on this. If you want a quick fix for now, you can just leverage the `$EBROOT*` variables that are set for every EB module:
```
export CMAKE_PREFIX_PATH=$(env | grep EBROOT | sed  s/=/\ /|awk '{print $2}'|tr '\n' ':')$CMAKE_PREFIX_PATH
```
Having configurable modules has been discussed before but noone has gotten around to an implementation.
The idea of having a default module file template that you can customize has indeed been discussed before.

It shouldn't be terribly hard to implement either, I think it's mostly because of a lack of time and the fact that nobody has had a strong use case for it yet...

To make it concrete: someone implementing this would have to dive into `easybuild/framework/easyblock.py` and make `make_module_step` use a template of a module file rather than concatenating a set of strings obtained by calling the different `make_module*` methods.
First stab at adding support for using a custom module file template in #2639 .

For @ax3l's use case, `--modules-footer` is still a good option though:

```
$ cat $HOME/my-modules-footer.lua
prepend_path("CMAKE_PREFIX_PATH", "%(installdir)s")
```

```
$ eb --modules-footer $HOME/my-modules-footer.lua ...

# or

$ export EASYBUILD_MODULES_FOOTER=$HOME/my-modules-footer.lua
$ eb ...
```
This is getting old now. How about adding `prepend_path("CMAKE_PREFIX_PATH", "%(installdir)s")` to the default EB?
There are actually now two solutions to this issue, the one from @boegel above and a more general one using [hooks](https://easybuild.readthedocs.io/en/latest/Hooks.html).
I probably agree that this is a good idea (@akesandgren is, I think, more familiar with CMake than me) but it potentially has a big impact on users. Even if we thought it was a good idea to force this, it could have enough of an impact that we would have to default it off until the next major release.
This actually came up (more or less) in the [confcall on Wednesday](https://github.com/easybuilders/easybuild/wiki/Conference-call-notes-20191211).
For EB itself this shouldn't be a problem. We already set CMAKE_INCLUDE_PATH and CMAKE_LIBRARY_PATH in the cmakemake.py easyblock to the combined list of the toolchain includes and libdirs.
CMAKE_PREFIX_PATH is just another variable it will use to do the same things as the other two.

The caveat here is that CMAKE_PREFIX_PATH looks in <prefix>/lib/<arch> if CMAKE_LIBRARY_ARCHITECTURE is set and in <prefix>/lib otherwise.
Not all libs are installed in <prefix>/lib, on some systems they end up in <prefix>/lib64, so we would still have to set CMAKE_LIBRARY_PATH.

We could also make sure to set CMAKE_PROGRAM_PATH to point to the [s]bin dir of installed software. but CMAKE_PREFIX_PATH is already used to look in <prefix>/[s]bin.

So, for most, but not all, packages setting CMAKE_PREFIX_PATH to $EBROOTXXX is usually enough. Also setting CMAKE_LIBRARY_PATH to the actual libdir would be required.

So, from a user perspective (not using EB itself that is) having them set is likely a good idea.

(Unfortunately, too many cmake based codes are using NO_DEFAULT_PATH when doing find_xxx which makes all this useless)
> The caveat here is that CMAKE_PREFIX_PATH looks in `/lib/<arch>` if CMAKE_LIBRARY_ARCHITECTURE is set and in /lib otherwise.

No, you misunderstood the docs. There is no **otherwise**, it looks in `/lib/<arch>` before `/lib` if the arch is set. (Otherwise it just looks in the second) See https://cmake.org/cmake/help/latest/command/find_library.html

> So, for most, but not all, packages setting CMAKE_PREFIX_PATH to $EBROOTXXX is usually enough. Also setting CMAKE_LIBRARY_PATH to the actual libdir would be required.

IMO `CMAKE_PREFIX_PATH` is enough, `CMAKE_LIBRARY_PATH` is not required.

> So, from a user perspective (not using EB itself that is) having them set is likely a good idea.

Very much so!

> (Unfortunately, too many cmake based codes are using NO_DEFAULT_PATH when doing find_xxx which makes all this useless)

Well, make it painful to use stuff "wrong". This may lead to better codes in the future. Nothing wrong with `NO_DEFAULT_PATH` if that find call is followed by another without though.
Yeah, sorry didn't mean to write otherwise. But it does not look in lib64 ever, which is where, on some distros, the libraries will be installed. Thus we need CMAKE_LIBRARY_PATH too.
Unless FIND_LIBRARY_USE_LIB64_PATHS is set... missed that one (again)

However I don't know if it looks in both lib and lib64 if that is set or only in lib64.

It's still safer to set CMAKE_LIBRARY_PATH too.
> But it does not look in lib64 ever, which is where, on some distros, the libraries will be installed. Thus we need CMAKE_LIBRARY_PATH too.

Can you provide an example? Because I'd expect `CMAKE_LIBRARY_ARCHITECTURE` to be set on such distros: https://cmake.org/cmake/help/v3.4/variable/CMAKE_LIBRARY_ARCHITECTURE.html -> https://cmake.org/cmake/help/v3.4/variable/CMAKE_LANG_LIBRARY_ARCHITECTURE.html

```
Target architecture library directory name detected for <LANG>.

 If the <LANG> compiler passes to the linker an architecture-specific system library search directory such as <prefix>/lib/<arch> this variable contains the <arch> name if/as detected by CMake.
```

> However I don't know if it looks in both lib and lib64 if that is set or only in lib64.

Both:

`If the FIND_LIBRARY_USE_LIB64_PATHS global property is set all search paths will be tested as normal, with 64/ appended, and with all matches of lib/ replaced with lib64/. This property is automatically set for the platforms that are known to need it if at least one of the languages supported by the project() command is enabled.`
On our Ubuntu Xenial, some packages put stuff in lib64 and some in lib. This has nothing to do with CMake or the OS distro as such. 

For instance binary packages like cuDNN only have lib64 while most stuff is in lib.
Just seen that `CMAKE_LIBRARY_ARCHITECTURE` is unrelated. It appends a path, not change lib->lib64

Just checked CMake sources. It does set FIND_LIBRARY_USE_LIB64_PATHS by default: https://github.com/Kitware/CMake/blob/7574e16096abe57d44cde4e6380a22d940cde98c/Modules/Platform/UnixPaths.cmake#L91 but disabled for debians: https://github.com/Kitware/CMake/blob/7574e16096abe57d44cde4e6380a22d940cde98c/Modules/Platform/Linux.cmake#L58 with a comment `Debian has lib32 and lib64 paths only for compatibility so they should not be searched.`. So those package structures could probably be considered wrong. But well, you are right.

Then I'd say: prepend the lib64 path for a module if it exists, else dont touch `CMAKE_LIBRARY_PATH` to keep the impact minimal.
Yeah, that's what I had in mind above.
So who's up for translating this in a PR that also updates `$CMAKE_PREFIX_PATH` (+ `$CMAKE_LIBRARY_PATH` if needed)?

Resolving patch:
diff --git a/easybuild/framework/easyblock.py b/easybuild/framework/easyblock.py
--- a/easybuild/framework/easyblock.py
+++ b/easybuild/framework/easyblock.py
@@ -1282,47 +1282,69 @@ def make_module_req(self):
 
         lines = ['\n']
         if os.path.isdir(self.installdir):
-            change_dir(self.installdir)
+            old_dir = change_dir(self.installdir)
+        else:
+            old_dir = None
 
+        if self.dry_run:
+            self.dry_run_msg("List of paths that would be searched and added to module file:\n")
+            note = "note: glob patterns are not expanded and existence checks "
+            note += "for paths are skipped for the statements below due to dry run"
+            lines.append(self.module_generator.comment(note))
+
+        # for these environment variables, the corresponding subdirectory must include at least one file
+        keys_requiring_files = set(('PATH', 'LD_LIBRARY_PATH', 'LIBRARY_PATH', 'CPATH',
+                                    'CMAKE_PREFIX_PATH', 'CMAKE_LIBRARY_PATH'))
+
+        for key, reqs in sorted(requirements.items()):
+            if isinstance(reqs, string_type):
+                self.log.warning("Hoisting string value %s into a list before iterating over it", reqs)
+                reqs = [reqs]
             if self.dry_run:
-                self.dry_run_msg("List of paths that would be searched and added to module file:\n")
-                note = "note: glob patterns are not expanded and existence checks "
-                note += "for paths are skipped for the statements below due to dry run"
-                lines.append(self.module_generator.comment(note))
-
-            # for these environment variables, the corresponding subdirectory must include at least one file
-            keys_requiring_files = ('CPATH', 'LD_LIBRARY_PATH', 'LIBRARY_PATH', 'PATH')
-
-            for key in sorted(requirements):
-                if self.dry_run:
-                    self.dry_run_msg(" $%s: %s" % (key, ', '.join(requirements[key])))
-                reqs = requirements[key]
-                if isinstance(reqs, string_type):
-                    self.log.warning("Hoisting string value %s into a list before iterating over it", reqs)
-                    reqs = [reqs]
-
-                for path in reqs:
-                    # only use glob if the string is non-empty
-                    if path and not self.dry_run:
-                        paths = sorted(glob.glob(path))
-                        if paths and key in keys_requiring_files:
-                            # only retain paths that contain at least one file
-                            retained_paths = [
-                                path for path in paths
-                                if os.path.isdir(os.path.join(self.installdir, path))
-                                and dir_contains_files(os.path.join(self.installdir, path))
-                            ]
-                            self.log.info("Only retaining paths for %s that contain at least one file: %s -> %s",
-                                          key, paths, retained_paths)
-                            paths = retained_paths
-                    else:
-                        # empty string is a valid value here (i.e. to prepend the installation prefix, cfr $CUDA_HOME)
-                        paths = [path]
+                self.dry_run_msg(" $%s: %s" % (key, ', '.join(reqs)))
+                # Don't expand globs or do any filtering below for dry run
+                paths = sorted(reqs)
+            else:
+                # Expand globs but only if the string is non-empty
+                # empty string is a valid value here (i.e. to prepend the installation prefix, cfr $CUDA_HOME)
+                paths = sorted(sum((glob.glob(path) if path else [path] for path in reqs), []))  # sum flattens to list
+
+                # If lib64 is just a symlink to lib we fixup the paths to avoid duplicates
+                lib64_is_symlink = (all(os.path.isdir(path) for path in ['lib', 'lib64'])
+                                    and os.path.samefile('lib', 'lib64'))
+                if lib64_is_symlink:
+                    fixed_paths = []
+                    for path in paths:
+                        if (path + os.path.sep).startswith('lib64' + os.path.sep):
+                            # We only need CMAKE_LIBRARY_PATH if there is a separate lib64 path, so skip symlink
+                            if key == 'CMAKE_LIBRARY_PATH':
+                                continue
+                            path = path.replace('lib64', 'lib', 1)
+                        fixed_paths.append(path)
+                    if fixed_paths != paths:
+                        self.log.info("Fixed symlink lib64 in paths for %s: %s -> %s", key, paths, fixed_paths)
+                        paths = fixed_paths
+                # Use a set to remove duplicates, e.g. by having lib64 and lib which get fixed to lib and lib above
+                paths = sorted(set(paths))
+                if key in keys_requiring_files:
+                    # only retain paths that contain at least one file
+                    retained_paths = [
+                        path for path in paths
+                        if os.path.isdir(os.path.join(self.installdir, path))
+                        and dir_contains_files(os.path.join(self.installdir, path))
+                    ]
+                    if retained_paths != paths:
+                        self.log.info("Only retaining paths for %s that contain at least one file: %s -> %s",
+                                      key, paths, retained_paths)
+                        paths = retained_paths
+
+            if paths:
+                lines.append(self.module_generator.prepend_paths(key, paths))
+        if self.dry_run:
+            self.dry_run_msg('')
 
-                    lines.append(self.module_generator.prepend_paths(key, paths))
-            if self.dry_run:
-                self.dry_run_msg('')
-            change_dir(self.orig_workdir)
+        if old_dir is not None:
+            change_dir(old_dir)
 
         return ''.join(lines)
 
@@ -1342,6 +1364,8 @@ def make_module_req_guess(self):
             'CLASSPATH': ['*.jar'],
             'XDG_DATA_DIRS': ['share'],
             'GI_TYPELIB_PATH': [os.path.join(x, 'girepository-*') for x in lib_paths],
+            'CMAKE_PREFIX_PATH': [''],
+            'CMAKE_LIBRARY_PATH': ['lib64'],  # lib and lib32 are searched through the above
         }
 
     def load_module(self, mod_paths=None, purge=True, extra_modules=None):


Test patch:
diff --git a/test/framework/easyblock.py b/test/framework/easyblock.py
--- a/test/framework/easyblock.py
+++ b/test/framework/easyblock.py
@@ -51,7 +51,7 @@
 from easybuild.tools.modules import reset_module_caches
 from easybuild.tools.utilities import time2str
 from easybuild.tools.version import get_git_revision, this_is_easybuild
-
+from easybuild.tools.py2vs3 import string_type
 
 class EasyBlockTest(EnhancedTestCase):
     """ Baseclass for easyblock testcases """
@@ -318,11 +318,10 @@ def test_make_module_req(self):
         os.makedirs(eb.installdir)
         open(os.path.join(eb.installdir, 'foo.jar'), 'w').write('foo.jar')
         open(os.path.join(eb.installdir, 'bla.jar'), 'w').write('bla.jar')
-        os.mkdir(os.path.join(eb.installdir, 'bin'))
-        os.mkdir(os.path.join(eb.installdir, 'bin', 'testdir'))
-        os.mkdir(os.path.join(eb.installdir, 'sbin'))
-        os.mkdir(os.path.join(eb.installdir, 'share'))
-        os.mkdir(os.path.join(eb.installdir, 'share', 'man'))
+        for path in ('bin', ('bin', 'testdir'), 'sbin', 'share', ('share', 'man'), 'lib', 'lib64'):
+            if isinstance(path, string_type):
+                path = (path, )
+            os.mkdir(os.path.join(eb.installdir, *path))
         # this is not a path that should be picked up
         os.mkdir(os.path.join(eb.installdir, 'CPATH'))
 
@@ -332,6 +331,7 @@ def test_make_module_req(self):
             self.assertTrue(re.search(r"^prepend-path\s+CLASSPATH\s+\$root/bla.jar$", guess, re.M))
             self.assertTrue(re.search(r"^prepend-path\s+CLASSPATH\s+\$root/foo.jar$", guess, re.M))
             self.assertTrue(re.search(r"^prepend-path\s+MANPATH\s+\$root/share/man$", guess, re.M))
+            self.assertTrue(re.search(r"^prepend-path\s+CMAKE_PREFIX_PATH\s+\$root$", guess, re.M))
             # bin/ is not added to $PATH if it doesn't include files
             self.assertFalse(re.search(r"^prepend-path\s+PATH\s+\$root/bin$", guess, re.M))
             self.assertFalse(re.search(r"^prepend-path\s+PATH\s+\$root/sbin$", guess, re.M))
@@ -341,6 +341,7 @@ def test_make_module_req(self):
             self.assertTrue(re.search(r'^prepend_path\("CLASSPATH", pathJoin\(root, "bla.jar"\)\)$', guess, re.M))
             self.assertTrue(re.search(r'^prepend_path\("CLASSPATH", pathJoin\(root, "foo.jar"\)\)$', guess, re.M))
             self.assertTrue(re.search(r'^prepend_path\("MANPATH", pathJoin\(root, "share/man"\)\)$', guess, re.M))
+            self.assertTrue('prepend_path("CMAKE_PREFIX_PATH", root)' in guess)
             # bin/ is not added to $PATH if it doesn't include files
             self.assertFalse(re.search(r'^prepend_path\("PATH", pathJoin\(root, "bin"\)\)$', guess, re.M))
             self.assertFalse(re.search(r'^prepend_path\("PATH", pathJoin\(root, "sbin"\)\)$', guess, re.M))
@@ -361,6 +362,41 @@ def test_make_module_req(self):
         else:
             self.assertTrue(False, "Unknown module syntax: %s" % get_module_syntax())
 
+        # Check that lib64 is only added to CMAKE_LIBRARY_PATH if there are files in there
+        # but only if it is not a symlink to lib
+        # -- No Files
+        if get_module_syntax() == 'Tcl':
+            self.assertFalse(re.search(r"^prepend-path\s+CMAKE_LIBRARY_PATH\s+\$root/lib64$", guess, re.M))
+        elif get_module_syntax() == 'Lua':
+            self.assertFalse('prepend_path("CMAKE_LIBRARY_PATH", pathJoin(root, "lib64"))' in guess)
+        # -- With files
+        open(os.path.join(eb.installdir, 'lib64', 'libfoo.so'), 'w').write('test')
+        guess = eb.make_module_req()
+        if get_module_syntax() == 'Tcl':
+            self.assertTrue(re.search(r"^prepend-path\s+CMAKE_LIBRARY_PATH\s+\$root/lib64$", guess, re.M))
+        elif get_module_syntax() == 'Lua':
+            self.assertTrue('prepend_path("CMAKE_LIBRARY_PATH", pathJoin(root, "lib64"))' in guess)
+        # -- With files in lib and lib64 symlinks to lib
+        open(os.path.join(eb.installdir, 'lib', 'libfoo.so'), 'w').write('test')
+        shutil.rmtree(os.path.join(eb.installdir, 'lib64'))
+        os.symlink('lib', os.path.join(eb.installdir, 'lib64'))
+        guess = eb.make_module_req()
+        if get_module_syntax() == 'Tcl':
+            self.assertFalse(re.search(r"^prepend-path\s+CMAKE_LIBRARY_PATH\s+\$root/lib64$", guess, re.M))
+        elif get_module_syntax() == 'Lua':
+            self.assertFalse('prepend_path("CMAKE_LIBRARY_PATH", pathJoin(root, "lib64"))' in guess)
+
+        # With files in /lib and /lib64 symlinked to /lib there should be exactly 1 entry for (LD_)LIBRARY_PATH
+        # pointing to /lib
+        for var in ('LIBRARY_PATH', 'LD_LIBRARY_PATH'):
+            if get_module_syntax() == 'Tcl':
+                self.assertFalse(re.search(r"^prepend-path\s+%s\s+\$root/lib64$" % var, guess, re.M))
+                self.assertEqual(len(re.findall(r"^prepend-path\s+%s\s+\$root/lib$" % var, guess, re.M)), 1)
+            elif get_module_syntax() == 'Lua':
+                self.assertFalse(re.search(r'^prepend_path\("%s", pathJoin\(root, "lib64"\)\)$' % var, guess, re.M))
+                self.assertEqual(len(re.findall(r'^prepend_path\("%s", pathJoin\(root, "lib"\)\)$' % var,
+                                                guess, re.M)), 1)
+
         # check for behavior when a string value is used as dict value by make_module_req_guesses
         eb.make_module_req_guess = lambda: {'PATH': 'bin'}
         txt = eb.make_module_req()
diff --git a/test/framework/toy_build.py b/test/framework/toy_build.py
--- a/test/framework/toy_build.py
+++ b/test/framework/toy_build.py
@@ -1231,6 +1231,7 @@ def test_toy_module_fulltxt(self):
                 r'',
                 r'conflict\("toy"\)',
                 r'',
+                r'prepend_path\("CMAKE_PREFIX_PATH", root\)',
                 r'prepend_path\("LD_LIBRARY_PATH", pathJoin\(root, "lib"\)\)',
                 r'prepend_path\("LIBRARY_PATH", pathJoin\(root, "lib"\)\)',
                 r'prepend_path\("PATH", pathJoin\(root, "bin"\)\)',
@@ -1268,6 +1269,7 @@ def test_toy_module_fulltxt(self):
                 r'',
                 r'conflict toy',
                 r'',
+                r'prepend-path	CMAKE_PREFIX_PATH		\$root',
                 r'prepend-path	LD_LIBRARY_PATH		\$root/lib',
                 r'prepend-path	LIBRARY_PATH		\$root/lib',
                 r'prepend-path	PATH		\$root/bin',