Repository: easybuilders/easybuild-framework

Issue:
Ordering bug when making module file in v4.2.x
We ran into problems with a recent install of `impi`, when investigating we discovered that the order of the prepends to `LD_LIBRARY_PATH` has changed between v4.1.2 and v4.2.x:
```
[ocais1@jrl11 ~]$ module load EasyBuild/4.1.2 

The following have been reloaded with a version change:
  1) EasyBuild/4.2.1 => EasyBuild/4.1.2

[ocais1@jrl11 ~]$ eb  impi-2019.7.217-iccifort-2019.5.281-GCC-8.3.0.eb --module-only --force 
== temporary log file in case of crash /tmp/eb-qfskkD/easybuild-YNVdbn.log
== resolving dependencies ...
== processing EasyBuild easyconfig /usr/local/software/jureca/Stages/Devel-2019a/eb_repo/impi/impi-2019.7.217-iccifort-2019.5.281-GCC-8.3.0.eb
== building and installing Compiler/mpi/intel/2019-GCC-8.3.0/IntelMPI/2019.7.217...
== fetching files [skipped]
== creating build dir, resetting environment...
== backup of existing module file stored at /usr/local/software/jureca/Stages/Devel-2019a/modules/all/Compiler/mpi/intel/2019-GCC-8.3.0/IntelMPI/2019.7.217.bak_20200616164837_28214
== unpacking [skipped]
== patching [skipped]
== preparing...
== configuring [skipped]
== building [skipped]
== testing [skipped]
== installing [skipped]
== taking care of extensions [skipped]
== restore after iterating...
== postprocessing [skipped]
== sanity checking [skipped]
== cleaning up [skipped]
== creating module...
== comparing module file with backup /usr/local/software/jureca/Stages/Devel-2019a/modules/all/Compiler/mpi/intel/2019-GCC-8.3.0/IntelMPI/2019.7.217.bak_20200616164837_28214; diff is:
--- /usr/local/software/jureca/Stages/Devel-2019a/modules/all/Compiler/mpi/intel/2019-GCC-8.3.0/IntelMPI/2019.7.217.bak_20200616164837_28214
+++ /usr/local/software/jureca/Stages/Devel-2019a/modules/all/Compiler/mpi/intel/2019-GCC-8.3.0/IntelMPI/2019.7.217.lua
@@ -32,13 +32,13 @@
 
 prepend_path("CPATH", pathJoin(root, "intel64/include"))
 prepend_path("FI_PROVIDER_PATH", pathJoin(root, "intel64/libfabric/lib/prov"))
+prepend_path("LD_LIBRARY_PATH", pathJoin(root, "intel64/lib/release_mt"))
+prepend_path("LD_LIBRARY_PATH", pathJoin(root, "intel64/lib/release"))
 prepend_path("LD_LIBRARY_PATH", pathJoin(root, "intel64/lib"))
-prepend_path("LD_LIBRARY_PATH", pathJoin(root, "intel64/lib/release"))
-prepend_path("LD_LIBRARY_PATH", pathJoin(root, "intel64/lib/release_mt"))
 prepend_path("LD_LIBRARY_PATH", pathJoin(root, "intel64/libfabric/lib"))
+prepend_path("LIBRARY_PATH", pathJoin(root, "intel64/lib/release_mt"))
+prepend_path("LIBRARY_PATH", pathJoin(root, "intel64/lib/release"))
 prepend_path("LIBRARY_PATH", pathJoin(root, "intel64/lib"))
-prepend_path("LIBRARY_PATH", pathJoin(root, "intel64/lib/release"))
-prepend_path("LIBRARY_PATH", pathJoin(root, "intel64/lib/release_mt"))
 prepend_path("LIBRARY_PATH", pathJoin(root, "intel64/libfabric/lib"))
 prepend_path("MANPATH", pathJoin(root, "man"))
 prepend_path("PATH", pathJoin(root, "intel64/bin"))
@@ -59,7 +59,7 @@
 set_alias("mpiicc", "mpiicc -cc=icc")
 set_alias("mpiicpc", "mpiicpc -cxx=icpc")
 set_alias("mpiifort", "mpiifort -f90=ifort")
--- Built with EasyBuild version 4.2.1
+-- Built with EasyBuild version 4.1.2

``` 


Discussion:
I *think* this may be a result of https://github.com/easybuilders/easybuild-framework/pull/3176/files#diff-a1bcbbc1aab6b59688b219f298fddb56R1328 , it's definitely something like that where sorting was introduced without taking into account that the order may actually matter.
Nope, tested that and that wasn't it.
Ok, I found it, there are a couple of agressive sortings going on that were introduced in that PR. I'll make a PR to revert them.
The precedence of `release_mt` over `release` was introduced by commit https://github.com/easybuilders/easybuild-easyblocks/commit/de4aa4cd7a9b2cbaf98535c3c7cbd0b393b11614 (line 200) in the easyblock of `impi`

This change was introduced in release v3.8.0. I guess that you are seeing this now because `impi` v2019 has not been part of a toolchain before last release.
According to the [Developers Guide for Intel MPI v2019](https://software.intel.com/content/www/us/en/develop/download/intel-mpi-library-for-linux-os-developer-guide-2019-update-1.html), `release` is the default with this version. Moreover, `release` is also a multi-threaded library as it is `release_mt`, they differ in how the multi-threading works. So I would fix the `impi` easyblock to follow the default behaviour.

Another option is to enable `multi-ep` but that comes with several implications regarding support from the software linking to MPI.

Resolving patch:
diff --git a/easybuild/framework/easyblock.py b/easybuild/framework/easyblock.py
--- a/easybuild/framework/easyblock.py
+++ b/easybuild/framework/easyblock.py
@@ -1374,11 +1374,11 @@ def make_module_req(self):
             if self.dry_run:
                 self.dry_run_msg(" $%s: %s" % (key, ', '.join(reqs)))
                 # Don't expand globs or do any filtering below for dry run
-                paths = sorted(reqs)
+                paths = reqs
             else:
                 # Expand globs but only if the string is non-empty
                 # empty string is a valid value here (i.e. to prepend the installation prefix, cfr $CUDA_HOME)
-                paths = sorted(sum((glob.glob(path) if path else [path] for path in reqs), []))  # sum flattens to list
+                paths = sum((glob.glob(path) if path else [path] for path in reqs), [])  # sum flattens to list
 
                 # If lib64 is just a symlink to lib we fixup the paths to avoid duplicates
                 lib64_is_symlink = (all(os.path.isdir(path) for path in ['lib', 'lib64'])
@@ -1395,8 +1395,13 @@ def make_module_req(self):
                     if fixed_paths != paths:
                         self.log.info("Fixed symlink lib64 in paths for %s: %s -> %s", key, paths, fixed_paths)
                         paths = fixed_paths
-                # Use a set to remove duplicates, e.g. by having lib64 and lib which get fixed to lib and lib above
-                paths = sorted(set(paths))
+                # remove duplicate paths
+                # don't use 'set' here, since order in 'paths' is important!
+                uniq_paths = []
+                for path in paths:
+                    if path not in uniq_paths:
+                        uniq_paths.append(path)
+                paths = uniq_paths
                 if key in keys_requiring_files:
                     # only retain paths that contain at least one file
                     retained_paths = [


Test patch:
diff --git a/test/framework/easyblock.py b/test/framework/easyblock.py
--- a/test/framework/easyblock.py
+++ b/test/framework/easyblock.py
@@ -421,6 +421,31 @@ def test_make_module_req(self):
         else:
             self.assertTrue(False, "Unknown module syntax: %s" % get_module_syntax())
 
+        # check for correct order of prepend statements when providing a list (and that no duplicates are allowed)
+        eb.make_module_req_guess = lambda: {'LD_LIBRARY_PATH': ['lib/pathC', 'lib/pathA', 'lib/pathB', 'lib/pathA']}
+        for path in ['pathA', 'pathB', 'pathC']:
+            os.mkdir(os.path.join(eb.installdir, 'lib', path))
+            open(os.path.join(eb.installdir, 'lib', path, 'libfoo.so'), 'w').write('test')
+        txt = eb.make_module_req()
+        if get_module_syntax() == 'Tcl':
+            self.assertTrue(re.search(r"\nprepend-path\s+LD_LIBRARY_PATH\s+\$root/lib/pathC\n" +
+                                      r"prepend-path\s+LD_LIBRARY_PATH\s+\$root/lib/pathA\n" +
+                                      r"prepend-path\s+LD_LIBRARY_PATH\s+\$root/lib/pathB\n",
+                                      txt, re.M))
+            self.assertFalse(re.search(r"\nprepend-path\s+LD_LIBRARY_PATH\s+\$root/lib/pathB\n" +
+                                       r"prepend-path\s+LD_LIBRARY_PATH\s+\$root/lib/pathA\n",
+                                       txt, re.M))
+        elif get_module_syntax() == 'Lua':
+            self.assertTrue(re.search(r'\nprepend_path\("LD_LIBRARY_PATH", pathJoin\(root, "lib/pathC"\)\)\n' +
+                                      r'prepend_path\("LD_LIBRARY_PATH", pathJoin\(root, "lib/pathA"\)\)\n' +
+                                      r'prepend_path\("LD_LIBRARY_PATH", pathJoin\(root, "lib/pathB"\)\)\n',
+                                      txt, re.M))
+            self.assertFalse(re.search(r'\nprepend_path\("LD_LIBRARY_PATH", pathJoin\(root, "lib/pathB"\)\)\n' +
+                                       r'prepend_path\("LD_LIBRARY_PATH", pathJoin\(root, "lib/pathA"\)\)\n',
+                                       txt, re.M))
+        else:
+            self.assertTrue(False, "Unknown module syntax: %s" % get_module_syntax())
+
         # cleanup
         eb.close_log()
         os.remove(eb.logfile)