Repository: easybuilders/easybuild-framework

Issue:
Hierarchical Module Naming Scheme with Cray toolchains
I have tried following the tutorial https://easybuilders.github.io/easybuild-tutorial/hmns to implement the Hierarchical Module Naming Scheme (MNS) with a modified Cray toolchain using Lmod 8.3.1, but the compiler modulefile created under the `Core` folder does not update the `MODULEPATH` once loaded, therefore the modules created cannot be searched (not even with `module spider`).

I was wondering if it is at all possible to use the Hierarchical MNS with the way Cray toolchains are configured: for instance I cannot prepare the environment as suggested in the tutorial, using the commands below, or I would unload essential modules:
```
module purge
module unuse $MODULEPATH
```

Thanks for your advice!


Discussion:
@lucamar Can you share the command you run plus the contents of the `Core` module you obtained?
Thanks for your reply @boegel. I have followed the recommended way to install EasyBuild via "pip3 install" of the tutorial, but nothing changed with respect to a local EasyBuild module. I run the command below, for instance:
```
eb Bison-3.7.5-cpeGNU-21.02.eb --hidden -Dr
 * [x] $CFGS/c/cpeGNU/cpeGNU-21.02.eb (module: Core | cpeGNU/21.02)
 * [x] $CFGS/m/M4/M4-1.4.18-cpeGNU-21.02.eb (module: Toolchain/cpeGNU/21.02 | M4/.1.4.18)
 * [ ] $CFGS/b/Bison/Bison-3.7.5-cpeGNU-21.02.eb (module: Toolchain/cpeGNU/21.02 | Bison/.3.7.5)
```
I have previously defined `EASYBUILD_PREFIX=/apps/pilatus/UES/sandbox/lucamar/easybuild`, therefore the folders `Core` and `Toolchain` are created under `/apps/pilatus/UES/sandbox/lucamar/easybuild/modules/all`. The build of `cpeGNU` and `M4` is successful, but `Bison` fails since the dependency `M4` cannot be loaded and the error reported by `Lmod` is the following:
```
== FAILED: Installation ended unsuccessfully (build directory: /run/user/21827/easybuild/build/Bison/3.7.5/cpeGNU-21.02): build failed (first 300 chars): Module command '/usr/share/lmod/lmod/libexec/lmod python load M4/.1.4.18' failed with exit code 1; stderr: Lmod has detected the following error: The following module(s) are unknown: "M4/.1.4.18"

Please check the spelling or version number. Also try "module spider ..."
It is also possible your cache file is out-of-date; it may help to try:
  $ module --ignore-cache load "M4/.1.4.18"
Also make sure that all modulefiles written in TCL start with the string #%Module
; stdout: _mlstatus = False
```

The toolchain `cpeGNU` is a modified CrayGNU toolchain that pins the versions of Cray dependencies and loads the module `cpe-gnu` instead of `PrgEnv-gnu`, which does not exist any longer: the easyconfig file is https://github.com/lucamar/production/blob/pilatus/easybuild/easyconfigs/c/cpeGNU/cpeGNU-21.02.eb and the custom easyblock used is https://github.com/eth-cscs/production/blob/master/easybuild/easyblocks/cpetoolchain.py (`cpe.py` and `cpegnu.py` are added as local files).

If I add the command `prepend_path("MODULEPATH",root:gsub("software","modules/all/Toolchain"))` at the end of the Lua modulefile `cpeGNU/21.02.lua`, then the `Toolchain` path is added to `MODULEPATH` after the toolchain modulefile is loaded and I can load the modules as expected. Please find enclosed the PDF printout of the Lua modulefile: [21.02.lua.pdf](https://github.com/easybuilders/easybuild-framework/files/6004057/21.02.lua.pdf).
 
You'll see that I load the Cray dependencies using `modluafooter` instead of `dependencies`: I had to do that to avoid the use of the function `swap` in the Lua modulefile `cpeGNU/21.02.lua` created by EasyBuild, since it raised a Lua error at the sanity check in some cases:
```
== 2021-02-18 10:41:00,170 build_log.py:169 ERROR EasyBuild crashed with an error (at easybuild/base/exceptions.py:124 in __init__): Module command '/usr/share/lmod/lmod/libexec/lmod python load cpeGNU/21.02' failed with exit code 1; stderr: Lmod has detected the following error: Unable to load module because of error when evaluating modulefile:
     /tmp/eb-l61bgz4f/tmpk9aiog8r/all/Core/cpeGNU/21.02.lua: [string "help([==[..."]:29: attempt to call a nil value (global
'swap')
     Please check the modulefile and especially if there is a the line number specified in the above message 
While processing the following module(s):
    Module fullname  Module Filename
    ---------------  ---------------
    cpeGNU/21.02     /tmp/eb-l61bgz4f/tmpk9aiog8r/all/Core/cpeGNU/21.02.lua

; stdout: _mlstatus = False
 (at easybuild/tools/modules.py:814 in run_module)
```

Since the Lmod installation complains often, I wonder if the issues are rather due to the Cray PE Lmod 8.3.1 : thanks for your advice!

edit: separate issue was opened for the `swap` problem: #3626
@lucamar This is using the standard `HierarchicalMNS` that is included with EasyBuild, correct?
If so, that's not very surprising, since the Cray toolchains are quite different from a regular toolchain...

`HierarchicalMNS` determines which modules should expand `$MODULEPATH` based on the `moduleclass` (both `compiler` and `mpi` are treated special there).

So in this case I think you'll need a small custom hierarchical module naming scheme specific to Cray toolchains...
That should be *a lot* easier to implement that `HierarchicalMNS` though, which has to take into account all types of special cases.

What type of module hierarchy are you interested in, just a single-level one based on toolchain version?
Thanks for your feedback @boegel : yes, I have used the standard `Hierarchical MNS` included with EasyBuild and a hierarchy based on the toolchain name and version only. 
For instance, I have created the toolchain modulefile `$EASYBUILD_PREFIX/modules/all/Core/cpeGNU/21.02.lua` (new toolchain `cpeGNU`, release `21.02`) and loading it will add the folder `$EASYBUILD_PREFIX/modules/all/Toolchain/cpeGNU/21.02` to the `MODULEPATH`, with all the modules created with the toolchain `cpeGNU-21.02`.
So, at present I have a working tree of hierarchical modules, using the quick and dirty workaround with the `modluafooter` that I pasted above `prepend_path("MODULEPATH",root:gsub("software","modules/all/Toolchain"))`.

Resolving patch:
diff --git a/easybuild/tools/module_naming_scheme/hierarchical_mns.py b/easybuild/tools/module_naming_scheme/hierarchical_mns.py
--- a/easybuild/tools/module_naming_scheme/hierarchical_mns.py
+++ b/easybuild/tools/module_naming_scheme/hierarchical_mns.py
@@ -70,6 +70,12 @@
     'xlc,xlf': ('xlcxlf', '%(xlc)s'),
 }
 
+# possible prefixes for Cray toolchain names
+# example: CrayGNU, CrayCCE, cpeGNU, cpeCCE, ...;
+# important for determining $MODULEPATH extensions in det_modpath_extensions,
+# cfr. https://github.com/easybuilders/easybuild-framework/issues/3575
+CRAY_TOOLCHAIN_NAME_PREFIXES = ('Cray', 'cpe')
+
 
 class HierarchicalMNS(ModuleNamingScheme):
     """Class implementing an example hierarchical module naming scheme."""
@@ -236,8 +242,9 @@ def det_modpath_extensions(self, ec):
                 paths.append(os.path.join(MPI, tc_comp_name, tc_comp_ver, ec['name'], fullver))
 
         # special case for Cray toolchains
-        elif modclass == MODULECLASS_TOOLCHAIN and tc_comp_info is None and ec.name.startswith('Cray'):
-            paths.append(os.path.join(TOOLCHAIN, ec.name, ec.version))
+        elif modclass == MODULECLASS_TOOLCHAIN and tc_comp_info is None:
+            if any(ec.name.startswith(x) for x in CRAY_TOOLCHAIN_NAME_PREFIXES):
+                paths.append(os.path.join(TOOLCHAIN, ec.name, ec.version))
 
         return paths
 


Test patch:
diff --git a/test/framework/easyconfigs/test_ecs/c/cpeGNU/cpeGNU-21.04.eb b/test/framework/easyconfigs/test_ecs/c/cpeGNU/cpeGNU-21.04.eb
new file mode 100644
--- /dev/null
+++ b/test/framework/easyconfigs/test_ecs/c/cpeGNU/cpeGNU-21.04.eb
@@ -0,0 +1,16 @@
+easyblock = 'Toolchain'
+
+name = 'cpeGNU'
+version = '21.04'
+
+homepage = 'https://pubs.cray.com'
+description = """Toolchain using Cray compiler wrapper with gcc module (CPE release: %(version)s)."""
+
+toolchain = SYSTEM
+
+dependencies = [
+   ('PrgEnv-gnu', EXTERNAL_MODULE),
+   ('cpe/%(version)s', EXTERNAL_MODULE),
+]
+
+moduleclass = 'toolchain'
diff --git a/test/framework/filetools.py b/test/framework/filetools.py
--- a/test/framework/filetools.py
+++ b/test/framework/filetools.py
@@ -2071,7 +2071,7 @@ def test_index_functions(self):
         # test with specified path with and without trailing '/'s
         for path in [test_ecs, test_ecs + '/', test_ecs + '//']:
             index = ft.create_index(path)
-            self.assertEqual(len(index), 84)
+            self.assertEqual(len(index), 85)
 
             expected = [
                 os.path.join('b', 'bzip2', 'bzip2-1.0.6-GCC-4.9.2.eb'),
diff --git a/test/framework/module_generator.py b/test/framework/module_generator.py
--- a/test/framework/module_generator.py
+++ b/test/framework/module_generator.py
@@ -1326,6 +1326,10 @@ def test_ec(ecfile, short_modname, mod_subdir, modpath_exts, user_modpath_exts,
                                   ['Toolchain/CrayCCE/5.1.29'],
                                   ['Toolchain/CrayCCE/5.1.29'],
                                   ['Core']),
+            'cpeGNU-21.04.eb': ('cpeGNU/21.04', 'Core',
+                                ['Toolchain/cpeGNU/21.04'],
+                                ['Toolchain/cpeGNU/21.04'],
+                                ['Core']),
             'HPL-2.1-CrayCCE-5.1.29.eb': ('HPL/2.1', 'Toolchain/CrayCCE/5.1.29', [], [], ['Core']),
         }
         for ecfile, mns_vals in test_ecs.items():