Repository: easybuilders/easybuild-framework

Issue:
Java/11 not considered present with HMNS on develop
Att: @Flamefire 

The changes in #3216 seem to have caused a regression on my Ubuntu 18.04 machine, where something depending on `Java/11` fails to notice the installed module.

```
$ eb R-4.0.0-foss-2020a.eb -rD
*snip*
 * [x] $CFGS/j/Java/Java-11.0.2.eb (module: Core | Java/11.0.2)
 * [ ] $CFGS/j/Java/Java-11.eb (module: Core | Java/11)
 * [x] $CFGS/n/NLopt/NLopt-2.6.1-GCCcore-9.3.0.eb (module: Compiler/GCCcore/9.3.0 | NLopt/2.6.1)
 * [x] $CFGS/l/libsndfile/libsndfile-1.0.28-GCCcore-9.3.0.eb (module: Compiler/GCCcore/9.3.0 | libsndfile/1.0.28)
*snip*
 * [x] $CFGS/s/ScaLAPACK/ScaLAPACK-2.1.0-gompi-2020a.eb (module: MPI/GCC/9.3.0/OpenMPI/4.0.3 | ScaLAPACK/2.1.0)
 * [x] $CFGS/f/foss/foss-2020a.eb (module: Core | foss/2020a)
 * [ ] $CFGS/r/R/R-4.0.0-foss-2020a.eb (module: MPI/GCC/9.3.0/OpenMPI/4.0.3 | R/4.0.0)
```

This works fine on 4.2.0, but fails on 4.2.0 w/ 3216 applied and on current develop.

The build attempts to find `Core/Java/11` which doesn't resolve with my module path, while `Java/11` does.


Discussion:
Sorry I don't know what HMNS is exactly and how modules are supposed to be named. If the build attempts to find `Core/Java/11` and it does not exist, then this is not an issue with #3216 but with HMNS outputting the wrong name or with your setup. My change only fixes the `module_exists(module_name)` function to match behavior of the module tool, i.e. if `module_exists` returns True then a `module load module_name` is possible. It sounds like that is working as expected so I'll leave this to someone familiar with HMNS
Sorry, HierarchicalMNS, the module naming system where there's a hierarchy of modules.

I can `ml Java/11` and `ml show Java/11`. I don't know what EB used to do, but with the PR EB attempts to `ml show Core/Java/11` which doesn't resolve.

You can see this in this log: https://gist.github.com/zao/6c90bab96218fcbe8584918b09a0444b

Regardless of if this is something underlying broken thing that was exposed by changing the method with which things were looked up, it's a regression.
The existence check in 4.2.0 produced True in the same situation:

https://gist.github.com/zao/1d41be949440435974a3d6c5c8b3953d
> Regardless of if this is something underlying broken thing that was exposed by changing the method with which things were looked up, it's a regression.

True. I was just pointing out that exactly this is the case: Something is broken trying to use `Core/Java/11` as the module name which is not a valid module but previously a (hacky, incomplete and sometimes wrong) fallback was used which did recognize this as a valid module.

Hence me relaying it to someone familiar with HMNS to sort this out.

From your log I see:

> == 2020-05-12 15:20:37,413 utilities.py:102 DEBUG Module name Core/Java/11.0.2 validated
== 2020-05-12 15:20:37,413 easyconfig.py:2501 DEBUG Obtained valid full module name Core/Java/11.0.2
== 2020-05-12 15:20:37,413 easyconfig.py:2522 DEBUG Determining short module name for <easybuild.framework.easyconfig.easyconfig.EasyConfig object at 0x7fbfc7dc7c50> (force_visible: False)
== 2020-05-12 15:20:37,413 easyconfig.py:2470 DEBUG No alternative software name specified to determine module name with
== 2020-05-12 15:20:37,413 utilities.py:102 DEBUG Module name Java/11.0.2 validated
== 2020-05-12 15:20:37,413 easyconfig.py:2524 DEBUG Obtained valid short module name Java/11.0.2

I.e. see the `Module name Java/11.0.2 validated` using the non Core-prefix

And 

> Determining full module name for {'external_module_metadata': {}, 'short_mod_name': 'Java/11', 'toolchain': {'version': '', 'name': 'system'}, 'name': 'Java', 'full_mod_name': 'Core/Java/11', 'system': True, 'build_only': False, 'versionsuffix': '', 'version': '11', 'toolchain_inherited': False, 'hidden': False, 'external_module': False}

So I guess the `short_mod_name` should be used. This is supported by

> == 2020-05-12 15:20:54,332 modules.py:1591 DEBUG Paths for 'module show' key '('MODULEPATH=/eb/modules/all:/eb/develop/modules:/eb/modules/all/Core:/opt/Lmod-7.8.8/modulefiles/Linux:/opt/Lmod-7.8.8/modulefiles/Core:/opt/Lmod-7.8.8/lmod/lmod/modulefiles/Core', 'lmod', 'Core/Java/11')': ['/eb/modules/all', '/eb/develop/modules', '/eb/modules/all/Core', '/opt/Lmod-7.8.8/modulefiles/Linux', '/opt/Lmod-7.8.8/modulefiles/Core', '/opt/Lmod-7.8.8/lmod/lmod/modulefiles/Core']

which shows that `Core` s a subdirectory and hence must not be part of the module name, and by the output of `module avail` which lists `'Core/Java/11.0.2', 'Java/11', 'Java/11.0.2',` but no `Core/Java/11`
When using HierarchicalMNS, EasyBuild checks for the existence of modules with the "full" name (`Core/Java/11` in this case), not the short name (`Java/11`).

The reason for that is that the module may not be visible without loading other modules first, since that's how the module hierarchy works (loading a `GCCcore` module makes all modules built with it visible).

If this got broken, we should really fix it, even if it means re-introducing the fallback.

I'm a bit surprised by this regression though, I could've sworn this was covered by the tests...
This seems to only apply to aliases. See my last sentence how the underlying module exists with and without Core but the alias only exists without Core.

> I'm a bit surprised by this regression though, I could've sworn this was covered by the tests...

The tests are (now) done so as `module show <foo>` equals `module_exists(<foo>)` and as `ml show Core/Java/11` does not work, False is the correct result. So I'd suggest to re-verify the reasoning for using full-module-names. Maybe checking for the short module name first solves this already. But as mentioned I don't know enough about HMNS to tell for sure

Resolving patch:
diff --git a/easybuild/tools/modules.py b/easybuild/tools/modules.py
--- a/easybuild/tools/modules.py
+++ b/easybuild/tools/modules.py
@@ -487,10 +487,7 @@ def module_wrapper_exists(self, mod_name, modulerc_fn='.modulerc', mod_wrapper_r
         """
         Determine whether a module wrapper with specified name exists.
         Only .modulerc file in Tcl syntax is considered here.
-        DEPRECATED. Use exists()
         """
-        self.log.deprecated('module_wrapper_exists is unreliable and should no longer be used. ' +
-                            'Use exists instead to check for an existing module or alias.', '5.0')
 
         if mod_wrapper_regex_template is None:
             mod_wrapper_regex_template = "^[ ]*module-version (?P<wrapped_mod>[^ ]*) %s$"
@@ -590,6 +587,19 @@ def mod_exists_via_show(mod_name):
                 self.log.debug("checking whether hidden module %s exists via 'show'..." % mod_name)
                 mod_exists = mod_exists_via_show(mod_name)
 
+            # if no module file was found, check whether specified module name can be a 'wrapper' module...
+            # this fallback mechanism is important when using a hierarchical module naming scheme,
+            # where "full" module names (like Core/Java/11) are used to check whether modules exist already;
+            # Lmod will report module wrappers as non-existent when full module name is used,
+            # see https://github.com/TACC/Lmod/issues/446
+            if not mod_exists:
+                self.log.debug("Module %s not found via module avail/show, checking whether it is a wrapper", mod_name)
+                wrapped_mod = self.module_wrapper_exists(mod_name)
+                if wrapped_mod is not None:
+                    # module wrapper only really exists if the wrapped module file is also available
+                    mod_exists = wrapped_mod in avail_mod_names or mod_exists_via_show(wrapped_mod)
+                    self.log.debug("Result for existence check of wrapped module %s: %s", wrapped_mod, mod_exists)
+
             self.log.debug("Result for existence check of %s module: %s", mod_name, mod_exists)
 
             mods_exist.append(mod_exists)
@@ -1407,7 +1417,7 @@ def prepend_module_path(self, path, set_mod_paths=True, priority=None):
     def module_wrapper_exists(self, mod_name):
         """
         Determine whether a module wrapper with specified name exists.
-        DEPRECATED. Use exists()
+        First check for wrapper defined in .modulerc.lua, fall back to also checking .modulerc (Tcl syntax).
         """
         res = None
 


Test patch:
diff --git a/test/framework/modules.py b/test/framework/modules.py
--- a/test/framework/modules.py
+++ b/test/framework/modules.py
@@ -318,12 +318,11 @@ def test_exist(self):
 
         self.assertTrue('Core/Java/1.8.0_181' in self.modtool.available())
         self.assertEqual(self.modtool.exist(['Core/Java/1.8.0_181']), [True])
-        # module-version only works for EnvironmentModules(C) as LMod and EnvironmentModulesTcl would need updating
-        # to full path, see https://github.com/TACC/Lmod/issues/446
-        if isinstance(self.modtool, Lmod) or self.modtool.__class__ == EnvironmentModulesTcl:
-            self.assertEqual(self.modtool.exist(['Core/Java/1.8', 'Core/Java/site_default']), [False, False])
-        else:
-            self.assertEqual(self.modtool.exist(['Core/Java/1.8', 'Core/Java/site_default']), [True, True])
+        # there's a workaround to ensure that module wrappers/aliases are recognized when they're
+        # being checked with the full module name (see https://github.com/TACC/Lmod/issues/446);
+        # that's necessary when using a hierarchical module naming scheme,
+        # see https://github.com/easybuilders/easybuild-framework/issues/3335
+        self.assertEqual(self.modtool.exist(['Core/Java/1.8', 'Core/Java/site_default']), [True, True])
 
         # also check with .modulerc.lua for Lmod 7.8 or newer
         if isinstance(self.modtool, Lmod) and StrictVersion(self.modtool.version) >= StrictVersion('7.8'):
@@ -354,8 +353,8 @@ def test_exist(self):
             shutil.move(java_mod_dir, os.path.join(self.test_prefix, 'Core', 'Java'))
             self.assertTrue('Core/Java/1.8.0_181' in self.modtool.available())
             self.assertEqual(self.modtool.exist(['Core/Java/1.8.0_181']), [True])
-            self.assertEqual(self.modtool.exist(['Core/Java/1.8']), [False])
-            self.assertEqual(self.modtool.exist(['Core/Java/site_default']), [False])
+            self.assertEqual(self.modtool.exist(['Core/Java/1.8']), [True])
+            self.assertEqual(self.modtool.exist(['Core/Java/site_default']), [True])
 
         # Test alias in home directory .modulerc
         if isinstance(self.modtool, Lmod) and StrictVersion(self.modtool.version) >= StrictVersion('7.0'):