Repository: easybuilders/easybuild-framework

Issue:
Java-1.8.eb modulerc file fails to install with EasyBuild-4.2.1 due to changes in modules.py
**Issue**

With EasyBuild-4.2.0 `eb Java-1.8.eb -f` worked without problems, but with EasyBuild-4.2.1 I get:

```
== 2020-07-01 13:12:01,272 modules.py:596 DEBUG Module Java/1.8.0_192 not found via module avail/show, checking whether it is a wrapper
== 2020-07-01 13:12:01,273 modules.py:502 DEBUG Found /home/casparl/.local/easybuild/RedHatEnterpriseServer7/2019/modulefiles/all/Java/.modulerc that may define Java/1.8.0_192 as a wrapper for a module file
== 2020-07-01 13:12:01,273 modules.py:502 DEBUG Found /sw/arch/RedHatEnterpriseServer7/EB_production/2019/modulefiles/lang/Java/.modulerc that may define Java/1.8.0_192 as a wrapper for a module file
== 2020-07-01 13:12:01,274 modules.py:603 DEBUG Result for existence check of Java/1.8.0_192 module: False
== 2020-07-01 13:12:01,382 build_log.py:169 ERROR EasyBuild crashed with an error (at easybuild/base/exceptions.py:124 in __init__): Can't get value from a non-existing module Java/1.8.0_192 (at easybuild/tools/modules.py:691 in get_value_from_modulefile)
```

I have pinpointed this to the `modules.py`, because if I copy the `modules.py` from 4.2.0 into my 4.2.1 installation, my EasyBuild 4.2.1 can suddenly complete `eb Java-1.8.eb -f` succesfully.

Quite a number of things changed in the `modules.py` with the last update, but most of it is logging. I suspect the change in `mod_exists_via_show(mod_name)` to be the breaking change. I will proceed to modify the modules.py to only roll back that change.




Discussion:
I changed the `mod_exists_via_show` function of EasyBuild-4.2.1 to:

```
def mod_exists_via_show(mod_name):
            """
            Helper function to check whether specified module name exists through 'module show'.

            :param mod_name: module name
            """

            # Old mod exists:
            mod_exists_regex_template_local = r'^\s*\S*/%s.*:\s*$'
            mod_exists_regex = mod_exists_regex_template_local % re.escape(mod_name)
            txt = self.show(mod_name)
            old_res = bool(re.search(mod_exists_regex, txt, re.M))
            self.log.debug("Old mod_exists_via_show would return %s" % old_res)

            stderr = self.show(mod_name)
            res = False
            # Parse the output:
            # - Skip whitespace
            # - Any error -> Module does not exist
            # - Check first non-whitespace line for something that looks like an absolute path terminated by a colon
            mod_exists_regex = r'\s*/.+:\s*'
            for line in stderr.split('\n'):
                if OUTPUT_MATCHES['whitespace'].search(line):
                    continue
                if OUTPUT_MATCHES['error'].search(line):
                    break
                if re.match(mod_exists_regex, line):
                    res = True
                break
            self.log.debug("New mod_exists_via_show will return %s" % res)
            return res
```
I.e. it first goes through the logic that was in EasyBuild 4.2.0 and prints whether the function would return true or false. Then, it goes through the EasyBuild v 4.2.1 logic, prints whether that would return true or false, and return that.

My (debugging) output now shows:

```
== 2020-07-01 13:37:29,132 modules.py:556 DEBUG Old mod_exists_via_show would return True
...
== 2020-07-01 13:37:29,133 modules.py:573 DEBUG New mod_exists_via_show will return False
```
Finally, changing the function further by returning `old_res` instead of `res`, the build now completes.

In other words: this piece of code is indeed the breaking change.
Running the original EasyBuild 4.2.1 `modules.py` with some extra debugging output:
```
mod_exists_regex = r'\s*/.+:\s*'
            for line in stderr.split('\n'):
                self.log.debug("Scanning line %s" % line)
                if OUTPUT_MATCHES['whitespace'].search(line):
                    self.log.debug("Output matches whitespace for module %s" % mod_name)
                    continue
                if OUTPUT_MATCHES['error'].search(line):
                    self.log.debug("Module show of module %s resulted in an error on the following line:" % mod_name)
                    self.log.debug("%s" % line)
                    break
                if re.match(mod_exists_regex, line):
                    self.log.debug("Module %s exists!)" % mod_name)
                    res = True
                break
            self.log.debug("New mod_exists_via_show will return %s" % res)
            return res
```
Resulted in:
```
== 2020-07-01 13:47:36,809 modules.py:566 DEBUG Scanning line module-version    Java/1.8.0_192 1.8
== 2020-07-01 13:47:36,810 modules.py:578 DEBUG New mod_exists_via_show will return False
```
In other words: none of the if-statements are true. The regexp doesn't match, even though I think it should (?).
Ok, I think I found it. This line, returns the stderr of a module show command:
```
stderr = self.show(mod_name)
```
Looking at the debug logs, this runs

```
[casparl@int2 TMP]$ tclsh /hpc/eb/modules-tcl-1.775//libexec/modulecmd.tcl python show Java/1.8.0_192
```
of which the stderr output is
```
module-version  Java/1.8.0_192 1.8
-------------------------------------------------------------------
/home/casparl/.local/easybuild/RedHatEnterpriseServer7/2019/modulefiles/all/Java/1.8.0_192:

module-whatis   Description: Java Platform, Standard Edition (Java SE) lets you develop and deploy
 Java applications on desktops and servers.
module-whatis   Homepage: http://java.com/
module-whatis   URL: http://java.com/
conflict        Java
prepend-path    CMAKE_PREFIX_PATH       /home/casparl/.local/easybuild/RedHatEnterpriseServer7/2019/software/Java/1.8.0_192
prepend-path    CPATH   /home/casparl/.local/easybuild/RedHatEnterpriseServer7/2019/software/Java/1.8.0_192/include
prepend-path    LD_LIBRARY_PATH /home/casparl/.local/easybuild/RedHatEnterpriseServer7/2019/software/Java/1.8.0_192/lib
prepend-path    LIBRARY_PATH    /home/casparl/.local/easybuild/RedHatEnterpriseServer7/2019/software/Java/1.8.0_192/lib
prepend-path    SURFSARA_LIBRARY_PATH   /home/casparl/.local/easybuild/RedHatEnterpriseServer7/2019/software/Java/1.8.0_192/lib
prepend-path    MANPATH /home/casparl/.local/easybuild/RedHatEnterpriseServer7/2019/software/Java/1.8.0_192/man
prepend-path    PATH    /home/casparl/.local/easybuild/RedHatEnterpriseServer7/2019/software/Java/1.8.0_192/bin
setenv          EBROOTJAVA      /home/casparl/.local/easybuild/RedHatEnterpriseServer7/2019/software/Java/1.8.0_192
setenv          EBVERSIONJAVA   1.8.0_192
setenv          EBDEVELJAVA     /home/casparl/.local/easybuild/RedHatEnterpriseServer7/2019/software/Java/1.8.0_192/easybuild/Java-1.8.0_192-easybuild-devel
prepend-path    PATH    /home/casparl/.local/easybuild/RedHatEnterpriseServer7/2019/software/Java/1.8.0_192
setenv          JAVA_HOME       /home/casparl/.local/easybuild/RedHatEnterpriseServer7/2019/software/Java/1.8.0_192
-------------------------------------------------------------------
```
The new mod_exists_via_show function has this comment:
```
# - Check first non-whitespace line for something that looks like an absolute path terminated by a colon
```
That is indeed what the logic with the three if's and then a break does. This is also what causes my problem: for Tcl, the full path to the modulefile is on the third line, not the first! Probably, this only happens for modules that are wrapped by a `.modulerc`, since I only see this issue when trying to install `Java-1.8.eb`.

Now the question is: how should one solve this. I'm happy to remove the `break` statement so that it goes on to the second and third line, but it was probably put in there to make the regexp more robust against matching other paths... At the very least it means we need to put the module name back in the regexp, because now it would match ANY path that ends in a colon with `r'\s*/.+:\s*'`.
@Flamefire Since you wrote this piece of code, do you have a suggestion? Should we simply go back to the original regexp that was matched in EasyBuild <= 4.2.0? Or did you move away from that to solve a problem on your side?
> of which the stderr output is 

I'm confused by that. Are you 100% sure that the shown output is the stderr and not stdout + stderr? This is tested on CI so I'm surprised to see it failing.

If it is, then on to solving:
- Is there any chance to find out why tclsh is outputting the (I suppose) alias command from the modulerc and avoid that?
- the `---` lines are treated as whitespace, hence we need to check the second line

> I'm happy to remove the break statement so that it goes on to the second and third line, but it was probably put in there to make the regexp more robust against matching other paths...

I'd rather suggest to ignore the first line starting with `module-` to keep the strength

> At the very least it means we need to put the module name back in the regexp, because now it would match ANY path that ends in a colon with r'\s*/.+:\s*'.

This is by design. A module alias may alias a completely different name. E.g. imagine `Java` as an alias for `PrettySystemJRE` because of reasons. This is fine and should be supported.

So quick fix: Ignore the first `module-` line (maybe only on tclsh)

To strengthen it: Come up with a test for this. Again I thought this is already covered, maybe @boegel might know why it didn't come up on CI
@casparvl Which modules tool are you using here?

Resolving patch:
diff --git a/easybuild/tools/modules.py b/easybuild/tools/modules.py
--- a/easybuild/tools/modules.py
+++ b/easybuild/tools/modules.py
@@ -547,6 +547,7 @@ def mod_exists_via_show(mod_name):
 
             :param mod_name: module name
             """
+            self.log.debug("Checking whether %s exists based on output of 'module show'", mod_name)
             stderr = self.show(mod_name)
             res = False
             # Parse the output:
@@ -555,13 +556,38 @@ def mod_exists_via_show(mod_name):
             # - Check first non-whitespace line for something that looks like an absolute path terminated by a colon
             mod_exists_regex = r'\s*/.+:\s*'
             for line in stderr.split('\n'):
+
+                self.log.debug("Checking line '%s' to determine whether %s exists...", line, mod_name)
+
+                # skip whitespace lines
                 if OUTPUT_MATCHES['whitespace'].search(line):
+                    self.log.debug("Treating line '%s' as whitespace, so skipping it", line)
                     continue
+
+                # if any errors occured, conclude that module doesn't exist
                 if OUTPUT_MATCHES['error'].search(line):
+                    self.log.debug("Line '%s' looks like an error, so concluding that %s doesn't exist",
+                                   line, mod_name)
                     break
-                if re.match(mod_exists_regex, line):
-                    res = True
+
+                # skip warning lines, which may be produced by modules tool but should not be used
+                # to determine whether a module file exists
+                if line.startswith('WARNING: '):
+                    self.log.debug("Skipping warning line '%s'", line)
+                    continue
+
+                # skip lines that start with 'module-' (like 'module-version'),
+                # see https://github.com/easybuilders/easybuild-framework/issues/3376
+                if line.startswith('module-'):
+                    self.log.debug("Skipping line '%s' since it starts with 'module-'", line)
+                    continue
+
+                # if line matches pattern that indicates an existing module file, the module file exists
+                res = bool(re.match(mod_exists_regex, line))
+                self.log.debug("Result for existence check of %s based on 'module show' output line '%s': %s",
+                               mod_name, line, res)
                 break
+
             return res
 
         if skip_avail:
@@ -577,14 +603,19 @@ def mod_exists_via_show(mod_name):
 
         mods_exist = []
         for (mod_name, visible) in mod_names:
+            self.log.info("Checking whether %s exists...", mod_name)
             if visible:
                 mod_exists = mod_name in avail_mod_names
                 # module name may be partial, so also check via 'module show' as fallback
-                if not mod_exists and maybe_partial:
+                if mod_exists:
+                    self.log.info("Module %s exists (found in list of available modules)", mod_name)
+                elif maybe_partial:
+                    self.log.info("Module %s not found in list of available modules, checking via 'module show'...",
+                                  mod_name)
                     mod_exists = mod_exists_via_show(mod_name)
             else:
                 # hidden modules are not visible in 'avail', need to use 'show' instead
-                self.log.debug("checking whether hidden module %s exists via 'show'..." % mod_name)
+                self.log.info("Checking whether hidden module %s exists via 'show'..." % mod_name)
                 mod_exists = mod_exists_via_show(mod_name)
 
             # if no module file was found, check whether specified module name can be a 'wrapper' module...
@@ -593,14 +624,14 @@ def mod_exists_via_show(mod_name):
             # Lmod will report module wrappers as non-existent when full module name is used,
             # see https://github.com/TACC/Lmod/issues/446
             if not mod_exists:
-                self.log.debug("Module %s not found via module avail/show, checking whether it is a wrapper", mod_name)
+                self.log.info("Module %s not found via module avail/show, checking whether it is a wrapper", mod_name)
                 wrapped_mod = self.module_wrapper_exists(mod_name)
                 if wrapped_mod is not None:
                     # module wrapper only really exists if the wrapped module file is also available
                     mod_exists = wrapped_mod in avail_mod_names or mod_exists_via_show(wrapped_mod)
                     self.log.debug("Result for existence check of wrapped module %s: %s", wrapped_mod, mod_exists)
 
-            self.log.debug("Result for existence check of %s module: %s", mod_name, mod_exists)
+            self.log.info("Result for existence check of %s module: %s", mod_name, mod_exists)
 
             mods_exist.append(mod_exists)
 


Test patch:
diff --git a/test/framework/modules.py b/test/framework/modules.py
--- a/test/framework/modules.py
+++ b/test/framework/modules.py
@@ -310,6 +310,34 @@ def test_exist(self):
         if self.modtool.__class__ != EnvironmentModulesC:
             self.assertEqual(self.modtool.exist(['Java/Alias', 'Java/NonExist']), [True, False])
 
+        # set 'module avail' cache entries to empty lists,
+        # to enforce fallback to 'module show'
+        import easybuild.tools.modules
+        for key in easybuild.tools.modules.MODULE_AVAIL_CACHE:
+            easybuild.tools.modules.MODULE_AVAIL_CACHE[key] = []
+
+        # clear 'module show' cache, to keep control below
+        easybuild.tools.modules.MODULE_SHOW_CACHE.clear()
+        self.assertEqual(self.modtool.exist(['Java/1.8', 'Java/1.8.0_181']), [True, True])
+
+        # mimic more verbose stderr output produced by old Tmod version,
+        # including a warning produced when multiple .modulerc files are being picked up
+        # see https://github.com/easybuilders/easybuild-framework/issues/3376
+        ml_show_java18_stderr = '\n'.join([
+            "module-version    Java/1.8.0_181 1.8",
+            "WARNING: Duplicate version symbol '1.8' found",
+            "module-version  Java/1.8.0_181 1.8",
+            "-------------------------------------------------------------------",
+            "/modulefiles/lang/Java/1.8.0_181:",
+            "-------------------------------------------------------------------",
+        ])
+
+        # overwrite 'module show' cache entries with output that includes extra lines
+        for key in easybuild.tools.modules.MODULE_SHOW_CACHE:
+            easybuild.tools.modules.MODULE_SHOW_CACHE[key] = ml_show_java18_stderr
+
+        self.assertEqual(self.modtool.exist(['Java/1.8', 'Java/1.8.0_181']), [True, True])
+
         reset_module_caches()
 
         # what if we're in an HMNS setting...