Repository: easybuilders/easybuild-framework

Issue:
Error creating Cray toolchain modulefiles with Lua syntax  
I report here as a placeholder for further discussion the issue with the `swap` function mentioned in #3575.
The error is raised at the sanity check when the Lua syntax is used to create the Cray toolchain modulefiles:
```
== 2021-02-18 10:41:00,170 build_log.py:169 ERROR EasyBuild crashed with an error (at easybuild/base/exceptions.py:124 in __init__): Module command '/usr/share/lmod/lmod/libexec/lmod python load cpeGNU/21.02' failed with exit code 1; stderr: Lmod has detected the following error: Unable to load module because of error when evaluating modulefile:
     /tmp/eb-l61bgz4f/tmpk9aiog8r/all/Core/cpeGNU/21.02.lua: [string "help([==[..."]:29: attempt to call a nil value (global
'swap')
     Please check the modulefile and especially if there is a the line number specified in the above message 
While processing the following module(s):
    Module fullname  Module Filename
    ---------------  ---------------
    cpeGNU/21.02     /tmp/eb-l61bgz4f/tmpk9aiog8r/all/Core/cpeGNU/21.02.lua

; stdout: _mlstatus = False
 (at easybuild/tools/modules.py:814 in run_module)
```
The framework scripts involved are `easyblocks/generic/craytoolchain.py` and `tools/module_generator.py`. The workaround is to list Cray external modules using `module unlod/load` commands in `modluafooter` instead of `dependencies`, to avoid the use of the function `swap` in the Lua modulefile created by EasyBuild for the Cray toolchains. 
In the example below I make use of the custom easyblock [cpeToolchain](https://github.com/eth-cscs/production/blob/master/easybuild/easyblocks/cpetoolchain.py) adapted from `craytoolchain.py`:
```
# Compiler toolchain for Cray EX Programming Environment GNU compiler (cpe-gnu)
easyblock = 'cpeToolchain'

name = 'cpeGNU'
version = "21.02"

homepage = 'https://pubs.cray.com'
description = """Toolchain using Cray compiler wrapper with cpe-gnu module (CPE release: %s).\n""" % version

toolchain = SYSTEM

dependencies = [
   ('cpe-gnu', EXTERNAL_MODULE)
]

modluafooter = '''
if not ( isloaded("cray-dsmml/0.1.3") ) then
    load("cray-dsmml/0.1.3")
end
if not ( isloaded("cray-libsci/20.12.1.2") ) then
    load("cray-libsci/20.12.1.2")
end
if not ( isloaded("cray-mpich/8.1.2") ) then
    load("cray-mpich/8.1.2")
end
if not ( isloaded("craype/2.7.5") ) then
    load("craype/2.7.5")
end
if not ( isloaded("craype-x86-rome") ) then
    load("craype-x86-rome")
end
if not ( isloaded("craype-network-ofi") ) then
    load("craype-network-ofi")
end
if not ( isloaded("gcc/10.2.0") ) then
    load("gcc/10.2.0")
end
if not ( isloaded("libfabric/1.11.0.0.233") ) then
    load("libfabric/1.11.0.0.233")
end
if not ( isloaded("perftools-base/21.02.0") ) then
    load("perftools-base/21.02.0")
end
if not ( isloaded("xpmem/2.2.35-7.0.1.0_1.9__gd50fabf.shasta") ) then
    load("xpmem/2.2.35-7.0.1.0_1.9__gd50fabf.shasta")
end
prepend_path("MODULEPATH",root:gsub("software","modules/all/Toolchain"))
'''
moduleclass = 'toolchain'
``` 


Discussion:
I took a quick look at this, and it's easy to reproduce with a simple test module:

```
$ find /tmp/$USER/testmods -type f
/tmp/testmods/swapit/1.0.lua
/tmp/testmods/foo/1.2.lua
/tmp/testmods/foo/1.0.lua

$ cat /tmp/testmods/swapit/1.0.lua
swap("foo", "foo/1.2")
```

This triggers the same error:

```
$ module list

Currently Loaded Modules:
  1) foo/1.0

$ module load swapit
Lmod has detected the following error:  Unable to load module because of error when evaluating modulefile:
     /tmp/testmods/swapit/1.0.lua: [string "swap("foo", "foo/1.2")..."]:1: attempt to call a nil value (global 'swap')
     Please check the modulefile and especially if there is a the line number specified in the above message
While processing the following module(s):
    Module fullname  Module Filename
    ---------------  ---------------
    swapit/1.0       /tmp/testmods/swapit/1.0.lua
```

So apparently the `swap` statement that EasyBuild has been generating since Feb'17 in Lua module files is simply not supported by Lmod at all, and we never noticed?!

@rtmclay Can you confirm that `swap` is not supported in Lua module files (while `module swap` in Tcl module files is)?
The module swap command in TCL modulefiles just does an unload() followed by a load().


Resolving patch:
diff --git a/easybuild/tools/module_generator.py b/easybuild/tools/module_generator.py
--- a/easybuild/tools/module_generator.py
+++ b/easybuild/tools/module_generator.py
@@ -1430,7 +1430,10 @@ def swap_module(self, mod_name_out, mod_name_in, guarded=True):
         :param mod_name_in: name of module to load (swap in)
         :param guarded: guard 'swap' statement, fall back to 'load' if module being swapped out is not loaded
         """
-        body = 'swap("%s", "%s")' % (mod_name_out, mod_name_in)
+        body = '\n'.join([
+            'unload("%s")' % mod_name_out,
+            'load("%s")' % mod_name_in,
+        ])
         if guarded:
             alt_body = self.LOAD_TEMPLATE % {'mod_name': mod_name_in}
             swap_statement = [self.conditional_statement(self.is_loaded(mod_name_out), body, else_body=alt_body)]


Test patch:
diff --git a/test/framework/module_generator.py b/test/framework/module_generator.py
--- a/test/framework/module_generator.py
+++ b/test/framework/module_generator.py
@@ -616,7 +616,8 @@ def test_swap(self):
         else:
             expected = '\n'.join([
                 '',
-                'swap("foo", "bar")',
+                'unload("foo")',
+                'load("bar")',
                 '',
             ])
 
@@ -637,7 +638,8 @@ def test_swap(self):
             expected = '\n'.join([
                 '',
                 'if isloaded("foo") then',
-                '    swap("foo", "bar")',
+                '    unload("foo")',
+                '    load("bar")',
                 'else',
                 '    load("bar")',
                 'end',
@@ -1460,6 +1462,48 @@ def test_det_installdir(self):
 
         self.assertEqual(self.modgen.det_installdir(test_modfile), expected)
 
+    def test_generated_module_file_swap(self):
+        """Test loading a generated module file that includes swap statements."""
+
+        if self.MODULE_GENERATOR_CLASS == ModuleGeneratorLua:
+            mod_ext = '.lua'
+
+            if not isinstance(self.modtool, Lmod):
+                # Lua module files are only supported by Lmod,
+                # so early exit if that's not the case in the test setup
+                return
+
+        else:
+            mod_ext = ''
+
+        # empty test modules
+        for modname in ('one/1.0', 'one/1.1'):
+            modfile = os.path.join(self.test_prefix, modname + mod_ext)
+            write_file(modfile, self.modgen.MODULE_SHEBANG)
+
+        modulepath = os.getenv('MODULEPATH')
+        if modulepath:
+            self.modtool.unuse(modulepath)
+
+        test_mod = os.path.join(self.test_prefix, 'test', '1.0' + mod_ext)
+        test_mod_txt = '\n'.join([
+            self.modgen.MODULE_SHEBANG,
+            self.modgen.swap_module('one', 'one/1.1'),
+        ])
+        write_file(test_mod, test_mod_txt)
+
+        # prepare environment for loading test module
+        self.modtool.use(self.test_prefix)
+        self.modtool.load(['one/1.0'])
+
+        self.modtool.load(['test/1.0'])
+
+        # check whether resulting environment is correct
+        loaded_mods = self.modtool.list()
+        self.assertEqual(loaded_mods[-1]['mod_name'], 'test/1.0')
+        # one/1.0 module was swapped for one/1.1
+        self.assertEqual(loaded_mods[-2]['mod_name'], 'one/1.1')
+
 
 class TclModuleGeneratorTest(ModuleGeneratorTest):
     """Test for module_generator module for Tcl syntax."""