Repository: easybuilders/easybuild-framework

Issue:
Possible bug with eb --list--toolchains in v1.6? kindly verify
Hi,

it could well be a configuration issue on our end, yet the following tells likely it's not:

```
fgeorgatos@gaia-59:~: $ module load HPCBIOS/20130601 # define MODULEPATH
fgeorgatos@gaia-59:~: $ module av Easy

---------------------------- /opt/apps/HPCBIOS.20130601/modules/all ----------------------------
EasyBuild/1.5.0 EasyBuild/1.6.0
fgeorgatos@gaia-59:~: $ module load EasyBuild
fgeorgatos@gaia-59:~: $ eb --list-toolchains
Traceback (most recent call last):
  File "/opt/apps/HPCBIOS.20130601/software/EasyBuild/1.6.0/lib/python2.6/site-packages/easybuild_framework-1.6.0-py2.6.egg/easybuild/main.py", line 1295, in <module>
    main()
  File "/opt/apps/HPCBIOS.20130601/software/EasyBuild/1.6.0/lib/python2.6/site-packages/easybuild_framework-1.6.0-py2.6.egg/easybuild/main.py", line 122, in main
    eb_go = eboptions.parse_options(args=args)
  File "/opt/apps/HPCBIOS.20130601/software/EasyBuild/1.6.0/lib/python2.6/site-packages/easybuild_framework-1.6.0-py2.6.egg/easybuild/tools/options.py", line 540, in parse_options
    go_args=args,
  File "/opt/apps/HPCBIOS.20130601/software/EasyBuild/1.6.0/lib/python2.6/site-packages/easybuild_framework-1.6.0-py2.6.egg/vsc/utils/generaloption.py", line 615, in __init__
    self.postprocess()
  File "/opt/apps/HPCBIOS.20130601/software/EasyBuild/1.6.0/lib/python2.6/site-packages/easybuild_framework-1.6.0-py2.6.egg/easybuild/tools/options.py", line 317, in postprocess
    self._postprocess_list_avail()
  File "/opt/apps/HPCBIOS.20130601/software/EasyBuild/1.6.0/lib/python2.6/site-packages/easybuild_framework-1.6.0-py2.6.egg/easybuild/tools/options.py", line 364, in _postprocess_list_avail
    msg += self.avail_toolchains()
  File "/opt/apps/HPCBIOS.20130601/software/EasyBuild/1.6.0/lib/python2.6/site-packages/easybuild_framework-1.6.0-py2.6.egg/easybuild/tools/options.py", line 483, in avail_toolchains
    tc = tcc(version='1.2.3')  # version doesn't matter here, but something needs to be there
  File "/opt/apps/HPCBIOS.20130601/software/EasyBuild/1.6.0/lib/python2.6/site-packages/easybuild_framework-1.6.0-py2.6.egg/easybuild/tools/toolchain/compiler.py", line 131, in __init__
    super(Compiler, self).__init__(*args, **kwargs)
  File "/opt/apps/HPCBIOS.20130601/software/EasyBuild/1.6.0/lib/python2.6/site-packages/easybuild_framework-1.6.0-py2.6.egg/easybuild/tools/toolchain/toolchain.py", line 88, in __init__
    self.modules_tool = modules_tool()
  File "/opt/apps/HPCBIOS.20130601/software/EasyBuild/1.6.0/lib/python2.6/site-packages/easybuild_framework-1.6.0-py2.6.egg/easybuild/tools/modules.py", line 606, in modules_tool
    modules_tool_class = avail_modules_tools().get(get_modules_tool())
  File "/opt/apps/HPCBIOS.20130601/software/EasyBuild/1.6.0/lib/python2.6/site-packages/easybuild_framework-1.6.0-py2.6.egg/easybuild/tools/config.py", line 395, in get_modules_tool
    return variables['modules_tool']
  File "/opt/apps/HPCBIOS.20130601/software/EasyBuild/1.6.0/lib/python2.6/site-packages/easybuild_framework-1.6.0-py2.6.egg/easybuild/tools/config.py", line 155, in __getitem__
    return super(ConfigurationVariables, self).__getitem__(self._check_oldstyle(key))
KeyError: 'modules_tool'
fgeorgatos@gaia-59:~: $ module swap EasyBuild EasyBuild/1.5.0
fgeorgatos@gaia-59:~: $ eb --list-toolchains
List of known toolchains (toolchainname: module[,module...]):
    ClangGCC: Clang, GCC
    GCC: GCC
    cgmpich: ClangGCC, MPICH
    cgmpolf: BLACS, ClangGCC, FFTW, MPICH, OpenBLAS, ScaLAPACK
    cgmvapich2: ClangGCC, MVAPICH2
    cgmvolf: BLACS, ClangGCC, FFTW, MVAPICH2, OpenBLAS, ScaLAPACK
    cgompi: ClangGCC, OpenMPI
    cgoolf: BLACS, ClangGCC, FFTW, OpenBLAS, OpenMPI, ScaLAPACK
    dummy:
    gcccuda: CUDA, GCC
    gimkl: GCC, imkl, impi
    gmacml: ACML, BLACS, FFTW, GCC, MVAPICH2, ScaLAPACK
    gmpich2: GCC, MPICH2
    gmvapich2: GCC, MVAPICH2
    gmvolf: BLACS, FFTW, GCC, MVAPICH2, OpenBLAS, ScaLAPACK
    goalf: ATLAS, BLACS, FFTW, GCC, OpenMPI, ScaLAPACK
    gompi: GCC, OpenMPI
    goolf: BLACS, FFTW, GCC, OpenBLAS, OpenMPI, ScaLAPACK
    goolfc: BLACS, CUDA, FFTW, GCC, OpenBLAS, OpenMPI, ScaLAPACK
    gqacml: ACML, BLACS, FFTW, GCC, QLogicMPI, ScaLAPACK
    iccifort: icc, ifort
    ictce: icc, ifort, imkl, impi
    iiqmpi: QLogicMPI, icc, ifort
    iomkl: OpenMPI, icc, ifort, imkl
    iqacml: ACML, BLACS, FFTW, QLogicMPI, ScaLAPACK, icc, ifort
    ismkl: MPICH2, icc, ifort, imkl
fgeorgatos@gaia-59:~: $
```

Perhaps during code addition to support lmod over v1.6, this modules_tool fiddled with the mentioned feature.

Also, when I look at the very last line mentioned in the stack trace, I notice code like the following:

```
def get_user_easybuild_dir():
    """Return the per-user easybuild dir (e.g. to store config files)"""
    return os.path.join(os.path.expanduser('~'), ".easybuild")
```

A hint I have here is: `~` does not always expand as nicely as `$HOME` (weird, hm?), therefore the latter should be preferred for stability in a piece of code; I know because I had found out the hard way, just don't remember now the circumstances!

If you tell us if this works or not upstream, that's already a bit of progress for our end,
because we've tried it on 3 different systems with the same result...

tia,
Fotis



Discussion:
Confirmed, I can easily reproduce this.

Will fix, PR coming up!


Resolving patch:
diff --git a/easybuild/tools/config.py b/easybuild/tools/config.py
--- a/easybuild/tools/config.py
+++ b/easybuild/tools/config.py
@@ -392,7 +392,8 @@ def get_modules_tool():
     """
     Return modules tool (EnvironmentModulesC, Lmod, ...)
     """
-    return variables['modules_tool']
+    # 'modules_tool' key will only be present if EasyBuild config is initialized
+    return variables.get('modules_tool', None)
 
 
 def log_file_format(return_directory=False):
diff --git a/easybuild/tools/modules.py b/easybuild/tools/modules.py
--- a/easybuild/tools/modules.py
+++ b/easybuild/tools/modules.py
@@ -599,12 +599,18 @@ def avail_modules_tools():
         del class_dict['Modules']
     return class_dict
 
+
 def modules_tool(mod_paths=None):
     """
     Return interface to modules tool (environment modules, lmod)
     """
-    modules_tool_class = avail_modules_tools().get(get_modules_tool())
-    return modules_tool_class(mod_paths=mod_paths)
+    # get_modules_tool might return none (e.g. if config was not initialized yet)
+    modules_tool = get_modules_tool()
+    if modules_tool is not None:
+        modules_tool_class = avail_modules_tools().get(modules_tool)
+        return modules_tool_class(mod_paths=mod_paths)
+    else:
+        return None
 
 
 # provide Modules class for backward compatibility (e.g., in easyblocks)
diff --git a/easybuild/tools/options.py b/easybuild/tools/options.py
--- a/easybuild/tools/options.py
+++ b/easybuild/tools/options.py
@@ -43,9 +43,9 @@
 from easybuild.framework.easyconfig.templates import template_documentation
 from easybuild.framework.easyconfig.tools import get_paths_for
 from easybuild.framework.extension import Extension
+from easybuild.tools import config, filetools  # @UnusedImport make sure config is always initialized!
 from easybuild.tools.config import get_default_configfiles, get_pretend_installpath
 from easybuild.tools.config import get_default_oldstyle_configfile_defaults, DEFAULT_MODULECLASSES
-from easybuild.tools import filetools
 from easybuild.tools.modules import avail_modules_tools
 from easybuild.tools.ordereddict import OrderedDict
 from easybuild.tools.toolchain.utilities import search_toolchain


Test patch:
diff --git a/test/framework/easyconfig.py b/test/framework/easyconfig.py
--- a/test/framework/easyconfig.py
+++ b/test/framework/easyconfig.py
@@ -53,13 +53,13 @@ class EasyConfigTest(TestCase):
     contents = None
     eb_file = ''
 
-    # initialize configuration so config.get_modules_tool function works
-    eb_go = eboptions.parse_options()
-    config.init(eb_go.options, eb_go.get_options_by_section('config'))
-    del eb_go
-
     def setUp(self):
-        """ create temporary easyconfig file """
+        """Set up everything for running a unit test."""
+
+        # initialize configuration so config.get_modules_tool function works
+        eb_go = eboptions.parse_options()
+        config.init(eb_go.options, eb_go.get_options_by_section('config'))
+
         self.log = fancylogger.getLogger("EasyConfigTest", fname=False)
         self.cwd = os.getcwd()
         self.all_stops = [x[0] for x in EasyBlock.get_steps()]
diff --git a/test/framework/modules.py b/test/framework/modules.py
--- a/test/framework/modules.py
+++ b/test/framework/modules.py
@@ -40,15 +40,14 @@
 
 
 class ModulesTest(TestCase):
-    """ small test for Modules """
-
-    # initialize configuration so modules_tool function works
-    eb_go = eboptions.parse_options()
-    config.init(eb_go.options, eb_go.get_options_by_section('config'))
-    del eb_go
+    """Test cases for modules."""
 
     def setUp(self):
-        """setup"""
+        """set up everything for a unit test."""
+        # initialize configuration so config.get_modules_tool function works
+        eb_go = eboptions.parse_options()
+        config.init(eb_go.options, eb_go.get_options_by_section('config'))
+
         self.cwd = os.getcwd()
 
     def test_load(self):
diff --git a/test/framework/options.py b/test/framework/options.py
--- a/test/framework/options.py
+++ b/test/framework/options.py
@@ -376,7 +376,9 @@ def run_test(custom=None, extra_params=[]):
         # restore original Python search path
         sys.path = orig_sys_path
 
-    def test_list_toolchains(self):
+    # double underscore to make sure it runs first, which is required to detect certain types of bugs,
+    # e.g. running with non-initialized EasyBuild config (truly mimicing 'eb --list-toolchains')
+    def test__list_toolchains(self):
         """Test listing known compiler toolchains."""
 
         fd, dummylogfn = tempfile.mkstemp(prefix='easybuild-dummy', suffix='.log')
diff --git a/test/framework/robot.py b/test/framework/robot.py
--- a/test/framework/robot.py
+++ b/test/framework/robot.py
@@ -60,13 +60,12 @@ def mock_module(mod_paths=None):
 class RobotTest(TestCase):
     """ Testcase for the robot dependency resolution """
 
-    # initialize configuration so get_modules_tool function works
-    eb_go = eboptions.parse_options()
-    config.init(eb_go.options, eb_go.get_options_by_section('config'))
-    del eb_go
-
     def setUp(self):
-        """ dynamically replace Modules class with MockModule """
+        """Set up everything for a unit test."""
+        # initialize configuration so config.get_modules_tool function works
+        eb_go = eboptions.parse_options()
+        config.init(eb_go.options, eb_go.get_options_by_section('config'))
+
         # replace Modules class with something we have control over
         config.modules_tool = mock_module
         main.modules_tool = mock_module
diff --git a/test/framework/suite.py b/test/framework/suite.py
--- a/test/framework/suite.py
+++ b/test/framework/suite.py
@@ -25,7 +25,7 @@
 # #
 """
 This script is a collection of all the testcases.
-Usage: "python -m easybuild.test.suite.py" or "./easybuild/test/suite.py"
+Usage: "python -m test.framework.suite" or "python test/framework/suite.py"
 
 @author: Toon Willems (Ghent University)
 @author: Kenneth Hoste (Ghent University)
@@ -39,19 +39,19 @@
 
 # toolkit should be first to allow hacks to work
 import test.framework.asyncprocess as a
+import test.framework.config as c
+import test.framework.easyblock as b
 import test.framework.easyconfig as e
+import test.framework.filetools as f
+import test.framework.github as g
 import test.framework.modulegenerator as mg
 import test.framework.modules as m
-import test.framework.filetools as f
+import test.framework.options as o
 import test.framework.repository as r
 import test.framework.robot as robot
-import test.framework.easyblock as b
-import test.framework.variables as v
-import test.framework.github as g
-import test.framework.toolchainvariables as tcv
 import test.framework.toolchain as tc
-import test.framework.options as o
-import test.framework.config as c
+import test.framework.toolchainvariables as tcv
+import test.framework.variables as v
 
 
 # initialize logger for all the unit tests
@@ -63,7 +63,8 @@
 log.setLevelName('DEBUG')
 
 # call suite() for each module and then run them all
-SUITE = unittest.TestSuite([x.suite() for x in [r, e, mg, m, f, a, robot, b, v, g, tcv, tc, o, c]])
+# note: make sure the options unit tests run first, to avoid running some of them with a readily initialized config
+SUITE = unittest.TestSuite([x.suite() for x in [o, r, e, mg, m, f, a, robot, b, v, g, tcv, tc, c]])
 
 # uses XMLTestRunner if possible, so we can output an XML file that can be supplied to Jenkins
 xml_msg = ""
diff --git a/test/framework/toolchain.py b/test/framework/toolchain.py
--- a/test/framework/toolchain.py
+++ b/test/framework/toolchain.py
@@ -42,11 +42,6 @@
 class ToolchainTest(TestCase):
     """ Baseclass for toolchain testcases """
 
-    # initialize configuration so config.get_modules_tool function works
-    eb_go = eboptions.parse_options()
-    config.init(eb_go.options, eb_go.get_options_by_section('config'))
-    del eb_go
-
     def assertErrorRegex(self, error, regex, call, *args):
         """ convenience method to match regex with the error message """
         try:
@@ -55,7 +50,11 @@ def assertErrorRegex(self, error, regex, call, *args):
             self.assertTrue(re.search(regex, err.msg))
 
     def setUp(self):
-        """Setup for tests."""
+        """Set up everything for a unit test."""
+        # initialize configuration so config.get_modules_tool function works
+        eb_go = eboptions.parse_options()
+        config.init(eb_go.options, eb_go.get_options_by_section('config'))
+
         # make sure path with modules for testing is added to MODULEPATH
         self.orig_modpath = os.environ.get('MODULEPATH', '')
         os.environ['MODULEPATH'] = find_full_path(os.path.join('test', 'framework', 'modules'))