Repository: easybuilders/easybuild-framework

Issue:
hierarchical module naming scheme, apps loading compilers and mpi
Applications and libraries incorrectly loads:
modules/all> grep -E "GCC|MPI" -R . | grep load
./Core/gompi/2.9.11:module load GCC/4.9.1
./Core/gompi/2.9.11:module load OpenMPI/1.8.1
./Core/goolf/2.9.11:module load GCC/4.9.1
./Core/goolf/2.9.11:module load OpenMPI/1.8.1
./MPI/GCC/4.9.1/OpenMPI/1.8.1/FFTW/3.3.4:module load GCC/4.9.1
./MPI/GCC/4.9.1/OpenMPI/1.8.1/FFTW/3.3.4:module load OpenMPI/1.8.1
./MPI/GCC/4.9.1/OpenMPI/1.8.1/ScaLAPACK/2.0.2:module load GCC/4.9.1
./MPI/GCC/4.9.1/OpenMPI/1.8.1/ScaLAPACK/2.0.2:module load OpenMPI/1.8.1
./MPI/GCC/4.9.1/OpenMPI/1.8.1/Python/2.7.8:module load GCC/4.9.1
./MPI/GCC/4.9.1/OpenMPI/1.8.1/Python/2.7.8:module load OpenMPI/1.8.1

hierarchical mns needs to be extended to understand that Compiler/MPI traverses the hierarchy.

This creates modules not compatible with Lmod hierarchy.



Discussion:
@olavks: the loads for GCC and MPI in the `gompi` and `goolf` toolchains are OK, although it doesn't make a lot of sense to still have toolchain modules aren't in a hierarchy. 
The ones in the 'application' modules (FFTW, ScaLAPACK, Python) are an issue though.

Can you outline the kind of issues you've run into w.r.t. unloading application modules like this with Lmod?

Yes, so just to reiterate from chat:
so only the toolchain itself should have a load, the apps should not.
so compiler and mpi depdendencies must be seperated from library dependencies

boegel said:
the problem is that unloading e.g. Python will also trigger an unload of GCC/OpenMPI.
 thus screwing up your $MODULEPATH.
which also means swapping is broken, which is one of the major advantages of a HMNS.
right now, EB 'unrolls' the "module load goolf" that is usually included.
using a "load goolf" doesn't make sense with a HMNS, since that triggers an icc/impi load => same problem.in fact, avoiding this situation is part of why we added the unroll support, but we failed to handle this particular issue.
we need to query the tc being used for compiler/MPI, determine the (short) module names for those, and filter them out.

you only load short modules, so doing a swap triggers a reload of the short modules since they change. and you swap out the compiler for all subsequent modules -- that's the cool part that is desired.

libraries / apps loading a _specific_ compiler module ruins that feature.

See #1016 for a fix. I'm closing, this should be discussed further in #1016.


Resolving patch:
diff --git a/easybuild/framework/easyblock.py b/easybuild/framework/easyblock.py
--- a/easybuild/framework/easyblock.py
+++ b/easybuild/framework/easyblock.py
@@ -57,7 +57,7 @@
 from easybuild.tools.build_log import EasyBuildError, print_error, print_msg
 from easybuild.tools.config import build_option, build_path, get_log_filename, get_repository, get_repositorypath
 from easybuild.tools.config import install_path, log_path, read_only_installdir, source_paths
-from easybuild.tools.environment import modify_env
+from easybuild.tools.environment import restore_env
 from easybuild.tools.filetools import DEFAULT_CHECKSUM
 from easybuild.tools.filetools import adjust_permissions, apply_patch, convert_name, download_file, encode_class_name
 from easybuild.tools.filetools import extract_file, mkdir, read_file, rmtree2
@@ -997,7 +997,7 @@ def clean_up_fake_module(self, fake_mod_data):
             self.log.warning("Not unloading module, since self.full_mod_name is not set.")
 
         # restore original environment
-        modify_env(os.environ, orig_env)
+        restore_env(orig_env)
 
     def load_dependency_modules(self):
         """Load dependency modules."""
@@ -1851,7 +1851,7 @@ def build_and_install_one(module, orig_environ):
     # restore original environment
     _log.info("Resetting environment")
     filetools.errors_found_in_log = 0
-    modify_env(os.environ, orig_environ)
+    restore_env(orig_environ)
 
     cwd = os.getcwd()
 
@@ -2054,7 +2054,7 @@ def perform_step(step, obj, method, logfile):
 
             # start with a clean slate
             os.chdir(base_dir)
-            modify_env(os.environ, base_env)
+            restore_env(base_env)
 
             steps = EasyBlock.get_steps(iteration_count=app.det_iter_cnt())
 
diff --git a/easybuild/tools/environment.py b/easybuild/tools/environment.py
--- a/easybuild/tools/environment.py
+++ b/easybuild/tools/environment.py
@@ -102,7 +102,6 @@ def restore_env_vars(env_keys):
     """
     Restore the environment by setting the keys in the env_keys dict again with their old value
     """
-
     for key in env_keys:
         if env_keys[key] is not None:
             _log.info("Restoring environment variable %s (value: %s)" % (key, env_keys[key]))
@@ -150,3 +149,10 @@ def modify_env(old, new):
             _log.debug("Key in old environment found that is not in new one: %s (%s)" % (key, old[key]))
             os.unsetenv(key)
             del os.environ[key]
+
+
+def restore_env(env):
+    """
+    Restore active environment based on specified dictionary.
+    """
+    modify_env(os.environ, env)
diff --git a/easybuild/tools/modules.py b/easybuild/tools/modules.py
--- a/easybuild/tools/modules.py
+++ b/easybuild/tools/modules.py
@@ -47,7 +47,7 @@
 
 from easybuild.tools.build_log import EasyBuildError
 from easybuild.tools.config import build_option, get_modules_tool, install_path
-from easybuild.tools.environment import modify_env
+from easybuild.tools.environment import restore_env
 from easybuild.tools.filetools import convert_name, mkdir, read_file, path_matches, which
 from easybuild.tools.module_naming_scheme import DEVEL_MODULE_SUFFIX
 from easybuild.tools.run import run_cmd
@@ -377,7 +377,7 @@ def load(self, modules, mod_paths=None, purge=False, orig_env=None):
             self.purge()
             # restore original environment if provided
             if orig_env is not None:
-                modify_env(os.environ, orig_env)
+                restore_env(orig_env)
 
         # make sure $MODULEPATH is set correctly after purging
         self.check_module_path()
@@ -590,6 +590,8 @@ def modpath_extensions_for(self, mod_names):
         Determine dictionary with $MODULEPATH extensions for specified modules.
         Modules with an empty list of $MODULEPATH extensions are included.
         """
+        self.log.debug("Determining $MODULEPATH extensions for modules %s" % mod_names)
+
         # copy environment so we can restore it
         orig_env = os.environ.copy()
 
@@ -599,6 +601,7 @@ def modpath_extensions_for(self, mod_names):
             useregex = re.compile(r"^\s*module\s+use\s+(\S+)", re.M)
             exts = useregex.findall(modtxt)
 
+            self.log.debug("Found $MODULEPATH extensions for %s: %s" % (mod_name, exts))
             modpath_exts.update({mod_name: exts})
 
             if exts:
@@ -607,7 +610,7 @@ def modpath_extensions_for(self, mod_names):
                 self.load([mod_name])
 
         # restore original environment (modules may have been loaded above)
-        modify_env(os.environ, orig_env)
+        restore_env(orig_env)
 
         return modpath_exts
 
@@ -656,7 +659,8 @@ def path_to_top_of_module_tree(self, top_paths, mod_name, full_mod_subdir, deps,
             modpath_exts = dict([(k, v) for k, v in self.modpath_extensions_for(deps).items() if v])
             self.log.debug("Non-empty lists of module path extensions for dependencies: %s" % modpath_exts)
 
-        path = []
+        mods_to_top = []
+        full_mod_subdirs = []
         for dep in modpath_exts:
             # if a $MODULEPATH extension is identical to where this module will be installed, we have a hit
             # use os.path.samefile when comparing paths to avoid issues with resolved symlinks
@@ -664,32 +668,32 @@ def path_to_top_of_module_tree(self, top_paths, mod_name, full_mod_subdir, deps,
             if path_matches(full_mod_subdir, full_modpath_exts):
                 # full path to module subdir of dependency is simply path to module file without (short) module name
                 full_mod_subdir = self.modulefile_path(dep)[:-len(dep)-1]
+                full_mod_subdirs.append(full_mod_subdir)
 
-                path.append(dep)
+                mods_to_top.append(dep)
                 tup = (dep, full_mod_subdir, full_modpath_exts)
                 self.log.debug("Found module to top of module tree: %s (subdir: %s, modpath extensions %s)" % tup)
 
-                # no need to continue further, we found the module that extends $MODULEPATH with module subdir
-                break
-
             if full_modpath_exts:
                 # load module for this dependency, since it may extend $MODULEPATH to make dependencies available
                 # this is required to obtain the corresponding module file paths (via 'module show')
                 self.load([dep])
 
-        if path:
-            # remove retained dependency from the list, since we're climbing up the module tree
-            modpath_exts.pop(path[-1])
+        # restore original environment (modules may have been loaded above)
+        restore_env(orig_env)
+
+        path = mods_to_top[:]
+        if mods_to_top:
+            # remove retained dependencies from the list, since we're climbing up the module tree
+            remaining_modpath_exts = dict([m for m in modpath_exts.items() if not m[0] in mods_to_top])
 
-            self.log.debug("Path to top from %s extended to %s, so recursing to find way to the top" % (mod_name, path))
-            path.extend(self.path_to_top_of_module_tree(top_paths, path[-1], full_mod_subdir, None,
-                                                        modpath_exts=modpath_exts))
+            self.log.debug("Path to top from %s extended to %s, so recursing to find way to the top" % (mod_name, mods_to_top))
+            for mod_name, full_mod_subdir in zip(mods_to_top, full_mod_subdirs):
+                path.extend(self.path_to_top_of_module_tree(top_paths, mod_name, full_mod_subdir, None,
+                                                            modpath_exts=remaining_modpath_exts))
         else:
             self.log.debug("Path not extended, we must have reached the top of the module tree")
 
-        # restore original environment (modules may have been loaded above)
-        modify_env(os.environ, orig_env)
-
         self.log.debug("Path to top of module tree from %s: %s" % (mod_name, path))
         return path
 


Test patch:
diff --git a/test/framework/easyblock.py b/test/framework/easyblock.py
--- a/test/framework/easyblock.py
+++ b/test/framework/easyblock.py
@@ -44,7 +44,9 @@
 from easybuild.framework.extensioneasyblock import ExtensionEasyBlock
 from easybuild.tools import config
 from easybuild.tools.build_log import EasyBuildError
-from easybuild.tools.filetools import mkdir, write_file
+from easybuild.tools.environment import modify_env
+from easybuild.tools.filetools import mkdir, read_file, write_file
+from easybuild.tools.modules import modules_tool
 
 
 class EasyBlockTest(EnhancedTestCase):
@@ -465,6 +467,47 @@ def test_check_readiness(self):
 
         shutil.rmtree(tmpdir)
 
+    def test_exclude_path_to_top_of_module_tree(self):
+        """
+        Make sure that modules under the HierarchicalMNS are correct,
+        w.r.t. not including any load statements for modules that build up the path to the top of the module tree.
+        """
+        self.orig_module_naming_scheme = config.get_module_naming_scheme()
+        test_ecs_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'easyconfigs')
+        all_stops = [x[0] for x in EasyBlock.get_steps()]
+        build_options = {
+            'check_osdeps': False,
+            'robot_path': [test_ecs_path],
+            'valid_stops': all_stops,
+            'validate': False,
+        }
+        os.environ['EASYBUILD_MODULE_NAMING_SCHEME'] = 'HierarchicalMNS'
+        init_config(build_options=build_options)
+        self.setup_hierarchical_modules()
+        modtool = modules_tool()
+
+        ec = EasyConfig(os.path.join(test_ecs_path, 'imkl-11.1.2.144-iimpi-5.5.3-GCC-4.8.3.eb'))
+        eb = EasyBlock(ec)
+
+        modfile_prefix = os.path.join(self.test_installpath, 'modules', 'all')
+        mkdir(os.path.join(modfile_prefix, 'Compiler', 'GCC', '4.8.3'), parents=True)
+        mkdir(os.path.join(modfile_prefix, 'MPI', 'intel', '2013.5.192', 'impi', '4.1.3.049'), parents=True)
+        eb.toolchain.prepare()
+        modpath = eb.make_module_step()
+        modfile_path = os.path.join(modpath, 'MPI', 'intel', '2013.5.192', 'impi', '4.1.3.049', 'imkl', '11.1.2.144')
+        modtxt = read_file(modfile_path)
+
+        # for imkl on top of iimpi toolchain with HierarchicalMNS, no module load statements should be included at all
+        # not for the toolchain or any of the toolchain components,
+        # since both icc/ifort and impi form the path to the top of the module tree
+        for imkl_dep in ['icc', 'ifort', 'impi', 'iccifort', 'iimpi']:
+            tup = (imkl_dep, modfile_path, modtxt)
+            failmsg = "No 'module load' statement found for '%s' not found in module %s: %s" % tup
+            self.assertFalse(re.search("module load %s" % imkl_dep, modtxt), failmsg)
+
+        os.environ['EASYBUILD_MODULE_NAMING_SCHEME'] = self.orig_module_naming_scheme
+        init_config(build_options=build_options)
+
     def tearDown(self):
         """ make sure to remove the temporary file """
         super(EasyBlockTest, self).tearDown()
diff --git a/test/framework/easyconfigs/GCC-4.8.3.eb b/test/framework/easyconfigs/GCC-4.8.3.eb
new file mode 100644
--- /dev/null
+++ b/test/framework/easyconfigs/GCC-4.8.3.eb
@@ -0,0 +1,28 @@
+name = "GCC"
+version = '4.8.3'
+
+homepage = 'http://gcc.gnu.org/'
+description = """The GNU Compiler Collection includes front ends for C, C++, Objective-C, Fortran, Java, and Ada,
+ as well as libraries for these languages (libstdc++, libgcj,...)."""
+
+toolchain = {'name': 'dummy', 'version': 'dummy'}
+
+source_urls = [
+    'http://ftpmirror.gnu.org/%(namelower)s/%(namelower)s-%(version)s',  # GCC auto-resolving HTTP mirror
+    'http://ftpmirror.gnu.org/gmp',  # idem for GMP
+    'http://ftpmirror.gnu.org/mpfr',  # idem for MPFR
+    'http://www.multiprecision.org/mpc/download',  # MPC official
+]
+sources = [
+    SOURCELOWER_TAR_GZ,
+    'gmp-5.1.3.tar.bz2',
+    'mpfr-3.1.2.tar.gz',
+    'mpc-1.0.1.tar.gz',
+]
+
+languages = ['c', 'c++', 'fortran', 'lto']
+
+# building GCC sometimes fails if make parallelism is too high, so let's limit it
+maxparallel = 4
+
+moduleclass = 'compiler'
diff --git a/test/framework/easyconfigs/icc-2013.5.192-GCC-4.8.3.eb b/test/framework/easyconfigs/icc-2013.5.192-GCC-4.8.3.eb
new file mode 100644
--- /dev/null
+++ b/test/framework/easyconfigs/icc-2013.5.192-GCC-4.8.3.eb
@@ -0,0 +1,23 @@
+name = 'icc'
+version = '2013.5.192'
+
+homepage = 'http://software.intel.com/en-us/intel-compilers/'
+description = "C and C++ compiler from Intel"
+
+toolchain = {'name': 'dummy', 'version': 'dummy'}
+
+sources = ['l_ccompxe_%(version)s.tgz']
+
+gcc = 'GCC'
+gccver = '4.8.3'
+versionsuffix = '-%s-%s' % (gcc, gccver)
+
+dependencies = [(gcc, gccver)]
+
+dontcreateinstalldir = 'True'
+
+# license file
+import os
+license_file = os.path.join(os.getenv('HOME'), "licenses", "intel", "license.lic")
+
+moduleclass = 'compiler'
diff --git a/test/framework/easyconfigs/iccifort-2013.5.192-GCC-4.8.3.eb b/test/framework/easyconfigs/iccifort-2013.5.192-GCC-4.8.3.eb
new file mode 100644
--- /dev/null
+++ b/test/framework/easyconfigs/iccifort-2013.5.192-GCC-4.8.3.eb
@@ -0,0 +1,17 @@
+easyblock = "Toolchain"
+
+name = 'iccifort'
+version = '2013.5.192'
+versionsuffix = '-GCC-4.8.3'
+
+homepage = 'http://software.intel.com/en-us/intel-cluster-toolkit-compiler/'
+description = """Intel C, C++ and Fortran compilers"""
+
+toolchain = {'name': 'dummy', 'version': 'dummy'}
+
+dependencies = [
+    ('icc', version, versionsuffix),
+    ('ifort', version, versionsuffix),
+]
+
+moduleclass = 'toolchain'
diff --git a/test/framework/easyconfigs/ifort-2013.5.192-GCC-4.8.3.eb b/test/framework/easyconfigs/ifort-2013.5.192-GCC-4.8.3.eb
new file mode 100644
--- /dev/null
+++ b/test/framework/easyconfigs/ifort-2013.5.192-GCC-4.8.3.eb
@@ -0,0 +1,23 @@
+name = 'ifort'
+version = '2013.5.192'
+
+homepage = 'http://software.intel.com/en-us/intel-compilers/'
+description = "Fortran compiler from Intel"
+
+toolchain = {'name': 'dummy', 'version': 'dummy'}
+
+sources = ['l_fcompxe_%(version)s.tgz']
+
+gcc = 'GCC'
+gccver = '4.8.3'
+versionsuffix = '-%s-%s' % (gcc, gccver)
+
+dependencies = [(gcc, gccver)]
+
+dontcreateinstalldir = 'True'
+
+# license file
+import os
+license_file = os.path.join(os.getenv('HOME'), "licenses", "intel", "license.lic")
+
+moduleclass = 'compiler'
diff --git a/test/framework/easyconfigs/iimpi-5.5.3-GCC-4.8.3.eb b/test/framework/easyconfigs/iimpi-5.5.3-GCC-4.8.3.eb
new file mode 100644
--- /dev/null
+++ b/test/framework/easyconfigs/iimpi-5.5.3-GCC-4.8.3.eb
@@ -0,0 +1,21 @@
+easyblock = "Toolchain"
+
+name = 'iimpi'
+version = '5.5.3'
+versionsuffix = '-GCC-4.8.3'
+
+homepage = 'http://software.intel.com/en-us/intel-cluster-toolkit-compiler/'
+description = """Intel C/C++ and Fortran compilers, alongside Intel MPI."""
+
+toolchain = {'name': 'dummy', 'version': 'dummy'}
+
+suff = '5.192'
+compver = '2013.%s' % suff
+
+dependencies = [                                                                 # version/released
+    ('icc', compver, versionsuffix),                                             # 28 Apr 2014
+    ('ifort', compver, versionsuffix),                                           # 28 Apr 2014
+    ('impi', '4.1.3.049', '', ('iccifort', '%s%s' % (compver, versionsuffix))),  # 06 Mar 2014
+]
+
+moduleclass = 'toolchain'
diff --git a/test/framework/easyconfigs/imkl-11.1.2.144-iimpi-5.5.3-GCC-4.8.3.eb b/test/framework/easyconfigs/imkl-11.1.2.144-iimpi-5.5.3-GCC-4.8.3.eb
new file mode 100644
--- /dev/null
+++ b/test/framework/easyconfigs/imkl-11.1.2.144-iimpi-5.5.3-GCC-4.8.3.eb
@@ -0,0 +1,22 @@
+name = 'imkl'
+version = '11.1.2.144'
+
+homepage = 'http://software.intel.com/en-us/intel-mkl/'
+description = """Intel Math Kernel Library is a library of highly optimized,
+ extensively threaded math routines for science, engineering, and financial
+ applications that require maximum performance. Core math functions include
+ BLAS, LAPACK, ScaLAPACK, Sparse Solvers, Fast Fourier Transforms, Vector Math, and more."""
+
+toolchain = {'name': 'iimpi', 'version': '5.5.3-GCC-4.8.3'}
+
+sources = ['l_mkl_%(version)s.tgz']
+
+dontcreateinstalldir = 'True'
+
+interfaces = True
+
+# license file
+import os
+license_file = os.path.join(os.getenv('HOME'), "licenses", "intel", "license.lic")
+
+moduleclass = 'numlib'
diff --git a/test/framework/easyconfigs/impi-4.1.3.049-iccifort-2013.5.192-GCC-4.8.3.eb b/test/framework/easyconfigs/impi-4.1.3.049-iccifort-2013.5.192-GCC-4.8.3.eb
new file mode 100644
--- /dev/null
+++ b/test/framework/easyconfigs/impi-4.1.3.049-iccifort-2013.5.192-GCC-4.8.3.eb
@@ -0,0 +1,19 @@
+name = 'impi'
+version = '4.1.3.049'
+
+homepage = 'http://software.intel.com/en-us/intel-mpi-library/'
+description = """The Intel(R) MPI Library for Linux* OS is a multi-fabric message
+ passing library based on ANL MPICH2 and OSU MVAPICH2. The Intel MPI Library for
+ Linux OS implements the Message Passing Interface, version 2 (MPI-2) specification."""
+
+toolchain = {'name': 'iccifort', 'version': '2013.5.192-GCC-4.8.3'}
+
+sources = ['l_mpi_p_%(version)s.tgz']
+
+dontcreateinstalldir = 'True'
+
+# license file
+import os
+license_file = os.path.join(os.getenv('HOME'), "licenses", "intel", "license.lic")
+
+moduleclass = 'mpi'
diff --git a/test/framework/module_generator.py b/test/framework/module_generator.py
--- a/test/framework/module_generator.py
+++ b/test/framework/module_generator.py
@@ -419,6 +419,7 @@ def test_ec(ecfile, short_modname, mod_subdir, modpath_exts, init_modpaths):
         for ecfile, mns_vals in test_ecs.items():
             test_ec(ecfile, *mns_vals)
 
+
 def suite():
     """ returns all the testcases in this module """
     return TestLoader().loadTestsFromTestCase(ModuleGeneratorTest)
diff --git a/test/framework/modules.py b/test/framework/modules.py
--- a/test/framework/modules.py
+++ b/test/framework/modules.py
@@ -43,7 +43,7 @@
 
 
 # number of modules included for testing purposes
-TEST_MODULES_COUNT = 44
+TEST_MODULES_COUNT = 50
 
 
 class ModulesTest(EnhancedTestCase):
@@ -283,6 +283,7 @@ def test_path_to_top_of_module_tree(self):
         path = modtool.path_to_top_of_module_tree(init_modpaths, 'FFTW/3.3.3', full_mod_subdir, deps)
         self.assertEqual(path, ['OpenMPI/1.6.4', 'GCC/4.7.2'])
 
+
 def suite():
     """ returns all the testcases in this module """
     return TestLoader().loadTestsFromTestCase(ModulesTest)
diff --git a/test/framework/modules/Compiler/intel/2013.5.192/impi/4.1.3.049 b/test/framework/modules/Compiler/intel/2013.5.192/impi/4.1.3.049
new file mode 100644
--- /dev/null
+++ b/test/framework/modules/Compiler/intel/2013.5.192/impi/4.1.3.049
@@ -0,0 +1,30 @@
+#%Module
+
+proc ModulesHelp { } {
+    puts stderr {   The Intel(R) MPI Library for Linux* OS is a multi-fabric message
+ passing library based on ANL MPICH2 and OSU MVAPICH2. The Intel MPI Library for
+ Linux OS implements the Message Passing Interface, version 2 (MPI-2) specification. - Homepage: http://software.intel.com/en-us/intel-mpi-library/
+    }
+}
+
+module-whatis {Description: The Intel(R) MPI Library for Linux* OS is a multi-fabric message
+ passing library based on ANL MPICH2 and OSU MVAPICH2. The Intel MPI Library for
+ Linux OS implements the Message Passing Interface, version 2 (MPI-2) specification. - Homepage: http://software.intel.com/en-us/intel-mpi-library/}
+
+set root    /tmp/software/Compiler/intel/2013.5.192/impi/4.1.3.049
+
+conflict impi
+module use /tmp/modules/all/MPI/intel/2013.5.192/impi/4.1.3.049
+prepend-path	CPATH		$root/include64
+prepend-path	LD_LIBRARY_PATH		$root/lib64
+prepend-path	LIBRARY_PATH		$root/lib64
+prepend-path	PATH		$root/bin64
+
+setenv	EBROOTIMPI		"$root"
+setenv	EBVERSIONIMPI		"4.1.3.049"
+setenv	EBDEVELIMPI		"$root/easybuild/Compiler-intel-2013.5.192-impi-4.1.3.049-easybuild-devel"
+
+prepend-path	INTEL_LICENSE_FILE		/tmp/license.lic
+setenv	I_MPI_ROOT		$root
+
+# Built with EasyBuild version 1.16.0dev
diff --git a/test/framework/modules/Core/GCC/4.8.3 b/test/framework/modules/Core/GCC/4.8.3
new file mode 100644
--- /dev/null
+++ b/test/framework/modules/Core/GCC/4.8.3
@@ -0,0 +1,30 @@
+#%Module
+
+proc ModulesHelp { } {
+    puts stderr {   The GNU Compiler Collection includes front ends for C, C++, Objective-C, Fortran, Java, and Ada,
+ as well as libraries for these languages (libstdc++, libgcj,...). - Homepage: http://gcc.gnu.org/
+    }
+}
+
+module-whatis {Description: The GNU Compiler Collection includes front ends for C, C++, Objective-C, Fortran, Java, and Ada,
+ as well as libraries for these languages (libstdc++, libgcj,...). - Homepage: http://gcc.gnu.org/}
+
+set root    /tmp/software/Core/GCC/4.8.3
+
+conflict GCC
+module use /tmp/modules/all/Compiler/GCC/4.8.3
+prepend-path	CPATH		$root/include
+prepend-path	LD_LIBRARY_PATH		$root/lib
+prepend-path	LD_LIBRARY_PATH		$root/lib64
+prepend-path	LD_LIBRARY_PATH		$root/lib/gcc/x86_64-unknown-linux-gnu/4.8.3
+prepend-path	LIBRARY_PATH		$root/lib
+prepend-path	LIBRARY_PATH		$root/lib64
+prepend-path	MANPATH		$root/share/man
+prepend-path	PATH		$root/bin
+
+setenv	EBROOTGCC		"$root"
+setenv	EBVERSIONGCC		"4.8.3"
+setenv	EBDEVELGCC		"$root/easybuild/Core-GCC-4.8.3-easybuild-devel"
+
+
+# Built with EasyBuild version 1.16.0dev
diff --git a/test/framework/modules/Core/icc/2013.5.192-GCC-4.8.3 b/test/framework/modules/Core/icc/2013.5.192-GCC-4.8.3
new file mode 100644
--- /dev/null
+++ b/test/framework/modules/Core/icc/2013.5.192-GCC-4.8.3
@@ -0,0 +1,34 @@
+#%Module
+
+proc ModulesHelp { } {
+    puts stderr {   C and C++ compiler from Intel - Homepage: http://software.intel.com/en-us/intel-compilers/
+    }
+}
+
+module-whatis {Description: C and C++ compiler from Intel - Homepage: http://software.intel.com/en-us/intel-compilers/}
+
+set root    /tmp/software/Core/icc/2013.5.192-GCC-4.8.3
+
+conflict icc
+module use /tmp/modules/all/Compiler/intel/2013.5.192
+
+if { ![is-loaded GCC/4.8.3] } {
+    module load GCC/4.8.3
+}
+
+prepend-path	IDB_HOME		$root/bin/intel64
+prepend-path	LD_LIBRARY_PATH		$root/compiler/lib
+prepend-path	LD_LIBRARY_PATH		$root/compiler/lib/intel64
+prepend-path	MANPATH		$root/man
+prepend-path	MANPATH		$root/man/en_US
+prepend-path	PATH		$root/bin
+prepend-path	PATH		$root/bin/intel64
+
+setenv	EBROOTICC		"$root"
+setenv	EBVERSIONICC		"2013.5.192"
+setenv	EBDEVELICC		"$root/easybuild/Core-icc-2013.5.192-GCC-4.8.3-easybuild-devel"
+
+prepend-path	INTEL_LICENSE_FILE		/tmp/license.lic
+prepend-path	NLSPATH		$root/idb/intel64/locale/%l_%t/%N
+
+# Built with EasyBuild version 1.16.0dev
diff --git a/test/framework/modules/Core/iccifort/2013.5.192-GCC-4.8.3 b/test/framework/modules/Core/iccifort/2013.5.192-GCC-4.8.3
new file mode 100644
--- /dev/null
+++ b/test/framework/modules/Core/iccifort/2013.5.192-GCC-4.8.3
@@ -0,0 +1,28 @@
+#%Module
+
+proc ModulesHelp { } {
+    puts stderr {   Intel C, C++ and Fortran compilers - Homepage: http://software.intel.com/en-us/intel-cluster-toolkit-compiler/
+    }
+}
+
+module-whatis {Description: Intel C, C++ and Fortran compilers - Homepage: http://software.intel.com/en-us/intel-cluster-toolkit-compiler/}
+
+set root    /tmp/software/Core/iccifort/2013.5.192-GCC-4.8.3
+
+conflict iccifort
+
+if { ![is-loaded icc/2013.5.192-GCC-4.8.3] } {
+    module load icc/2013.5.192-GCC-4.8.3
+}
+
+if { ![is-loaded ifort/2013.5.192-GCC-4.8.3] } {
+    module load ifort/2013.5.192-GCC-4.8.3
+}
+
+
+setenv	EBROOTICCIFORT		"$root"
+setenv	EBVERSIONICCIFORT		"2013.5.192"
+setenv	EBDEVELICCIFORT		"$root/easybuild/Core-iccifort-2013.5.192-GCC-4.8.3-easybuild-devel"
+
+
+# Built with EasyBuild version 1.16.0dev
diff --git a/test/framework/modules/Core/ifort/2013.5.192-GCC-4.8.3 b/test/framework/modules/Core/ifort/2013.5.192-GCC-4.8.3
new file mode 100644
--- /dev/null
+++ b/test/framework/modules/Core/ifort/2013.5.192-GCC-4.8.3
@@ -0,0 +1,34 @@
+#%Module
+
+proc ModulesHelp { } {
+    puts stderr {   Fortran compiler from Intel - Homepage: http://software.intel.com/en-us/intel-compilers/
+    }
+}
+
+module-whatis {Description: Fortran compiler from Intel - Homepage: http://software.intel.com/en-us/intel-compilers/}
+
+set root    /tmp/software/Core/ifort/2013.5.192-GCC-4.8.3
+
+conflict ifort
+module use /tmp/modules/all/Compiler/intel/2013.5.192
+
+if { ![is-loaded GCC/4.8.3] } {
+    module load GCC/4.8.3
+}
+
+prepend-path	IDB_HOME		$root/bin/intel64
+prepend-path	LD_LIBRARY_PATH		$root/compiler/lib
+prepend-path	LD_LIBRARY_PATH		$root/compiler/lib/intel64
+prepend-path	MANPATH		$root/man
+prepend-path	MANPATH		$root/man/en_US
+prepend-path	PATH		$root/bin
+prepend-path	PATH		$root/bin/intel64
+
+setenv	EBROOTIFORT		"$root"
+setenv	EBVERSIONIFORT		"2013.5.192"
+setenv	EBDEVELIFORT		"$root/easybuild/Core-ifort-2013.5.192-GCC-4.8.3-easybuild-devel"
+
+prepend-path	INTEL_LICENSE_FILE		/tmp/license.lic
+prepend-path	NLSPATH		$root/idb/intel64/locale/%l_%t/%N
+
+# Built with EasyBuild version 1.16.0dev
diff --git a/test/framework/modules/Core/iimpi/5.5.3-GCC-4.8.3 b/test/framework/modules/Core/iimpi/5.5.3-GCC-4.8.3
new file mode 100644
--- /dev/null
+++ b/test/framework/modules/Core/iimpi/5.5.3-GCC-4.8.3
@@ -0,0 +1,32 @@
+#%Module
+
+proc ModulesHelp { } {
+    puts stderr {   Intel C/C++ and Fortran compilers, alongside Intel MPI. - Homepage: http://software.intel.com/en-us/intel-cluster-toolkit-compiler/
+    }
+}
+
+module-whatis {Description: Intel C/C++ and Fortran compilers, alongside Intel MPI. - Homepage: http://software.intel.com/en-us/intel-cluster-toolkit-compiler/}
+
+set root    /tmp/software/Core/iimpi/5.5.3-GCC-4.8.3
+
+conflict iimpi
+
+if { ![is-loaded icc/2013.5.192-GCC-4.8.3] } {
+    module load icc/2013.5.192-GCC-4.8.3
+}
+
+if { ![is-loaded ifort/2013.5.192-GCC-4.8.3] } {
+    module load ifort/2013.5.192-GCC-4.8.3
+}
+
+if { ![is-loaded impi/4.1.3.049] } {
+    module load impi/4.1.3.049
+}
+
+
+setenv	EBROOTIIMPI		"$root"
+setenv	EBVERSIONIIMPI		"5.5.3"
+setenv	EBDEVELIIMPI		"$root/easybuild/Core-iimpi-5.5.3-GCC-4.8.3-easybuild-devel"
+
+
+# Built with EasyBuild version 1.16.0dev
diff --git a/test/framework/scripts.py b/test/framework/scripts.py
--- a/test/framework/scripts.py
+++ b/test/framework/scripts.py
@@ -65,13 +65,13 @@ def test_generate_software_list(self):
         out, ec = run_cmd(cmd, simple=False)
 
         # make sure output is kind of what we expect it to be
-        regex = r"Supported Packages \(12 "
+        regex = r"Supported Packages \(17 "
         self.assertTrue(re.search(regex, out), "Pattern '%s' found in output: %s" % (regex, out))
         per_letter = {
             'F': '1',  # FFTW
             'G': '4',  # GCC, gompi, goolf, gzip
             'H': '1',  # hwloc
-            'I': '2',  # ictce, impi
+            'I': '7',  # icc, iccifort, ictce, ifort, iimpi, imkl, impi
             'O': '2',  # OpenMPI, OpenBLAS
             'S': '1',  # ScaLAPACK
             'T': '1',  # toy
diff --git a/test/framework/utilities.py b/test/framework/utilities.py
--- a/test/framework/utilities.py
+++ b/test/framework/utilities.py
@@ -202,13 +202,17 @@ def setup_hierarchical_modules(self):
 
         # make sure only modules in a hierarchical scheme are available, mixing modules installed with
         # a flat scheme like EasyBuildMNS and a hierarhical one like HierarchicalMNS doesn't work
-        self.reset_modulepath([os.path.join(mod_prefix, 'Core')])
+        self.reset_modulepath([mod_prefix, os.path.join(mod_prefix, 'Core')])
 
-        # tweak use statements in GCC/OpenMPI modules to ensure correct paths
+        # tweak use statements in modules to ensure correct paths
         mpi_pref = os.path.join(mod_prefix, 'MPI', 'GCC', '4.7.2', 'OpenMPI', '1.6.4')
         for modfile in [
             os.path.join(mod_prefix, 'Core', 'GCC', '4.7.2'),
+            os.path.join(mod_prefix, 'Core', 'GCC', '4.8.3'),
+            os.path.join(mod_prefix, 'Core', 'icc', '2013.5.192-GCC-4.8.3'),
+            os.path.join(mod_prefix, 'Core', 'ifort', '2013.5.192-GCC-4.8.3'),
             os.path.join(mod_prefix, 'Compiler', 'GCC', '4.7.2', 'OpenMPI', '1.6.4'),
+            os.path.join(mod_prefix, 'Compiler', 'intel', '2013.5.192', 'impi', '4.1.3.049'),
             os.path.join(mpi_pref, 'FFTW', '3.3.3'),
             os.path.join(mpi_pref, 'OpenBLAS', '0.2.6-LAPACK-3.4.2'),
             os.path.join(mpi_pref, 'ScaLAPACK', '2.0.2-OpenBLAS-0.2.6-LAPACK-3.4.2'),