Repository: easybuilders/easybuild-framework

Issue:
fatal error Invalidating the module cache with 2.8
Hi,
   I am hitting a Python Error and traceback at the end of creating a module with EasyBuild 2.8. I retested and 2.7 doesn't hit the issue so it appears to be new in 2.8. The package and module seem to build fine and indeed it doesn't appear to depend on what software you are actually building as the error happens after the module is created. This is on a Cray XE system with the traditional modules install. Below is the end of the debug log. The full debug log is attached and the traceback is below. I can get past it by commenting out the invalidate cache code, which occasionally means the cache is out of date, but otherwise works. However, I am not sure how to fix it properly.

Thanks,
Brett

== 2016-05-20 13:02:18,852 build_log.py:152 ERROR Traceback (most recent call last):
  File "/mnt/abc/u/staff/bbode/EasyBuild-2.8/software/EasyBuild/2.8.0/lib64/python2.6/site-packages/easybuild_framework-2.8.0-py2.6.egg/easybuild/main.py", line 114, in build_and_install_software
    (ec_res['success'], app_log, err) = build_and_install_one(ec, init_env)
  File "/u/staff/bbode/EasyBuild-2.8/software/EasyBuild/2.8.0/lib64/python2.6/site-packages/easybuild_framework-2.8.0-py2.6.egg/easybuild/framework/easyblock.py", line 2358, in build_and_install_one
    result = app.run_all_steps(run_test_cases=run_test_cases)
  File "/u/staff/bbode/EasyBuild-2.8/software/EasyBuild/2.8.0/lib64/python2.6/site-packages/easybuild_framework-2.8.0-py2.6.egg/easybuild/framework/easyblock.py", line 2274, in run_all_steps
    self.run_step(step_name, step_methods)
  File "/u/staff/bbode/EasyBuild-2.8/software/EasyBuild/2.8.0/lib64/python2.6/site-packages/easybuild_framework-2.8.0-py2.6.egg/easybuild/framework/easyblock.py", line 2153, in run_step
    step_method(self)()
  File "/u/staff/bbode/EasyBuild-2.8/software/EasyBuild/2.8.0/lib64/python2.6/site-packages/easybuild_framework-2.8.0-py2.6.egg/easybuild/framework/easyblock.py", line 2005, in make_module_step
    invalidate_module_caches_for(path)
  File "/u/staff/bbode/EasyBuild-2.8/software/EasyBuild/2.8.0/lib64/python2.6/site-packages/easybuild_framework-2.8.0-py2.6.egg/easybuild/tools/modules.py", line 1150, in invalidate_module_caches_for
    key, subcmd, path, cache[key])
KeyError: ('MODULEPATH=/u/staff/bbode/EasyBuild-2.8/modules/all:/u/staff/bbode/EasyBuild/modules/all/:/sw/bw/modulefiles:/sw/xe/modulefiles:/usr/local/modulefiles:/opt/cray/craype/2.5.4/modulefiles:/opt/cray/gem/modulefiles:/opt/cray/modulefiles:/opt/modulefiles', 'modulecmd', '')
 (at easybuild/main.py:147 in build_and_install_software)

[easybuild-naHPSm.txt](https://github.com/hpcugent/easybuild-framework/files/275268/easybuild-naHPSm.txt)



Discussion:
Thank you for the detailed bug report @brettbode!

The caching mechanism is indeed new in EasyBuild 2.8, it was introduced via https://github.com/hpcugent/easybuild-framework/pull/1742 to speed up things.

I think the problem is that your $MODULEPATH includes two entries that point to the same location on the filesystem, probably <prefix>/EasyBuild-2.8/modules/all and <prefix>/EasyBuild/modules/all.

This setup is not taken into account by the invalidate_module_caches_for function, making it delete a cache entry twice, which results in the traceback you're seeing.vof course, that's a bug, it shouldn't be trying to delete a cache entry that was just evicted again...

I think the fix is pretty straightforward: just add a 'break' statement below the 'del' (same indent level), to break out of the innermost loop.

Can you confirm the symlink entry in your $MODULEPATH, and verify the proposed fix?


Resolving patch:
diff --git a/easybuild/tools/modules.py b/easybuild/tools/modules.py
--- a/easybuild/tools/modules.py
+++ b/easybuild/tools/modules.py
@@ -1149,6 +1149,7 @@ def invalidate_module_caches_for(path):
                     _log.debug("Entry '%s' in 'module %s' cache is evicted, marked as invalid via path '%s': %s",
                                key, subcmd, path, cache[key])
                     del cache[key]
+                    break
 
 
 class Modules(EnvironmentModulesC):


Test patch:
diff --git a/test/framework/modules.py b/test/framework/modules.py
--- a/test/framework/modules.py
+++ b/test/framework/modules.py
@@ -536,6 +536,13 @@ def test_module_caches(self):
         self.assertTrue(any([os.path.samefile(nonpath, mp) for mp in modulepaths]))
         shutil.rmtree(nonpath)
 
+        # create symlink to entry in $MODULEPATH we're going to use, and add it to $MODULEPATH
+        # invalidate_module_caches_for should be able to deal with this
+        test_mods_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'modules')
+        mods_symlink = os.path.join(self.test_prefix, 'modules_symlink')
+        os.symlink(test_mods_path, mods_symlink)
+        self.modtool.use(mods_symlink)
+
         # no caching for 'avail' commands with an argument
         self.assertTrue(self.modtool.available('GCC'))
         self.assertEqual(mod.MODULE_AVAIL_CACHE, {})
@@ -565,14 +572,13 @@ def test_module_caches(self):
         self.assertTrue(self.modtool.show('FFTW') is show_res_fftw)
 
         # invalidate caches with correct path
-        modpath = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'modules')
         modulepaths = [p for p in os.environ.get('MODULEPATH', '').split(os.pathsep) if p]
-        self.assertTrue(any([os.path.exists(mp) and os.path.samefile(modpath, mp) for mp in modulepaths]))
+        self.assertTrue(any([os.path.exists(mp) and os.path.samefile(test_mods_path, mp) for mp in modulepaths]))
         paths_in_key = [p for p in avail_cache_key[0].split('=')[1].split(os.pathsep) if p]
-        self.assertTrue(any([os.path.exists(p) and os.path.samefile(modpath, p) for p in paths_in_key]))
+        self.assertTrue(any([os.path.exists(p) and os.path.samefile(test_mods_path, p) for p in paths_in_key]))
 
         # verify cache invalidation, caches should be empty again
-        invalidate_module_caches_for(modpath)
+        invalidate_module_caches_for(test_mods_path)
         self.assertEqual(mod.MODULE_AVAIL_CACHE, {})
         self.assertEqual(mod.MODULE_SHOW_CACHE, {})