Repository: easybuilders/easybuild-framework

Issue:
`NotImplementedError` on `--fetch` with `external_module` used in easyconfig
After a lengthy discussion in Slack and a longer search for the issue I think this is a bug:

```
eb EP-Stability-WF-1.0.4-intel-2020b-DD-3.37.0.eb --fetch -f
== Temporary log file in case of crash /tmp/eb-064p5o3j/easybuild-m72q5q32.log
== found valid index for /home/.local/easybuild/easyconfigs, so using it...
Traceback (most recent call last):
  File "/opt/compilers/python/3.9.4/none/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/compilers/python/3.9.4/none/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/.local/lib/python3.9/site-packages/easybuild/main.py", line 597, in <module>
    main()
  File "/home/.local/lib/python3.9/site-packages/easybuild/main.py", line 424, in main
    easyconfigs, generated_ecs = parse_easyconfigs(paths, validate=not options.inject_checksums)
  File "/home/.local/lib/python3.9/site-packages/easybuild/framework/easyconfig/tools.py", line 405, in parse_easyconfigs
    easyconfigs.extend(process_easyconfig(ec_file, **kwargs))
  File "/home/.local/lib/python3.9/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 2069, in process_easyconfig
    ec = EasyConfig(spec, build_specs=build_specs, validate=validate, hidden=hidden)
  File "/home/.local/lib/python3.9/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 514, in __init__
    self.parse()
  File "/home/.local/lib/python3.9/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 750, in parse
    self['dependencies'] = remove_false_versions(self._parse_dependency(dep) for dep in self['dependencies'])
  File "/home/.local/lib/python3.9/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 748, in remove_false_versions
    return [dep for dep in deps if not (isinstance(dep, dict) and dep['version'] is False)]
  File "/home/.local/lib/python3.9/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 748, in <listcomp>
    return [dep for dep in deps if not (isinstance(dep, dict) and dep['version'] is False)]
  File "/home/.local/lib/python3.9/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 750, in <genexpr>
    self['dependencies'] = remove_false_versions(self._parse_dependency(dep) for dep in self['dependencies'])
  File "/home/.local/lib/python3.9/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 1558, in _parse_dependency
    dependency.update(self.handle_external_module_metadata(dep[0]))
  File "/home/.local/lib/python3.9/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 1408, in handle_external_module_metadata
    probed_metadata = self.probe_external_module_metadata(mod_name, existing_metadata=metadata)
  File "/home/.local/lib/python3.9/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 1350, in probe_external_module_metadata
    prefix = self.modules_tool.get_setenv_value_from_modulefile(mod_name, prefix_var_name)
  File "/home/.local/lib/python3.9/site-packages/easybuild/tools/modules.py", line 1180, in get_setenv_value_from_modulefile
    raise NotImplementedError
NotImplementedError
```

What I found:
- Error is triggered because the (dummy) `NoModulesTool` class is used which doesn't implement `get_setenv_value_from_modulefile`
- An external module is used in the dependencies which triggers https://github.com/easybuilders/easybuild-framework/blob/fca8b932f04957ad73e5f82c3798eb1d21a7db37/easybuild/framework/easyconfig/easyconfig.py#L1562
- That calls the `probe_*` function at https://github.com/easybuilders/easybuild-framework/blob/fca8b932f04957ad73e5f82c3798eb1d21a7db37/easybuild/framework/easyconfig/easyconfig.py#L1418 using the modules-tool functionality at https://github.com/easybuilders/easybuild-framework/blob/fca8b932f04957ad73e5f82c3798eb1d21a7db37/easybuild/framework/easyconfig/easyconfig.py#L1354-L1355
- `--fetch` is even documented as ignoring the modules tool at https://github.com/easybuilders/easybuild-framework/blob/fca8b932f04957ad73e5f82c3798eb1d21a7db37/easybuild/tools/options.py#L410
- It does so at https://github.com/easybuilders/easybuild-framework/blob/fca8b932f04957ad73e5f82c3798eb1d21a7db37/easybuild/tools/options.py#L1190
- with `modules_tool==None` the class `NoModulesTool` is used at https://github.com/easybuilders/easybuild-framework/blob/fca8b932f04957ad73e5f82c3798eb1d21a7db37/easybuild/tools/modules.py#L1680 leading to the failure

So in summary: `--fetch` must not ignore the modules tool as resolving EC dependencies may not be possible without it.
Don't ignore modules tool when using `--fetch`
The modules tool is required when resolving (at least) external dependencies.

Fixes #4298

I guess the intention was to ignore installed modules for `--fetch` as I had to change the condition for `keep_available_modules` for the test to pass as it would otherwise not fetch for installed modules which I'd guess is not intended.

For that I also changed the `keep_available_modules = keep_available_modules or ...` style to `any([...])` which I think is easier to read and format. The 2 dry-run and inject-checksum options are on the same line on purpose


Discussion:
`--fetch` was deliberately implemented in #2457 such that no modules tool is required, so `eb --fetch` can also be used on your laptop or another system where no modules tool may be available.
With that in mind, I don't think we should start requiring a modules tool to use `--fetch`.

Question is then how this problem can be fixed in another way.

Maybe `get_setenv_value_from_modulefile` should be implemented generically in `ModulesTool`, without requiring an actual modules tool (so without the call to `get_value_from_modulefile` which runs `module show`)?
> Question is then how this problem can be fixed in another way.
> 
> Maybe `get_setenv_value_from_modulefile` should be implemented generically in `ModulesTool`, without requiring an actual modules tool (so without the call to `get_value_from_modulefile` which runs `module show`)?

I have no idea how this would be possible: How could you `get_*_from_modulefile` without a "modulefile"?

Maybe we can rather not call `probe_external_module_metadata` if there is no modules tool? Not sure what `external_module_metadata` is used for and if it is really required.

But it all boils down to: `--fetch` needs to resolve dependencies which isn't possible without a modules-tool, is it?    
Maybe we can add a flag to ignore the modules tool and only consider dependencies defined in ECs and download everything for that use case?
> > Question is then how this problem can be fixed in another way.
> > Maybe `get_setenv_value_from_modulefile` should be implemented generically in `ModulesTool`, without requiring an actual modules tool (so without the call to `get_value_from_modulefile` which runs `module show`)?
> 
> I have no idea how this would be possible: How could you `get_*_from_modulefile` without a "modulefile"?

We can get the contents of a module file without running `module show`, just by reading the module file directly instead?

> Maybe we can rather not call `probe_external_module_metadata` if there is no modules tool? Not sure what `external_module_metadata` is used for and if it is really required.

See https://docs.easybuild.io/using-external-modules/#using_external_modules_metadata w.r.t. metadata for external modules.

It definitely not required for just fetching sources.
In fact, external modules are totally irrelevant when doing `--fetch`.


> But it all boils down to: `--fetch` needs to resolve dependencies which isn't possible without a modules-tool, is it? Maybe we can add a flag to ignore the modules tool and only consider dependencies defined in ECs and download everything for that use case?

Resolving dependencies doesn't require a modules tool, that's purely done based on `*dependencies` in easyconfigs.
Checking which easyconfigs are installed already is what requires a modules tool (since that implies using `module avail`).
> We can get the contents of a module file without running `module show`, just by reading the module file directly instead?

But how do you find the module file? Especially finding the correct module file would required duplicating logic `module show` does (inside e.g. lmod).

> Resolving dependencies doesn't require a modules tool, that's purely done based on `*dependencies` in easyconfigs. Checking which easyconfigs are installed already is what requires a modules tool (since that implies using `module avail`).

Ah ok, so installed modules are ignored and sources for all ECs in the dependency chain are collected even if they are installed, aren't they?

If so I'd say that `probe_external_module_metadata` can simply return an empty dict when "NoModulesTool" is used as-if it wasn't found.
The intention of ignoring the modules tool is that `--fetch` can be used even if no modules tool is available at all (for example on your laptop), see also the original implementation in #2457

Resolving patch:
diff --git a/easybuild/framework/easyconfig/easyconfig.py b/easybuild/framework/easyconfig/easyconfig.py
--- a/easybuild/framework/easyconfig/easyconfig.py
+++ b/easybuild/framework/easyconfig/easyconfig.py
@@ -73,7 +73,7 @@
 from easybuild.tools.module_naming_scheme.mns import DEVEL_MODULE_SUFFIX
 from easybuild.tools.module_naming_scheme.utilities import avail_module_naming_schemes, det_full_ec_version
 from easybuild.tools.module_naming_scheme.utilities import det_hidden_modname, is_valid_module_name
-from easybuild.tools.modules import modules_tool
+from easybuild.tools.modules import modules_tool, NoModulesTool
 from easybuild.tools.py2vs3 import OrderedDict, create_base_metaclass, string_type
 from easybuild.tools.systemtools import check_os_dependency, pick_dep_version
 from easybuild.tools.toolchain.toolchain import SYSTEM_TOOLCHAIN_NAME, is_system_toolchain
@@ -1306,6 +1306,9 @@ def probe_external_module_metadata(self, mod_name, existing_metadata=None):
         :param existing_metadata: already available metadata for this external module (if any)
         """
         res = {}
+        if isinstance(self.modules_tool, NoModulesTool):
+            self.log.debug('Ignoring request for external module data for %s as no modules tool is active', mod_name)
+            return res
 
         if existing_metadata is None:
             existing_metadata = {}


Test patch:
diff --git a/test/framework/options.py b/test/framework/options.py
--- a/test/framework/options.py
+++ b/test/framework/options.py
@@ -5073,19 +5073,22 @@ def test_fetch(self):
         lock_path = os.path.join(self.test_installpath, 'software', '.locks', lock_fn)
         mkdir(lock_path, parents=True)
 
-        args = ['toy-0.0.eb', '--fetch']
-        stdout, stderr = self._run_mock_eb(args, raise_error=True, strip=True, testing=False)
+        # Run for a "regular" EC and one with an external module dependency
+        # which might trip up the dependency resolution (see #4298)
+        for ec in ('toy-0.0.eb', 'toy-0.0-deps.eb'):
+            args = [ec, '--fetch']
+            stdout, stderr = self._run_mock_eb(args, raise_error=True, strip=True, testing=False)
 
-        patterns = [
-            r"^== fetching files\.\.\.$",
-            r"^== COMPLETED: Installation STOPPED successfully \(took .* secs?\)$",
-        ]
-        for pattern in patterns:
-            regex = re.compile(pattern, re.M)
-            self.assertTrue(regex.search(stdout), "Pattern '%s' not found in: %s" % (regex.pattern, stdout))
+            patterns = [
+                r"^== fetching files\.\.\.$",
+                r"^== COMPLETED: Installation STOPPED successfully \(took .* secs?\)$",
+            ]
+            for pattern in patterns:
+                regex = re.compile(pattern, re.M)
+                self.assertTrue(regex.search(stdout), "Pattern '%s' not found in: %s" % (regex.pattern, stdout))
 
-        regex = re.compile(r"^== creating build dir, resetting environment\.\.\.$")
-        self.assertFalse(regex.search(stdout), "Pattern '%s' found in: %s" % (regex.pattern, stdout))
+            regex = re.compile(r"^== creating build dir, resetting environment\.\.\.$")
+            self.assertFalse(regex.search(stdout), "Pattern '%s' found in: %s" % (regex.pattern, stdout))
 
     def test_parse_external_modules_metadata(self):
         """Test parse_external_modules_metadata function."""