Repository: easybuilders/easybuild-framework

Issue:
Easybuild lists all dependencies of a package as unresolved
Easybuild lists all dependencies of a package as unresolved, in the case that the easyconfig of one of them does not exist.

When I create the missing easyconfig and add the --robot option, the package is built with no problems.

I believe that only the unresolved dependency should be listed.



Discussion:
When running with the `-l` option, eb shows the missing dependency, no?

The problem is that:
Assume that a package X has 3 dependencies A,B and C
B and C have their easyconfigs and are ready to be built, but A does not have an easyconfig.

When I try to build X

```
eb X -r
```

It gives me the Error

```
Unresolved dependencies A,B,C
```

Instead of

```
Unresolved dependencies A
```

Which version are you using? Because I get a nice dependency resolve failed for package A in your example:

```
ERROR: EasyBuild crashed with an error (at easybuild/main.py:556 in resolve_dependencies):
Dependencies not met. Cannot resolve ["Eigen/3.1.7-ictce-5.5.0 for {'dummy': False, 'versionsuffix': '', 
'version': '3.1.7', 'toolchain': {'version': '5.5.0', 'name': 'ictce'}, 'name': 'Eigen'}"]
```

(There were 4 deps and 1 is missing)

I am using EasyBuild 1.8.2 (framework: 1.8.2, easyblocks: 1.8.2).

The problem appears when the rest of the dependencies are not built but their easyconfigs exist. Since I am using the --robot option I would expect to report as unresolved only the one with the missing easyconfig. This will help me to identify which easyconfigs I should implement myself and which are already implemented.

@apanteli: You're right, `eb` could do better there, and I often get confused about it myself.

The problem is that it stops resolving dependencies as soon as one thing is missing, and then spits out whatever isn't resolved yet. But, that doesn't mean none of them could be resolved anymore, it just stops hard.
Also, it assumes that if A is missing, B is missing too if it depends on A, even though there might be an easyconfig available for B.

I'll try and look into making it spit out a better error message, only showing the relevant easyconfigs (and trimming down the output a bit, it's quite verbose now, even for a single missing item).

Thank you Kenneth!
On Nov 14, 2013 4:09 PM, "Kenneth Hoste" notifications@github.com wrote:

> @apanteli https://github.com/apanteli: You're right, eb could do better
> there, and I often get confused about it myself.
> 
> The problem is that it stops resolving dependencies as soon as one thing
> is missing, and then spits out whatever isn't resolved yet. But, that
> doesn't mean none of them could be resolved anymore, it just stops hard.
> Also, it assumes that if A is missing, B is missing too if it depends on
> A, even though there might be an easyconfig available for B.
> 
> I'll try and look into making it spit out a better error message, only
> showing the relevant easyconfigs (and trimming down the output a bit, it's
> quite verbose now, even for a single missing item).
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/hpcugent/easybuild-framework/issues/763#issuecomment-28486217
> .

example of current output for a single missing dep:

```
EasyBuildError: 'EasyBuild crashed with an error (at easybuild/main.py:594 in resolve_dependencies):
Dependencies not met. Cannot resolve ["HDF5/1.8.10-ictce-4.1.13-parallel-gpfs for
{\'dummy\': False, \'toolchain\': {\'version\': \'4.1.13\', \'name\': \'ictce\'},
\'name\': \'HDF5\', \'versionsuffix\': \'-parallel-gpfs\', \'version\': \'1.8.10\'}"]'
```

example of current output with only one missing dependency, but multiple ones being reported (`CGAL` depends on `Qt`):

```
EasyBuildError: 'EasyBuild crashed with an error (at easybuild/main.py:594 in resolve_dependencies):
Dependencies not met. Cannot resolve [
"CGAL/4.0-ictce-4.0.6-Python-2.7.3 for {\'dummy\': False, \'toolchain\': {\'version\': \'4.0.6\', \'name\': \'ictce\'},
\'name\': \'CGAL\', \'versionsuffix\': \'-Python-2.7.3\', \'version\': \'4.0\'}",
"CGAL/4.0-goalf-1.1.0-no-OFED-Python-2.7.3 for {\'dummy\': False, \'toolchain\': {\'version\': \'1.1.0-no-OFED\',
\'name\': \'goalf\'}, \'name\': \'CGAL\', \'versionsuffix\': \'-Python-2.7.3\', \'version\': \'4.0\'}",
"Qt/4.8.4-ictce-4.0.6 for {\'dummy\': False, \'versionsuffix\': \'\', \'version\': \'4.8.4\',
\'toolchain\': {\'version\': \'4.0.6\', \'name\': \'ictce\'}, \'name\': \'Qt\'}",
"Qt/4.8.4-ictce-5.3.0 for {\'dummy\': False, \'versionsuffix\': \'\', \'version\': \'4.8.4\',
\'toolchain\': {\'version\': \'5.3.0\', \'name\': \'ictce\'}, \'name\': \'Qt\'}",
"Qt/4.8.4-goalf-1.1.0-no-OFED for {\'dummy\': False, \'versionsuffix\': \'\', \'version\': \'4.8.4\',
\'toolchain\': {\'version\': \'1.1.0-no-OFED\', \'name\': \'goalf\'}, \'name\': \'Qt\'}"]'
```


Resolving patch:
diff --git a/easybuild/main.py b/easybuild/main.py
--- a/easybuild/main.py
+++ b/easybuild/main.py
@@ -514,7 +514,7 @@ def skip_available(easyconfigs, testing=False):
 def resolve_dependencies(unprocessed, robot, force=False, check_osdeps=True, specs=None):
     """
     Work through the list of easyconfigs to determine an optimal order
-    @param unprocessed: llist of easyconfigs
+    @param unprocessed: list of easyconfigs
     @param robot: robot path
     @param force: retain all dependencies and skip validation of easyconfigs (default: False)
     @param check_osdeps: check OS dependencies (default: True)
@@ -523,49 +523,51 @@ def resolve_dependencies(unprocessed, robot, force=False, check_osdeps=True, spe
 
     if force:
         # assume that no modules are available when forced
-        available_modules = []
+        avail_modules = []
         _log.info("Forcing all dependencies to be retained.")
     else:
         # Get a list of all available modules (format: [(name, installversion), ...])
-        available_modules = modules_tool().available()
+        avail_modules = modules_tool().available()
 
-        if len(available_modules) == 0:
+        if len(avail_modules) == 0:
             _log.warning("No installed modules. Your MODULEPATH is probably incomplete: %s" % os.getenv('MODULEPATH'))
 
     ordered_ecs = []
-    # All available modules can be used for resolving dependencies except
-    # those that will be installed
+    # all available modules can be used for resolving dependencies except those that will be installed
     being_installed = [p['module'] for p in unprocessed]
-    processed = [m for m in available_modules if not m in being_installed]
+    avail_modules = [m for m in avail_modules if not m in being_installed]
 
     _log.debug('unprocessed before resolving deps: %s' % unprocessed)
 
-    # as long as there is progress in processing the modules, keep on trying
+    # resolve all dependencies, put a safeguard in place to avoid an infinite loop (shouldn't occur though)
+    irresolvable = []
     loopcnt = 0
     maxloopcnt = 10000
-    robot_add_dep = True
-    while robot_add_dep:
-
-        robot_add_dep = False
-
+    while unprocessed:
         # make sure this stops, we really don't want to get stuck in an infinite loop
         loopcnt += 1
         if loopcnt > maxloopcnt:
-            msg = "Maximum loop cnt %s reached, so quitting." % maxloopcnt
+            tup = (maxloopcnt, unprocessed, irresolvable)
+            msg = "Maximum loop cnt %s reached, so quitting (unprocessed: %s, irresolvable: %s)" % tup
             _log.error(msg)
 
         # first try resolving dependencies without using external dependencies
+        # find_resolved_modules may extend 'avail_modules', hence the while loop;
+        # it will also update 'unprocessed' to only leave in entries with one or more unresolved dependencies
         last_processed_count = -1
-        while len(processed) > last_processed_count:
-            last_processed_count = len(processed)
-            ordered_ecs.extend(find_resolved_modules(unprocessed, processed))
+        while len(avail_modules) > last_processed_count:
+            last_processed_count = len(avail_modules)
+            # note: find_resolved_modules may append entries to avail_modules (and modify unprocessed as well)
+            for ec in find_resolved_modules(unprocessed, avail_modules):
+                if not ec['module'] in [x['module'] for x in ordered_ecs]:
+                    ordered_ecs.append(ec)
 
-        # robot: look for an existing dependency, add one
-        if robot and len(unprocessed) > 0:
+        # robot: look for existing dependencies, add them
+        if robot and unprocessed:
 
             being_installed = [det_full_module_name(p['ec'], eb_ns=True) for p in unprocessed]
 
-            for entry in unprocessed:
+            for i, entry in enumerate(unprocessed):
                 # do not choose an entry that is being installed in the current run
                 # if they depend, you probably want to rebuild them using the new dependency
                 deps = entry['dependencies']
@@ -574,61 +576,56 @@ def resolve_dependencies(unprocessed, robot, force=False, check_osdeps=True, spe
                     cand_dep = candidates[0]
                     # find easyconfig, might not find any
                     _log.debug("Looking for easyconfig for %s" % str(cand_dep))
+                    # note: robot_find_easyconfig may return None
                     path = robot_find_easyconfig(robot, cand_dep['name'], det_full_ec_version(cand_dep))
 
+                    if path is None:
+                        # no easyconfig found for dependency, add to list of irresolvable dependencies
+                        if cand_dep not in irresolvable:
+                            irresolvable.append(cand_dep)
+                        # remove irresolvable dependency from list of dependencies so we can continue
+                        entry['dependencies'].remove(cand_dep)
+                    else:
+                        _log.info("Robot: resolving dependency %s with %s" % (cand_dep, path))
+                        processed_ecs = process_easyconfig(path, validate=(not force), check_osdeps=check_osdeps,
+                                                           specs=specs)
+
+                        # ensure that selected easyconfig provides required dependency
+                        mods = [det_full_module_name(spec['ec']) for spec in processed_ecs]
+                        dep_mod_name = det_full_module_name(cand_dep)
+                        if not dep_mod_name in mods:
+                            tup = (path, dep_mod_name, mods)
+                            _log.error("easyconfig file %s does not contain module %s (mods: %s)" % tup)
+
+                        for ec in processed_ecs:
+                            if not ec in unprocessed:
+                                unprocessed.append(ec)
+                                _log.debug("Added %s as dependency of %s" % (ec, entry))
                 else:
-                    path = None
                     mod_name = det_full_module_name(entry['ec'], eb_ns=True)
                     _log.debug("No more candidate dependencies to resolve for %s" % mod_name)
 
-                if path is not None:
-                    cand_dep = candidates[0]
-                    _log.info("Robot: resolving dependency %s with %s" % (cand_dep, path))
-
-                    processed_ecs = process_easyconfig(path, validate=(not force), check_osdeps=check_osdeps,
-                                                       specs=specs)
-
-                    # ensure that selected easyconfig provides required dependency
-                    mods = [det_full_module_name(spec['ec']) for spec in processed_ecs]
-                    dep_mod_name = det_full_module_name(cand_dep)
-                    if not dep_mod_name in mods:
-                        tup = (path, dep_mod_name, mods)
-                        _log.error("easyconfig file %s does not contain module %s (mods: %s)" % tup)
-
-                    unprocessed.extend(processed_ecs)
-                    robot_add_dep = True
-                    break
-
-    _log.debug('unprocessed after resolving deps: %s' % unprocessed)
-
-    # there are dependencies that cannot be resolved
-    if len(unprocessed) > 0:
-        _log.debug("List of unresolved dependencies: %s" % unprocessed)
-        missing_dependencies = []
-        for ec in unprocessed:
-            for dep in ec['dependencies']:
-                missing_dependencies.append('%s for %s' % (det_full_module_name(dep, eb_ns=True), dep))
-
-        msg = "Dependencies not met. Cannot resolve %s" % missing_dependencies
-        _log.error(msg)
+    if irresolvable:
+        irresolvable_mod_deps = [(det_full_module_name(dep, eb_ns=True), dep) for dep in irresolvable]
+        _log.error('Irresolvable dependencies encountered: %s' % irresolvable_mod_deps)
 
     _log.info("Dependency resolution complete, building as follows:\n%s" % ordered_ecs)
     return ordered_ecs
 
 
-def find_resolved_modules(unprocessed, processed):
+def find_resolved_modules(unprocessed, avail_modules):
     """
-    Find easyconfigs in unprocessed which can be fully resolved using easyconfigs in processed
+    Find easyconfigs in 1st argument which can be fully resolved using modules specified in 2nd argument
     """
     ordered_ecs = []
 
     for ec in unprocessed:
-        ec['dependencies'] = [d for d in ec['dependencies'] if not det_full_module_name(d) in processed]
+        ec['dependencies'] = [d for d in ec['dependencies'] if not det_full_module_name(d) in avail_modules]
 
         if len(ec['dependencies']) == 0:
             _log.debug("Adding easyconfig %s to final list" % ec['spec'])
             ordered_ecs.append(ec)
-            processed.append(ec['module'])
+            avail_modules.append(ec['module'])
 
     unprocessed[:] = [m for m in unprocessed if len(m['dependencies']) > 0]
 


Test patch:
diff --git a/test/framework/easyblock.py b/test/framework/easyblock.py
--- a/test/framework/easyblock.py
+++ b/test/framework/easyblock.py
@@ -35,12 +35,12 @@
 import shutil
 import tempfile
 import sys
+from unittest import TestCase, TestLoader, main
 
 import easybuild.tools.options as eboptions
 from easybuild.framework.easyblock import EasyBlock
 from easybuild.framework.extension import Extension
 from easybuild.tools import config
-from unittest import TestCase, TestLoader, main
 from easybuild.tools.build_log import EasyBuildError
 from easybuild.tools.filetools import write_file
 from easybuild.tools.module_generator import det_full_module_name
diff --git a/test/framework/robot.py b/test/framework/robot.py
--- a/test/framework/robot.py
+++ b/test/framework/robot.py
@@ -30,7 +30,7 @@
 
 import os
 from copy import deepcopy
-from unittest import TestCase, TestSuite
+from unittest import TestCase, TestLoader
 from unittest import main as unittestmain
 from vsc import fancylogger
 
@@ -79,7 +79,7 @@ def setUp(self):
         self.base_easyconfig_dir = find_full_path(os.path.join("test", "framework", "easyconfigs"))
         self.assertTrue(self.base_easyconfig_dir)
 
-    def runTest(self):
+    def test_resolve_dependencies(self):
         """ Test with some basic testcases (also check if he can find dependencies inside the given directory """
         easyconfig = {
             'spec': '_',
@@ -145,7 +145,7 @@ def tearDown(self):
 
 def suite():
     """ returns all the testcases in this module """
-    return TestSuite([RobotTest()])
+    return TestLoader().loadTestsFromTestCase(RobotTest)
 
 if __name__ == '__main__':
     unittestmain()