Repository: easybuilders/easybuild-framework

Issue:
GC3Pie backend submits jobs even if dependency build failed
When robot-building a bunch of software via the GC3Pie job backend with SLURM, GC3Pie happily submits follow-up jobs even if building of one (or more) of the dependencies failed.  I am not entirely sure whether this is an issue in EasyBuild, in GC3Pie, or in the way the two are working together.  I also don't know whether this also happens with other job schedulers.

Steps to reproduce:
- Start with an empty installdir
- Edit `M4-1.4.17.eb` and add `preconfigopts='false && '`
- Run `eb GCC-5.1.0.eb -r --job` using GC3Pie with SLURM
- Both jobs will be submitted, both fail (the first for obvious reasons, the second because the M4 module can't be found)

GC3Pie backend submits jobs even if dependency build failed
When robot-building a bunch of software via the GC3Pie job backend with SLURM, GC3Pie happily submits follow-up jobs even if building of one (or more) of the dependencies failed.  I am not entirely sure whether this is an issue in EasyBuild, in GC3Pie, or in the way the two are working together.  I also don't know whether this also happens with other job schedulers.

Steps to reproduce:
- Start with an empty installdir
- Edit `M4-1.4.17.eb` and add `preconfigopts='false && '`
- Run `eb GCC-5.1.0.eb -r --job` using GC3Pie with SLURM
- Both jobs will be submitted, both fail (the first for obvious reasons, the second because the M4 module can't be found)



Discussion:
@geimer pbs will submit all build jobs at once with a dependency between them. So in the PBS case, both jobs will also be submitted and fail.

It would be rather simple to add this kind of checks to the EB->GC3Pie interface, _provided_ `eb` always exits with a non-zero exit code when a build fails.

Is there (still) interest in this?

@riccardomurri `eb` does indeed exit with non-zero exit code in case a build fails, so you should be able to rely on that

@riccardomurri IMHO this feature would still be useful.  Why submit jobs that are already known to fail?

@riccardomurri ping on this? It was brought up again during today's EB conf call.
@riccardomurri Re-ping? :)
Sorry, completely fell off my radar ... :-(
@geimer pbs will submit all build jobs at once with a dependency between them. So in the PBS case, both jobs will also be submitted and fail.

It would be rather simple to add this kind of checks to the EB->GC3Pie interface, _provided_ `eb` always exits with a non-zero exit code when a build fails.

Is there (still) interest in this?

@riccardomurri `eb` does indeed exit with non-zero exit code in case a build fails, so you should be able to rely on that

@riccardomurri IMHO this feature would still be useful.  Why submit jobs that are already known to fail?

@riccardomurri ping on this? It was brought up again during today's EB conf call.
@riccardomurri Re-ping? :)
Sorry, completely fell off my radar ... :-(

Resolving patch:
diff --git a/easybuild/tools/job/gc3pie.py b/easybuild/tools/job/gc3pie.py
--- a/easybuild/tools/job/gc3pie.py
+++ b/easybuild/tools/job/gc3pie.py
@@ -51,7 +51,7 @@
     from gc3libs import Application, Run, create_engine
     from gc3libs.core import Engine
     from gc3libs.quantity import hours as hr
-    from gc3libs.workflow import DependentTaskCollection
+    from gc3libs.workflow import AbortOnError, DependentTaskCollection
 
     # inject EasyBuild logger into GC3Pie
     gc3libs.log = fancylogger.getLogger('gc3pie', fname=False)
@@ -61,6 +61,16 @@
     # instruct GC3Pie to not ignore errors, but raise exceptions instead
     gc3libs.UNIGNORE_ALL_ERRORS = True
 
+    # note: order of class inheritance is important!
+    class _BuildTaskCollection(AbortOnError, DependentTaskCollection):
+        """
+        A `DependentTaskCollection`:class: that aborts execution upon error.
+
+        This is used to stop the build process in case some dependency
+        fails.  See also `<https://github.com/easybuilders/easybuild-framework/issues/1441>`_
+        """
+        pass
+
 except ImportError as err:
     _log.debug("Failed to import gc3libs from GC3Pie."
                " Silently ignoring, this is a real issue only when GC3Pie is used as backend for --job")
@@ -124,7 +134,7 @@ def init(self):
             self.config_files.append(cfgfile)
 
         self.output_dir = build_option('job_output_dir')
-        self.jobs = DependentTaskCollection(output_dir=self.output_dir)
+        self.jobs = _BuildTaskCollection(output_dir=self.output_dir)
         self.job_cnt = 0
 
         # after polling for job status, sleep for this time duration
@@ -135,23 +145,20 @@ def make_job(self, script, name, env_vars=None, hours=None, cores=None):
         """
         Create and return a job object with the given parameters.
 
-        First argument `server` is an instance of the corresponding
-        `JobBackend` class, i.e., a `GC3Pie`:class: instance in this case.
-
-        Second argument `script` is the content of the job script
+        Argument *script* is the content of the job script
         itself, i.e., the sequence of shell commands that will be
         executed.
 
-        Third argument `name` sets the job human-readable name.
+        Argument *name* sets the job's human-readable name.
 
-        Fourth (optional) argument `env_vars` is a dictionary with
+        Optional argument *env_vars* is a dictionary with
         key-value pairs of environment variables that should be passed
         on to the job.
 
-        Fifth and sixth (optional) arguments `hours` and `cores` should be
+        Optional arguments *hours* and *cores* should be
         integer values:
-        * hours must be in the range 1 .. MAX_WALLTIME;
-        * cores depends on which cluster the job is being run.
+        - *hours* must be in the range 1 .. ``MAX_WALLTIME``;
+        - *cores* depends on which cluster the job is being run.
         """
         named_args = {
             'jobname': name, # job name in GC3Pie
@@ -236,7 +243,7 @@ def complete(self):
         # the application.
         self._engine.add(self.jobs)
 
-        # in case you want to select a specific resource, call
+        # select a specific execution resource?
         target_resource = build_option('job_target_resource')
         if target_resource:
             res = self._engine.select_resource(target_resource)


Test patch: