Repository: Parsl/parsl

Issue:
Consider renaming `lazy_errors` and re-think behavior
In implementing the class-based configuration, I am writing docstring entries for things that we currently just have a longer description in the docs for. One of these is `lazyErrors`. In the [docs](http://parsl.readthedocs.io/en/latest/userguide/exceptions.html?highlight=lazyErrors#lazy-fail), we have this description titled `Lazy Fail` (even though the option is called 'lazyErrors'):
> While Retries address resiliency at the level of Apps, lazy failure adds resiliency at the workflow level. When lazy failures are enabled, the workflow does not halt as soon as it encounters a failure, but continues execution of every app that is unaffected. Lazy failures is the default behavior in parsl, with the expectation that when running production workflows, individual app failures can be deferred until the end of the workflow. During the development and testing of workflows, failing immediately on any failure is often preferred and this behavior is possible by setting lazyErrors=False.

I do not favor the name `lazy_fail` or `lazy_errors`. I'm not sure the lazy part is a perfect description of what we're doing, and I think it relies on background knowledge unnecessarily. I propose `halt_on_error` or `halt_on_failure`. In that way, the name would describe what is being done without assuming prior knowledge or a mental mapping between concepts. The docstring entry could be (I'm open to help tweaking this):
> Halt workflow processing and exit immediately if any apps fails.

We've been discussing batching all of the user-facing changes together, which is why I wanted to address this now.


Discussion:
More about the description in general rather than specifically the name:

i) better describe the trade-off between the two modes: non-lazy mode is useful in production *if* you can make use of the other stuff that is being run - for example, if you have sufficient checkpointing in place to usefully resume in a way that uses that later stuff, or because you have some other thing in your app which makes that useful. It's not clear to me at the moment that that is always (or even often) the case (even though we'd like it to be).

ii) The pasted text talks about "exit immediately" - is that really what happens?  or does a Python exception get thrown to the user's workflow code? and if so, where/by whom?

iii) I think my preference for name would be something along the lines of "continue as much as possible" - describing the unusual behaviour that parsl will provide *despite* failure.
> i) better describe the trade-off between the two modes: non-lazy mode is useful in production if you can make use of the other stuff that is being run - for example, if you have sufficient checkpointing in place to usefully resume in a way that uses that later stuff, or because you have some other thing in your app which makes that useful. It's not clear to me at the moment that that is always (or even often) the case (even though we'd like it to be).

Agreed.

> ii) The pasted text talks about "exit immediately" - is that really what happens? or does a Python exception get thrown to the user's workflow code? and if so, where/by whom?

Agreed. I will clarify this and post an updated proposal soon.

> iii) I think my preference for name would be something along the lines of "continue as much as possible" - describing the unusual behaviour that parsl will provide despite failure.

This is interesting. I think both of our instincts about what this should be called are informed by our assumptions about what constitutes "unusual" behavior. For me, continuing as much as possible would be my default assumption, and the entire workflow exiting if one task fails would be unusual behavior. Continuing as much as possible has been the default with other workflow systems I've worked with: Lobster, CRAB, Condor DAGMan, etc. What was the default in Swift? I suppose we should favor whatever the most common default assumption is that users would make, but I'm not sure how to determine that. Thinking about this more, maybe `halt_workflow_on_task_failure` would be a more precise (but verbose) version of my original proposal.

@danielskatz @kylechard @yadudoc @mjwilde Do you have any preferences here?
(After other discussion elsewhere:)

My view of how non-lazy errors should appear to the library user seems different to how the present implementation tries to present them (which is also different from what is really happening).

My view is that non-lazy errors are a request that errors be presented to the user "earlier rather than later" but it shouldn't otherwise affect what parsl is doing, or otherwise affect the enclosing app.

In lazy mode, exceptions are presented to the enclosing python code at fairly well defined points: mostly, when you ask a future for its result. At that point, you might get back an exception saying that some other piece of work has failed - not directly related to that specific future (a dependency exception).

In non-lazy mode, I can imagine the same thing happening:

When a task fails, parsl stops scheduling new tasks, perhaps tries to kill any tasks that are running now...

... but leaves the DFK usable in future, executors running, and reports the error only at the point that the enclosing app asks for a result. (a variant of a dependency exception).

If the app decides that it wants to abort back to a shell prompt, then it can do that (by not catching the exception, by default!) and *that* is when DFK shutdown happens, and  DFK-related cleanups like shutting down managed executors should happen there.

I think this is substantially less "fail hard" than at least what @yadudoc had been aiming for.
I support @benclifford's proposal.
I'm most likely repeating stuff that has been recorded above, but I want to put in information from a discussion : 
`lazy_errors=False` is ambiguous, and it might be simpler to use `halt_workflow/execution` to clarify the behavior that the user is enabling. ie. by enabling (`halt_workflow_on_failure=True`) any task failure would trigger the shutdown of the workflow including propagation of the exceptions involved if any new tasks submission is attempted.

There is the issue of raising the exception to the user when `halt_workflow_on_failure` is enabled. We are working with the assumption that if this option is enabled, we do not let the user handle the error and prevent the halting of the workflow. In addition, when an exception from a task is handled by "handle_update", we trigger cleanup shutting down provisioned resources. To prevent new tasks we might need to add functionality to submit and/or launch interfaces to prevent new tasks from being accepted by the dfk.
see #645 for some code working towards this

Resolving patch:
diff --git a/docs/userguide/examples/config.py b/docs/userguide/examples/config.py
--- a/docs/userguide/examples/config.py
+++ b/docs/userguide/examples/config.py
@@ -3,6 +3,5 @@
 from parsl.executors.threads import ThreadPoolExecutor
 
 local_threads = Config(
-  executors=[ThreadPoolExecutor(max_threads=4)],
-  lazy_errors=True
+  executors=[ThreadPoolExecutor(max_threads=4)]
 )
diff --git a/parsl/config.py b/parsl/config.py
--- a/parsl/config.py
+++ b/parsl/config.py
@@ -37,9 +37,6 @@ class Config(RepresentationMixin):
         Default is 10.
     monitoring : MonitoringHub, optional
         The config to use for database monitoring. Default is None which does not log to a database.
-    lazy_errors : bool, optional
-        If True, errors from task failures will not be raised until `future.result()` is called. Otherwise, they will
-        be raised as soon as the task returns. Default is True.
     retries : int, optional
         Set the number of retries in case of failure. Default is 0.
     run_dir : str, optional
@@ -71,7 +68,6 @@ def __init__(self,
                  checkpoint_mode: Optional[str] = None,
                  checkpoint_period: Optional[str] = None,
                  data_management_max_threads: int = 10,
-                 lazy_errors: bool = True,
                  retries: int = 0,
                  run_dir: str = 'runinfo',
                  strategy: Optional[str] = 'simple',
@@ -98,7 +94,6 @@ def __init__(self,
             checkpoint_period = "00:30:00"
         self.checkpoint_period = checkpoint_period
         self.data_management_max_threads = data_management_max_threads
-        self.lazy_errors = lazy_errors
         self.retries = retries
         self.run_dir = run_dir
         self.strategy = strategy
diff --git a/parsl/dataflow/dflow.py b/parsl/dataflow/dflow.py
--- a/parsl/dataflow/dflow.py
+++ b/parsl/dataflow/dflow.py
@@ -184,7 +184,7 @@ def __init__(self, config=Config()):
 
         atexit.register(self.atexit_cleanup)
 
-    def _create_task_log_info(self, task_id, fail_mode):
+    def _create_task_log_info(self, task_id):
         """
         Create the dictionary that will be included in the log.
         """
@@ -224,7 +224,6 @@ def _create_task_log_info(self, task_id, fail_mode):
         if self.tasks[task_id]['time_returned'] is not None:
             task_log_info['task_elapsed_time'] = (self.tasks[task_id]['time_returned'] -
                                                   self.tasks[task_id]['time_submitted']).total_seconds()
-        task_log_info['task_fail_mode'] = fail_mode
         return task_log_info
 
     def _count_deps(self, depends):
@@ -274,14 +273,6 @@ def handle_exec_update(self, task_id, future):
             self.tasks[task_id]['fail_history'].append(str(e))
             self.tasks[task_id]['fail_count'] += 1
 
-            if not self._config.lazy_errors:
-                logger.exception("Eager fail, skipping retry logic")
-                self.tasks[task_id]['status'] = States.failed
-                if self.monitoring:
-                    task_log_info = self._create_task_log_info(task_id, 'eager')
-                    self.monitoring.send(MessageType.TASK_INFO, task_log_info)
-                return
-
             if self.tasks[task_id]['status'] == States.dep_fail:
                 logger.info("Task {} failed due to dependency failure so skipping retries".format(task_id))
             elif self.tasks[task_id]['fail_count'] <= self._config.retries:
@@ -308,7 +299,7 @@ def handle_exec_update(self, task_id, future):
             logger.info("Standard error for task {} available at {}".format(task_id, self.tasks[task_id]['app_fu'].stderr))
 
         if self.monitoring:
-            task_log_info = self._create_task_log_info(task_id, 'lazy')
+            task_log_info = self._create_task_log_info(task_id)
             self.monitoring.send(MessageType.TASK_INFO, task_log_info)
 
         # it might be that in the course of the update, we've gone back to being
@@ -436,7 +427,7 @@ def launch_if_ready(self, task_id):
                 self.tasks_dep_fail_count += 1
 
                 if self.monitoring is not None:
-                    task_log_info = self._create_task_log_info(task_id, 'lazy')
+                    task_log_info = self._create_task_log_info(task_id)
                     self.monitoring.send(MessageType.TASK_INFO, task_log_info)
 
                 self.tasks[task_id]['retries_left'] = 0
@@ -505,7 +496,7 @@ def launch_task(self, task_id, executable, *args, **kwargs):
             exec_fu = executor.submit(executable, *args, **kwargs)
         self.tasks[task_id]['status'] = States.launched
         if self.monitoring is not None:
-            task_log_info = self._create_task_log_info(task_id, 'lazy')
+            task_log_info = self._create_task_log_info(task_id)
             self.monitoring.send(MessageType.TASK_INFO, task_log_info)
 
         self.tasks[task_id]['retries_left'] = self._config.retries - \


Test patch:
diff --git a/parsl/tests/test_threads/test_immediate_error.py b/parsl/tests/test_threads/test_immediate_error.py
deleted file mode 100644
--- a/parsl/tests/test_threads/test_immediate_error.py
+++ /dev/null
@@ -1,39 +0,0 @@
-import parsl
-import pytest
-from parsl import python_app
-from parsl.tests.configs.local_threads import config
-
-
-@pytest.mark.local
-@pytest.mark.skip("Broke somewhere between PR #525 and PR #652")
-def test_non_lazy_behavior():
-    """Testing non lazy errors to work"""
-
-    parsl.clear()
-    config.lazy_errors = False
-    parsl.load(config)
-
-    @python_app
-    def divide(a, b):
-        return a / b
-
-    try:
-        items = []
-        for i in range(0, 1):
-            items.append(divide(10, i))
-
-        while True:
-            if items[0].done:
-                break
-
-    except Exception as e:
-        assert isinstance(e, ZeroDivisionError), "Expected ZeroDivisionError, got: {}".format(e)
-    else:
-        raise Exception("Expected ZeroDivisionError, got nothing")
-
-    return
-
-
-if __name__ == "__main__":
-
-    test_non_lazy_behavior()
diff --git a/parsl/tests/test_threads/test_lazy_errors.py b/parsl/tests/test_threads/test_lazy_errors.py
--- a/parsl/tests/test_threads/test_lazy_errors.py
+++ b/parsl/tests/test_threads/test_lazy_errors.py
@@ -9,7 +9,6 @@ def test_lazy_behavior():
     """Testing that lazy errors work"""
 
     config = fresh_config()
-    config.lazy_errors = True
     parsl.load(config)
 
     @python_app