Repository: Parsl/parsl

Issue:
Failing stageout does not cause any failure, and apps complete even with failed stageout
AppFutures and DataFutures are marked as finished before output stage-out happens.

DataFutures marked as successfully should probably mean that stage-out has happened successfully.

Possibly AppFutures should not be marked as successfully complete until stage out has happened either.

I've tested this with the below code, running to my own Globus Connect Server installation, to demonstrate this - the code reports that the test has completed successfully, and then: in the case that stageout doesn't work due to permissions, hangs rather than exiting (see #762); or in the case of a successful transfer, performs the transfer after the test has reported successfully.

Stage out jobs are submitted as separate tasks at the point that an app completes. Probably all of the stage out jobs should be joined on, and that join point is when the App and DataFutures report themselves as complete.

If stageout fails, then this is not reported except in debug logs. In particular, the test suite does not detect failing stageouts.


Discussion:
Just for context, I think the thought process at the time was that, we planned two distinct types of data movement:
1. Wide-area transfers with globus, usually onto a persistent-node.
2. Data transfer between transient nodes (worker-login)

Globus and such transfer modes run from the parsl-client side and can be restarted/retried and data loss is unlikely. Here decoupling AppFutures and DataFutures made some sense.

In the second mode (which we don't support yet), staging in/out is packaged together with the task, and the task completely only when the data has been moved out. We assume data is not yet available until it is moved to client-side.

Resolving patch:
diff --git a/parsl/app/bash.py b/parsl/app/bash.py
--- a/parsl/app/bash.py
+++ b/parsl/app/bash.py
@@ -2,7 +2,6 @@
 from inspect import signature, Parameter
 
 from parsl.app.errors import wrap_error
-from parsl.app.futures import DataFuture
 from parsl.app.app import AppBase
 from parsl.dataflow.dflow import DataFlowKernelLoader
 
@@ -163,8 +162,4 @@ def __call__(self, *args, **kwargs):
                              cache=self.cache,
                              **self.kwargs)
 
-        out_futs = [DataFuture(app_fut, o, tid=app_fut.tid)
-                    for o in kwargs.get('outputs', [])]
-        app_fut._outputs = out_futs
-
         return app_fut
diff --git a/parsl/app/python.py b/parsl/app/python.py
--- a/parsl/app/python.py
+++ b/parsl/app/python.py
@@ -3,7 +3,6 @@
 import tblib.pickling_support
 tblib.pickling_support.install()
 
-from parsl.app.futures import DataFuture
 from parsl.app.app import AppBase
 from parsl.app.errors import wrap_error
 from parsl.dataflow.dflow import DataFlowKernelLoader
@@ -72,10 +71,4 @@ def __call__(self, *args, **kwargs):
                              cache=self.cache,
                              **kwargs)
 
-        # logger.debug("App[{}] assigned Task[{}]".format(self.func.__name__,
-        #                                                 app_fut.tid))
-        out_futs = [DataFuture(app_fut, o, tid=app_fut.tid)
-                    for o in kwargs.get('outputs', [])]
-        app_fut._outputs = out_futs
-
         return app_fut
diff --git a/parsl/dataflow/dflow.py b/parsl/dataflow/dflow.py
--- a/parsl/dataflow/dflow.py
+++ b/parsl/dataflow/dflow.py
@@ -21,6 +21,7 @@
 
 import parsl
 from parsl.app.errors import RemoteExceptionWrapper
+from parsl.app.futures import DataFuture
 from parsl.config import Config
 from parsl.data_provider.data_manager import DataManager
 from parsl.data_provider.files import File
@@ -488,17 +489,26 @@ def _add_input_deps(self, executor, args, kwargs):
 
         return tuple(newargs), kwargs
 
-    def _add_output_deps(self, executor, args, kwargs, app_fu):
+    def _add_output_deps(self, executor, args, kwargs, app_fut):
         logger.debug("Adding output dependencies")
-
-        if not self.check_staging_inhibited(kwargs):
-            outputs = kwargs.get('outputs', [])
-            for f in outputs:
-                if isinstance(f, File) and f.is_remote():
-                    logger.debug("Submitting stage out job for output file {}".format(f))
-                    self.data_manager.stage_out(f, executor, app_fu)
-                else:
-                    logger.debug("Skipping stageout for output {}".format(f))
+        inhibit = self.check_staging_inhibited(kwargs)
+        outputs = kwargs.get('outputs', [])
+        app_fut._outputs = []
+        for f in outputs:
+            if isinstance(f, File) and f.is_remote() and not inhibit:
+                # with remote stageout, the DataFuture will be complete when
+                # the staging task is complete
+                logger.debug("Submitting stage out job for output file {}".format(f))
+                stageout_fut = self.data_manager.stage_out(f, executor, app_fut)
+                app_fut._outputs.append(DataFuture(stageout_fut, f, tid=app_fut.tid))
+            else:
+                # with local stageout, the DataFuture will be complete when
+                # the core task future is complete
+                # with remote URLs, if staging is inhibited, the app future completing
+                # will signal that stage-out is complete - because that app *is* the
+                # stageout task
+                logger.debug("Skipping stageout for output {}".format(f))
+                app_fut._outputs.append(DataFuture(app_fut, f, tid=app_fut.tid))
 
     def _gather_all_deps(self, args, kwargs):
         """Count the number of unresolved futures on which a task depends.


Test patch: