Repository: Parsl/parsl

Issue:
Sporadic globus staging error
I get the below error sporadically on my laptop - I haven't seen it before today, but I've been ramping up my globus testing recently so I can't tell if it is a recent error or not.

```
$ python parsl/tests/test_staging/test_implicit_staging_globus.py  --debug
2019-07-30 11:20:54 parsl.dataflow.dflow:694 [INFO]  Task 1 submitted for App wrapper, waiting on tasks []
2019-07-30 11:20:54 parsl.dataflow.dflow:704 [DEBUG]  Task 1 set to pending state with AppFuture: <AppFuture super=<AppFuture at 0x7f6c07090e10 state=pending> parent=None>
2019-07-30 11:20:54 parsl.dataflow.dflow:470 [INFO]  Task 1 launched on executor data_manager
2019-07-30 11:20:54 parsl.app.futures:73 [DEBUG]  Creating DataFuture with parent: <AppFuture super=<AppFuture at 0x7f6c07090e10 state=pending> parent=<Future at 0x7f6c07090e80 state=running>>
2019-07-30 11:20:54 parsl.app.futures:74 [DEBUG]  Filepath: /home/benc/parsl/src/parsl/unsorted.txt
2019-07-30 11:20:54 parsl.dataflow.dflow:694 [INFO]  Task 0 submitted for App sort_strings, waiting on tasks [1]
2019-07-30 11:20:54 parsl.dataflow.dflow:704 [DEBUG]  Task 0 set to pending state with AppFuture: <AppFuture super=<AppFuture at 0x7f6c07090dd8 state=pending> parent=None>
2019-07-30 11:20:54 parsl.app.futures:73 [DEBUG]  Creating DataFuture with parent: <AppFuture super=<AppFuture at 0x7f6c07090dd8 state=pending> parent=None>
2019-07-30 11:20:54 parsl.app.futures:74 [DEBUG]  Filepath: sorted.txt
2019-07-30 11:21:57 parsl.app.errors:158 [DEBUG]  Reraising exception of type <class 'IndexError'>
2019-07-30 11:21:57 parsl.dataflow.dflow:257 [ERROR]  Task 1 failed
Traceback (most recent call last):
  File "/home/benc/parsl/src/parsl/parsl/dataflow/dflow.py", line 254, in handle_exec_update
    res.reraise()
  File "/home/benc/parsl/src/parsl/parsl/app/errors.py", line 163, in reraise
    reraise(t, v, tb)
  File "/home/benc/parsl/virtualenv/lib/python3.5/site-packages/six.py", line 692, in reraise
    raise value.with_traceback(tb)
  File "/home/benc/parsl/src/parsl/parsl/app/errors.py", line 172, in wrapper
    return func(*args, **kwargs)
  File "/home/benc/parsl/src/parsl/parsl/data_provider/globus.py", line 255, in _globus_stage_in
    file.path, dst_path)
  File "/home/benc/parsl/src/parsl/parsl/data_provider/globus.py", line 101, in transfer_file
    event = events.data[0]
IndexError: list index out of range
2019-07-30 11:21:57 parsl.dataflow.dflow:278 [INFO]  Task 1 failed after 0 retry attempts
```


Discussion:
In some testing, I found that `events.data`, a list of events, was changing in value as this section of globus.py executes, including having an event when `task_event_list` returns, but then having no events by the time `events.data[0]` tries to extract the event data.

I'm unclear what the concurrent behaviour of that event list is supposed to be.
@chard points out that `events` is an iterable which will only return results when iterated over the first time, and subsequent iterations will return no results.

This probably explains some of the behaviour I was seeing in debugging the original error.

Resolving patch:
diff --git a/parsl/data_provider/globus.py b/parsl/data_provider/globus.py
--- a/parsl/data_provider/globus.py
+++ b/parsl/data_provider/globus.py
@@ -98,7 +98,11 @@ def transfer_file(cls, src_ep, dst_ep, src_path, dst_path):
             task = tc.get_task(task['task_id'])
             # Get the last error Globus event
             events = tc.task_event_list(task['task_id'], num_results=1, filter='is_error:1')
-            event = events.data[0]
+            try:
+                event = next(events)
+            # No error reported,  the transfer is still running
+            except StopIteration:
+                continue
             # Print the error event to stderr and Parsl file log if it was not yet printed
             if event['time'] != last_event_time:
                 last_event_time = event['time']


Test patch: