Repository: Parsl/parsl

Issue:
Parsl internal bugs silently squashed then hangs
@benclifford reported an alarming bug in which a error in the parsl code does not throw an exception but is silently squashed. This is followed by the code hanging and not making progress. To make the situation even worse the strategy thread continues to run, giving the impression that parsl is doing something. 

My suspicion is the the exception silently kills some thread, and that goes unreported.

The string formatting bug added here (in memoization code) -> https://github.com/Parsl/parsl/tree/pr238-related-example reproduces this issue with a test script that hits this memoization code. 

Some recommendations:
1. Better exception handling
2. Logs to show thread-ids so that we can see when threads have gone missing


Discussion:
I had a bit of a poke with stack traces, and I think exceptions thrown inside a future callback such as `handle_update` probably hit this:

https://docs.python.org/3/library/concurrent.futures.html

> If the callable raises an Exception subclass, it will be logged and ignored. 

... and indeed with more logging turned on by this: (to log everything, not just parsl logs)

```
parsl.set_stream_logger(name=None)
```

... I see this message now on the log output:

```
2018-05-21 12:00:11 concurrent.futures:299 [ERROR]  exception calling callback for <Future at 0x7ff1dd136cf8 state=finished returned int>
Traceback (most recent call last):
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 297, in _invoke_callbacks
    callback(self)
  File "/home/benc/parsl/src/parsl/parsl/dataflow/dflow.py", line 235, in handle_update
    self.checkpoint(tasks=[task_id])
  File "/home/benc/parsl/src/parsl/parsl/dataflow/dflow.py", line 629, in checkpoint
    raise ValueError("BENC: deliberate checkpoint failure")
ValueError: BENC: deliberate checkpoint failure
```

A short term workaround might be to attach logging to `concurrent.futures` so that we can see this as log output; it doesn't fix the broader problem of parsl ignoring the exception.

Resolving patch:
diff --git a/parsl/__init__.py b/parsl/__init__.py
--- a/parsl/__init__.py
+++ b/parsl/__init__.py
@@ -71,6 +71,12 @@ def set_stream_logger(name='parsl', level=logging.DEBUG, format_string=None):
     handler.setFormatter(formatter)
     logger.addHandler(handler)
 
+    # Concurrent.futures errors are also of interest, as exceptions
+    # which propagate out of the top of a callback are logged this way
+    # and then discarded. (see #240)
+    futures_logger = logging.getLogger("concurrent.futures")
+    futures_logger.addHandler(handler)
+
 
 def set_file_logger(filename, name='parsl', level=logging.DEBUG, format_string=None):
     """Add a stream log handler.
@@ -95,6 +101,11 @@ def set_file_logger(filename, name='parsl', level=logging.DEBUG, format_string=N
     handler.setFormatter(formatter)
     logger.addHandler(handler)
 
+    # see note in set_stream_logger for notes about logging
+    # concurrent.futures
+    futures_logger = logging.getLogger("concurrent.futures")
+    futures_logger.addHandler(handler)
+
 
 class NullHandler(logging.Handler):
     """Setup default logging to /dev/null since this is library."""


Test patch: