Repository: dask/dask

Issue:
`dask.async.remote_exception` fails for some exception types
```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-2-197e1ec89913> in <module>()
----> 1 model = build.build_all()

/Users/henriquealves/Code/nu/lusa-model/src/lusa/build.py in build_all()
     24     exceptions.CAPTURE_EXCEPTIONS = False
     25
---> 26     return build_model('score_time', mordor=mordor)

/Users/henriquealves/Code/nu/lusa-model/src/lusa/build.py in build_model(score_time_col, mordor)
      7
      8 def build_model(score_time_col, mordor):
----> 9     return dask_cached.DaskRunner().run(train.full_pipeline(mordor, score_time_col))
     10
     11

/Users/henriquealves/Code/nu/common-models/src/common_models/dask_cached.pyc in run(self, obj, prev)
     44         result = self.build(obj, prev)
     45         with dask.set_options(get=self.get_function):
---> 46             return result.compute(num_workers=self.num_workers)

/Users/henriquealves/anaconda/envs/lusa/lib/python2.7/site-packages/dask/base.pyc in compute(self, **kwargs)
     32
     33     def compute(self, **kwargs):
---> 34         return compute(self, **kwargs)[0]
     35
     36     @classmethod

/Users/henriquealves/anaconda/envs/lusa/lib/python2.7/site-packages/dask/base.pyc in compute(*args, **kwargs)
    100                 for opt, val in groups.items()])
    101     keys = [arg._keys() for arg in args]
--> 102     results = get(dsk, keys, **kwargs)
    103     return tuple(a._finalize(a, r) for a, r in zip(args, results))
    104

/Users/henriquealves/anaconda/envs/lusa/lib/python2.7/site-packages/dask/threaded.pyc in get(dsk, result, cache, num_workers, **kwargs)
     55     results = get_async(pool.apply_async, len(pool._pool), dsk, result,
     56                         cache=cache, queue=queue, get_id=_thread_get_id,
---> 57                         **kwargs)
     58
     59     return results

/Users/henriquealves/anaconda/envs/lusa/lib/python2.7/site-packages/dask/async.pyc in get_async(apply_async, num_workers, dsk, result, cache, queue, get_id, raise_on_exception, rerun_exceptions_locally, callbacks, **kwargs)
    481                 _execute_task(task, data)  # Re-execute locally
    482             else:
--> 483                 raise(remote_exception(res, tb))
    484         state['cache'][key] = res
    485         finish_task(dsk, key, state, results, keyorder.get)

/Users/henriquealves/anaconda/envs/lusa/lib/python2.7/site-packages/dask/async.pyc in remote_exception(exc, tb)
    585         typ = type(exc.__class__.__name__,
    586                    (RemoteException, type(exc)),
--> 587                    {'exception_type': type(exc)})
    588         exceptions[type(exc)] = typ
    589     return typ(exc, tb)

TypeError: Cannot create a consistent method resolution
order (MRO) for bases AssertionError, RemoteException
```



Discussion:
Hrm, I'm not sure I understand what about the class hierarchies here is causing the error

``` python
In [1]: from dask.async import RemoteException

In [2]: RemoteException.mro()
Out[2]: [dask.async.RemoteException, Exception, BaseException, object]

In [3]: AssertionError.mro()  # Python 2
Out[3]: [AssertionError, StandardError, Exception, BaseException, object]

In [3]: AssertionError.mro()  # Python 3
Out[3]: [AssertionError, Exception, BaseException, object]
```

I added a quick test and using `remote_exception` on `AssertionError`s seems to work fine in my simple case.  Can you post a small reproducible example?

I can't seem to reproduce locally. Using python 2.7:

``` python
>>> from dask.async import remote_exception
>>> exc = AssertionError()
>>> tb = "traceback"
>>> ra = remote_exception(exc, tb)
>>> ra
dask.async.AssertionError()
```

The original code is simply raising AssertionError. I managed to get the correct original traceback print by using the `rerun_exceptions_locally=True` option for `compute()`, although this is not optimal (runs twice).

```
<SNIP>
/Users/henriquealves/anaconda/envs/lusa/lib/python2.7/site-packages/dask/threaded.pyc in get(dsk, result, cache, num_workers, **kwargs)
     55     results = get_async(pool.apply_async, len(pool._pool), dsk, result,
     56                         cache=cache, queue=queue, get_id=_thread_get_id,
---> 57                         **kwargs)
     58
     59     return results

/Users/henriquealves/anaconda/envs/lusa/lib/python2.7/site-packages/dask/async.pyc in get_async(apply_async, num_workers, dsk, result, cache, queue, get_id, raise_on_exception, rerun_exceptions_locally, callbacks, **kwargs)
    481                 _execute_task(task, data)  # Re-execute locally
    482             else:
--> 483                 raise(remote_exception(res, tb))
    484         state['cache'][key] = res
    485         finish_task(dsk, key, state, results, keyorder.get)

AssertionError: `args` must be a list of arguments or a valid EDN string, got type <type 'datetime.datetime'>

Traceback
---------
  File "/Users/henriquealves/anaconda/envs/lusa/lib/python2.7/site-packages/dask/async.py", line 263, in execute_task
    result = _execute_task(task, data)
  File "/Users/henriquealves/anaconda/envs/lusa/lib/python2.7/site-packages/dask/async.py", line 245, in _execute_task
    return func(*args2)
  File "/Users/henriquealves/anaconda/envs/lusa/lib/python2.7/site-packages/imordor/imordor.py", line 248, in <lambda>
    **kwargs)
  File "/Users/henriquealves/anaconda/envs/lusa/lib/python2.7/site-packages/imordor/mordor_client.py", line 164, in ad_hoc
    payload = query_to_edn(query, args, timeout, since, as_of, history)
  File "/Users/henriquealves/anaconda/envs/lusa/lib/python2.7/site-packages/imordor/adapters/query.py", line 119, in query_to_edn
    args and join(':args', args_to_edn(args))])))
  File "/Users/henriquealves/anaconda/envs/lusa/lib/python2.7/site-packages/imordor/adapters/query.py", line 95, in args_to_edn
    raise AssertionError("`args` must be a list of arguments or a valid EDN string, got type {}".format(type(args)))
```

Depending on the `get` function I use, I see different error handling behavior. The above is using `dask.threaded`. What's the expected behavior?

If the function raises an `AssertionError` it should be wrapped up and presented to the user as follows:

``` python
In [1]: def f():
   ...:     assert False
   ...:     

In [2]: dsk = {'x': (f,)}

In [3]: import dask

In [4]: dask.threaded.get(dsk, 'x')

Traceback
---------
  File "/home/mrocklin/workspace/dask/dask/async.py", line 264, in execute_task
    result = _execute_task(task, data)
  File "/home/mrocklin/workspace/dask/dask/async.py", line 246, in _execute_task
    return func(*args2)
  File "<ipython-input-1-0dd086228188>", line 2, in f
    assert False
```

In your case it identifies that there is an `AssertionError` but gets confused as it packages it up and tries to present it to the user.  In our comments above we were diagnosing the way in which it became confused.  We were unable to reproduce this particular issue.

Its easy to add a fix for this but it's concerning that it's happening on your machine and we don't know why.  Are you sure that your `AssertionError` is a typical normal `AssertionError` as would be created by `assert False`?


Resolving patch:
diff --git a/dask/async.py b/dask/async.py
--- a/dask/async.py
+++ b/dask/async.py
@@ -580,9 +580,13 @@ def remote_exception(exc, tb):
     """ Metaclass that wraps exception type in RemoteException """
     if type(exc) in exceptions:
         typ = exceptions[type(exc)]
+        return typ(exc, tb)
     else:
-        typ = type(exc.__class__.__name__,
-                   (RemoteException, type(exc)),
-                   {'exception_type': type(exc)})
-        exceptions[type(exc)] = typ
-    return typ(exc, tb)
+        try:
+            typ = type(exc.__class__.__name__,
+                       (RemoteException, type(exc)),
+                       {'exception_type': type(exc)})
+            exceptions[type(exc)] = typ
+            return typ(exc, tb)
+        except TypeError:
+            return exc


Test patch: