Repository: joblib/joblib

Issue:
Regression in 0.12: different exception type than PicklingError when arguments of Parallel can not be pickled
Moved from https://github.com/joblib/joblib/issues/706#issuecomment-400189831. Probably because of the switch from pickle to cloudpickle.

```py
import numpy as np

import pymc3 as pm

rng = np.random.RandomState(0)
count_data = rng.randint(1, 70, 100).astype(dtype=np.float64)
n_count_data = len(count_data)

with pm.Model() as model:
    alpha = 1.0/count_data.mean()  # Recall count_data is the
                                   # variable that holds our txt counts
    lambda_1 = pm.Exponential("lambda_1", alpha)
    lambda_2 = pm.Exponential("lambda_2", alpha)
    
    tau = pm.DiscreteUniform("tau", lower=0, upper=n_count_data - 1)

with model:
    step = pm.Metropolis()
    trace = pm.sample(5, tune=10,step=step)
```

More context from https://github.com/joblib/joblib/issues/706#issuecomment-400187390:
It seems like the problem comes from the fact that  pymc3 catches `PickleError` to fall-back to single threaded mode:

```pytb
~/miniconda3/lib/python3.6/site-packages/pymc3/sampling.py in sample(draws, step, init, n_init, start, trace, chain_idx, chains, cores, tune, nuts_kwargs, step_kwargs, progressbar, model, random_seed, live_plot, discard_tuned_samples, live_plot_kwargs, compute_convergence_checks, use_mmap, **kwargs)
    440         _print_step_hierarchy(step)
    441         try:
--> 442             trace = _mp_sample(**sample_args)
    443         except pickle.PickleError:
    444             _log.warning("Could not pickle model, sampling singlethreaded.")```
```

But now that joblib uses cloudpickle the error you get is a `TypeError`. Not 100% sure what's the best way to fix that, maybe catch all errors produced by cloudpickle.dump and reraise a PickleError?

Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/joblib/joblib/pull/708?src=pr&el=h1) Report
> Merging [#708](https://codecov.io/gh/joblib/joblib/pull/708?src=pr&el=desc) into [master](https://codecov.io/gh/joblib/joblib/commit/bb027ab7dfd01ce618c04a9bbac12f7fa1fe2857?src=pr&el=desc) will **decrease** coverage by `1.4%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/joblib/joblib/pull/708/graphs/tree.svg?token=gA6LF5DGTW&width=650&src=pr&height=150)](https://codecov.io/gh/joblib/joblib/pull/708?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #708      +/-   ##
==========================================
- Coverage    95.1%   93.69%   -1.41%     
==========================================
  Files          40       40              
  Lines        5746     5746              
==========================================
- Hits         5465     5384      -81     
- Misses        281      362      +81
```


| [Impacted Files](https://codecov.io/gh/joblib/joblib/pull/708?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [joblib/test/common.py](https://codecov.io/gh/joblib/joblib/pull/708/diff?src=pr&el=tree#diff-am9ibGliL3Rlc3QvY29tbW9uLnB5) | `74.57% <0%> (-13.56%)` | :arrow_down: |
| [joblib/testing.py](https://codecov.io/gh/joblib/joblib/pull/708/diff?src=pr&el=tree#diff-am9ibGliL3Rlc3RpbmcucHk=) | `87.5% <0%> (-7.5%)` | :arrow_down: |
| [joblib/backports.py](https://codecov.io/gh/joblib/joblib/pull/708/diff?src=pr&el=tree#diff-am9ibGliL2JhY2twb3J0cy5weQ==) | `89.58% <0%> (-6.25%)` | :arrow_down: |
| [joblib/pool.py](https://codecov.io/gh/joblib/joblib/pull/708/diff?src=pr&el=tree#diff-am9ibGliL3Bvb2wucHk=) | `85.34% <0%> (-6.04%)` | :arrow_down: |
| [joblib/func\_inspect.py](https://codecov.io/gh/joblib/joblib/pull/708/diff?src=pr&el=tree#diff-am9ibGliL2Z1bmNfaW5zcGVjdC5weQ==) | `91.42% <0%> (-3.43%)` | :arrow_down: |
| [joblib/test/test\_parallel.py](https://codecov.io/gh/joblib/joblib/pull/708/diff?src=pr&el=tree#diff-am9ibGliL3Rlc3QvdGVzdF9wYXJhbGxlbC5weQ==) | `93.71% <0%> (-2.92%)` | :arrow_down: |
| [joblib/numpy\_pickle.py](https://codecov.io/gh/joblib/joblib/pull/708/diff?src=pr&el=tree#diff-am9ibGliL251bXB5X3BpY2tsZS5weQ==) | `96.05% <0%> (-2.47%)` | :arrow_down: |
| [joblib/numpy\_pickle\_utils.py](https://codecov.io/gh/joblib/joblib/pull/708/diff?src=pr&el=tree#diff-am9ibGliL251bXB5X3BpY2tsZV91dGlscy5weQ==) | `91.39% <0%> (-2.16%)` | :arrow_down: |
| [joblib/test/test\_numpy\_pickle.py](https://codecov.io/gh/joblib/joblib/pull/708/diff?src=pr&el=tree#diff-am9ibGliL3Rlc3QvdGVzdF9udW1weV9waWNrbGUucHk=) | `96.32% <0%> (-1.51%)` | :arrow_down: |
| [joblib/parallel.py](https://codecov.io/gh/joblib/joblib/pull/708/diff?src=pr&el=tree#diff-am9ibGliL3BhcmFsbGVsLnB5) | `95.94% <0%> (-1.45%)` | :arrow_down: |
| ... and [7 more](https://codecov.io/gh/joblib/joblib/pull/708/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/joblib/joblib/pull/708?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/joblib/joblib/pull/708?src=pr&el=footer). Last update [bb027ab...1732ea8](https://codecov.io/gh/joblib/joblib/pull/708?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/joblib/externals/loky/__init__.py b/joblib/externals/loky/__init__.py
index 0935619..332adfe 100644
--- a/joblib/externals/loky/__init__.py
+++ b/joblib/externals/loky/__init__.py
@@ -9,4 +9,4 @@ from .process_executor import BrokenProcessPool  # noqa: F401
 
 from .backend.context import cpu_count  # noqa: F401
 
-__version__ = '2.1.3'
+__version__ = '2.1.4'
diff --git a/joblib/externals/loky/backend/utils.py b/joblib/externals/loky/backend/utils.py
index d4cc11a..28cf689 100644
--- a/joblib/externals/loky/backend/utils.py
+++ b/joblib/externals/loky/backend/utils.py
@@ -42,9 +42,9 @@ def _recursive_terminate(pid):
                 ["taskkill", "/F", "/T", "/PID", str(pid)],
                 stderr=None)
         except subprocess.CalledProcessError as e:
-            # In windows, taskkill return 1 for permission denied and 128 for
-            # no process found.
-            if e.returncode not in [1, 128]:
+            # In windows, taskkill return 1 for permission denied and 128, 255
+            # for no process found.
+            if e.returncode not in [1, 128, 255]:
                 raise
             elif e.returncode == 1:
                 # Try to kill the process without its descendants if taskkill
diff --git a/joblib/externals/loky/process_executor.py b/joblib/externals/loky/process_executor.py
index a788024..f537fce 100644
--- a/joblib/externals/loky/process_executor.py
+++ b/joblib/externals/loky/process_executor.py
@@ -69,6 +69,7 @@ import traceback
 import threading
 import multiprocessing as mp
 from functools import partial
+from pickle import PicklingError
 
 from . import _base
 from .backend import get_context
@@ -285,16 +286,19 @@ class _SafeQueue(Queue):
     def _on_queue_feeder_error(self, e, obj):
         if isinstance(obj, _CallItem):
             # fromat traceback only on python3
+            pickling_error = PicklingError(
+                "Could not pickle the task to send it to the workers.")
             tb = traceback.format_exception(
                 type(e), e, getattr(e, "__traceback__", None))
-            e.__cause__ = _RemoteTraceback('\n"""\n{}"""'.format(''.join(tb)))
+            pickling_error.__cause__ = _RemoteTraceback(
+                '\n"""\n{}"""'.format(''.join(tb)))
             work_item = self.pending_work_items.pop(obj.work_id, None)
             self.running_work_items.remove(obj.work_id)
             # work_item can be None if another process terminated. In this
             # case, the queue_manager_thread fails all work_items with
             # BrokenProcessPool
             if work_item is not None:
-                work_item.future.set_exception(e)
+                work_item.future.set_exception(pickling_error)
                 del work_item
             self.thread_wakeup.wakeup()
         else:


Test patch:
diff --git a/joblib/test/test_parallel.py b/joblib/test/test_parallel.py
index bdb6b42..3ec929c 100644
--- a/joblib/test/test_parallel.py
+++ b/joblib/test/test_parallel.py
@@ -14,6 +14,7 @@ import threading
 from traceback import format_exception
 from math import sqrt
 from time import sleep
+from pickle import PicklingError
 from multiprocessing import TimeoutError
 
 import joblib
@@ -28,14 +29,8 @@ from joblib._compat import PY3_OR_LATER, PY27
 
 try:
     import cPickle as pickle
-    PickleError = TypeError
 except ImportError:
     import pickle
-    PickleError = pickle.PicklingError
-
-
-if PY3_OR_LATER:
-    PickleError = pickle.PicklingError
 
 try:
     from queue import Queue
@@ -291,11 +286,11 @@ def test_parallel_pickling():
     """ Check that pmap captures the errors when it is passed an object
         that cannot be pickled.
     """
-    class UnpicklableObject:
+    class UnpicklableObject(object):
         def __reduce__(self):
             raise RuntimeError()
 
-    with raises(RuntimeError):
+    with raises(PicklingError):
         Parallel(n_jobs=2)(delayed(id)(UnpicklableObject()) for _ in range(10))