Repository: sktime/sktime

Issue:
[BUG] HIVECOTEV2 with n_jobs>1 crashes on internal dataset
**Describe the bug**
I want to fit HIVECOTEV2 on an internal dataset in "df-list" format with two classes. Running it in a singe thread `HIVECOTEV2(time_limit_in_minutes=1)` works as expected and outputs no warnings or errors. However I want to speed it up with multiprocessing `HIVECOTEV2(time_limit_in_minutes=1, n_jobs=14)` but this creates `NumbaPendingDeprecationWarning`:s, overflow warnings and then crashes with a `PicklingError`.

**To Reproduce**
Unfortunate I can't share the data since it contains sensitive personal information, please tell me if I should check something in the data. Hopefully you can figure something out from the logs.

Load and verify data:
```python
samples = load_data()

check_raise(samples[0], "pd.DataFrame")
check_raise(samples, "df-list")

print(any([df.isnull().values.any() for df in samples]))  # False
print(any([~np.isfinite(df.values).all() for df in samples]))  # False
```

Fit single process creates a working model without any warnings or errors.
```python
model = HIVECOTEV2(time_limit_in_minutes=1)
model.fit(samples, np.array(targets, dtype=int))
```

Fit multiprocessing shows warnings and then crashes.
```python
model = HIVECOTEV2(time_limit_in_minutes=1, n_jobs=14)
model.fit(samples, np.array(targets, dtype=int))
```
```
/home/johan/dev/vAll/lib/python3.10/site-packages/numba/core/ir_utils.py:2147: NumbaPendingDeprecationWarning: 
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'sorted_indicies' of function '_online_shapelet_distance'.

For more information visit https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../../../vAll/lib/python3.10/site-packages/sktime/transformations/panel/shapelet_transform.py", line 1473:
@njit(fastmath=True, cache=True)
def _online_shapelet_distance(series, shapelet, sorted_indicies, position, length):
^

  warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))
/home/johan/dev/vAll/lib/python3.10/site-packages/numba/core/ir_utils.py:2147: NumbaPendingDeprecationWarning: 
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'sorted_indicies' of function '_online_shapelet_distance'.

For more information visit https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../../../vAll/lib/python3.10/site-packages/sktime/transformations/panel/shapelet_transform.py", line 1473:
@njit(fastmath=True, cache=True)
def _online_shapelet_distance(series, shapelet, sorted_indicies, position, length):
^

  warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))
/home/johan/dev/vAll/lib/python3.10/site-packages/numba/core/ir_utils.py:2147: NumbaPendingDeprecationWarning: 
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'sorted_indicies' of function '_online_shapelet_distance'.

For more information visit https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../../../vAll/lib/python3.10/site-packages/sktime/transformations/panel/shapelet_transform.py", line 1473:
@njit(fastmath=True, cache=True)
def _online_shapelet_distance(series, shapelet, sorted_indicies, position, length):
^

  warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))
/home/johan/dev/vAll/lib/python3.10/site-packages/numba/core/ir_utils.py:2147: NumbaPendingDeprecationWarning: 
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'sorted_indicies' of function '_online_shapelet_distance'.

For more information visit https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../../../vAll/lib/python3.10/site-packages/sktime/transformations/panel/shapelet_transform.py", line 1473:
@njit(fastmath=True, cache=True)
def _online_shapelet_distance(series, shapelet, sorted_indicies, position, length):
^

  warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))
/home/johan/dev/vAll/lib/python3.10/site-packages/numba/core/ir_utils.py:2147: NumbaPendingDeprecationWarning: 
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'sorted_indicies' of function '_online_shapelet_distance'.

For more information visit https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../../../vAll/lib/python3.10/site-packages/sktime/transformations/panel/shapelet_transform.py", line 1473:
@njit(fastmath=True, cache=True)
def _online_shapelet_distance(series, shapelet, sorted_indicies, position, length):
^

  warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))
/home/johan/dev/vAll/lib/python3.10/site-packages/numba/core/ir_utils.py:2147: NumbaPendingDeprecationWarning: 
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'sorted_indicies' of function '_online_shapelet_distance'.

For more information visit https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../../../vAll/lib/python3.10/site-packages/sktime/transformations/panel/shapelet_transform.py", line 1473:
@njit(fastmath=True, cache=True)
def _online_shapelet_distance(series, shapelet, sorted_indicies, position, length):
^

  warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))
/home/johan/dev/vAll/lib/python3.10/site-packages/numba/core/ir_utils.py:2147: NumbaPendingDeprecationWarning: 
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'sorted_indicies' of function '_online_shapelet_distance'.

For more information visit https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../../../vAll/lib/python3.10/site-packages/sktime/transformations/panel/shapelet_transform.py", line 1473:
@njit(fastmath=True, cache=True)
def _online_shapelet_distance(series, shapelet, sorted_indicies, position, length):
^

  warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))
/home/johan/dev/vAll/lib/python3.10/site-packages/numba/core/ir_utils.py:2147: NumbaPendingDeprecationWarning: 
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'sorted_indicies' of function '_online_shapelet_distance'.

For more information visit https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../../../vAll/lib/python3.10/site-packages/sktime/transformations/panel/shapelet_transform.py", line 1473:
@njit(fastmath=True, cache=True)
def _online_shapelet_distance(series, shapelet, sorted_indicies, position, length):
^

  warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))
/home/johan/dev/vAll/lib/python3.10/site-packages/numba/core/ir_utils.py:2147: NumbaPendingDeprecationWarning: 
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'sorted_indicies' of function '_online_shapelet_distance'.

For more information visit https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../../../vAll/lib/python3.10/site-packages/sktime/transformations/panel/shapelet_transform.py", line 1473:
@njit(fastmath=True, cache=True)
def _online_shapelet_distance(series, shapelet, sorted_indicies, position, length):
^

  warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))
/home/johan/dev/vAll/lib/python3.10/site-packages/numba/core/ir_utils.py:2147: NumbaPendingDeprecationWarning: 
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'sorted_indicies' of function '_online_shapelet_distance'.

For more information visit https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../../../vAll/lib/python3.10/site-packages/sktime/transformations/panel/shapelet_transform.py", line 1473:
@njit(fastmath=True, cache=True)
def _online_shapelet_distance(series, shapelet, sorted_indicies, position, length):
^

  warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))
/home/johan/dev/vAll/lib/python3.10/site-packages/numba/core/ir_utils.py:2147: NumbaPendingDeprecationWarning: 
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'sorted_indicies' of function '_online_shapelet_distance'.

For more information visit https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../../../vAll/lib/python3.10/site-packages/sktime/transformations/panel/shapelet_transform.py", line 1473:
@njit(fastmath=True, cache=True)
def _online_shapelet_distance(series, shapelet, sorted_indicies, position, length):
^

  warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))
/home/johan/dev/vAll/lib/python3.10/site-packages/numba/core/ir_utils.py:2147: NumbaPendingDeprecationWarning: 
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'sorted_indicies' of function '_online_shapelet_distance'.

For more information visit https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../../../vAll/lib/python3.10/site-packages/sktime/transformations/panel/shapelet_transform.py", line 1473:
@njit(fastmath=True, cache=True)
def _online_shapelet_distance(series, shapelet, sorted_indicies, position, length):
^

  warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))
/home/johan/dev/vAll/lib/python3.10/site-packages/numba/core/ir_utils.py:2147: NumbaPendingDeprecationWarning: 
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'sorted_indicies' of function '_online_shapelet_distance'.

For more information visit https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../../../vAll/lib/python3.10/site-packages/sktime/transformations/panel/shapelet_transform.py", line 1473:
@njit(fastmath=True, cache=True)
def _online_shapelet_distance(series, shapelet, sorted_indicies, position, length):
^

  warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))
/home/johan/dev/vAll/lib/python3.10/site-packages/numba/core/ir_utils.py:2147: NumbaPendingDeprecationWarning: 
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'sorted_indicies' of function '_online_shapelet_distance'.

For more information visit https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../../../vAll/lib/python3.10/site-packages/sktime/transformations/panel/shapelet_transform.py", line 1473:
@njit(fastmath=True, cache=True)
def _online_shapelet_distance(series, shapelet, sorted_indicies, position, length):
^

  warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))
/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/interval_based/_drcif.py:517: RuntimeWarning: overflow encountered in multiply
  transformed_x = transformed_x.round(8)
/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/interval_based/_drcif.py:517: RuntimeWarning: overflow encountered in multiply
  transformed_x = transformed_x.round(8)
/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/interval_based/_drcif.py:517: RuntimeWarning: overflow encountered in multiply
  transformed_x = transformed_x.round(8)
/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/interval_based/_drcif.py:517: RuntimeWarning: overflow encountered in multiply
  transformed_x = transformed_x.round(8)
/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/interval_based/_drcif.py:517: RuntimeWarning: overflow encountered in multiply
  transformed_x = transformed_x.round(8)
/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/interval_based/_drcif.py:517: RuntimeWarning: overflow encountered in multiply
  transformed_x = transformed_x.round(8)
/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/interval_based/_drcif.py:517: RuntimeWarning: overflow encountered in multiply
  transformed_x = transformed_x.round(8)
joblib.externals.loky.process_executor._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/externals/loky/backend/queues.py", line 125, in _feed
    obj_ = dumps(obj, reducers=reducers)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/externals/loky/backend/reduction.py", line 211, in dumps
    dump(obj, buf, reducers=reducers, protocol=protocol)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/externals/loky/backend/reduction.py", line 204, in dump
    _LokyPickler(file, reducers=reducers, protocol=protocol).dump(obj)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/externals/cloudpickle/cloudpickle_fast.py", line 632, in dump
    return Pickler.dump(self, obj)
TypeError: cannot pickle '_nrt_python._MemInfo' object
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/johan/dev/app-drug-detection/app_drug_detection/approaches/johan_sktime/main.py", line 130, in <module>
    main()
  File "/home/johan/dev/app-drug-detection/app_drug_detection/approaches/johan_sktime/main.py", line 126, in main
    evaluator.evaluate_predictor(DummyBaseline, RydeEvaluator.NOT_IMPAIRED_THRESHOLD, RydeEvaluator.IMPAIRED_THRESHOLD)
  File "/home/johan/dev/app-drug-detection/app_drug_detection/common/evaluation.py", line 190, in evaluate_predictor
    pred = predictor.predict(metadata, timeseries)
  File "/home/johan/dev/app-drug-detection/app_drug_detection/approaches/johan_sktime/main.py", line 119, in predict
    pred = self.model.predict([timeseries])
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/base.py", line 228, in predict
    return self._predict(X)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/hybrid/_hivecote_v2.py", line 295, in _predict
    for prob in self.predict_proba(X)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/base.py", line 262, in predict_proba
    return self._predict_proba(X)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/hybrid/_hivecote_v2.py", line 331, in _predict_proba
    tde_probas = self._tde.predict_proba(X)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/base.py", line 262, in predict_proba
    return self._predict_proba(X)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/dictionary_based/_tde.py", line 411, in _predict_proba
    preds = clf.predict(X)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/base.py", line 228, in predict
    return self._predict(X)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/dictionary_based/_tde.py", line 879, in _predict
    classes = Parallel(n_jobs=self._threads_to_use)(
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/parallel.py", line 1098, in __call__
    self.retrieve()
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/parallel.py", line 975, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/_parallel_backends.py", line 567, in wrap_future_result
    return future.result(timeout=timeout)
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
_pickle.PicklingError: Could not pickle the task to send it to the workers.
)
```
**Versions**

System:
    python: 3.10.6 (main, Nov  2 2022, 18:53:38) [GCC 11.3.0]
executable: /home/johan/dev/vAll/bin/python
   machine: Linux-5.15.0-52-generic-x86_64-with-glibc2.35

Python dependencies:
          pip: 22.0.2
   setuptools: 59.6.0
      sklearn: 1.1.2
       sktime: 0.14.0
  statsmodels: 0.13.2
        numpy: 1.22.4
        scipy: 1.8.1
       pandas: 1.4.3
   matplotlib: 3.5.3
       joblib: 1.2.0
        numba: 0.56.3
     pmdarima: None
      tsfresh: None

Thanks!
[BUG] HIVECOTEV2 with n_jobs>1 crashes on internal dataset
**Describe the bug**
I want to fit HIVECOTEV2 on an internal dataset in "df-list" format with two classes. Running it in a singe thread `HIVECOTEV2(time_limit_in_minutes=1)` works as expected and outputs no warnings or errors. However I want to speed it up with multiprocessing `HIVECOTEV2(time_limit_in_minutes=1, n_jobs=14)` but this creates `NumbaPendingDeprecationWarning`:s, overflow warnings and then crashes with a `PicklingError`.

**To Reproduce**
Unfortunate I can't share the data since it contains sensitive personal information, please tell me if I should check something in the data. Hopefully you can figure something out from the logs.

Load and verify data:
```python
samples = load_data()

check_raise(samples[0], "pd.DataFrame")
check_raise(samples, "df-list")

print(any([df.isnull().values.any() for df in samples]))  # False
print(any([~np.isfinite(df.values).all() for df in samples]))  # False
```

Fit single process creates a working model without any warnings or errors.
```python
model = HIVECOTEV2(time_limit_in_minutes=1)
model.fit(samples, np.array(targets, dtype=int))
```

Fit multiprocessing shows warnings and then crashes.
```python
model = HIVECOTEV2(time_limit_in_minutes=1, n_jobs=14)
model.fit(samples, np.array(targets, dtype=int))
```
```
/home/johan/dev/vAll/lib/python3.10/site-packages/numba/core/ir_utils.py:2147: NumbaPendingDeprecationWarning: 
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'sorted_indicies' of function '_online_shapelet_distance'.

For more information visit https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../../../vAll/lib/python3.10/site-packages/sktime/transformations/panel/shapelet_transform.py", line 1473:
@njit(fastmath=True, cache=True)
def _online_shapelet_distance(series, shapelet, sorted_indicies, position, length):
^

  warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))
/home/johan/dev/vAll/lib/python3.10/site-packages/numba/core/ir_utils.py:2147: NumbaPendingDeprecationWarning: 
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'sorted_indicies' of function '_online_shapelet_distance'.

For more information visit https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../../../vAll/lib/python3.10/site-packages/sktime/transformations/panel/shapelet_transform.py", line 1473:
@njit(fastmath=True, cache=True)
def _online_shapelet_distance(series, shapelet, sorted_indicies, position, length):
^

  warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))
/home/johan/dev/vAll/lib/python3.10/site-packages/numba/core/ir_utils.py:2147: NumbaPendingDeprecationWarning: 
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'sorted_indicies' of function '_online_shapelet_distance'.

For more information visit https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../../../vAll/lib/python3.10/site-packages/sktime/transformations/panel/shapelet_transform.py", line 1473:
@njit(fastmath=True, cache=True)
def _online_shapelet_distance(series, shapelet, sorted_indicies, position, length):
^

  warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))
/home/johan/dev/vAll/lib/python3.10/site-packages/numba/core/ir_utils.py:2147: NumbaPendingDeprecationWarning: 
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'sorted_indicies' of function '_online_shapelet_distance'.

For more information visit https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../../../vAll/lib/python3.10/site-packages/sktime/transformations/panel/shapelet_transform.py", line 1473:
@njit(fastmath=True, cache=True)
def _online_shapelet_distance(series, shapelet, sorted_indicies, position, length):
^

  warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))
/home/johan/dev/vAll/lib/python3.10/site-packages/numba/core/ir_utils.py:2147: NumbaPendingDeprecationWarning: 
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'sorted_indicies' of function '_online_shapelet_distance'.

For more information visit https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../../../vAll/lib/python3.10/site-packages/sktime/transformations/panel/shapelet_transform.py", line 1473:
@njit(fastmath=True, cache=True)
def _online_shapelet_distance(series, shapelet, sorted_indicies, position, length):
^

  warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))
/home/johan/dev/vAll/lib/python3.10/site-packages/numba/core/ir_utils.py:2147: NumbaPendingDeprecationWarning: 
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'sorted_indicies' of function '_online_shapelet_distance'.

For more information visit https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../../../vAll/lib/python3.10/site-packages/sktime/transformations/panel/shapelet_transform.py", line 1473:
@njit(fastmath=True, cache=True)
def _online_shapelet_distance(series, shapelet, sorted_indicies, position, length):
^

  warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))
/home/johan/dev/vAll/lib/python3.10/site-packages/numba/core/ir_utils.py:2147: NumbaPendingDeprecationWarning: 
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'sorted_indicies' of function '_online_shapelet_distance'.

For more information visit https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../../../vAll/lib/python3.10/site-packages/sktime/transformations/panel/shapelet_transform.py", line 1473:
@njit(fastmath=True, cache=True)
def _online_shapelet_distance(series, shapelet, sorted_indicies, position, length):
^

  warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))
/home/johan/dev/vAll/lib/python3.10/site-packages/numba/core/ir_utils.py:2147: NumbaPendingDeprecationWarning: 
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'sorted_indicies' of function '_online_shapelet_distance'.

For more information visit https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../../../vAll/lib/python3.10/site-packages/sktime/transformations/panel/shapelet_transform.py", line 1473:
@njit(fastmath=True, cache=True)
def _online_shapelet_distance(series, shapelet, sorted_indicies, position, length):
^

  warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))
/home/johan/dev/vAll/lib/python3.10/site-packages/numba/core/ir_utils.py:2147: NumbaPendingDeprecationWarning: 
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'sorted_indicies' of function '_online_shapelet_distance'.

For more information visit https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../../../vAll/lib/python3.10/site-packages/sktime/transformations/panel/shapelet_transform.py", line 1473:
@njit(fastmath=True, cache=True)
def _online_shapelet_distance(series, shapelet, sorted_indicies, position, length):
^

  warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))
/home/johan/dev/vAll/lib/python3.10/site-packages/numba/core/ir_utils.py:2147: NumbaPendingDeprecationWarning: 
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'sorted_indicies' of function '_online_shapelet_distance'.

For more information visit https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../../../vAll/lib/python3.10/site-packages/sktime/transformations/panel/shapelet_transform.py", line 1473:
@njit(fastmath=True, cache=True)
def _online_shapelet_distance(series, shapelet, sorted_indicies, position, length):
^

  warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))
/home/johan/dev/vAll/lib/python3.10/site-packages/numba/core/ir_utils.py:2147: NumbaPendingDeprecationWarning: 
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'sorted_indicies' of function '_online_shapelet_distance'.

For more information visit https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../../../vAll/lib/python3.10/site-packages/sktime/transformations/panel/shapelet_transform.py", line 1473:
@njit(fastmath=True, cache=True)
def _online_shapelet_distance(series, shapelet, sorted_indicies, position, length):
^

  warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))
/home/johan/dev/vAll/lib/python3.10/site-packages/numba/core/ir_utils.py:2147: NumbaPendingDeprecationWarning: 
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'sorted_indicies' of function '_online_shapelet_distance'.

For more information visit https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../../../vAll/lib/python3.10/site-packages/sktime/transformations/panel/shapelet_transform.py", line 1473:
@njit(fastmath=True, cache=True)
def _online_shapelet_distance(series, shapelet, sorted_indicies, position, length):
^

  warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))
/home/johan/dev/vAll/lib/python3.10/site-packages/numba/core/ir_utils.py:2147: NumbaPendingDeprecationWarning: 
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'sorted_indicies' of function '_online_shapelet_distance'.

For more information visit https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../../../vAll/lib/python3.10/site-packages/sktime/transformations/panel/shapelet_transform.py", line 1473:
@njit(fastmath=True, cache=True)
def _online_shapelet_distance(series, shapelet, sorted_indicies, position, length):
^

  warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))
/home/johan/dev/vAll/lib/python3.10/site-packages/numba/core/ir_utils.py:2147: NumbaPendingDeprecationWarning: 
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'sorted_indicies' of function '_online_shapelet_distance'.

For more information visit https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../../../vAll/lib/python3.10/site-packages/sktime/transformations/panel/shapelet_transform.py", line 1473:
@njit(fastmath=True, cache=True)
def _online_shapelet_distance(series, shapelet, sorted_indicies, position, length):
^

  warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))
/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/interval_based/_drcif.py:517: RuntimeWarning: overflow encountered in multiply
  transformed_x = transformed_x.round(8)
/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/interval_based/_drcif.py:517: RuntimeWarning: overflow encountered in multiply
  transformed_x = transformed_x.round(8)
/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/interval_based/_drcif.py:517: RuntimeWarning: overflow encountered in multiply
  transformed_x = transformed_x.round(8)
/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/interval_based/_drcif.py:517: RuntimeWarning: overflow encountered in multiply
  transformed_x = transformed_x.round(8)
/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/interval_based/_drcif.py:517: RuntimeWarning: overflow encountered in multiply
  transformed_x = transformed_x.round(8)
/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/interval_based/_drcif.py:517: RuntimeWarning: overflow encountered in multiply
  transformed_x = transformed_x.round(8)
/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/interval_based/_drcif.py:517: RuntimeWarning: overflow encountered in multiply
  transformed_x = transformed_x.round(8)
joblib.externals.loky.process_executor._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/externals/loky/backend/queues.py", line 125, in _feed
    obj_ = dumps(obj, reducers=reducers)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/externals/loky/backend/reduction.py", line 211, in dumps
    dump(obj, buf, reducers=reducers, protocol=protocol)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/externals/loky/backend/reduction.py", line 204, in dump
    _LokyPickler(file, reducers=reducers, protocol=protocol).dump(obj)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/externals/cloudpickle/cloudpickle_fast.py", line 632, in dump
    return Pickler.dump(self, obj)
TypeError: cannot pickle '_nrt_python._MemInfo' object
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/johan/dev/app-drug-detection/app_drug_detection/approaches/johan_sktime/main.py", line 130, in <module>
    main()
  File "/home/johan/dev/app-drug-detection/app_drug_detection/approaches/johan_sktime/main.py", line 126, in main
    evaluator.evaluate_predictor(DummyBaseline, RydeEvaluator.NOT_IMPAIRED_THRESHOLD, RydeEvaluator.IMPAIRED_THRESHOLD)
  File "/home/johan/dev/app-drug-detection/app_drug_detection/common/evaluation.py", line 190, in evaluate_predictor
    pred = predictor.predict(metadata, timeseries)
  File "/home/johan/dev/app-drug-detection/app_drug_detection/approaches/johan_sktime/main.py", line 119, in predict
    pred = self.model.predict([timeseries])
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/base.py", line 228, in predict
    return self._predict(X)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/hybrid/_hivecote_v2.py", line 295, in _predict
    for prob in self.predict_proba(X)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/base.py", line 262, in predict_proba
    return self._predict_proba(X)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/hybrid/_hivecote_v2.py", line 331, in _predict_proba
    tde_probas = self._tde.predict_proba(X)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/base.py", line 262, in predict_proba
    return self._predict_proba(X)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/dictionary_based/_tde.py", line 411, in _predict_proba
    preds = clf.predict(X)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/base.py", line 228, in predict
    return self._predict(X)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/dictionary_based/_tde.py", line 879, in _predict
    classes = Parallel(n_jobs=self._threads_to_use)(
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/parallel.py", line 1098, in __call__
    self.retrieve()
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/parallel.py", line 975, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/_parallel_backends.py", line 567, in wrap_future_result
    return future.result(timeout=timeout)
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
_pickle.PicklingError: Could not pickle the task to send it to the workers.
)
```
**Versions**

System:
    python: 3.10.6 (main, Nov  2 2022, 18:53:38) [GCC 11.3.0]
executable: /home/johan/dev/vAll/bin/python
   machine: Linux-5.15.0-52-generic-x86_64-with-glibc2.35

Python dependencies:
          pip: 22.0.2
   setuptools: 59.6.0
      sklearn: 1.1.2
       sktime: 0.14.0
  statsmodels: 0.13.2
        numpy: 1.22.4
        scipy: 1.8.1
       pandas: 1.4.3
   matplotlib: 3.5.3
       joblib: 1.2.0
        numba: 0.56.3
     pmdarima: None
      tsfresh: None

Thanks!


Discussion:
I can reproduce the deprecation errors but not the `PicklingError` on a small sktime dataset. Do you have any information about your data i.e. number of cases, dimensions and series length?

Either way we should fix the deprecated stuff.
Sure:
- number of samples: 248
- number of dimensions: 70
- timeseries length: 1831
thanks @EyescannerJE we saw something similar to this with TDE, it is to do with the Parallel threading model writing to local memory. We will look into this now 
probably related to this https://github.com/sktime/sktime/issues/3582
I tried with TemporalDictionaryEnsemble now with the same result. `TemporalDictionaryEnsemble(time_limit_in_minutes=1)` works as expected but `TemporalDictionaryEnsemble(time_limit_in_minutes=1, n_jobs=14)` causes the exact same error as above.

```
joblib.externals.loky.process_executor._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/externals/loky/backend/queues.py", line 125, in _feed
    obj_ = dumps(obj, reducers=reducers)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/externals/loky/backend/reduction.py", line 211, in dumps
    dump(obj, buf, reducers=reducers, protocol=protocol)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/externals/loky/backend/reduction.py", line 204, in dump
    _LokyPickler(file, reducers=reducers, protocol=protocol).dump(obj)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/externals/cloudpickle/cloudpickle_fast.py", line 632, in dump
    return Pickler.dump(self, obj)
TypeError: cannot pickle '_nrt_python._MemInfo' object
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/johan/dev/app-drug-detection/app_drug_detection/approaches/johan_sktime/main.py", line 130, in <module>
    main()
  File "/home/johan/dev/app-drug-detection/app_drug_detection/approaches/johan_sktime/main.py", line 126, in main
    evaluator.evaluate_predictor(DummyBaseline, RydeEvaluator.NOT_IMPAIRED_THRESHOLD, RydeEvaluator.IMPAIRED_THRESHOLD)
  File "/home/johan/dev/app-drug-detection/app_drug_detection/common/evaluation.py", line 190, in evaluate_predictor
    pred = predictor.predict(metadata, timeseries)
  File "/home/johan/dev/app-drug-detection/app_drug_detection/approaches/johan_sktime/main.py", line 119, in predict
    pred = self.model.predict([timeseries])
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/base.py", line 228, in predict
    return self._predict(X)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/dictionary_based/_tde.py", line 384, in _predict
    for prob in self._predict_proba(X)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/dictionary_based/_tde.py", line 411, in _predict_proba
    preds = clf.predict(X)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/base.py", line 228, in predict
    return self._predict(X)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/dictionary_based/_tde.py", line 879, in _predict
    classes = Parallel(n_jobs=self._threads_to_use)(
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/parallel.py", line 1098, in __call__
    self.retrieve()
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/parallel.py", line 975, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/_parallel_backends.py", line 567, in wrap_future_result
    return future.result(timeout=timeout)
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
_pickle.PicklingError: Could not pickle the task to send it to the workers.
```
think its a Parallel thing, but it could be an edge case for short contract interact with Parallel. Could you try it with a longer contract and fewer jobs? thanks. 
I ran it again with a time limit of 60 minutes and only 2 jobs. It did run for 60 minutes but then crashed with the same error.
```python
HIVECOTEV2(time_limit_in_minutes=60, n_jobs=2)
```
```
/home/johan/dev/vAll/lib/python3.10/site-packages/numba/core/ir_utils.py:2147: NumbaPendingDeprecationWarning: 
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'sorted_indicies' of function '_online_shapelet_distance'.

For more information visit https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../vAll/lib/python3.10/site-packages/sktime/transformations/panel/shapelet_transform.py", line 1473:
@njit(fastmath=True, cache=True)
def _online_shapelet_distance(series, shapelet, sorted_indicies, position, length):
^

  warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))
/home/johan/dev/vAll/lib/python3.10/site-packages/numba/core/ir_utils.py:2147: NumbaPendingDeprecationWarning: 
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'sorted_indicies' of function '_online_shapelet_distance'.

For more information visit https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../vAll/lib/python3.10/site-packages/sktime/transformations/panel/shapelet_transform.py", line 1473:
@njit(fastmath=True, cache=True)
def _online_shapelet_distance(series, shapelet, sorted_indicies, position, length):
^

  warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))
/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/interval_based/_drcif.py:517: RuntimeWarning: overflow encountered in multiply
  transformed_x = transformed_x.round(8)
/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/interval_based/_drcif.py:517: RuntimeWarning: overflow encountered in multiply
  transformed_x = transformed_x.round(8)
/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/interval_based/_drcif.py:517: RuntimeWarning: overflow encountered in multiply
  transformed_x = transformed_x.round(8)
/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/interval_based/_drcif.py:517: RuntimeWarning: overflow encountered in multiply
  transformed_x = transformed_x.round(8)
/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/interval_based/_drcif.py:517: RuntimeWarning: overflow encountered in multiply
  transformed_x = transformed_x.round(8)
/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/externals/loky/process_executor.py:700: UserWarning: A worker stopped while some jobs were given to the executor. This can be caused by a too short worker timeout or by a memory leak.
  warnings.warn(
Evaluating:   0%|                                                               | 0/3 [1:02:02<?, ?it/s]
joblib.externals.loky.process_executor._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/externals/loky/backend/queues.py", line 125, in _feed
    obj_ = dumps(obj, reducers=reducers)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/externals/loky/backend/reduction.py", line 211, in dumps
    dump(obj, buf, reducers=reducers, protocol=protocol)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/externals/loky/backend/reduction.py", line 204, in dump
    _LokyPickler(file, reducers=reducers, protocol=protocol).dump(obj)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/externals/cloudpickle/cloudpickle_fast.py", line 632, in dump
    return Pickler.dump(self, obj)
TypeError: cannot pickle '_nrt_python._MemInfo' object
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/johan/dev/app-drug-detection/app_drug_detection/approaches/johan_sktime/main.py", line 130, in <module>
    main()
  File "/home/johan/dev/app-drug-detection/app_drug_detection/approaches/johan_sktime/main.py", line 126, in main
    evaluator.evaluate_predictor(DummyBaseline, RydeEvaluator.NOT_IMPAIRED_THRESHOLD, RydeEvaluator.IMPAIRED_THRESHOLD)
  File "/home/johan/dev/app-drug-detection/app_drug_detection/common/evaluation.py", line 190, in evaluate_predictor
    pred = predictor.predict(metadata, timeseries)
  File "/home/johan/dev/app-drug-detection/app_drug_detection/approaches/johan_sktime/main.py", line 119, in predict
    pred = self.model.predict([timeseries])
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/base.py", line 228, in predict
    return self._predict(X)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/hybrid/_hivecote_v2.py", line 295, in _predict
    for prob in self.predict_proba(X)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/base.py", line 262, in predict_proba
    return self._predict_proba(X)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/hybrid/_hivecote_v2.py", line 331, in _predict_proba
    tde_probas = self._tde.predict_proba(X)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/base.py", line 262, in predict_proba
    return self._predict_proba(X)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/dictionary_based/_tde.py", line 411, in _predict_proba
    preds = clf.predict(X)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/base.py", line 228, in predict
    return self._predict(X)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/dictionary_based/_tde.py", line 879, in _predict
    classes = Parallel(n_jobs=self._threads_to_use)(
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/parallel.py", line 1098, in __call__
    self.retrieve()
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/parallel.py", line 975, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/_parallel_backends.py", line 567, in wrap_future_result
    return future.result(timeout=timeout)
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
_pickle.PicklingError: Could not pickle the task to send it to the workers.
```
thank you for doing that, we will look into this
hi, so we are tracking this down, there is some kind of interference in TDE predict_prob. This crashes 

```
def _debug_threaded_tde_3788_numpy(n_cases=100, n_dims=2, series_length=100):
    from sktime.classification.dictionary_based import TemporalDictionaryEnsemble
    trainX = np.random.rand(n_cases, n_dims, series_length)
    trainY = np.random.randint(0, 2, n_cases)
    tde = TemporalDictionaryEnsemble(n_jobs=2, time_limit_in_minutes=1)
    print(" beginning fit numpy")
    tde.fit(trainX, trainY)
    print(" beginning predict numpy")
    preds = tde.predict(trainX)
```
with _debug_threaded_tde_3788_numpy()
but not with _debug_threaded_tde_3788_numpy(n_dims=1)
I can reproduce the deprecation errors but not the `PicklingError` on a small sktime dataset. Do you have any information about your data i.e. number of cases, dimensions and series length?

Either way we should fix the deprecated stuff.
Sure:
- number of samples: 248
- number of dimensions: 70
- timeseries length: 1831
thanks @EyescannerJE we saw something similar to this with TDE, it is to do with the Parallel threading model writing to local memory. We will look into this now 
probably related to this https://github.com/sktime/sktime/issues/3582
I tried with TemporalDictionaryEnsemble now with the same result. `TemporalDictionaryEnsemble(time_limit_in_minutes=1)` works as expected but `TemporalDictionaryEnsemble(time_limit_in_minutes=1, n_jobs=14)` causes the exact same error as above.

```
joblib.externals.loky.process_executor._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/externals/loky/backend/queues.py", line 125, in _feed
    obj_ = dumps(obj, reducers=reducers)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/externals/loky/backend/reduction.py", line 211, in dumps
    dump(obj, buf, reducers=reducers, protocol=protocol)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/externals/loky/backend/reduction.py", line 204, in dump
    _LokyPickler(file, reducers=reducers, protocol=protocol).dump(obj)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/externals/cloudpickle/cloudpickle_fast.py", line 632, in dump
    return Pickler.dump(self, obj)
TypeError: cannot pickle '_nrt_python._MemInfo' object
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/johan/dev/app-drug-detection/app_drug_detection/approaches/johan_sktime/main.py", line 130, in <module>
    main()
  File "/home/johan/dev/app-drug-detection/app_drug_detection/approaches/johan_sktime/main.py", line 126, in main
    evaluator.evaluate_predictor(DummyBaseline, RydeEvaluator.NOT_IMPAIRED_THRESHOLD, RydeEvaluator.IMPAIRED_THRESHOLD)
  File "/home/johan/dev/app-drug-detection/app_drug_detection/common/evaluation.py", line 190, in evaluate_predictor
    pred = predictor.predict(metadata, timeseries)
  File "/home/johan/dev/app-drug-detection/app_drug_detection/approaches/johan_sktime/main.py", line 119, in predict
    pred = self.model.predict([timeseries])
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/base.py", line 228, in predict
    return self._predict(X)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/dictionary_based/_tde.py", line 384, in _predict
    for prob in self._predict_proba(X)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/dictionary_based/_tde.py", line 411, in _predict_proba
    preds = clf.predict(X)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/base.py", line 228, in predict
    return self._predict(X)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/dictionary_based/_tde.py", line 879, in _predict
    classes = Parallel(n_jobs=self._threads_to_use)(
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/parallel.py", line 1098, in __call__
    self.retrieve()
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/parallel.py", line 975, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/_parallel_backends.py", line 567, in wrap_future_result
    return future.result(timeout=timeout)
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
_pickle.PicklingError: Could not pickle the task to send it to the workers.
```
think its a Parallel thing, but it could be an edge case for short contract interact with Parallel. Could you try it with a longer contract and fewer jobs? thanks. 
I ran it again with a time limit of 60 minutes and only 2 jobs. It did run for 60 minutes but then crashed with the same error.
```python
HIVECOTEV2(time_limit_in_minutes=60, n_jobs=2)
```
```
/home/johan/dev/vAll/lib/python3.10/site-packages/numba/core/ir_utils.py:2147: NumbaPendingDeprecationWarning: 
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'sorted_indicies' of function '_online_shapelet_distance'.

For more information visit https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../vAll/lib/python3.10/site-packages/sktime/transformations/panel/shapelet_transform.py", line 1473:
@njit(fastmath=True, cache=True)
def _online_shapelet_distance(series, shapelet, sorted_indicies, position, length):
^

  warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))
/home/johan/dev/vAll/lib/python3.10/site-packages/numba/core/ir_utils.py:2147: NumbaPendingDeprecationWarning: 
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'sorted_indicies' of function '_online_shapelet_distance'.

For more information visit https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../vAll/lib/python3.10/site-packages/sktime/transformations/panel/shapelet_transform.py", line 1473:
@njit(fastmath=True, cache=True)
def _online_shapelet_distance(series, shapelet, sorted_indicies, position, length):
^

  warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))
/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/interval_based/_drcif.py:517: RuntimeWarning: overflow encountered in multiply
  transformed_x = transformed_x.round(8)
/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/interval_based/_drcif.py:517: RuntimeWarning: overflow encountered in multiply
  transformed_x = transformed_x.round(8)
/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/interval_based/_drcif.py:517: RuntimeWarning: overflow encountered in multiply
  transformed_x = transformed_x.round(8)
/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/interval_based/_drcif.py:517: RuntimeWarning: overflow encountered in multiply
  transformed_x = transformed_x.round(8)
/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/interval_based/_drcif.py:517: RuntimeWarning: overflow encountered in multiply
  transformed_x = transformed_x.round(8)
/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/externals/loky/process_executor.py:700: UserWarning: A worker stopped while some jobs were given to the executor. This can be caused by a too short worker timeout or by a memory leak.
  warnings.warn(
Evaluating:   0%|                                                               | 0/3 [1:02:02<?, ?it/s]
joblib.externals.loky.process_executor._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/externals/loky/backend/queues.py", line 125, in _feed
    obj_ = dumps(obj, reducers=reducers)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/externals/loky/backend/reduction.py", line 211, in dumps
    dump(obj, buf, reducers=reducers, protocol=protocol)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/externals/loky/backend/reduction.py", line 204, in dump
    _LokyPickler(file, reducers=reducers, protocol=protocol).dump(obj)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/externals/cloudpickle/cloudpickle_fast.py", line 632, in dump
    return Pickler.dump(self, obj)
TypeError: cannot pickle '_nrt_python._MemInfo' object
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/johan/dev/app-drug-detection/app_drug_detection/approaches/johan_sktime/main.py", line 130, in <module>
    main()
  File "/home/johan/dev/app-drug-detection/app_drug_detection/approaches/johan_sktime/main.py", line 126, in main
    evaluator.evaluate_predictor(DummyBaseline, RydeEvaluator.NOT_IMPAIRED_THRESHOLD, RydeEvaluator.IMPAIRED_THRESHOLD)
  File "/home/johan/dev/app-drug-detection/app_drug_detection/common/evaluation.py", line 190, in evaluate_predictor
    pred = predictor.predict(metadata, timeseries)
  File "/home/johan/dev/app-drug-detection/app_drug_detection/approaches/johan_sktime/main.py", line 119, in predict
    pred = self.model.predict([timeseries])
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/base.py", line 228, in predict
    return self._predict(X)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/hybrid/_hivecote_v2.py", line 295, in _predict
    for prob in self.predict_proba(X)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/base.py", line 262, in predict_proba
    return self._predict_proba(X)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/hybrid/_hivecote_v2.py", line 331, in _predict_proba
    tde_probas = self._tde.predict_proba(X)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/base.py", line 262, in predict_proba
    return self._predict_proba(X)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/dictionary_based/_tde.py", line 411, in _predict_proba
    preds = clf.predict(X)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/base.py", line 228, in predict
    return self._predict(X)
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/sktime/classification/dictionary_based/_tde.py", line 879, in _predict
    classes = Parallel(n_jobs=self._threads_to_use)(
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/parallel.py", line 1098, in __call__
    self.retrieve()
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/parallel.py", line 975, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "/home/johan/dev/vAll/lib/python3.10/site-packages/joblib/_parallel_backends.py", line 567, in wrap_future_result
    return future.result(timeout=timeout)
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
_pickle.PicklingError: Could not pickle the task to send it to the workers.
```
thank you for doing that, we will look into this
hi, so we are tracking this down, there is some kind of interference in TDE predict_prob. This crashes 

```
def _debug_threaded_tde_3788_numpy(n_cases=100, n_dims=2, series_length=100):
    from sktime.classification.dictionary_based import TemporalDictionaryEnsemble
    trainX = np.random.rand(n_cases, n_dims, series_length)
    trainY = np.random.randint(0, 2, n_cases)
    tde = TemporalDictionaryEnsemble(n_jobs=2, time_limit_in_minutes=1)
    print(" beginning fit numpy")
    tde.fit(trainX, trainY)
    print(" beginning predict numpy")
    preds = tde.predict(trainX)
```
with _debug_threaded_tde_3788_numpy()
but not with _debug_threaded_tde_3788_numpy(n_dims=1)

Resolving patch:
diff --git a/sktime/classification/dictionary_based/_tde.py b/sktime/classification/dictionary_based/_tde.py
--- a/sktime/classification/dictionary_based/_tde.py
+++ b/sktime/classification/dictionary_based/_tde.py
@@ -465,7 +465,7 @@ def _get_train_probs(self, X, y, train_estimate_method="loocv") -> np.ndarray:
                 preds = (
                     clf._train_predictions
                     if self.save_train_predictions
-                    else Parallel(n_jobs=self._threads_to_use)(
+                    else Parallel(n_jobs=self._threads_to_use, prefer="threads")(
                         delayed(clf._train_predict)(
                             i,
                         )
@@ -510,7 +510,7 @@ def _individual_train_acc(self, tde, y, train_size, lowest_acc):
         required_correct = int(lowest_acc * train_size)
 
         if self._threads_to_use > 1:
-            c = Parallel(n_jobs=self._threads_to_use)(
+            c = Parallel(n_jobs=self._threads_to_use, prefer="threads")(
                 delayed(tde._train_predict)(
                     i,
                 )
@@ -878,7 +878,7 @@ def _predict(self, X):
             test_bags = self._transformers[0].transform(X)
             test_bags = test_bags[0]
 
-        classes = Parallel(n_jobs=self._threads_to_use)(
+        classes = Parallel(n_jobs=self._threads_to_use, prefer="threads")(
             delayed(self._test_nn)(
                 test_bag,
             )


Test patch: