Repository: sktime/sktime

Issue:
[BUG] Numba error when setting n_jobs
**Describe the bug**
<!--
-->
Numba error when setting n_jobs

**To Reproduce**
<!--
-->
How can i upload my dataset? It has to be related to my dataset because it works fine with sktime datasets.

```python
estimators = [("oscillator", DrCIF(n_jobs=6), oscillator_columns)]
ensemble_model = ColumnEnsembleClassifier(estimators=estimators, remainder="passthrough")
ensemble_model.fit(exog, endog)
```

**Additional context**
<!--
-->
I get the following error when I fit the classifier and set n_jobs (to 6 for exemple). I have double check my dataset, everything seems fine (ive tried applying np.median to all entries and no problem with that).

```
numba.core.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
No implementation of function Function(<function median at 0x7fecff6e0160>) found for signature:
 
 >>> median(readonly array(float64, 1d, C))
 
There are 2 candidate implementations:
  - Of which 2 did not match due to:
  Overload in function 'np_median': File: numba/np/arraymath.py: Line 1447.
    With argument(s): '(readonly array(float64, 1d, C))':
   Rejected as the implementation raised a specific error:
     TypingError: Failed in nopython mode pipeline (step: nopython frontend)
   No implementation of function Function(<function _median_inner at 0x7fecdee57820>) found for signature:
    
    >>> _median_inner(readonly array(float64, 1d, C), int64)
    
   There are 2 candidate implementations:
         - Of which 2 did not match due to:
         Overload in function 'register_jitable.<locals>.wrap.<locals>.ov_wrap': File: numba/core/extending.py: Line 159.
           With argument(s): '(readonly array(float64, 1d, C), int64)':
          Rejected as the implementation raised a specific error:
            TypingError: Failed in nopython mode pipeline (step: nopython frontend)
          No implementation of function Function(<function _select_two at 0x7fecdee57700>) found for signature:
           
           >>> _select_two(readonly array(float64, 1d, C), int64, Literal[int](0), int64)
           
          There are 2 candidate implementations:
                - Of which 1 did not match due to:
                Overload in function 'register_jitable.<locals>.wrap.<locals>.ov_wrap': File: numba/core/extending.py: Line 159.
                  With argument(s): '(readonly array(float64, 1d, C), int64, int64, int64)':
                 Rejected as the implementation raised a specific error:
                   TypingError: Failed in nopython mode pipeline (step: nopython frontend)
                 No implementation of function Function(<function _partition_factory.<locals>._partition at 0x7fecdee57280>) found for signature:
                  
                  >>> _partition(readonly array(float64, 1d, C), int64, int64)
                  
                 There are 2 candidate implementations:
                       - Of which 2 did not match due to:
                       Overload in function 'register_jitable.<locals>.wrap.<locals>.ov_wrap': File: numba/core/extending.py: Line 159.
                         With argument(s): '(readonly array(float64, 1d, C), int64, int64)':
                        Rejected as the implementation raised a specific error:
                          TypingError: Failed in nopython mode pipeline (step: nopython frontend)
                        No implementation of function Function(<built-in function setitem>) found for signature:
                         
                         >>> setitem(readonly array(float64, 1d, C), int64, float64)
                         
                        There are 16 candidate implementations:
                              - Of which 14 did not match due to:
                              Overload of function 'setitem': File: <numerous>: Line N/A.
                                With argument(s): '(readonly array(float64, 1d, C), int64, float64)':
                               No match.
                              - Of which 2 did not match due to:
                              Overload in function 'SetItemBuffer.generic': File: numba/core/typing/arraydecl.py: Line 176.
                                With argument(s): '(readonly array(float64, 1d, C), int64, float64)':
                               Rejected as the implementation raised a specific error:
                                 NumbaTypeError: Cannot modify readonly array of type: readonly array(float64, 1d, C)
                          raised from /home/henry/anaconda3/lib/python3.9/site-packages/numba/core/typing/arraydecl.py:183
                        
                        During: typing of setitem at /home/henry/anaconda3/lib/python3.9/site-packages/numba/np/arraymath.py (1354)
                        
                        File "../../anaconda3/lib/python3.9/site-packages/numba/np/arraymath.py", line 1354:
                            def _partition(A, low, high):
                                <source elided>
                                if pivotimpl(A[mid], A[low]):
                                    A[low], A[mid] = A[mid], A[low]
                                    ^
                 
                   raised from /home/henry/anaconda3/lib/python3.9/site-packages/numba/core/typeinfer.py:1086
                 
                 During: resolving callee type: Function(<function _partition_factory.<locals>._partition at 0x7fecdee57280>)
                 During: typing of call at /home/henry/anaconda3/lib/python3.9/site-packages/numba/np/arraymath.py (1416)
                 
                 
                 File "../../anaconda3/lib/python3.9/site-packages/numba/np/arraymath.py", line 1416:
                 def _select_two(arry, k, low, high):
                     <source elided>
                         assert high > low  # by construction
                         i = _partition(arry, low, high)
                         ^
          
            raised from /home/henry/anaconda3/lib/python3.9/site-packages/numba/core/typeinfer.py:1086
                - Of which 1 did not match due to:
                Overload in function 'register_jitable.<locals>.wrap.<locals>.ov_wrap': File: numba/core/extending.py: Line 159.
                  With argument(s): '(readonly array(float64, 1d, C), int64, Literal[int](0), int64)':
                 Rejected as the implementation raised a specific error:
                   TypingError: Failed in nopython mode pipeline (step: nopython frontend)
                 No implementation of function Function(<function _partition_factory.<locals>._partition at 0x7fecdee57280>) found for signature:
                  
                  >>> _partition(readonly array(float64, 1d, C), Literal[int](0), int64)
                  
                 There are 2 candidate implementations:
                       - Of which 1 did not match due to:
                       Overload in function 'register_jitable.<locals>.wrap.<locals>.ov_wrap': File: numba/core/extending.py: Line 159.
                         With argument(s): '(readonly array(float64, 1d, C), int64, int64)':
                        Rejected as the implementation raised a specific error:
                          TypingError: Failed in nopython mode pipeline (step: nopython frontend)
                        No implementation of function Function(<built-in function setitem>) found for signature:
                         
                         >>> setitem(readonly array(float64, 1d, C), int64, float64)
                         
                        There are 16 candidate implementations:
                              - Of which 14 did not match due to:
                              Overload of function 'setitem': File: <numerous>: Line N/A.
                                With argument(s): '(readonly array(float64, 1d, C), int64, float64)':
                               No match.
                              - Of which 2 did not match due to:
                              Overload in function 'SetItemBuffer.generic': File: numba/core/typing/arraydecl.py: Line 176.
                                With argument(s): '(readonly array(float64, 1d, C), int64, float64)':
                               Rejected as the implementation raised a specific error:
                                 NumbaTypeError: Cannot modify readonly array of type: readonly array(float64, 1d, C)
                          raised from /home/henry/anaconda3/lib/python3.9/site-packages/numba/core/typing/arraydecl.py:183
                        
                        During: typing of setitem at /home/henry/anaconda3/lib/python3.9/site-packages/numba/np/arraymath.py (1354)
                        
                        File "../../anaconda3/lib/python3.9/site-packages/numba/np/arraymath.py", line 1354:
                            def _partition(A, low, high):
                                <source elided>
                                if pivotimpl(A[mid], A[low]):
                                    A[low], A[mid] = A[mid], A[low]
                                    ^
                 
                   raised from /home/henry/anaconda3/lib/python3.9/site-packages/numba/core/typeinfer.py:1086
                       - Of which 1 did not match due to:
                       Overload in function 'register_jitable.<locals>.wrap.<locals>.ov_wrap': File: numba/core/extending.py: Line 159.
                         With argument(s): '(readonly array(float64, 1d, C), Literal[int](0), int64)':
                        Rejected as the implementation raised a specific error:
                          TypingError: Failed in nopython mode pipeline (step: nopython frontend)
                        No implementation of function Function(<built-in function setitem>) found for signature:
                         
                         >>> setitem(readonly array(float64, 1d, C), Literal[int](0), float64)
                         
                        There are 16 candidate implementations:
                              - Of which 14 did not match due to:
                              Overload of function 'setitem': File: <numerous>: Line N/A.
                                With argument(s): '(readonly array(float64, 1d, C), int64, float64)':
                               No match.
                              - Of which 2 did not match due to:
                              Overload in function 'SetItemBuffer.generic': File: numba/core/typing/arraydecl.py: Line 176.
                                With argument(s): '(readonly array(float64, 1d, C), int64, float64)':
                               Rejected as the implementation raised a specific error:
                                 NumbaTypeError: Cannot modify readonly array of type: readonly array(float64, 1d, C)
                          raised from /home/henry/anaconda3/lib/python3.9/site-packages/numba/core/typing/arraydecl.py:183
                        
                        During: typing of setitem at /home/henry/anaconda3/lib/python3.9/site-packages/numba/np/arraymath.py (1354)
                        
                        File "../../anaconda3/lib/python3.9/site-packages/numba/np/arraymath.py", line 1354:
                            def _partition(A, low, high):
                                <source elided>
                                if pivotimpl(A[mid], A[low]):
                                    A[low], A[mid] = A[mid], A[low]
                                    ^
                 
                   raised from /home/henry/anaconda3/lib/python3.9/site-packages/numba/core/typeinfer.py:1086
                 
                 During: resolving callee type: Function(<function _partition_factory.<locals>._partition at 0x7fecdee57280>)
                 During: typing of call at /home/henry/anaconda3/lib/python3.9/site-packages/numba/np/arraymath.py (1416)
                 
                 
                 File "../../anaconda3/lib/python3.9/site-packages/numba/np/arraymath.py", line 1416:
                 def _select_two(arry, k, low, high):
                     <source elided>
                         assert high > low  # by construction
                         i = _partition(arry, low, high)
                         ^
          
            raised from /home/henry/anaconda3/lib/python3.9/site-packages/numba/core/typeinfer.py:1086
          
          During: resolving callee type: Function(<function _select_two at 0x7fecdee57700>)
          During: typing of call at /home/henry/anaconda3/lib/python3.9/site-packages/numba/np/arraymath.py (1441)
          
          
          File "../../anaconda3/lib/python3.9/site-packages/numba/np/arraymath.py", line 1441:
          def _median_inner(temp_arry, n):
              <source elided>
              if n & 1 == 0:
                  a, b = _select_two(temp_arry, half - 1, low, high)
                  ^
   
     raised from /home/henry/anaconda3/lib/python3.9/site-packages/numba/core/typeinfer.py:1086
   
   During: resolving callee type: Function(<function _median_inner at 0x7fecdee57820>)
   During: typing of call at /home/henry/anaconda3/lib/python3.9/site-packages/numba/np/arraymath.py (1457)
   
   
   File "../../anaconda3/lib/python3.9/site-packages/numba/np/arraymath.py", line 1457:
       def median_impl(a):
           <source elided>
           n = temp_arry.shape[0]
           return _median_inner(temp_arry, n)
           ^

  raised from /home/henry/anaconda3/lib/python3.9/site-packages/numba/core/typeinfer.py:1086

During: resolving callee type: Function(<function median at 0x7fecff6e0160>)
During: typing of call at /home/henry/anaconda3/lib/python3.9/site-packages/sktime/utils/numba/stats.py (17)


File "../../anaconda3/lib/python3.9/site-packages/sktime/utils/numba/stats.py", line 17:
def median(X):
    <source elided>
    """Numba median function for a single time series."""
    return np.median(X)
```

<!--
Please run the following code snippet and paste the output here:
 
from sktime import show_versions; show_versions()
-->

System:
    python: 3.9.12 (main, Apr  5 2022, 06:56:58)  [GCC 7.5.0]
executable: /home/henry/anaconda3/bin/python
   machine: Linux-5.15.0-50-generic-x86_64-with-glibc2.31

Python dependencies:
          pip: 21.2.4
   setuptools: 61.2.0
      sklearn: 1.0.2
       sktime: 0.13.4
  statsmodels: 0.13.2
        numpy: 1.21.5
        scipy: 1.7.3
       pandas: 1.4.2
   matplotlib: 3.5.1
       joblib: 1.1.0
        numba: 0.55.1
     pmdarima: None
      tsfresh: None

<!-- Thanks for contributing! -->



Discussion:
Hi, I don't mind taking a look at this. If you are allowed to share your data (which you indicate towards in your issue) you can send it to me at m.middlehurst@uea.ac.uk, else some data characteristics such as the size would be helpful. How are you loading/formatting the data prior to fitting?
Could it be because the dataset is too big?

By the way, I'd like to ask will the model be trained exactly the same way if we fit it with successive chunks of the dataset vs. fitting directly the the whole dataset in one call function call?
(My question applies to these especially: TimeSeriesForestClassifier, SupervisedTimeSeriesForest, CanonicalIntervalForest, DrCIF, RandomIntervalSpectralEnsemble)
Calling fit will overwrite any previous fit calls. Probably some quirk with threading/n_jobs>1 and numba, but not taken a look yet.
Okay I have sent you the dataset and notebook to reproduce the issue by email. thanks!
Seems to be a numba issue related to readonly arrays: https://github.com/numba/numba/issues/3070#issue-337198698

The solution might be:

```python
x = np.array(...)
x.setflags(write=True)
```
thanks @patrickzib, we were just looking at this and were wondering if it were just np.median/numba related or a wider issue with interference with the disk writing used in mmap.  One solution (from @dguijo) was to set 
fit = Parallel(n_jobs=self._threads_to_use, mmap_mode='w+')

I'm just concerned that by making the mmap writable by all threads, they will interfere. This would happen if, for example, first process had not finished writing to the temp file before second looks to see if the mmap is there. It also requires changing a lot of files for a bug localised on median (it now seems, thanks to that thread). 

Another solution is to just not make median a numba operation. 
some possible solutions:
1. set the memory map to be writable everywhere.  mmap_mode='w+'. Dangerous and requires changes to a large number of files 
2. Dont use numba for finding the median until this is (hopefully) fixed. Some performance hit for what is not perhaps a common  use case: parallel run on large data (although I am doing this myself atm!).
3. Always set arrays to be writeable in the median function, as @patrickzib suggests x.setflags(write=True). Seems a bit dangerous and would not really meet the user expectations? 
4. Clone data in median if passed a non writable array (@MatthewMiddlehurst). e.g.
```
def median(X):
    if X.flags.writeable:
        return _median(X)
    else:
        X = X.copy()
        X.setflags(write=True)
        return _median(X)


@njit(fastmath=True, cache=True)
def _median(X):
    """Numba median function for a single time series."""
    return np.median(X)
``` 
Introduces a time and memory overhead, but safer than (3) 

I dont like (1) as a widespread solution. Perhaps we should profile (2), (3) and (4) and try quantify the hit, (4) seems safest
 
Just to confirm, replacing the `median` method in `sktime\utils\numba\stats.py` with the above code allows me to run `DrCIF` with `n_jobs=4` on the data provided by @henrydeclety. With the previous method, this crashed with the same error as the original issue.

Resolving patch:
diff --git a/sktime/classification/sklearn/_continuous_interval_tree.py b/sktime/classification/sklearn/_continuous_interval_tree.py
--- a/sktime/classification/sklearn/_continuous_interval_tree.py
+++ b/sktime/classification/sklearn/_continuous_interval_tree.py
@@ -18,7 +18,7 @@
 from sklearn.utils import check_random_state
 
 from sktime.exceptions import NotFittedError
-from sktime.utils.numba.stats import iqr, mean, median, numba_max, numba_min, slope, std
+from sktime.utils.numba.stats import iqr, mean, numba_max, numba_min, slope, std
 from sktime.utils.validation.panel import check_X
 
 
@@ -670,7 +670,7 @@ def _summary_stat(X, att):
     elif att == 24:
         function = slope
     elif att == 25:
-        function = median
+        function = np.median
     elif att == 26:
         function = iqr
     elif att == 27:
diff --git a/sktime/utils/numba/stats.py b/sktime/utils/numba/stats.py
--- a/sktime/utils/numba/stats.py
+++ b/sktime/utils/numba/stats.py
@@ -11,12 +11,6 @@ def mean(X):
     return np.mean(X)
 
 
-@njit(fastmath=True, cache=True)
-def median(X):
-    """Numba median function for a single time series."""
-    return np.median(X)
-
-
 @njit(fastmath=True, cache=True)
 def std(X):
     """Numba standard deviation function for a single time series."""


Test patch: