Repository: sktime/sktime

Issue:
[BUG] parallelism in `evaluate` breaks if deprecated `kwargs` are used, in 0.24.0 version
**Describe the bug**
<!--
A clear and concise description of what the bug is.
-->
0.24.0 release introduced the bug in parallelism related to the DASK library. The same code running with 0.23.0 doesn't fail.

**To Reproduce**
<!--
Add a Minimal, Complete, and Verifiable example (for more details, see e.g. https://stackoverflow.com/help/mcve

If the code is too long, feel free to put it in a public gist and link it in the issue: https://gist.github.com
-->

```python
from sktime.datasets import load_airline
import pandas as pd
from sktime.forecasting.naive import NaiveForecaster
from sktime.forecasting.compose import (
    make_reduction,
    TransformedTargetForecaster,
)
from sktime.transformations.series.detrend.mstl import MSTL
from sktime.transformations.series.impute import Imputer
from sktime.forecasting.model_evaluation import evaluate
from sktime.forecasting.model_selection import ExpandingWindowSplitter
from sklearn.ensemble import RandomForestRegressor

y = load_airline()
y.index = pd.PeriodIndex(y.index, freq="d")
data = pd.DataFrame(y)

for i in range(1, 4):
    data['Lag_' + str(i)] = data[y.name].shift(i)
data.dropna(inplace=True)
    
X = data.drop(columns=y.name)
y = data[y.name]

pipe = TransformedTargetForecaster(steps=[
    ("imputer", Imputer(method="mean")),
    ("forecaster", NaiveForecaster(strategy="drift")),
])

cv = ExpandingWindowSplitter(initial_window=15, step_length=6, fh=[1, 2, 3])
# ls = ["dask", "loky", "multiprocessing", "threading"]
# for l in ls:
results = evaluate(forecaster=pipe, X=X, y=y, cv=cv,  strategy='refit', return_data=True, n_jobs = 5,backend='dask') 
n=results.shape[0]
df_y_pred = pd.concat([results['y_pred'][i] for i in range(n)])
```
**Error 1**
 cv = ExpandingWindowSplitter(initial_window=15, step_length=6, fh=[1, 2, 3])
     [31](vscode-notebook-cell:/home/artur/src/model/play_test/testsktime.ipynb#W1sZmlsZQ%3D%3D?line=30) # ls = ["dask", "loky", "multiprocessing", "threading"]
     [32](vscode-notebook-cell:/home/artur/src/model/play_test/testsktime.ipynb#W1sZmlsZQ%3D%3D?line=31) # for l in ls:
---> [33](vscode-notebook-cell:/home/artur/src/model/play_test/testsktime.ipynb#W1sZmlsZQ%3D%3D?line=32) results = evaluate(forecaster=pipe, X=X, y=y, cv=cv,  strategy='refit', return_data=True, n_jobs = 5,backend='dask') 
     [34](vscode-notebook-cell:/home/artur/src/model/play_test/testsktime.ipynb#W1sZmlsZQ%3D%3D?line=33) n=results.shape[0]
     [35](vscode-notebook-cell:/home/artur/src/model/play_test/testsktime.ipynb#W1sZmlsZQ%3D%3D?line=34) df_y_pred = pd.concat([results['y_pred'][i] for i in range(n)])

File [~/src/model/venv/lib/python3.11/site-packages/sktime/forecasting/model_evaluation/_functions.py:516](https://file+.vscode-resource.vscode-cdn.net/home/artur/src/model/play_test/~/src/model/venv/lib/python3.11/site-packages/sktime/forecasting/model_evaluation/_functions.py:516), in evaluate(forecaster, cv, y, X, strategy, scoring, return_data, error_score, backend, compute, cv_X, backend_params, **kwargs)
    514 if backend in ["dask", "dask_lazy"]:
    515     if not _check_soft_dependencies("dask", severity="none"):
--> 516         raise RuntimeError(
    517             "running evaluate with backend='dask' requires the dask package "
    518             "installed, but dask is not present in the python environment"
    519         )
    521 # todo 0.25.0: remove kwargs and this warning
    522 if kwargs != {}:

RuntimeError: running evaluate with backend='dask' requires the dask package installed, but dask is not present in the python environment

**Expected behavior**
<!--
A clear and concise description of what you expected to happen.
-->
The same behaviour as it was with 0.23.0

**Additional context**

If I manually install dask (pip install dask), then the first error disappears the code goes through but fails with a new error, see below.
**Error 2**


---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
[...) Cell 2 line 3
     [33](...) results = evaluate(forecaster=pipe, X=X, y=y, cv=cv,  strategy='refit', return_data=True, n_jobs = 5,backend='dask') 
     [34](...) n=results.shape[0]
---> [35](...) df_y_pred = pd.concat([results['y_pred'][i] for i in range(n)])

TypeError: 'Delayed' object cannot be interpreted as an integer

**Versions**
<details>
sktime 0.24.0

System:
    python: 3.11.4 (main, Jul  5 2023, 14:15:25) [GCC 11.2.0]
executable: [/home/artur/src/model/venv/bin/python](https://file+.vscode-resource.vscode-cdn.net/home/artur/src/model/venv/bin/python)
   machine: Linux-6.2.0-34-generic-x86_64-with-glibc2.37

Python dependencies:
          pip: 23.1.2
       sktime: 0.24.0
      sklearn: 1.3.0
       skbase: 0.6.0
        numpy: 1.25.2
        scipy: 1.11.1
       pandas: 2.0.3
   matplotlib: 3.7.1
       joblib: 1.3.0
        numba: None
  statsmodels: 0.14.0
     pmdarima: None
statsforecast: None
      tsfresh: None
      tslearn: None
        torch: None
   tensorflow: None
tensorflow_probability: None
</details>

<!-- Thanks for contributing! -->



Discussion:
The first error is since you do not have `dask` installed, so not a bug - it tells you to install `dask`.

The second seems to be coming from your own code - can you spell out what your expectation is?
FYI @hazrulakmal, is it possible that my refactor broke a type expectation on the fields of the return dataframe that is not tested?
> The first error is since you do not have `dask` installed, so not a bug - it tells you to install `dask`.
> 
> The second seems to be coming from your own code - can you spell out what your expectation is?

Absolutely! The expectation would be that n has an integer type as it were with 0.23.0, and I would be able to loop over the rows of the evaluate outputted dataset and save the y_pred to a dataframe. For me it's fine to work with 0.23.0 version for now but I wanted to mention the change.

e.g. with 0.23.0 I would receive an output.
df_y_pred.head()
1950-07-01    149.0
1950-07-02    149.0
1950-07-03    149.0
1951-01-01    140.0
1951-01-02    140.0
Freq: D, Name: Number of airline passengers, dtype: float64
What is it post-0.24.0 for you?

I.e., the precise type of the `DataFrame` entries in the `"y_pred"` column, if you do not mind the question?

Resolving patch:
diff --git a/sktime/forecasting/model_evaluation/_functions.py b/sktime/forecasting/model_evaluation/_functions.py
--- a/sktime/forecasting/model_evaluation/_functions.py
+++ b/sktime/forecasting/model_evaluation/_functions.py
@@ -523,11 +523,12 @@ def evaluate(
         warnings.warn(
             "in evaluate, kwargs will no longer be supported from sktime 0.25.0. "
             "to pass configuration arguments to the parallelization backend, "
-            "use backend_params instead.",
+            "use backend_params instead. "
+            f"The following kwargs were found: {kwargs.keys()}, pass these as "
+            "dict elements to backend_params instead.",
             DeprecationWarning,
             stacklevel=2,
         )
-        backend = "dask_lazy"
 
     # todo 0.25.0: remove compute argument and logic, and remove this warning
     if compute is not None:


Test patch: