Repository: sktime/sktime

Issue:
[ENH] Allow `return_n_best_forecasters` in  `ForecastingGridSearchCV ` to be -1 (select all forecasters)
**Is your feature request related to a problem? Please describe.**
I would like the class `ForecastingGridSearchCV ` to have a possibility to return ALL forecasters. Right now we have the `return_n_best_forecasters ` argument in the API which lets you choose how many forecasters you want to return

**Describe the solution you'd like**
I would like to pass -1 to `return_n_best_forecasters ` and automatically get back all the forecasters fit.

This is just an idea that i think would be helpful.


Discussion:
sounds like a good idea to me - we should do it in all the other forecasting tuners too, and not forget to update docstrings.

Marking this a good first issue since I think it's a nicely defined entry contribution.
Can I try to work on this?

I'm new here, so I don't know how assignments work here, or can anyone work on any issues?
> I'm new here, so I don't know how assignments work here

Semi-formal - you can consider yourself assigned if you post on the issue and a core dev comments.

We track assignments more strictly (kanban board, assigned developers) only for intensive workstreams, but these last typically only through the summer period, which starts with GSoC and internships.

So: all yours!
PS: you can even make a contribution or PR without announcement if you want, of course it is "safer" to open an issue and describe the proposed work before, to avoid things like duplication or design incompatibilities.

Where this concerns contribution of estimators, sktime has a very open "library contribution" policy https://www.sktime.net/en/stable/get_involved/governance.html#algorithm-inclusion-guidelines
this is streamlined as long as you adhere to the extension templates and documentation requirements.

Resolving patch:
diff --git a/sktime/forecasting/model_selection/_tune.py b/sktime/forecasting/model_selection/_tune.py
--- a/sktime/forecasting/model_selection/_tune.py
+++ b/sktime/forecasting/model_selection/_tune.py
@@ -267,7 +267,10 @@ def evaluate_candidates(candidate_params):
         # Select n best forecaster
         self.n_best_forecasters_ = []
         self.n_best_scores_ = []
-        for i in range(self.return_n_best_forecasters):
+        _forecasters_to_return = min(self.return_n_best_forecasters, len(results.index))
+        if _forecasters_to_return == -1:
+            _forecasters_to_return = len(results.index)
+        for i in range(_forecasters_to_return):
             params = results["params"].iloc[i]
             rank = results[f"rank_{scoring_name}"].iloc[i]
             rank = str(int(rank))
@@ -445,7 +448,9 @@ class ForecastingGridSearchCV(BaseGridSearch):
     verbose: int, optional (default=0)
     return_n_best_forecasters : int, default=1
         In case the n best forecaster should be returned, this value can be set
-        and the n best forecasters will be assigned to n_best_forecasters_
+        and the n best forecasters will be assigned to n_best_forecasters_.
+        Set return_n_best_forecasters to -1 to return all forecasters.
+
     error_score : numeric value or the str 'raise', optional (default=np.nan)
         The test score returned when a forecaster fails to be fitted.
     return_train_score : bool, optional (default=False)
@@ -773,7 +778,9 @@ class ForecastingRandomizedSearchCV(BaseGridSearch):
     verbose : int, optional (default=0)
     return_n_best_forecasters: int, default=1
         In case the n best forecaster should be returned, this value can be set
-        and the n best forecasters will be assigned to n_best_forecasters_
+        and the n best forecasters will be assigned to n_best_forecasters_.
+        Set return_n_best_forecasters to -1 to return all forecasters.
+
     random_state : int, RandomState instance or None, default=None
         Pseudo random number generator state used for random uniform sampling
         from lists of possible values instead of scipy.stats distributions.
@@ -1037,7 +1044,8 @@ class ForecastingSkoptSearchCV(BaseGridSearch):
         FitFailedWarning is raised.
     return_n_best_forecasters: int, default=1
         In case the n best forecaster should be returned, this value can be set
-        and the n best forecasters will be assigned to n_best_forecasters_
+        and the n best forecasters will be assigned to n_best_forecasters_.
+        Set return_n_best_forecasters to -1 to return all forecasters.
 
     backend : {"dask", "loky", "multiprocessing", "threading"}, by default "loky".
         Runs parallel evaluate if specified and ``strategy`` is set as "refit".


Test patch:
diff --git a/sktime/forecasting/model_selection/tests/test_tune.py b/sktime/forecasting/model_selection/tests/test_tune.py
--- a/sktime/forecasting/model_selection/tests/test_tune.py
+++ b/sktime/forecasting/model_selection/tests/test_tune.py
@@ -4,6 +4,10 @@
 
 __author__ = ["mloning", "fkiraly"]
 
+
+from functools import reduce
+from typing import Dict, List, Union
+
 import numpy as np
 import pytest
 from sklearn.model_selection import ParameterGrid, ParameterSampler
@@ -348,3 +352,76 @@ def test_gscv_backends(backend_set):
         backend_params=backend_params,
     )
     gscv.fit(y, X)
+
+
+TEST_PARAMS_DICT = PIPE_GRID
+
+TEST_PARAMS_LIST = [
+    {
+        "window_length": [1, 2, 3],
+        "strategy": ["last", "mean"],
+        "transformer__degree": [1, 2, 3],
+        "forecaster__strategy": ["last", "mean", "seasonal_last"],
+    },
+    {
+        "window_length": [4, 5, 6],
+        "forecaster__strategy": ["last", "mean"],
+    },
+]
+
+
+@pytest.mark.parametrize("return_n_best_forecasters", [-1, 0, 3])
+@pytest.mark.parametrize(
+    "Forecaster, kwargs",
+    [
+        (ForecastingGridSearchCV, {"param_grid": TEST_PARAMS_DICT}),
+        (ForecastingGridSearchCV, {"param_grid": TEST_PARAMS_LIST}),
+        (ForecastingRandomizedSearchCV, {"param_distributions": TEST_PARAMS_LIST}),
+        (
+            ForecastingRandomizedSearchCV,
+            {"param_distributions": TEST_PARAMS_LIST, "n_iter": 100},
+        ),
+    ],
+)
+def test_return_n_best_forecasters(Forecaster, return_n_best_forecasters, kwargs):
+    y, X = load_longley()
+    searchCV = Forecaster(
+        forecaster=PIPE,
+        cv=CVs[0],
+        **kwargs,
+        return_n_best_forecasters=return_n_best_forecasters,
+    )
+    searchCV.fit(y, X)
+    if return_n_best_forecasters == -1:
+
+        def calculate_total_combinations(param_grid: Union[List[Dict], Dict]):
+            if isinstance(param_grid, dict):
+                return reduce(lambda x, y: x * y, [len(x) for x in param_grid.values()])
+            elif isinstance(param_grid, list):
+                return sum(calculate_total_combinations(i) for i in param_grid)
+            else:
+                error_message = "`param_grid` must be a dict or a list[dict]"
+                raise ValueError(error_message)
+
+        if "param_grid" in kwargs:
+            total_combinations = calculate_total_combinations(kwargs["param_grid"])
+            assert len(searchCV.n_best_forecasters_) == total_combinations
+        else:
+            try:
+                assert len(searchCV.n_best_forecasters_) == searchCV.n_iter
+            except AssertionError:
+                total_combinations = calculate_total_combinations(
+                    kwargs["param_distributions"]
+                )
+                assert len(searchCV.n_best_forecasters_) == total_combinations
+    else:
+        try:
+            assert len(searchCV.n_best_forecasters_) == return_n_best_forecasters
+        except AssertionError:
+            key = (
+                "param_distributions"
+                if "param_distributions" in kwargs
+                else "param_grid"
+            )
+            total_combinations = calculate_total_combinations(kwargs[key])
+            assert len(searchCV.n_best_forecasters_) == total_combinations