Repository: nilearn/nilearn

Issue:
Decoders using `RidgeCV`/`RidgeClassifierCV` may not always be taking advantage of built-in CV capabilities
<!--Provide a brief description what you would like changes and why.-->

Some Decoder objects can use underlying `sklearn` models with built-in cross-validation (e.g., [`RidgeCV`](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.RidgeCV.html) and [`RidgeClassifierCV`](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.RidgeClassifierCV.html)). In those cases the `param_grid` input and `cv_params_` fitted attribute are handled in the same way as for the models without built-in CV, which leads to potentially suboptimal/unintuitive behaviour:
1. If a typical dict of sequences of hyperparameter values (e.g., `{'alphas': [0.1, 1, 10]}`) is passed to `param_grid`, then the Decoder will fit one `RidgeCV`/`RidgeClassifierCV` model per hyperparameter value, which means that the model's built-in CV will not be used. Instead, to take advantage of the built-in CV, the user has to wrap the sequence of hyperparameter values in another sequence (see code snippet below), but that is not intuitive and is inconsistent with the models without built-in CV.
2. If the user does pass in a `param_grid` like `{'alphas': [[0.1, 1, 10]]}` as described above, the `cv_params_` attribute of the fitted Decoder will show the full sequence (`[0.1, 1, 10]` here) for each CV iteration (see example below), which is unexpected because there should only be one value per outer CV iteration (per hyperparameter).
3. `cv_params_` will be empty if `param_grid` is not given, even though the model still does CV on the default `alphas` sequence.

Here is some code (adapted from [here](https://github.com/nilearn/nilearn/issues/2886#issuecomment-868760488)) illustrating some of my points above:
```python
from nilearn import datasets
from sklearn.model_selection import train_test_split
from nilearn.decoding import DecoderRegressor

n_subjects = 10 
oasis_dataset = datasets.fetch_oasis_vbm(n_subjects=n_subjects)
gray_matter_map_filenames = oasis_dataset.gray_matter_maps
age = oasis_dataset.ext_vars['age'].astype(float)
gm_imgs_train, gm_imgs_test, age_train, age_test = train_test_split(
    gray_matter_map_filenames, age, train_size=.6, random_state=0)

decoder = DecoderRegressor(
    estimator='ridge', 
    param_grid = {
        'alphas':[[0.1, 1, 10]], # list of list
    },
    cv=2,
)

decoder.fit(gm_imgs_train, age_train)
print(decoder.cv_params_) # {'beta': {'alphas': [[0.1, 1, 10], [0.1, 1, 10]]}}
```

I'd like to make a PR to address these points. My proposed changes for the case where the model is `RidgeCV` or `RidgeClassifierCV` are:
1. If `param_grid` is a sequence of values, wrap it in a list.
2. Within the [inner CV loop](https://github.com/nilearn/nilearn/blob/c33cca7a/nilearn/decoding/decoder.py#L220) (which iterates over hyperparameter values), set `best_param` to the single best value for each hyperparameter instead of the full list. This value should be available from the fitted estimator (e.g., in the `alpha_` attribute for `RidgeCV`/`RidgeClassifierCV`).
3. @jeromedockes also suggested we overwrite the default `alphas` value with a large list of 10-20 log-spaced values, so in most cases the user would not have to specify their own `param_grid`.

<!--Please fill in the following information, to the best of your ability.-->
### Benefits to the change
1. Users would be able to benefit from the built-in CV capabilities of the models even if they specify `param_grid` as they would for other models.
2. The `cv_params_` attribute of the fitted Decoder would be more informative. 

<!--
### Pseudocode for the new behavior, if applicable

```python

```
-->



Discussion:
Sounds great @michellewang!
sounds good, thanks for looking into this @michellewang ! note (i) the decoder does 2 loops: over CV splits and over hyperparameters. RidgeCV does both internally, so in theory we wouldn't need either of those loops. however we want to keep the loop over CV splits so that we can build the ensemble model by averaging the best models for each split. that is, if we have 5-fold CV and 10 alpha values, we will call RidgeCV.fit 5 times (ii) in a second step we might want to take advantage of the built-in CV capabilities of other estimators such as logisticregression, lasso
Thanks @jeromedockes! 

> (ii) in a second step we might want to take advantage of the built-in CV capabilities of other estimators such as logisticregression, lasso

Sounds good, this is also mentioned in #2200. Should this be in another PR or can it be the same one?

> Sounds good, this is also mentioned in #2200. Should this be in another PR or can it be the same one?

I would recommend doing it in another PR; this way the first PR will be
smaller and merged sooner. I just mentioned it here so that when you
make the changes for RidgeCV, you have in the back of your mind that we
may want to generalize this to other CV-able estimators too later on.


Resolving patch:
diff --git a/examples/02_decoding/plot_haxby_grid_search.py b/examples/02_decoding/plot_haxby_grid_search.py
--- a/examples/02_decoding/plot_haxby_grid_search.py
+++ b/examples/02_decoding/plot_haxby_grid_search.py
@@ -12,7 +12,7 @@
 might not maximize the score on left-out data.
 
 Thus using data to maximize a cross-validation score computed on that
-same data is likely to optimistic and lead to an overfit.
+same data is likely to be too optimistic and lead to an overfit.
 
 The proper approach is known as a "nested cross-validation". It consists
 in doing cross-validation loops to set the model parameters inside the
@@ -20,8 +20,8 @@
 parameters are set separately on each fold, never using the data used to
 measure performance.
 
-For decoding task, in nilearn, this can be done using the
-:class:`nilearn.decoding.Decoder` object, that will automatically select
+For decoding tasks, in nilearn, this can be done using the
+:class:`nilearn.decoding.Decoder` object, which will automatically select
 the best parameters of an estimator from a grid of parameter values.
 
 One difficulty is that the Decoder object is a composite estimator: a
@@ -74,6 +74,22 @@
 # on nested cross-validation.
 from nilearn.decoding import Decoder
 
+# We provide a grid of hyperparameter values to the Decoder's internal
+# cross-validation. If no param_grid is provided, the Decoder will use a
+# default grid with sensible values for the chosen estimator
+param_grid = [
+    {
+        "penalty": ["l2"],
+        "dual": [True],
+        "C": [100, 1000],
+    },
+    {
+        "penalty": ["l1"],
+        "dual": [False],
+        "C": [100, 1000],
+    },
+]
+
 # Here screening_percentile is set to 2 percent, meaning around 800
 # features will be selected with ANOVA.
 decoder = Decoder(
@@ -83,6 +99,7 @@
     smoothing_fwhm=4,
     standardize=True,
     screening_percentile=2,
+    param_grid=param_grid,
 )
 
 ###########################################################################
@@ -93,16 +110,24 @@
 # best parameters selected for each cross-validation fold. See
 # https://scikit-learn.org/stable/modules/cross_validation.html for an
 # excellent explanation of how cross-validation works.
-#
-# First we fit the Decoder
+
+# Fit the Decoder
 decoder.fit(fmri_niimgs, y)
-for i, (param, cv_score) in enumerate(
-    zip(decoder.cv_params_["shoe"]["C"], decoder.cv_scores_["shoe"])
+
+# Print the best parameters for each fold
+for i, (best_C, best_penalty, best_dual, cv_score) in enumerate(
+    zip(
+        decoder.cv_params_["shoe"]["C"],
+        decoder.cv_params_["shoe"]["penalty"],
+        decoder.cv_params_["shoe"]["dual"],
+        decoder.cv_scores_["shoe"],
+    )
 ):
     print(
-        "Fold %d | Best SVM parameter: %.1f with score: %.3f"
-        % (i + 1, param, cv_score)
+        f"Fold {i+1} | Best SVM parameters: C={best_C}"
+        f", penalty={best_penalty}, dual={best_dual} with score: {cv_score}"
     )
+
 # Output the prediction with Decoder
 y_pred = decoder.predict(fmri_niimgs)
 
@@ -123,6 +148,7 @@
         cv=3,
         standardize=True,
         screening_percentile=sp,
+        param_grid=param_grid,
     )
     decoder.fit(index_img(fmri_niimgs, session < 10), y[session < 10])
     cv_scores.append(np.mean(decoder.cv_scores_["bottle"]))
@@ -156,6 +182,7 @@
             cv=3,
             standardize=True,
             screening_percentile=sp,
+            param_grid=param_grid,
         )
         decoder.fit(index_img(fmri_niimgs, train), y_train)
         y_pred = decoder.predict(index_img(fmri_niimgs, test))
diff --git a/nilearn/decoding/decoder.py b/nilearn/decoding/decoder.py
--- a/nilearn/decoding/decoder.py
+++ b/nilearn/decoding/decoder.py
@@ -18,6 +18,7 @@
 
 import itertools
 import warnings
+from typing import Iterable
 
 import numpy as np
 from joblib import Parallel, delayed
@@ -70,11 +71,10 @@ def _check_param_grid(estimator, X, y, param_grid=None):
 
     Parameters
     ----------
-    estimator: str, optional
+    estimator: str
         The estimator to choose among:
         %(classifier_options)s
         %(regressor_options)s
-        Default 'svc'.
 
     X: list of Niimg-like objects
         See :ref:`extracting_data`.
@@ -106,43 +106,139 @@ def _check_param_grid(estimator, X, y, param_grid=None):
 
     """
     if param_grid is None:
-        param_grid = {}
+        param_grid = _default_param_grid(estimator, X, y)
+
+    else:
+        if isinstance(estimator, (RidgeCV, RidgeClassifierCV)):
+            param_grid = _wrap_param_grid(param_grid, "alphas")
+
+    return param_grid
+
+
+def _default_param_grid(estimator, X, y):
+    """Generate sensible default for param_grid.
+
+    Parameters
+    ----------
+    estimator: str
+        The estimator to choose among:
+        %(classifier_options)s
+        %(regressor_options)s
+
+    X: list of Niimg-like objects
+        See :ref:`extracting_data`.
+        Data on which model is to be fitted. If this is a list,
+        the affine is considered the same for all.
+
+    y: array or list of shape (n_samples)
+        The dependent variable (age, sex, IQ, yes/no, etc.).
+        Target variable to predict. Must have exactly as many elements as
+        3D images in niimg.
+
+    Returns
+    -------
+    param_grid: dict of str to sequence, or sequence of such. Sensible default
+    dict has size 1 for linear models.
+    """
+    param_grid = {}
+
+    # validate estimator
+    if isinstance(estimator, (DummyClassifier, DummyRegressor)):
+        if estimator.strategy in ["constant"]:
+            message = (
+                "Dummy classification implemented only for strategies"
+                ' "most_frequent", "prior", "stratified"'
+            )
+            raise NotImplementedError(message)
+    elif not isinstance(
+        estimator,
+        (LogisticRegression, LinearSVC, RidgeCV, RidgeClassifierCV, SVR),
+    ):
+        raise ValueError(
+            "Invalid estimator. The supported estimators are:"
+            f" {list(SUPPORTED_ESTIMATORS.keys())}"
+        )
+
+    # use l1_min_c to get lower bound for estimators with L1 penalty
+    if hasattr(estimator, "penalty") and (estimator.penalty == "l1"):
         # define loss function
         if isinstance(estimator, LogisticRegression):
             loss = "log"
-        elif isinstance(
-            estimator, (LinearSVC, RidgeCV, RidgeClassifierCV, SVR)
-        ):
+        elif isinstance(estimator, LinearSVC):
             loss = "squared_hinge"
-        elif isinstance(estimator, (DummyClassifier, DummyRegressor)):
-            if estimator.strategy in ["constant"]:
-                message = (
-                    "Dummy classification implemented only for strategies"
-                    ' "most_frequent", "prior", "stratified"'
-                )
-                raise NotImplementedError(message)
-        else:
-            raise ValueError(
-                "Invalid estimator. The supported estimators are:"
-                f" {list(SUPPORTED_ESTIMATORS.keys())}"
-            )
-        # define sensible default for different types of estimators
-        if hasattr(estimator, "penalty") and (estimator.penalty == "l1"):
-            min_c = l1_min_c(X, y, loss=loss)
-        else:
-            min_c = 0.5
 
-        if not isinstance(
-            estimator,
-            (RidgeCV, RidgeClassifierCV, DummyClassifier, DummyRegressor),
-        ):
-            param_grid["C"] = np.array([2, 20, 200]) * min_c
-        else:
-            param_grid = {}
+        min_c = l1_min_c(X, y, loss=loss)
+
+    # otherwise use 0.5 which will give param_grid["C"] = [1, 10, 100]
+    else:
+        min_c = 0.5
+
+    # define sensible default for different types of estimators
+    if isinstance(estimator, (RidgeCV, RidgeClassifierCV)):
+        param_grid["alphas"] = [np.geomspace(1e-3, 1e4, 8)]
+    elif isinstance(estimator, (LogisticRegression, LinearSVC, SVR)):
+        param_grid["C"] = np.array([2, 20, 200]) * min_c
+    else:
+        param_grid = {}
 
     return param_grid
 
 
+def _wrap_param_grid(param_grid, param_name):
+    """Wrap a parameter's sequence of values with an outer list.
+
+    This can be
+    desirable for models featuring built-in cross-validation, as it would leave
+    it to the model's internal (optimized) cross-validation to loop over
+    hyperparameter values. Does nothing if the parameter is already wrapped.
+
+    Parameters
+    ----------
+    param_grid : dict of str to sequence, or sequence of such
+        The parameter grid to wrap, as a dictionary mapping estimator
+        parameters to sequences of allowed values.
+    param_name : str
+        Name of parameter whose sequence of values should be wrapped
+
+    Returns
+    -------
+    param_grid_wrapped: dict of str to sequence, or sequence of such
+        The updated parameter grid
+    """
+    if param_grid is None:
+        return param_grid
+
+    # param_grid can be either a dict or a sequence of dicts
+    # we make sure that it is a sequence we can loop over
+    input_is_dict = isinstance(param_grid, dict)
+    if input_is_dict:
+        param_grid = [param_grid]
+
+    # process dicts one by one and add them to a new list
+    new_param_grid = []
+    for param_grid_item in param_grid:
+        if param_name in param_grid_item and not isinstance(
+            param_grid_item[param_name][0], Iterable
+        ):
+            warnings.warn(
+                f"parameter '{param_name}' should be a sequence of iterables"
+                f" (e.g., { {param_name:[[1, 10, 100]]} }) to benefit from"
+                " the built-in cross-validation of the estimator."
+                f" Wrapping {param_grid_item[param_name]} in an outer list."
+            )
+
+            param_grid_item = dict(param_grid_item)  # make a new dict
+            param_grid_item[param_name] = [param_grid_item[param_name]]
+
+        new_param_grid.append(param_grid_item)
+
+    # return a dict (not a list) if the original input was a dict
+    if input_is_dict:
+        new_param_grid = new_param_grid[0]
+
+    return new_param_grid
+
+
 def _check_estimator(estimator):
     if not isinstance(estimator, str):
         warnings.warn(
@@ -234,6 +330,9 @@ def _parallel_fit(
                     dummy_output = estimator.class_prior_
                 elif isinstance(estimator, DummyRegressor):
                     dummy_output = estimator.constant_
+
+            if isinstance(estimator, (RidgeCV, RidgeClassifierCV)):
+                param["best_alpha"] = estimator.alpha_
             best_param = param
 
     if best_coef is not None:
@@ -481,7 +580,11 @@ def fit(self, X, y, groups=None):
         `cv_params_` : dict of lists
             Best point in the parameter grid for each tested fold
             in the inner cross validation loop. The grid is empty
-            when Dummy estimators are provided.
+            when Dummy estimators are provided. Note: if the estimator used its
+            built-in cross-validation, this will include an additional key for
+            the single best value estimated by the built-in cross-validation
+            (e.g., 'best_alpha' for RidgeCV/RidgeClassifierCV), in addition to
+            the input list of values.
 
         'scorer_' : function
             Scorer function used on the held out data to choose the best


Test patch:
diff --git a/nilearn/decoding/tests/test_decoder.py b/nilearn/decoding/tests/test_decoder.py
--- a/nilearn/decoding/tests/test_decoder.py
+++ b/nilearn/decoding/tests/test_decoder.py
@@ -16,6 +16,8 @@
 #
 # License: simplified BSD
 
+import collections
+import numbers
 import warnings
 
 import numpy as np
@@ -30,6 +32,7 @@
     _check_estimator,
     _check_param_grid,
     _parallel_fit,
+    _wrap_param_grid,
 )
 from nilearn.decoding.tests.test_same_api import to_niimgs
 from nilearn.maskers import NiftiMasker
@@ -46,7 +49,7 @@
     r2_score,
     roc_auc_score,
 )
-from sklearn.model_selection import KFold, LeaveOneGroupOut
+from sklearn.model_selection import KFold, LeaveOneGroupOut, ParameterGrid
 from sklearn.preprocessing import StandardScaler
 from sklearn.svm import SVR, LinearSVC
 
@@ -102,7 +105,7 @@ def tiny_binary_classification_data():
 
 @pytest.fixture
 def binary_classification_data():
-    """Use for test where classifcication is actually performed."""
+    """Use for test where classification is actually performed."""
     return _make_binary_classification_test_data(n_samples=N_SAMPLES)
 
 
@@ -131,7 +134,7 @@ def multiclass_data():
 
 
 @pytest.mark.parametrize(
-    "regressor, param", [(RidgeCV(), []), (SVR(kernel="linear"), "C")]
+    "regressor, param", [(RidgeCV(), ["alphas"]), (SVR(kernel="linear"), "C")]
 )
 def test_check_param_grid_regression(regressor, param):
     """Test several estimators.
@@ -152,7 +155,7 @@ def test_check_param_grid_regression(regressor, param):
     [
         (LogisticRegression(penalty="l1"), "C"),
         (LogisticRegression(penalty="l2"), "C"),
-        (RidgeClassifierCV(), []),
+        (RidgeClassifierCV(), ["alphas"]),
     ],
 )
 def test_check_param_grid_classification(rand_X_Y, classifier, param):
@@ -187,6 +190,66 @@ def test_check_parameter_grid_is_empty(rand_X_Y):
     assert param_grid == {}
 
 
+@pytest.mark.parametrize(
+    "param_grid",
+    [
+        {"alphas": [1, 10, 100, 1000]},
+        {"alphas": [1, 10, 100, 1000], "fit_intercept": [True, False]},
+        {"fit_intercept": [True, False]},
+        {"alphas": [[1, 10, 100, 1000]]},
+        {"alphas": (1, 10, 100, 1000)},
+        {"alphas": [(1, 10, 100, 1000)]},
+        {"alphas": ((1, 10, 100, 1000),)},
+        {"alphas": np.array([1, 10, 100, 1000])},
+        {"alphas": [np.array([1, 10, 100, 1000])]},
+        [{"alphas": [1, 10]}, {"alphas": [[100, 1000]]}],
+        [{"alphas": [1, 10]}, {"fit_intercept": [True, False]}],
+    ],
+)
+def test_wrap_param_grid(param_grid):
+    param_name = "alphas"
+    original_grid = ParameterGrid(param_grid)
+    wrapped_grid = ParameterGrid(_wrap_param_grid(param_grid, param_name))
+    for grid_row in wrapped_grid:
+        if param_name in grid_row:
+            param_value = grid_row[param_name]
+            assert isinstance(param_value, collections.abc.Iterable)
+            assert all(
+                isinstance(item, numbers.Number) for item in param_value
+            )
+        else:
+            assert grid_row in original_grid
+
+
+@pytest.mark.parametrize(
+    "param_grid, need_wrap",
+    [
+        ({"alphas": [1, 10, 100, 1000]}, True),
+        ({"alphas": [[1, 10, 100, 1000]]}, False),
+    ],
+)
+def test_wrap_param_grid_warning(param_grid, need_wrap):
+    param_name = "alphas"
+    expected_warning_substring = "should be a sequence of iterables"
+
+    with warnings.catch_warnings(record=True) as raised_warnings:
+        _wrap_param_grid(param_grid, param_name)
+    warning_messages = [str(warning.message) for warning in raised_warnings]
+
+    found_warning = any(
+        expected_warning_substring in x for x in warning_messages
+    )
+
+    if need_wrap:
+        assert found_warning
+    else:
+        assert not found_warning
+
+
+def test_wrap_param_grid_is_none():
+    assert _wrap_param_grid(None, "alphas") is None
+
+
 @pytest.mark.parametrize(
     "model", [DecoderRegressor, Decoder, FREMRegressor, FREMClassifier]
 )
@@ -308,6 +371,76 @@ def test_parallel_fit(rand_X_Y):
             assert a == b
 
 
+@pytest.mark.parametrize(
+    "param_values",
+    (
+        [0.001, 0.01, 0.1, 1, 10, 100, 1000],
+        [[0.001, 0.01, 0.1, 1, 10, 100, 1000]],
+    ),
+)
+@pytest.mark.parametrize(
+    "estimator, param_name, fitted_param_name, is_classification",
+    [
+        (RidgeCV(), "alphas", "best_alpha", False),
+        (RidgeClassifierCV(), "alphas", "best_alpha", True),
+    ],
+)
+def test_parallel_fit_builtin_cv(
+    rand_X_Y,
+    estimator,
+    param_name,
+    fitted_param_name,
+    is_classification,
+    param_values,
+):
+    """Check that the `fitted_param_name` output of _parallel_fit is a single
+    value even if param_grid is wrapped in a list for models with built-in CV.
+    """
+    # y will be replaced if this is a classification
+    X, y = make_regression(
+        n_samples=N_SAMPLES,
+        n_features=20,
+        n_informative=5,
+        noise=0.2,
+        random_state=42,
+    )
+
+    # train/test indices
+    n_samples_train = int(0.8 * N_SAMPLES)
+    train = range(n_samples_train)
+    test = range(n_samples_train, N_SAMPLES)
+
+    # define a screening selector
+    selector = check_feature_screening(
+        screening_percentile=None, mask_img=None, is_classification=False
+    )
+
+    # create appropriate scorer and update y for classification
+    if is_classification:
+        scorer = check_scoring(estimator, "accuracy")
+        _, y = rand_X_Y
+    else:
+        scorer = check_scoring(estimator, "r2")
+
+    param_grid = {param_name: param_values}
+    _, _, _, best_param, _, _ = _parallel_fit(
+        estimator=estimator,
+        X=X,
+        y=y,
+        train=train,
+        test=test,
+        param_grid=param_grid,
+        is_classification=is_classification,
+        scorer=scorer,
+        mask_img=None,
+        class_index=1,
+        selector=selector,
+        clustering_percentile=100,
+    )
+
+    assert isinstance(best_param[fitted_param_name], (float, int))
+
+
 def test_decoder_binary_classification_with_masker_object(
     binary_classification_data,
 ):