Repository: scikit-learn/scikit-learn
Pull request: ENH Allow multiple scorers input to permutation_importance
<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs

Fixes #18701

<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.

Input argument `scoring` can be a list, tuple or dict with multiple metrics.
The permutation feature importance is computed iteratively for the different metrics and 
the result is a dict with `metric_name:Bunch`containing the feature importances, mean and std for each of the requested input metrics.


Problem statement:
Allow permutation_importance to accept multiple scorers in `scoring`
#### Describe the workflow you want to enable
We would like to compare permutation importance using different scorers and since our model is big, running multiple `permutation_importance` per scoring is not efficient. It would be nice if we can pass a list/tuple/dict to `scoring` arg and get multiple scores from one prediction.

#### Describe your proposed solution
Follow the same interface as `model_selection.cross_validate`, output a dict of each scorer result

Hints from issue discussion:
This seems to be an interesting feature. I think it could be done efficiently without requiring so much trick. It comes back to call `_weight_scorer` in a loop with different scorers.
Interesting idea but I am not sure what the output would look like: a dict of `Bunch`, one for each scorer?

The caching mechanism of `sklearn.metrics._scorer._MultimetricScorer` should be reusable. This can be constructed using `sklearn.metrics._scorer import _check_multimetric_scoring` as used in `sklearn/model_selection/_validation.py` or `sklearn/model_selection/_search.py`.
I think that the minimal example to use the multi-scorer would be the following:

```python
    scorers = {'a1': 'accuracy', 'a2': 'accuracy',
               'll1': 'neg_log_loss', 'll2': 'neg_log_loss',
               'ra1': 'roc_auc', 'ra2': 'roc_auc'}

    X, y = make_classification(random_state=0)

    clf = DecisionTreeClassifier()
    clf.fit(X, y)

    scorer_dict, _ = _check_multimetric_scoring(clf, scorers)
    multi_scorer = _MultimetricScorer(**scorer_dict)
    result = multi_scorer(clf, X, y)
```
My intuitions would be to replace `_weight_scorer` to handle both single-scorer and multi-scorer in the permutation importance code. This might induce to reorder the way we store the score once `_weight_scorer` will be called.

Commit message: bux fix for loop

Patch to review:
diff --git a/sklearn/inspection/_permutation_importance.py b/sklearn/inspection/_permutation_importance.py
index 688f6e9e68e03..8fb3b831921f6 100644
--- a/sklearn/inspection/_permutation_importance.py
+++ b/sklearn/inspection/_permutation_importance.py
@@ -3,6 +3,7 @@
 from joblib import Parallel
 
 from ..metrics import check_scoring
+from ..metrics._scorer import _check_multimetric_scoring
 from ..utils import Bunch
 from ..utils import check_random_state
 from ..utils import check_array
@@ -74,10 +75,21 @@ def permutation_importance(estimator, X, y, *, scoring=None, n_repeats=5,
     y : array-like or None, shape (n_samples, ) or (n_samples, n_classes)
         Targets for supervised or `None` for unsupervised.
 
-    scoring : string, callable or None, default=None
-        Scorer to use. It can be a single
-        string (see :ref:`scoring_parameter`) or a callable (see
-        :ref:`scoring`). If None, the estimator's default scorer is used.
+    scoring : str, callable, list, tuple, or dict, default=None
+        Scorer to use.
+        If `scoring` represents a single score, one can use:
+
+        - a single string (see :ref:`scoring_parameter`);
+        - a callable (see :ref:`scoring`) that returns a single value.
+
+        If `scoring` reprents multiple scores, one can use:
+
+        - a list or tuple of unique strings;
+        - a callable returning a dictionary where the keys are the metric
+          names and the values are the metric scores;
+        - a dictionary with metric names as keys and callables a values.
+
+        If None, the estimator's default scorer is used.
 
     n_repeats : int, default=5
         Number of times to permute a feature.
@@ -112,6 +124,10 @@ def permutation_importance(estimator, X, y, *, scoring=None, n_repeats=5,
         importances : ndarray, shape (n_features, n_repeats)
             Raw permutation importance scores.
 
+        If there are multiple scoring metrics in the scoring parameter
+        result is a dict with scorer names as keys (i.e., ``auc``) and
+        dictionary-like object like above as values.
+
     References
     ----------
     .. [BRE] L. Breiman, "Random Forests", Machine Learning, 45(1), 5-32,
@@ -143,14 +159,36 @@ def permutation_importance(estimator, X, y, *, scoring=None, n_repeats=5,
     random_state = check_random_state(random_state)
     random_seed = random_state.randint(np.iinfo(np.int32).max + 1)
 
-    scorer = check_scoring(estimator, scoring=scoring)
-    baseline_score = _weights_scorer(scorer, estimator, X, y, sample_weight)
-
-    scores = Parallel(n_jobs=n_jobs)(delayed(_calculate_permutation_scores)(
-        estimator, X, y, sample_weight, col_idx, random_seed, n_repeats, scorer
-    ) for col_idx in range(X.shape[1]))
-
-    importances = baseline_score - np.array(scores)
-    return Bunch(importances_mean=np.mean(importances, axis=1),
-                 importances_std=np.std(importances, axis=1),
-                 importances=importances)
+    if scoring is None or isinstance(scoring, str) or callable(scoring):
+        scorer = check_scoring(estimator, scoring=scoring)
+        baseline_score = _weights_scorer(scorer, estimator, X, y,
+                                         sample_weight)
+
+        scores = Parallel(n_jobs=n_jobs)(
+            delayed(_calculate_permutation_scores)(
+                estimator, X, y, sample_weight, col_idx, random_seed,
+                n_repeats, scorer
+            ) for col_idx in range(X.shape[1]))
+
+        importances = baseline_score - np.array(scores)
+        return Bunch(importances_mean=np.mean(importances, axis=1),
+                     importances_std=np.std(importances, axis=1),
+                     importances=importances)
+    else:
+        scorers = _check_multimetric_scoring(estimator, scoring)
+        ret = dict()
+        for name, scorer in scorers.items():
+            baseline_score = _weights_scorer(scorer, estimator, X, y,
+                                             sample_weight)
+
+            scores = Parallel(n_jobs=n_jobs)(
+                delayed(_calculate_permutation_scores)(
+                    estimator, X, y, sample_weight, col_idx, random_seed,
+                    n_repeats, scorer
+                ) for col_idx in range(X.shape[1]))
+
+            importances = baseline_score - np.array(scores)
+            ret[name] = Bunch(importances_mean=np.mean(importances, axis=1),
+                              importances_std=np.std(importances, axis=1),
+                              importances=importances)
+        return ret


Code review comments:
diff_hunk: @@ -143,14 +159,36 @@ def permutation_importance(estimator, X, y, *, scoring=None, n_repeats=5,
     random_state = check_random_state(random_state)
     random_seed = random_state.randint(np.iinfo(np.int32).max + 1)
 
-    scorer = check_scoring(estimator, scoring=scoring)
-    baseline_score = _weights_scorer(scorer, estimator, X, y, sample_weight)
-
-    scores = Parallel(n_jobs=n_jobs)(delayed(_calculate_permutation_scores)(
-        estimator, X, y, sample_weight, col_idx, random_seed, n_repeats, scorer
-    ) for col_idx in range(X.shape[1]))
-
-    importances = baseline_score - np.array(scores)
-    return Bunch(importances_mean=np.mean(importances, axis=1),
-                 importances_std=np.std(importances, axis=1),
-                 importances=importances)
+    if scoring is None or isinstance(scoring, str) or callable(scoring):
+        scorer = check_scoring(estimator, scoring=scoring)
+        baseline_score = _weights_scorer(scorer, estimator, X, y,
+                                         sample_weight)
+
+        scores = Parallel(n_jobs=n_jobs)(
+            delayed(_calculate_permutation_scores)(
+                estimator, X, y, sample_weight, col_idx, random_seed,
+                n_repeats, scorer
+            ) for col_idx in range(X.shape[1]))
+
+        importances = baseline_score - np.array(scores)
+        return Bunch(importances_mean=np.mean(importances, axis=1),
+                     importances_std=np.std(importances, axis=1),
+                     importances=importances)
+    else:
+        scorers = _check_multimetric_scoring(estimator, scoring)
path: sklearn/inspection/_permutation_importance.py
text: @user1:
I think that we need to create scorers using `scorers = _MultimetricScorer(scorers)` since the caching mechanism is within the class. `_check_multimetric_scoring` will only return a dict of validated scorer without cache implementation.

I think that the solution here is to make `_weight_scorer` accepting `_MultimetricScorer`. Currently, the code should not change, only the returned output will be slightly different (multiple score instead of a single score).

@user1:
I imagine that the outline of the code would look like:

```python
if scoring is None or isinstance(scoring, str) or callable(scoring):
    scorer = check_scoring(estimator, scoring=scoring)
else:
    scorers_dict = _check_multimetric_scoring(estimator, scoring)
    scorer = _MultimetricScoring(**scorers_dict)

    baseline_score = _weights_scorer(
        scorer, estimator, X, y, sample_weight=sample_weight)
    scores = Parallel(n_jobs=n_jobs)(
                delayed(_calculate_permutation_scores)(
                    estimator, X, y, sample_weight, col_idx, random_seed,
                    n_repeats, scorer
                ) for col_idx in range(X.shape[1]))

    if isinstance(baseline_score, dict):
         # unpack the scores since they come from multimetric scorer
    else:
        # a substraction is enough

     return importances 
```

@author:
yes, more or less the implementation done so far. I'll let you check.