Talip7/scikit-learn-issues-multilabel huggingface dataset:
Yeo-Johnson inverse_transform fails silently on extreme skew data

### Describe the bug

The Yeo-Johnson is not a surjective transformation for negative lambdas. Therefore, the inverse transformation returns `np.nan` when inverse transforming values outside the range of the transform. This failure is silent, so it took me quite a while of debugging to understand this behavior.

The problematic lines are

https://github.com/scikit-learn/scikit-learn/blob/8721245511de2f225ff5f9aa5f5fadce663cd4a3/sklearn/preprocessing/_data.py#L3390

and 

https://github.com/scikit-learn/scikit-learn/blob/8721245511de2f225ff5f9aa5f5fadce663cd4a3/sklearn/preprocessing/_data.py#L3386

in which we might compute `np.power(something_negative, not_integral_value)`, which of course returns `np.nan` as per https://numpy.org/doc/stable/reference/generated/numpy.power.html

### Steps/Code to Reproduce

To reproduce for positive values (there is a similar problem for negative values):

```python
import numpy as np
import sklearn.preprocessing
trans = sklearn.preprocessing.PowerTransformer(method='yeo-johnson')
x = np.array([1,1,1e10]).reshape(-1, 1) # extreme skew
trans.fit(x)
lmbda = trans.lambdas_[0] 
print(lmbda)
assert lmbda < 0 # == -0.096 negative value

# any value `psi` for which lambda*psi+1 <= 0 will result in nan due to lacking support, since the forwards transformation 
# is not surjective on negative lambdas. In this specific case, 10*-0.096 < 1
psi = np.array([10]).reshape(-1, 1)
x = trans.inverse_transform(psi).item()
print(x)
assert np.isnan(x)
```

### Expected Results

 The code should either:

1) validate its inputs and raise an exception
2) validate its inputs and raise a warning
3) fail silently, but have it documented behavior

### Actual Results

It just prints

```
-0.0962322261004418
nan
```

### Versions

```shell
System:
    python: 3.11.3 (main, Jan 18 2024, 19:07:12) [Clang 18.0.0 (https://github.com/llvm/llvm-project 75501f53624de92aafce2f1da698
executable: /home/pyodide/this.program
   machine: Emscripten-3.1.46-wasm32-32bit

Python dependencies:
      sklearn: 1.3.1
          pip: None
   setuptools: None
        numpy: 1.26.1
        scipy: 1.11.2
       Cython: None
       pandas: None
   matplotlib: None
       joblib: 1.3.2
threadpoolctl: 3.2.0

Built with OpenMP: False
```

COMMENT:
May I take this issue? I will trigger a warning.

Labels: Bug
Yeo-Johnson inverse_transform fails silently on extreme skew data

### Describe the bug

The Yeo-Johnson is not a surjective transformation for negative lambdas. Therefore, the inverse transformation returns `np.nan` when inverse transforming values outside the range of the transform. This failure is silent, so it took me quite a while of debugging to understand this behavior.

The problematic lines are

https://github.com/scikit-learn/scikit-learn/blob/8721245511de2f225ff5f9aa5f5fadce663cd4a3/sklearn/preprocessing/_data.py#L3390

and 

https://github.com/scikit-learn/scikit-learn/blob/8721245511de2f225ff5f9aa5f5fadce663cd4a3/sklearn/preprocessing/_data.py#L3386

in which we might compute `np.power(something_negative, not_integral_value)`, which of course returns `np.nan` as per https://numpy.org/doc/stable/reference/generated/numpy.power.html

### Steps/Code to Reproduce

To reproduce for positive values (there is a similar problem for negative values):

```python
import numpy as np
import sklearn.preprocessing
trans = sklearn.preprocessing.PowerTransformer(method='yeo-johnson')
x = np.array([1,1,1e10]).reshape(-1, 1) # extreme skew
trans.fit(x)
lmbda = trans.lambdas_[0] 
print(lmbda)
assert lmbda < 0 # == -0.096 negative value

# any value `psi` for which lambda*psi+1 <= 0 will result in nan due to lacking support, since the forwards transformation 
# is not surjective on negative lambdas. In this specific case, 10*-0.096 < 1
psi = np.array([10]).reshape(-1, 1)
x = trans.inverse_transform(psi).item()
print(x)
assert np.isnan(x)
```

### Expected Results

 The code should either:

1) validate its inputs and raise an exception
2) validate its inputs and raise a warning
3) fail silently, but have it documented behavior

### Actual Results

It just prints

```
-0.0962322261004418
nan
```

### Versions

```shell
System:
    python: 3.11.3 (main, Jan 18 2024, 19:07:12) [Clang 18.0.0 (https://github.com/llvm/llvm-project 75501f53624de92aafce2f1da698
executable: /home/pyodide/this.program
   machine: Emscripten-3.1.46-wasm32-32bit

Python dependencies:
      sklearn: 1.3.1
          pip: None
   setuptools: None
        numpy: 1.26.1
        scipy: 1.11.2
       Cython: None
       pandas: None
   matplotlib: None
       joblib: 1.3.2
threadpoolctl: 3.2.0

Built with OpenMP: False
```

COMMENT:
/take

Labels: Bug
Yeo-Johnson inverse_transform fails silently on extreme skew data

### Describe the bug

The Yeo-Johnson is not a surjective transformation for negative lambdas. Therefore, the inverse transformation returns `np.nan` when inverse transforming values outside the range of the transform. This failure is silent, so it took me quite a while of debugging to understand this behavior.

The problematic lines are

https://github.com/scikit-learn/scikit-learn/blob/8721245511de2f225ff5f9aa5f5fadce663cd4a3/sklearn/preprocessing/_data.py#L3390

and 

https://github.com/scikit-learn/scikit-learn/blob/8721245511de2f225ff5f9aa5f5fadce663cd4a3/sklearn/preprocessing/_data.py#L3386

in which we might compute `np.power(something_negative, not_integral_value)`, which of course returns `np.nan` as per https://numpy.org/doc/stable/reference/generated/numpy.power.html

### Steps/Code to Reproduce

To reproduce for positive values (there is a similar problem for negative values):

```python
import numpy as np
import sklearn.preprocessing
trans = sklearn.preprocessing.PowerTransformer(method='yeo-johnson')
x = np.array([1,1,1e10]).reshape(-1, 1) # extreme skew
trans.fit(x)
lmbda = trans.lambdas_[0] 
print(lmbda)
assert lmbda < 0 # == -0.096 negative value

# any value `psi` for which lambda*psi+1 <= 0 will result in nan due to lacking support, since the forwards transformation 
# is not surjective on negative lambdas. In this specific case, 10*-0.096 < 1
psi = np.array([10]).reshape(-1, 1)
x = trans.inverse_transform(psi).item()
print(x)
assert np.isnan(x)
```

### Expected Results

 The code should either:

1) validate its inputs and raise an exception
2) validate its inputs and raise a warning
3) fail silently, but have it documented behavior

### Actual Results

It just prints

```
-0.0962322261004418
nan
```

### Versions

```shell
System:
    python: 3.11.3 (main, Jan 18 2024, 19:07:12) [Clang 18.0.0 (https://github.com/llvm/llvm-project 75501f53624de92aafce2f1da698
executable: /home/pyodide/this.program
   machine: Emscripten-3.1.46-wasm32-32bit

Python dependencies:
      sklearn: 1.3.1
          pip: None
   setuptools: None
        numpy: 1.26.1
        scipy: 1.11.2
       Cython: None
       pandas: None
   matplotlib: None
       joblib: 1.3.2
threadpoolctl: 3.2.0

Built with OpenMP: False
```

COMMENT:
Hi @ogrisel, could you review the PR? Thanks!

Labels: Bug
Yeo-Johnson inverse_transform fails silently on extreme skew data

### Describe the bug

The Yeo-Johnson is not a surjective transformation for negative lambdas. Therefore, the inverse transformation returns `np.nan` when inverse transforming values outside the range of the transform. This failure is silent, so it took me quite a while of debugging to understand this behavior.

The problematic lines are

https://github.com/scikit-learn/scikit-learn/blob/8721245511de2f225ff5f9aa5f5fadce663cd4a3/sklearn/preprocessing/_data.py#L3390

and 

https://github.com/scikit-learn/scikit-learn/blob/8721245511de2f225ff5f9aa5f5fadce663cd4a3/sklearn/preprocessing/_data.py#L3386

in which we might compute `np.power(something_negative, not_integral_value)`, which of course returns `np.nan` as per https://numpy.org/doc/stable/reference/generated/numpy.power.html

### Steps/Code to Reproduce

To reproduce for positive values (there is a similar problem for negative values):

```python
import numpy as np
import sklearn.preprocessing
trans = sklearn.preprocessing.PowerTransformer(method='yeo-johnson')
x = np.array([1,1,1e10]).reshape(-1, 1) # extreme skew
trans.fit(x)
lmbda = trans.lambdas_[0] 
print(lmbda)
assert lmbda < 0 # == -0.096 negative value

# any value `psi` for which lambda*psi+1 <= 0 will result in nan due to lacking support, since the forwards transformation 
# is not surjective on negative lambdas. In this specific case, 10*-0.096 < 1
psi = np.array([10]).reshape(-1, 1)
x = trans.inverse_transform(psi).item()
print(x)
assert np.isnan(x)
```

### Expected Results

 The code should either:

1) validate its inputs and raise an exception
2) validate its inputs and raise a warning
3) fail silently, but have it documented behavior

### Actual Results

It just prints

```
-0.0962322261004418
nan
```

### Versions

```shell
System:
    python: 3.11.3 (main, Jan 18 2024, 19:07:12) [Clang 18.0.0 (https://github.com/llvm/llvm-project 75501f53624de92aafce2f1da698
executable: /home/pyodide/this.program
   machine: Emscripten-3.1.46-wasm32-32bit

Python dependencies:
      sklearn: 1.3.1
          pip: None
   setuptools: None
        numpy: 1.26.1
        scipy: 1.11.2
       Cython: None
       pandas: None
   matplotlib: None
       joblib: 1.3.2
threadpoolctl: 3.2.0

Built with OpenMP: False
```

COMMENT:
I'm stuck on "At least 1 approving review is required by reviewers with write access". What should I do?

Labels: Bug
DOC add an example on how to optimize a metric with a constraint in TunedThresholdClassifierCV

We merged `TunedThresholdClassifierCV` in #26120.
However, we don't expose any way to optimize a metric that is constrained by another as one would do when choosing a point on the ROC or PR curves.

We should have an example that shows how to do such optimization as discussed here:
https://github.com/scikit-learn/scikit-learn/pull/26120#pullrequestreview-2038175696

This would be a temporary trick until we settle on the best possible API regarding this constrained scorer.

COMMENT:
Probably also figure out a nice api that we can use for SearchCV as well

Labels: New Feature, Documentation
DOC add an example on how to optimize a metric with a constraint in TunedThresholdClassifierCV

We merged `TunedThresholdClassifierCV` in #26120.
However, we don't expose any way to optimize a metric that is constrained by another as one would do when choosing a point on the ROC or PR curves.

We should have an example that shows how to do such optimization as discussed here:
https://github.com/scikit-learn/scikit-learn/pull/26120#pullrequestreview-2038175696

This would be a temporary trick until we settle on the best possible API regarding this constrained scorer.

COMMENT:
I think that having a single ``scoring`` param able to handle constrained metrics as well would be ideal. In addition to make an example for that, we can maybe implement a helper for the main uses cases, precision at recall and tpr at tnr, that returns a ready-to-use scorer.

> Probably also figure out a nice api that we can use for SearchCV as well

Do you think it would make sense to search over constraint values ? My understanding was that such constraint is determined by an external factor that you have no flexibility over.

Labels: New Feature, Documentation
DOC add an example on how to optimize a metric with a constraint in TunedThresholdClassifierCV

We merged `TunedThresholdClassifierCV` in #26120.
However, we don't expose any way to optimize a metric that is constrained by another as one would do when choosing a point on the ROC or PR curves.

We should have an example that shows how to do such optimization as discussed here:
https://github.com/scikit-learn/scikit-learn/pull/26120#pullrequestreview-2038175696

This would be a temporary trick until we settle on the best possible API regarding this constrained scorer.

COMMENT:
At least in fairness related context, you might have a constraint on a fairness metric, and among those you want the best accuracy.

Or if you take the classical example, you want the model which has the least AIC/BIC within the 1 std of the test error of the best model.

Labels: New Feature, Documentation
DOC add an example on how to optimize a metric with a constraint in TunedThresholdClassifierCV

We merged `TunedThresholdClassifierCV` in #26120.
However, we don't expose any way to optimize a metric that is constrained by another as one would do when choosing a point on the ROC or PR curves.

We should have an example that shows how to do such optimization as discussed here:
https://github.com/scikit-learn/scikit-learn/pull/26120#pullrequestreview-2038175696

This would be a temporary trick until we settle on the best possible API regarding this constrained scorer.

COMMENT:
would be interested in contributing to this. my understanding is that a short example should be added to show the current api can be used (although in a bit of a tricky manner, using -np.inf) to optimize a metric according to some constraint. for where to add it, some potential spots i identified could be [here in 3.3.1.1](https://scikit-learn.org/dev/modules/classification_threshold.html#options-to-tune-the-decision-threshold), inside the note, or making a new subsection in [here in 3.4.1](https://scikit-learn.org/dev/modules/model_evaluation.html#using-multiple-metric-evaluation). perhaps most ambitiously it could be inserted in [this existing example](https://scikit-learn.org/dev/auto_examples/model_selection/plot_cost_sensitive_learning.html) on cost-sensitive learning.

Labels: New Feature, Documentation
DOC add an example on how to optimize a metric with a constraint in TunedThresholdClassifierCV

We merged `TunedThresholdClassifierCV` in #26120.
However, we don't expose any way to optimize a metric that is constrained by another as one would do when choosing a point on the ROC or PR curves.

We should have an example that shows how to do such optimization as discussed here:
https://github.com/scikit-learn/scikit-learn/pull/26120#pullrequestreview-2038175696

This would be a temporary trick until we settle on the best possible API regarding this constrained scorer.

COMMENT:
I think we should not complexify the example that you pointed out. However the [following one](https://scikit-learn.org/dev/auto_examples/model_selection/plot_tuned_decision_threshold.html) might be a better place where we can show the pattern and the associated ROC curve to understand the constraint.

Then, we can document in the user guide and refer to this example.

Labels: New Feature, Documentation
DOC add an example on how to optimize a metric with a constraint in TunedThresholdClassifierCV

We merged `TunedThresholdClassifierCV` in #26120.
However, we don't expose any way to optimize a metric that is constrained by another as one would do when choosing a point on the ROC or PR curves.

We should have an example that shows how to do such optimization as discussed here:
https://github.com/scikit-learn/scikit-learn/pull/26120#pullrequestreview-2038175696

This would be a temporary trick until we settle on the best possible API regarding this constrained scorer.

COMMENT:
added a draft PR for this example, still need to do a little more writing but the code and layout changes and overall shape is what I have in mind for reworking this example to include the constrained metric pattern. please let me know your thoughts

Labels: New Feature, Documentation
Rolling your own estimator

### Describe the issue linked to the documentation

The details on the Scikit-learn documentation page are at odds with the linked template.

According to the documentation, it suggests: 

```class TemplateClassifier(BaseEstimator, ClassifierMixin)```

https://scikit-learn.org/stable/developers/develop.html#rolling-your-own-estimator

While the template on GitHub recommends:

```
# Note that the mixin class should always be on the left of `BaseEstimator` to ensure
# the MRO works as expected.
class TemplateClassifier(ClassifierMixin, BaseEstimator)`
```

https://github.com/scikit-learn-contrib/project-template/blob/main/skltemplate/_template.py

### Suggest a potential alternative/fix

I'm unable to determine which method is correct, so I can't offer a suggestion.

COMMENT:
Recently solved in https://github.com/scikit-learn/scikit-learn/pull/28924

Labels: Documentation, Needs Triage
Allow for multiple scoring metrics in `RFECV`

### Workflow

In its current state, `RFECV` only allows for a single scoring metric. In my opinion, calculating multiple scores on each model using *k <= K* features would be extremely valuable.

For example, if I wanted to study how the precision and recall metrics of a binary classifier evolve as I feed less and less features to a model, I would have to run `RFECV` twice: one with `scoring='precision'` and another with `scoring='recall'`.
This is inefficient, as it implies running RFECV twice instead of once.

The `cv_results_` attribute of `GridSearchCV` returns one rank per metric used to evaluate each combination of hyperparameters. Replicating this behavior in `RFECV` would be extremely helpful.

### Proposed solution

#### Notation
- *K* is the number of folds used for cross-validation.
- *P* is the total number of features available.
- *p* is the number of features tried at each step. That is, an integer such that *`min_features_to_select` <= p <= P*.
- *m* is one of *M* performance metrics passed by the user (e.g., 'precision').

#### Solution
User can pass a list of strings representing *M* [predefined scoring metrics](https://scikit-learn.org/stable/modules/model_evaluation.html#common-cases-predefined-values) and at each step, the algorithm stores the performance metric of the *k* models trained with *p <= P* features.

The `cv_results_` attribute of the resulting `RFECV` would now include the following keys for each metric *m* and fold *k*:
- `'split{k}_test_{m}'`
- `mean_test_{m}`
- `'std_test_{m}'`
- `'rank_test_{m}'`

#### Example
```python
rfecv = RFECV(
    estimator=clf,  # Some classifier instance
    step=1,
    min_features_to_select=1,
    cv=10,
    scoring=['precision', 'recall', 'f1', 'roc_auc', 'accuracy']
)
```

##### Considerations
It is likely that `rank_test_{m1}` will differ from `rank_test_{m2}` for any pair of performance metrics *m1* and *m2*. Hence, adding this feature will no longer allow RFECV to automatically pick the best number of features, as the rankings can differ from one metric to another. This part of the workflow would be up to the user.

### Describe alternatives you've considered, if relevant

Running `RFECV` as many times as there are metrics.

### Additional context

I asked [this question](https://stackoverflow.com/questions/78420559/performing-recursive-feature-elimination-with-multiple-performance-metrics) on StackOverflow and the community agrees that the most viable way to do this is to run one RFECV per performance metric I need to evaluate.

COMMENT:
For your "Considerations" section, I wonder if it makes sense to also adopt the `refit` approach from the hyperparameter tuners: pick one metric to choose the best k, or even define a custom strategy callable.

(Maybe this should keep in mind #21291 too?)

Labels: New Feature
Allow for multiple scoring metrics in `RFECV`

### Workflow

In its current state, `RFECV` only allows for a single scoring metric. In my opinion, calculating multiple scores on each model using *k <= K* features would be extremely valuable.

For example, if I wanted to study how the precision and recall metrics of a binary classifier evolve as I feed less and less features to a model, I would have to run `RFECV` twice: one with `scoring='precision'` and another with `scoring='recall'`.
This is inefficient, as it implies running RFECV twice instead of once.

The `cv_results_` attribute of `GridSearchCV` returns one rank per metric used to evaluate each combination of hyperparameters. Replicating this behavior in `RFECV` would be extremely helpful.

### Proposed solution

#### Notation
- *K* is the number of folds used for cross-validation.
- *P* is the total number of features available.
- *p* is the number of features tried at each step. That is, an integer such that *`min_features_to_select` <= p <= P*.
- *m* is one of *M* performance metrics passed by the user (e.g., 'precision').

#### Solution
User can pass a list of strings representing *M* [predefined scoring metrics](https://scikit-learn.org/stable/modules/model_evaluation.html#common-cases-predefined-values) and at each step, the algorithm stores the performance metric of the *k* models trained with *p <= P* features.

The `cv_results_` attribute of the resulting `RFECV` would now include the following keys for each metric *m* and fold *k*:
- `'split{k}_test_{m}'`
- `mean_test_{m}`
- `'std_test_{m}'`
- `'rank_test_{m}'`

#### Example
```python
rfecv = RFECV(
    estimator=clf,  # Some classifier instance
    step=1,
    min_features_to_select=1,
    cv=10,
    scoring=['precision', 'recall', 'f1', 'roc_auc', 'accuracy']
)
```

##### Considerations
It is likely that `rank_test_{m1}` will differ from `rank_test_{m2}` for any pair of performance metrics *m1* and *m2*. Hence, adding this feature will no longer allow RFECV to automatically pick the best number of features, as the rankings can differ from one metric to another. This part of the workflow would be up to the user.

### Describe alternatives you've considered, if relevant

Running `RFECV` as many times as there are metrics.

### Additional context

I asked [this question](https://stackoverflow.com/questions/78420559/performing-recursive-feature-elimination-with-multiple-performance-metrics) on StackOverflow and the community agrees that the most viable way to do this is to run one RFECV per performance metric I need to evaluate.

COMMENT:
This sounds like a legitimate need that cannot be efficiently be addressed with the current API as far as I understand.

So we could indeed consider expanding `RFECV` to accept multiple metrics + passing a custom callable as done for GridSearchCV, see:

https://scikit-learn.org/stable/auto_examples/model_selection/plot_grid_search_digits.html

Labels: New Feature