Talip7/scikit-learn-issues-multilabel huggingface dataset:
Add verbose option to sklearn.inspection.permutation_importance()

### Describe the workflow you want to enable

Add the verbose parameter to sklearn.inspection.permutation_importance.
This will improve the user experience by allowing them to evaluate how long the process will take, which is especially useful when working models with a high number of features.

### Describe your proposed solution

My solution would be to add "verbose" as a paramter in sklearn.inspection.permutation_importance and than add verbose inside the function like this:

```
scores = Parallel(n_jobs=n_jobs, verbose=verbose)(
        delayed(_calculate_permutation_scores)(
            estimator,
            X,
            y,
            sample_weight,
            col_idx,
            random_seed,
            n_repeats,
            scorer,
            max_samples,
        )
        for col_idx in range(X.shape[1])
    )
```



### Describe alternatives you've considered, if relevant

_No response_

### Additional context

As someone who uses the library regularly, I was wondering if there's a reason why the 'verbose' parameter hasn't been implemented yet. I actually changed it locally and have been using it like this ever since!

COMMENT:
We could indeed have some feedback information. I am yet settled if we should have a verbose option or rather a progress bar with `rich` similarly to what we would like in https://github.com/scikit-learn/scikit-learn/issues/27592

Labels: New Feature
Add verbose option to sklearn.inspection.permutation_importance()

### Describe the workflow you want to enable

Add the verbose parameter to sklearn.inspection.permutation_importance.
This will improve the user experience by allowing them to evaluate how long the process will take, which is especially useful when working models with a high number of features.

### Describe your proposed solution

My solution would be to add "verbose" as a paramter in sklearn.inspection.permutation_importance and than add verbose inside the function like this:

```
scores = Parallel(n_jobs=n_jobs, verbose=verbose)(
        delayed(_calculate_permutation_scores)(
            estimator,
            X,
            y,
            sample_weight,
            col_idx,
            random_seed,
            n_repeats,
            scorer,
            max_samples,
        )
        for col_idx in range(X.shape[1])
    )
```



### Describe alternatives you've considered, if relevant

_No response_

### Additional context

As someone who uses the library regularly, I was wondering if there's a reason why the 'verbose' parameter hasn't been implemented yet. I actually changed it locally and have been using it like this ever since!

COMMENT:
@glemaitre Anything finalized if this should have a progress bar or a verbose option as @philippolfert mentioned? I would like to have a look at this.

Labels: New Feature
Reduce verbosity of error "Input X contains NaN" of PLSRegression

### Describe the workflow you want to enable

Currently if PLSRegression (and other algorithms, judging by google) gets any NaNs as input, it raises this long multiline exception:
```
ValueError: Input X contains NaN. 
PLSRegression does not accept missing values encoded as NaN natively. For supervised learning, you might want to consider sklearn.ensemble.HistGradientBoostingClassifier and Regressor which accept missing values encoded as NaNs natively. Alternatively, it is possible to preprocess the data, for instance by using an imputer transformer in a pipeline or drop samples with missing values. See https://scikit-learn.org/stable/modules/impute.html You can find a list of all estimators that handle NaN values at the following page: https://scikit-learn.org/stable/modules/impute.html#estimators-that-handle-nan-values
```

(source is here: https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/utils/validation.py#L158)

I am running the algorithm regularly at fixed intervals and unattended, so I am getting very verbose logs from time to time. It would be nice to have an option to reduce verbosity of this error.

### Describe your proposed solution

Add an option to suppress long exception description and only emit "Input X contains NaN" message. Or only emit verbose error once, and then switch to short message.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I don't think that we want to add a parameter to every estimator to control this. Then, we could maybe try something complex via the global config. However, I'm thinking that it would be better handle with a `try/except` in the user code to raise the desired error in this case.

@ogrisel do you have any thought?

Labels: New Feature
Reduce verbosity of error "Input X contains NaN" of PLSRegression

### Describe the workflow you want to enable

Currently if PLSRegression (and other algorithms, judging by google) gets any NaNs as input, it raises this long multiline exception:
```
ValueError: Input X contains NaN. 
PLSRegression does not accept missing values encoded as NaN natively. For supervised learning, you might want to consider sklearn.ensemble.HistGradientBoostingClassifier and Regressor which accept missing values encoded as NaNs natively. Alternatively, it is possible to preprocess the data, for instance by using an imputer transformer in a pipeline or drop samples with missing values. See https://scikit-learn.org/stable/modules/impute.html You can find a list of all estimators that handle NaN values at the following page: https://scikit-learn.org/stable/modules/impute.html#estimators-that-handle-nan-values
```

(source is here: https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/utils/validation.py#L158)

I am running the algorithm regularly at fixed intervals and unattended, so I am getting very verbose logs from time to time. It would be nice to have an option to reduce verbosity of this error.

### Describe your proposed solution

Add an option to suppress long exception description and only emit "Input X contains NaN" message. Or only emit verbose error once, and then switch to short message.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Hi, in my opinion this is a try/except situation as @glemaitre mentioned here. You can catch the specefic exception and log a more concise message.

Further you can customize the logging level (e.g., logging.error, logging.warning, logging.info) based on the  your verbosity requirement.

Labels: New Feature
Reduce verbosity of error "Input X contains NaN" of PLSRegression

### Describe the workflow you want to enable

Currently if PLSRegression (and other algorithms, judging by google) gets any NaNs as input, it raises this long multiline exception:
```
ValueError: Input X contains NaN. 
PLSRegression does not accept missing values encoded as NaN natively. For supervised learning, you might want to consider sklearn.ensemble.HistGradientBoostingClassifier and Regressor which accept missing values encoded as NaNs natively. Alternatively, it is possible to preprocess the data, for instance by using an imputer transformer in a pipeline or drop samples with missing values. See https://scikit-learn.org/stable/modules/impute.html You can find a list of all estimators that handle NaN values at the following page: https://scikit-learn.org/stable/modules/impute.html#estimators-that-handle-nan-values
```

(source is here: https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/utils/validation.py#L158)

I am running the algorithm regularly at fixed intervals and unattended, so I am getting very verbose logs from time to time. It would be nice to have an option to reduce verbosity of this error.

### Describe your proposed solution

Add an option to suppress long exception description and only emit "Input X contains NaN" message. Or only emit verbose error once, and then switch to short message.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Thank you for responses!
@glemaitre the code that emits this Exception is used by all estimators (and probably other code), so there will be one config anyway, not one for each estimator. 
Regarding try..except: since ValueError is rather generic error, I am interested in logging exception text, so I cannot just `raise Exception()` in my `except ValueError:`, because I'll loose actual reason for error. Do you suggest to extract exception message, take only 1st line of it and use only that line to log? That's how I currently solve it, but it seems hacky. 

If reducing verbosity of this error is undesirable, another solution would be to introduce specific `InputContainsNaNError(ValueError)`, for example, then it can be caught and dealt with without needing its message to get actual error reason.

Labels: New Feature
Reduce verbosity of error "Input X contains NaN" of PLSRegression

### Describe the workflow you want to enable

Currently if PLSRegression (and other algorithms, judging by google) gets any NaNs as input, it raises this long multiline exception:
```
ValueError: Input X contains NaN. 
PLSRegression does not accept missing values encoded as NaN natively. For supervised learning, you might want to consider sklearn.ensemble.HistGradientBoostingClassifier and Regressor which accept missing values encoded as NaNs natively. Alternatively, it is possible to preprocess the data, for instance by using an imputer transformer in a pipeline or drop samples with missing values. See https://scikit-learn.org/stable/modules/impute.html You can find a list of all estimators that handle NaN values at the following page: https://scikit-learn.org/stable/modules/impute.html#estimators-that-handle-nan-values
```

(source is here: https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/utils/validation.py#L158)

I am running the algorithm regularly at fixed intervals and unattended, so I am getting very verbose logs from time to time. It would be nice to have an option to reduce verbosity of this error.

### Describe your proposed solution

Add an option to suppress long exception description and only emit "Input X contains NaN" message. Or only emit verbose error once, and then switch to short message.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
> @glemaitre the code that emits this Exception is used by all estimators (and probably other code), so there will be one config anyway, not one for each estimator.

The code raising is in common but you need a parameter to control it. Therefore, each estimator needs an additional parameter just for this issue.

Labels: New Feature
Reduce verbosity of error "Input X contains NaN" of PLSRegression

### Describe the workflow you want to enable

Currently if PLSRegression (and other algorithms, judging by google) gets any NaNs as input, it raises this long multiline exception:
```
ValueError: Input X contains NaN. 
PLSRegression does not accept missing values encoded as NaN natively. For supervised learning, you might want to consider sklearn.ensemble.HistGradientBoostingClassifier and Regressor which accept missing values encoded as NaNs natively. Alternatively, it is possible to preprocess the data, for instance by using an imputer transformer in a pipeline or drop samples with missing values. See https://scikit-learn.org/stable/modules/impute.html You can find a list of all estimators that handle NaN values at the following page: https://scikit-learn.org/stable/modules/impute.html#estimators-that-handle-nan-values
```

(source is here: https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/utils/validation.py#L158)

I am running the algorithm regularly at fixed intervals and unattended, so I am getting very verbose logs from time to time. It would be nice to have an option to reduce verbosity of this error.

### Describe your proposed solution

Add an option to suppress long exception description and only emit "Input X contains NaN" message. Or only emit verbose error once, and then switch to short message.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
> egarding try..except: since ValueError is rather generic error, I am interested in logging exception text, so I cannot just raise Exception() in my except ValueError:, because I'll loose actual reason for error. 

You can parse the error message:

```python
try:
    model.fit(X, y)
except ValueError as exc:
    if "Input X contains NaN" in str(exc):
        # shorten the error message
        raise ValueError("Input X contains NaN")
    raise
```

Labels: New Feature
Reduce verbosity of error "Input X contains NaN" of PLSRegression

### Describe the workflow you want to enable

Currently if PLSRegression (and other algorithms, judging by google) gets any NaNs as input, it raises this long multiline exception:
```
ValueError: Input X contains NaN. 
PLSRegression does not accept missing values encoded as NaN natively. For supervised learning, you might want to consider sklearn.ensemble.HistGradientBoostingClassifier and Regressor which accept missing values encoded as NaNs natively. Alternatively, it is possible to preprocess the data, for instance by using an imputer transformer in a pipeline or drop samples with missing values. See https://scikit-learn.org/stable/modules/impute.html You can find a list of all estimators that handle NaN values at the following page: https://scikit-learn.org/stable/modules/impute.html#estimators-that-handle-nan-values
```

(source is here: https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/utils/validation.py#L158)

I am running the algorithm regularly at fixed intervals and unattended, so I am getting very verbose logs from time to time. It would be nice to have an option to reduce verbosity of this error.

### Describe your proposed solution

Add an option to suppress long exception description and only emit "Input X contains NaN" message. Or only emit verbose error once, and then switch to short message.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
That's exactly how I do it now, with `if "Input X contains NaN" in str(exc):` check. However, this relies on library's internal implementation specifics, so if you later change the string to something else ('e.g. 'Input has NaNs'), my check will suddenly stop working. Which I don't like.

Labels: New Feature
⚠️ CI failed on Linux_Nightly.pylatest_pip_scipy_dev ⚠️

**CI failed on [Linux_Nightly.pylatest_pip_scipy_dev](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=62150&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf)** (Jan 08, 2024)
- test_minibatch_sensible_reassign[34]

COMMENT:
## CI is no longer failing! ✅

[Successful run](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=62296&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf) on Jan 12, 2024

Labels: Needs Triage
IsolationForest should maybe check for duplicate y values that are given to ExtraTreeRegressor.

My understanding is that IsolationForest uses ExtraTreeRegressor, which in turn inherits from DecisionTreeRegressor, with random y values to ensure that all leafs correspond to a single point (required for IsolationForest algorithm). If DecisionTreeRegressor terminates when all y values at a node are the same, then the there needs to be a check for duplicate y values after this line:

https://github.com/scikit-learn/scikit-learn/blob/3f89022fa04d293152f1d32fbc2a5bdaaf2df364/sklearn/ensemble/_iforest.py#L297

If DecisionTreeRegressor for some reason still splits when all y values are the same, then this is a moot point.

COMMENT:
(yes, this bug is likely to never occur in the wild)

Labels: Needs Triage
IsolationForest should maybe check for duplicate y values that are given to ExtraTreeRegressor.

My understanding is that IsolationForest uses ExtraTreeRegressor, which in turn inherits from DecisionTreeRegressor, with random y values to ensure that all leafs correspond to a single point (required for IsolationForest algorithm). If DecisionTreeRegressor terminates when all y values at a node are the same, then the there needs to be a check for duplicate y values after this line:

https://github.com/scikit-learn/scikit-learn/blob/3f89022fa04d293152f1d32fbc2a5bdaaf2df364/sklearn/ensemble/_iforest.py#L297

If DecisionTreeRegressor for some reason still splits when all y values are the same, then this is a moot point.

COMMENT:
This is rather unlikely to happen.

Labels: Needs Triage
Regression Probability Distribution & Multi-Quantile Output API

### Describe the workflow you want to enable

Scikit-learn has a `predict` and `predict_proba` method for Classification classes but only a `predict` method for regression, with the option of quantile. Scikit-learn is adding more quantile output functionality [HistGradientBoostingRegressor](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.HistGradientBoostingRegressor.html) and [QuantileRegressor](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.QuantileRegressor.html#sklearn.linear_model.QuantileRegressor) - no doubt more will come in due course. The single `quantile` parameter is set at the class init step.

LightGBM and other packages also follow a similar API.

[MAPIE](https://mapie.readthedocs.io/en/latest/generated/mapie.regression.MapieQuantileRegressor.html#mapie.regression.MapieQuantileRegressor) allows alpha being set on class init and predict.

[XGBoost](https://xgboost.readthedocs.io/en/latest/python/examples/quantile_regression.html) also has this option but also allows multiple outputs with e.g. `alpha=np.array([0.05, 0.5, 0.95])`. Currently, this isn't documented in the scikit-learn documentation. Clearly, this is a far superior piece of functionality where possible.

Additionally, distributional regression packages like:
[XGBoostLSS](https://statmixedml.github.io/XGBoostLSS/) allow options on the `predict` method such as: `pred_type` = `quantiles`, `parameters`, `expectiles`. This returns a m x n array.

[PGBM](https://github.com/elephaint/pgbm) uses `predict` with just mean and an `return_std=True` option as a 1 x n or 2 x n array.

[XGBD](https://github.com/CDonnerer/xgboost-distribution?tab=readme-ov-file) returns the mean and std as a namedtuple.

[NGBoost](https://github.com/stanfordmlgroup/ngboost) has `predict` and `pred_dist` which return point predictions and the distribution parameters that can be passed to a scipy.stats distribution object. E.g. `normal`.

All of these packages use scikit learn style APIs or aim to add this as a feature.

### Describe your proposed solution

All this is to say, I think scikit-learn has the authority and opportunity to lead the way on unifying an API to cover both distributional outputs and quantile outputs. Therefore, I would like to open a discussion with the following points:

1) Should this be added to the core scikit learn API/package? Is this within scope?
2) If not, should it be an sckit-learn contrib package or is this something the "distributional python ML community" should sort out themselves?

3) If this is something scikit-learn would like to take a lead on, what should this API look like? If scikit-learn admin/owners think this is outside of scope but still have insights/opinions on this it would still be extremely valuable to hear them!

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I'll add any additional information from this to the previous issue. Thanks for checking back in

Labels: New Feature, Needs Triage
ENH: Random Forest Classifier oob scaling/parallel

My team, working on a bioinformatics problem with high feature count (columns/dimensions in `X`), noticed that the `RandomForestClassifier` out of bag scoring doesn't scale with `n_jobs`. To be fair, `n_jobs` clearly says what it does support, though I do wonder if the out of bag predictions under the hood might also benefit from parallel support. Someone on my team seems to have found that it does help, but implemented externally to sklearn using the exposed base estimators. I suppose it might be nice to have that internally at some point, if there are no design reasons not to?

Sample reproducer code with latest stable release (`1.3.2`) on 16 cores/x86_64 Linux box (`i9-13900K`) is below the fold, and the scaling plot is underneath that. We also use far more estimators and features than that, so the delta is much greater, but the scaling trend is the main observation in any case.

<details>

```python
from time import perf_counter
import numpy as np
# sklearn 1.3.2
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt


timings_oob = []
timings_base = []
feature_counts = np.linspace(10, 180_000, 20, dtype=np.int64)

for feature_count in feature_counts:
    X, y = make_classification(n_samples=1_000,
                               n_features=feature_count,
                               random_state=0)
    for use_oob, timing_list in zip([True, False], [timings_oob, timings_base]):
        start = perf_counter()
        clf = RandomForestClassifier(n_estimators=50,
                                     random_state=0,
                                     oob_score=use_oob,
                                     n_jobs=16)
        clf.fit(X, y)
        timing_list.append(perf_counter() - start)

fig, ax = plt.subplots(1, 1)
ax.set_title(f"Random Forest OOB scaling performance")
ax.plot(feature_counts,
        timings_oob,
        label="WITH OOB",
        marker=".")
ax.plot(feature_counts,
        timings_base,
        label="NO oob",
        marker=".")
ax.set_xlabel("num features")
ax.set_ylabel("Time (s)")
ax.legend()
fig.savefig("bench_feat.png", dpi=300)
```

</details>

![image](https://github.com/scikit-learn/scikit-learn/assets/7903078/d07a173a-d847-4980-aa6b-02bc0cfc0f9a)

COMMENT:
We are open to speeding this part of the algorithm. I think that we did not dedicate too much attention because this is the main usage but if the changes are straightforward and we get a decent speed-up, I don't see why we would not be inclined at including the changes.

Labels: Enhancement