Talip7/scikit-learn-issues-multilabel huggingface dataset:
Consolidation of the naming of `y_pred_proba`, `y_score` vs `probas_pred`

### Describe the issue linked to the documentation

I am trying to leverage the classification metrics that rely on a posterior probability (i.e. P(Y | X=x)). This is commonly named `y_pred_proba` in the sklearn API. 

However, I noticed a discrepancy in the naming of the argument for this in various metrics. For example:

- https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_curve.html#sklearn.metrics.precision_recall_curve names is `probas_pred`
- https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#sklearn.metrics.roc_auc_score names is `y_score`
- https://scikit-learn.org/stable/modules/generated/sklearn.metrics.brier_score_loss.html#sklearn.metrics.brier_score_loss is `y_prob`
- https://scikit-learn.org/stable/modules/generated/sklearn.metrics.top_k_accuracy_score.html#sklearn.metrics.top_k_accuracy_score is `y_score`

Based on the glossary, only `y_score` has anything related by ctrl+f. 

### Suggest a potential alternative/fix

Perhaps we can name them all `y_score` to be consistent? E.g. the following two metrics

- https://scikit-learn.org/stable/modules/generated/sklearn.metrics.brier_score_loss.html#sklearn.metrics.brier_score_loss
- https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_curve.html#sklearn.metrics.precision_recall_curve

COMMENT:
If I'm not mistaken, @glemaitre had touched on this, but not sure.

Labels: Documentation
Consolidation of the naming of `y_pred_proba`, `y_score` vs `probas_pred`

### Describe the issue linked to the documentation

I am trying to leverage the classification metrics that rely on a posterior probability (i.e. P(Y | X=x)). This is commonly named `y_pred_proba` in the sklearn API. 

However, I noticed a discrepancy in the naming of the argument for this in various metrics. For example:

- https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_curve.html#sklearn.metrics.precision_recall_curve names is `probas_pred`
- https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#sklearn.metrics.roc_auc_score names is `y_score`
- https://scikit-learn.org/stable/modules/generated/sklearn.metrics.brier_score_loss.html#sklearn.metrics.brier_score_loss is `y_prob`
- https://scikit-learn.org/stable/modules/generated/sklearn.metrics.top_k_accuracy_score.html#sklearn.metrics.top_k_accuracy_score is `y_score`

Based on the glossary, only `y_score` has anything related by ctrl+f. 

### Suggest a potential alternative/fix

Perhaps we can name them all `y_score` to be consistent? E.g. the following two metrics

- https://scikit-learn.org/stable/modules/generated/sklearn.metrics.brier_score_loss.html#sklearn.metrics.brier_score_loss
- https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_curve.html#sklearn.metrics.precision_recall_curve

COMMENT:
Nop, I did not but I would indeed like that we have a consistency there.

When it comes for estimators, I would like to have `y_proba` or `y_pred_proba` for the output of `predict_proba` to acknowledge that `y_proba.sum(axis=1) == 1`.

`y_score` would be reserved to `decision_function`. I would consider that `y_score` as less constraints than `y_proba`.

So when it comes to the metric, I would make it consistent and choose either `y_proba` or `y_score` depending on the type of prediction the functions take. If it take both the output of `decision_function` and `predict_proba`, then I would favor `y_score`.

In terms of deprecation cycle, I assume that those variables are not passed as keyword only. Therefore, we will need to rename the variable but still have a deprecation cycle for the old name (that we can put at the end of the signature).

Labels: Documentation
Consolidation of the naming of `y_pred_proba`, `y_score` vs `probas_pred`

### Describe the issue linked to the documentation

I am trying to leverage the classification metrics that rely on a posterior probability (i.e. P(Y | X=x)). This is commonly named `y_pred_proba` in the sklearn API. 

However, I noticed a discrepancy in the naming of the argument for this in various metrics. For example:

- https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_curve.html#sklearn.metrics.precision_recall_curve names is `probas_pred`
- https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#sklearn.metrics.roc_auc_score names is `y_score`
- https://scikit-learn.org/stable/modules/generated/sklearn.metrics.brier_score_loss.html#sklearn.metrics.brier_score_loss is `y_prob`
- https://scikit-learn.org/stable/modules/generated/sklearn.metrics.top_k_accuracy_score.html#sklearn.metrics.top_k_accuracy_score is `y_score`

Based on the glossary, only `y_score` has anything related by ctrl+f. 

### Suggest a potential alternative/fix

Perhaps we can name them all `y_score` to be consistent? E.g. the following two metrics

- https://scikit-learn.org/stable/modules/generated/sklearn.metrics.brier_score_loss.html#sklearn.metrics.brier_score_loss
- https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_curve.html#sklearn.metrics.precision_recall_curve

COMMENT:
I think we had a discussion on this like a decade ago, but I think ``proba`` instead of ``prob`` was a french-ism, and so I wonder if we should make it the standard. Though it would be consistent with ``predict_proba``...

Labels: Documentation
[RFC] Allow handling of NaNs in multi-task Random Forests

### Describe the workflow you want to enable

 Currently the RFR implementation is only capable of handling dense multi-task problems is there any scope to change the underlying algorithm to handle NaNs as a special case or would this break the API given that it might extend to allowing NaNs in the single class case? 

The hypothesis is that I may benefit from knowing the information about the available labels in the multi-task model even though I am missing labels for some tasks.

### Describe your proposed solution

boolean flag in the constructor that allows NaNs to be ignored when calculating the impurity.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

```python
>>> import numpy as np
>>> from sklearn.ensemble import RandomForestRegressor
>>> rfr=RandomForestRegressor()
>>> X = np.random.randn(5,10)
>>> y = np.random.randn(5,2)
>>> y[4,1]=np.nan
>>> rfr.fit(X, y)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/base.py", line 1151, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/ensemble/_forest.py", line 348, in fit
    X, y = self._validate_data(
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/base.py", line 621, in _validate_data
    X, y = check_X_y(X, y, **check_params)
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/utils/validation.py", line 1163, in check_X_y
    y = _check_y(y, multi_output=multi_output, y_numeric=y_numeric, estimator=estimator)
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/utils/validation.py", line 1173, in _check_y
    y = check_array(
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/utils/validation.py", line 959, in check_array
    _assert_all_finite(
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/utils/validation.py", line 124, in _assert_all_finite
    _assert_all_finite_element_wise(
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/utils/validation.py", line 173, in _assert_all_finite_element_wise
    raise ValueError(msg_err)
ValueError: Input y contains NaN.
```

COMMENT:
Hi, I would like to give a try at solving this issue. Would the instances with Nan values simply be ignored or would they be replaced with something like the median? And another question, wouldn't it be the programmer's responsability to impute the missing values from the start?

Labels: New Feature
[RFC] Allow handling of NaNs in multi-task Random Forests

### Describe the workflow you want to enable

 Currently the RFR implementation is only capable of handling dense multi-task problems is there any scope to change the underlying algorithm to handle NaNs as a special case or would this break the API given that it might extend to allowing NaNs in the single class case? 

The hypothesis is that I may benefit from knowing the information about the available labels in the multi-task model even though I am missing labels for some tasks.

### Describe your proposed solution

boolean flag in the constructor that allows NaNs to be ignored when calculating the impurity.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

```python
>>> import numpy as np
>>> from sklearn.ensemble import RandomForestRegressor
>>> rfr=RandomForestRegressor()
>>> X = np.random.randn(5,10)
>>> y = np.random.randn(5,2)
>>> y[4,1]=np.nan
>>> rfr.fit(X, y)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/base.py", line 1151, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/ensemble/_forest.py", line 348, in fit
    X, y = self._validate_data(
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/base.py", line 621, in _validate_data
    X, y = check_X_y(X, y, **check_params)
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/utils/validation.py", line 1163, in check_X_y
    y = _check_y(y, multi_output=multi_output, y_numeric=y_numeric, estimator=estimator)
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/utils/validation.py", line 1173, in _check_y
    y = check_array(
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/utils/validation.py", line 959, in check_array
    _assert_all_finite(
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/utils/validation.py", line 124, in _assert_all_finite
    _assert_all_finite_element_wise(
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/utils/validation.py", line 173, in _assert_all_finite_element_wise
    raise ValueError(msg_err)
ValueError: Input y contains NaN.
```

COMMENT:
@aventayman I believe it usually would be, personally it seems like a bad idea to auto-ignore data, since either if the x is nan or y is nan all of their counterparts would be ignored as well with no warning

Labels: New Feature
[RFC] Allow handling of NaNs in multi-task Random Forests

### Describe the workflow you want to enable

 Currently the RFR implementation is only capable of handling dense multi-task problems is there any scope to change the underlying algorithm to handle NaNs as a special case or would this break the API given that it might extend to allowing NaNs in the single class case? 

The hypothesis is that I may benefit from knowing the information about the available labels in the multi-task model even though I am missing labels for some tasks.

### Describe your proposed solution

boolean flag in the constructor that allows NaNs to be ignored when calculating the impurity.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

```python
>>> import numpy as np
>>> from sklearn.ensemble import RandomForestRegressor
>>> rfr=RandomForestRegressor()
>>> X = np.random.randn(5,10)
>>> y = np.random.randn(5,2)
>>> y[4,1]=np.nan
>>> rfr.fit(X, y)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/base.py", line 1151, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/ensemble/_forest.py", line 348, in fit
    X, y = self._validate_data(
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/base.py", line 621, in _validate_data
    X, y = check_X_y(X, y, **check_params)
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/utils/validation.py", line 1163, in check_X_y
    y = _check_y(y, multi_output=multi_output, y_numeric=y_numeric, estimator=estimator)
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/utils/validation.py", line 1173, in _check_y
    y = check_array(
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/utils/validation.py", line 959, in check_array
    _assert_all_finite(
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/utils/validation.py", line 124, in _assert_all_finite
    _assert_all_finite_element_wise(
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/utils/validation.py", line 173, in _assert_all_finite_element_wise
    raise ValueError(msg_err)
ValueError: Input y contains NaN.
```

COMMENT:
@CompRhys maybe a raised warning?

Labels: New Feature
[RFC] Allow handling of NaNs in multi-task Random Forests

### Describe the workflow you want to enable

 Currently the RFR implementation is only capable of handling dense multi-task problems is there any scope to change the underlying algorithm to handle NaNs as a special case or would this break the API given that it might extend to allowing NaNs in the single class case? 

The hypothesis is that I may benefit from knowing the information about the available labels in the multi-task model even though I am missing labels for some tasks.

### Describe your proposed solution

boolean flag in the constructor that allows NaNs to be ignored when calculating the impurity.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

```python
>>> import numpy as np
>>> from sklearn.ensemble import RandomForestRegressor
>>> rfr=RandomForestRegressor()
>>> X = np.random.randn(5,10)
>>> y = np.random.randn(5,2)
>>> y[4,1]=np.nan
>>> rfr.fit(X, y)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/base.py", line 1151, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/ensemble/_forest.py", line 348, in fit
    X, y = self._validate_data(
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/base.py", line 621, in _validate_data
    X, y = check_X_y(X, y, **check_params)
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/utils/validation.py", line 1163, in check_X_y
    y = _check_y(y, multi_output=multi_output, y_numeric=y_numeric, estimator=estimator)
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/utils/validation.py", line 1173, in _check_y
    y = check_array(
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/utils/validation.py", line 959, in check_array
    _assert_all_finite(
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/utils/validation.py", line 124, in _assert_all_finite
    _assert_all_finite_element_wise(
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/utils/validation.py", line 173, in _assert_all_finite_element_wise
    raise ValueError(msg_err)
ValueError: Input y contains NaN.
```

COMMENT:
where/why would you raise a warning if the user has to explicitly configure the model to ignore missing output values? either a maintainer will believe this is within the spirit of the API or it's outside the scope of the API and that users wanting this feature should implement themselves or go elsewhere. You could perhaps warn if all of some output y_i was NaN but more than that seems like arguing the case that this isn't really within the spirit of sklearn if you're warning for legitimate use.

Labels: New Feature
[RFC] Allow handling of NaNs in multi-task Random Forests

### Describe the workflow you want to enable

 Currently the RFR implementation is only capable of handling dense multi-task problems is there any scope to change the underlying algorithm to handle NaNs as a special case or would this break the API given that it might extend to allowing NaNs in the single class case? 

The hypothesis is that I may benefit from knowing the information about the available labels in the multi-task model even though I am missing labels for some tasks.

### Describe your proposed solution

boolean flag in the constructor that allows NaNs to be ignored when calculating the impurity.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

```python
>>> import numpy as np
>>> from sklearn.ensemble import RandomForestRegressor
>>> rfr=RandomForestRegressor()
>>> X = np.random.randn(5,10)
>>> y = np.random.randn(5,2)
>>> y[4,1]=np.nan
>>> rfr.fit(X, y)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/base.py", line 1151, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/ensemble/_forest.py", line 348, in fit
    X, y = self._validate_data(
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/base.py", line 621, in _validate_data
    X, y = check_X_y(X, y, **check_params)
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/utils/validation.py", line 1163, in check_X_y
    y = _check_y(y, multi_output=multi_output, y_numeric=y_numeric, estimator=estimator)
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/utils/validation.py", line 1173, in _check_y
    y = check_array(
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/utils/validation.py", line 959, in check_array
    _assert_all_finite(
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/utils/validation.py", line 124, in _assert_all_finite
    _assert_all_finite_element_wise(
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/utils/validation.py", line 173, in _assert_all_finite_element_wise
    raise ValueError(msg_err)
ValueError: Input y contains NaN.
```

COMMENT:
I missed that part before, if you explicitly configure it then there should be no warning

Labels: New Feature
[RFC] Allow handling of NaNs in multi-task Random Forests

### Describe the workflow you want to enable

 Currently the RFR implementation is only capable of handling dense multi-task problems is there any scope to change the underlying algorithm to handle NaNs as a special case or would this break the API given that it might extend to allowing NaNs in the single class case? 

The hypothesis is that I may benefit from knowing the information about the available labels in the multi-task model even though I am missing labels for some tasks.

### Describe your proposed solution

boolean flag in the constructor that allows NaNs to be ignored when calculating the impurity.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

```python
>>> import numpy as np
>>> from sklearn.ensemble import RandomForestRegressor
>>> rfr=RandomForestRegressor()
>>> X = np.random.randn(5,10)
>>> y = np.random.randn(5,2)
>>> y[4,1]=np.nan
>>> rfr.fit(X, y)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/base.py", line 1151, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/ensemble/_forest.py", line 348, in fit
    X, y = self._validate_data(
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/base.py", line 621, in _validate_data
    X, y = check_X_y(X, y, **check_params)
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/utils/validation.py", line 1163, in check_X_y
    y = _check_y(y, multi_output=multi_output, y_numeric=y_numeric, estimator=estimator)
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/utils/validation.py", line 1173, in _check_y
    y = check_array(
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/utils/validation.py", line 959, in check_array
    _assert_all_finite(
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/utils/validation.py", line 124, in _assert_all_finite
    _assert_all_finite_element_wise(
  File "/Users/chemix-rhys/opt/miniconda3/envs/chemix-py310/lib/python3.10/site-packages/sklearn/utils/validation.py", line 173, in _assert_all_finite_element_wise
    raise ValueError(msg_err)
ValueError: Input y contains NaN.
```

COMMENT:
After looking this over, I agree, I think there can be some benefit to the model that you describing. I can help with this. For imputing, should we offer various methods for imputing, like mean & median? We could also do something more sophisticated like K-means for imputing, although we will need to look into the efficacy of those.

Labels: New Feature