Talip7/scikit-learn-issues-multilabel huggingface dataset:
RFC Should cross-validation splitters validate that all classes are represented in each split?

This is a follow-up to the issue raised in https://github.com/scikit-learn/scikit-learn/issues/29554. However, I recall other issues raised for CV estimator in general.

So the context is the following: a CV estimator will use an internal cross-validation scheme. When we deal with a classifier, we don't have any safety mechanism in the CV to make sure that the classifier was at least trained on all classes. This could happen for two reasons on the top of the head: (i) the target is sorted and the training folds does not contain all classes and (ii) a class is potentially underrepresented and not selected.

In all cases, if the `fit` does not fail, we still obtain a broken estimator. If it breaks at `predict` at least this is not silently giving some wrong predictions but this is not a given. However, we don't provide a direct feedback to the user of what went wrong.

So I'm wondering if we should have a sort of mechanism in the CV strategies to ensure that at least all classes have been observed at `fit` time. I don't think that we should touch the estimators because we will repeat a lot of code and fundamentally the issue is raised because of the CV strategies.

NB: the same issue could happen with a simple classifier in a cross-validation to evaluate it. This is not necessarily a CV estimator.

COMMENT:
Do have a particular code snippet to illustrate this problem, including the current traceback and error message one would get as a result?

Labels: RFC
RFC Should cross-validation splitters validate that all classes are represented in each split?

This is a follow-up to the issue raised in https://github.com/scikit-learn/scikit-learn/issues/29554. However, I recall other issues raised for CV estimator in general.

So the context is the following: a CV estimator will use an internal cross-validation scheme. When we deal with a classifier, we don't have any safety mechanism in the CV to make sure that the classifier was at least trained on all classes. This could happen for two reasons on the top of the head: (i) the target is sorted and the training folds does not contain all classes and (ii) a class is potentially underrepresented and not selected.

In all cases, if the `fit` does not fail, we still obtain a broken estimator. If it breaks at `predict` at least this is not silently giving some wrong predictions but this is not a given. However, we don't provide a direct feedback to the user of what went wrong.

So I'm wondering if we should have a sort of mechanism in the CV strategies to ensure that at least all classes have been observed at `fit` time. I don't think that we should touch the estimators because we will repeat a lot of code and fundamentally the issue is raised because of the CV strategies.

NB: the same issue could happen with a simple classifier in a cross-validation to evaluate it. This is not necessarily a CV estimator.

COMMENT:
Wouldn't be an easy way to change `_parallel_build_trees` to properly handle zero-weighted classes in the training set?

Labels: RFC
RFC Should cross-validation splitters validate that all classes are represented in each split?

This is a follow-up to the issue raised in https://github.com/scikit-learn/scikit-learn/issues/29554. However, I recall other issues raised for CV estimator in general.

So the context is the following: a CV estimator will use an internal cross-validation scheme. When we deal with a classifier, we don't have any safety mechanism in the CV to make sure that the classifier was at least trained on all classes. This could happen for two reasons on the top of the head: (i) the target is sorted and the training folds does not contain all classes and (ii) a class is potentially underrepresented and not selected.

In all cases, if the `fit` does not fail, we still obtain a broken estimator. If it breaks at `predict` at least this is not silently giving some wrong predictions but this is not a given. However, we don't provide a direct feedback to the user of what went wrong.

So I'm wondering if we should have a sort of mechanism in the CV strategies to ensure that at least all classes have been observed at `fit` time. I don't think that we should touch the estimators because we will repeat a lot of code and fundamentally the issue is raised because of the CV strategies.

NB: the same issue could happen with a simple classifier in a cross-validation to evaluate it. This is not necessarily a CV estimator.

COMMENT:
Maybe we could even introduce a common test to check that all classifiers accept being fit with some zero weighted classes.

Labels: RFC
Sklearn metric module - mean squared error

### Describe the bug

```python
import matplotlib.pyplot as plt
import numpy as np
from sklearn import linear_model
from sklearn.metrics import mean_squared_error

axis_X = np.array([[1], [2], [3], [4], [5], [6], [7], [8], [9], [10]]).reshape(-1, 1)
axis_X_train = axis_X
axis_X_test = axis_X

axis_y_train = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
axis_y_test = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])

model = linear_model.LinearRegression()
model.fit(axis_X_train, axis_y_train)
axis_y_predicted = model.predict(axis_X_test)

print("Results")
print("Mean squared error is:", mean_squared_error(axis_y_test, axis_y_predicted))
print("Weights:", model.coef_)
print("Intercept:", model.intercept_)
print("axis_y_test", axis_y_test)
print("axis_y_predicted", axis_y_predicted)

plt.scatter(axis_X_test, axis_y_test)
plt.plot(axis_X_test, axis_y_predicted)
plt.show()
```

Above code is about the perfect Linear regression .Here , the Means squared value should be equal to 0 but on running the code the value is not equal to 0 . the output MSE is -8.8814...... . Hence ,the module
sklearn.metrics import mean_squared_error has a bug. kindly check the bug and resolve it asap

### Steps/Code to Reproduce

```python
import matplotlib.pyplot as plt
import numpy as np
from sklearn import linear_model
from sklearn.metrics import mean_squared_error

axis_X = np.array([[1], [2], [3], [4], [5], [6], [7], [8], [9], [10]]).reshape(-1, 1)
axis_X_train = axis_X
axis_X_test = axis_X

axis_y_train = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
axis_y_test = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])

model = linear_model.LinearRegression()
model.fit(axis_X_train, axis_y_train)
axis_y_predicted = model.predict(axis_X_test)

print("Results")
print("Mean squared error is:", mean_squared_error(axis_y_test, axis_y_predicted))
print("Weights:", model.coef_)
print("Intercept:", model.intercept_)
print("axis_y_test", axis_y_test)
print("axis_y_predicted", axis_y_predicted)

plt.scatter(axis_X_test, axis_y_test)
plt.plot(axis_X_test, axis_y_predicted)
plt.show()
```

### Expected Results

MSE should be 0

### Actual Results

MSE is not equal to 0

### Versions

```shell
Python 10
```

COMMENT:
I cannot reproduce:

```python
Results
Mean squared error is: 0.0
Weights: [1.]
Intercept: 0.0
axis_y_test [ 1  2  3  4  5  6  7  8  9 10]
axis_y_predicted [ 1.  2.  3.  4.  5.  6.  7.  8.  9. 10.]
Installed osx event loop hook.
```

> he output MSE is -8.8814...... 

I assume that the thing finish up with `1e-16` or a really small number. This is related to floating-point arithmetic and error. You can learn more here: https://docs.python.org/3/tutorial/floatingpoint.html

There is nothing that we can do.

On your side, if you are seeking if the error is zero, it means that you should not strictly check the equality with zero but check if this is close. That's why you have some function as `math.isclose` (https://docs.python.org/3/library/math.html#math.isclose) to check the closeness to floating number.

Labels: Bug, Needs Triage
RFECV cross-validation generator (`cv`) parameter

### Describe the issue linked to the documentation

Hello,  
if I'm not mistaken, I think that the [documentation of RFECV](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFECV.html) about the `cv` parameter might be incorrect regarding the choice of StratifiedKFold or KFold when the estimator is a classifier. The docs read:

> For integer/None inputs, if y is binary or multiclass, [StratifiedKFold](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.html#sklearn.model_selection.StratifiedKFold) is used. If the estimator is a classifier or if y is neither binary nor multiclass, **[KFold](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html#sklearn.model_selection.KFold)** is used. 

Which matches the [_rfe.py](https://github.com/scikit-learn/scikit-learn/blob/156ef1b7fe9bc0ee5b281634cfd56b9c54e83277/sklearn/feature_selection/_rfe.py#L527) file.

I believe that the correct phrasing of the second sentence is that it's when the estimator **is not** a classifier, then KFold will be used. When a classifier is used (and y is binary or multiclass), it's possible to perform StratifiedKFold.

In fact, code-wise, the `cv` parameter is processed by [check_cv](https://github.com/scikit-learn/scikit-learn/blob/156ef1b7fe9bc0ee5b281634cfd56b9c54e83277/sklearn/model_selection/_split.py#L2634), which states,

```
        For integer/None inputs, if classifier is True and ``y`` is either
        binary or multiclass, :class:`StratifiedKFold` is used. In all other
        cases, :class:`KFold` is used.
```
(here `classifier`  is a boolean that is True when the estimator is a classifier).  

According to the code of `check_cv()`, if `cv` is supplied as an integer (and so a cv method must be chosen), and the estimator is a classifier, and y is binary or multiclass, then the cv generator will indeed be a `StratifiedKFold`.
```
    cv = 5 if cv is None else cv
    if isinstance(cv, numbers.Integral):
        if (
            classifier
            and (y is not None)
            and (type_of_target(y, input_name="y") in ("binary", "multiclass"))
        ):
            return StratifiedKFold(cv)
        else:
            return KFold(cv)
```

Thank you!

### Suggest a potential alternative/fix

I suggest changing the phrasing "If the estimator is a classifier" to "If the estimator is not a classifier".

COMMENT:
Yes indeed. This is a typo. @bioruffo Do you wish to do a pull request?

Labels: Documentation
RFECV cross-validation generator (`cv`) parameter

### Describe the issue linked to the documentation

Hello,  
if I'm not mistaken, I think that the [documentation of RFECV](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFECV.html) about the `cv` parameter might be incorrect regarding the choice of StratifiedKFold or KFold when the estimator is a classifier. The docs read:

> For integer/None inputs, if y is binary or multiclass, [StratifiedKFold](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.html#sklearn.model_selection.StratifiedKFold) is used. If the estimator is a classifier or if y is neither binary nor multiclass, **[KFold](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html#sklearn.model_selection.KFold)** is used. 

Which matches the [_rfe.py](https://github.com/scikit-learn/scikit-learn/blob/156ef1b7fe9bc0ee5b281634cfd56b9c54e83277/sklearn/feature_selection/_rfe.py#L527) file.

I believe that the correct phrasing of the second sentence is that it's when the estimator **is not** a classifier, then KFold will be used. When a classifier is used (and y is binary or multiclass), it's possible to perform StratifiedKFold.

In fact, code-wise, the `cv` parameter is processed by [check_cv](https://github.com/scikit-learn/scikit-learn/blob/156ef1b7fe9bc0ee5b281634cfd56b9c54e83277/sklearn/model_selection/_split.py#L2634), which states,

```
        For integer/None inputs, if classifier is True and ``y`` is either
        binary or multiclass, :class:`StratifiedKFold` is used. In all other
        cases, :class:`KFold` is used.
```
(here `classifier`  is a boolean that is True when the estimator is a classifier).  

According to the code of `check_cv()`, if `cv` is supplied as an integer (and so a cv method must be chosen), and the estimator is a classifier, and y is binary or multiclass, then the cv generator will indeed be a `StratifiedKFold`.
```
    cv = 5 if cv is None else cv
    if isinstance(cv, numbers.Integral):
        if (
            classifier
            and (y is not None)
            and (type_of_target(y, input_name="y") in ("binary", "multiclass"))
        ):
            return StratifiedKFold(cv)
        else:
            return KFold(cv)
```

Thank you!

### Suggest a potential alternative/fix

I suggest changing the phrasing "If the estimator is a classifier" to "If the estimator is not a classifier".

COMMENT:
Yes I'll be happy to!

Labels: Documentation
RFECV cross-validation generator (`cv`) parameter

### Describe the issue linked to the documentation

Hello,  
if I'm not mistaken, I think that the [documentation of RFECV](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFECV.html) about the `cv` parameter might be incorrect regarding the choice of StratifiedKFold or KFold when the estimator is a classifier. The docs read:

> For integer/None inputs, if y is binary or multiclass, [StratifiedKFold](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.html#sklearn.model_selection.StratifiedKFold) is used. If the estimator is a classifier or if y is neither binary nor multiclass, **[KFold](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html#sklearn.model_selection.KFold)** is used. 

Which matches the [_rfe.py](https://github.com/scikit-learn/scikit-learn/blob/156ef1b7fe9bc0ee5b281634cfd56b9c54e83277/sklearn/feature_selection/_rfe.py#L527) file.

I believe that the correct phrasing of the second sentence is that it's when the estimator **is not** a classifier, then KFold will be used. When a classifier is used (and y is binary or multiclass), it's possible to perform StratifiedKFold.

In fact, code-wise, the `cv` parameter is processed by [check_cv](https://github.com/scikit-learn/scikit-learn/blob/156ef1b7fe9bc0ee5b281634cfd56b9c54e83277/sklearn/model_selection/_split.py#L2634), which states,

```
        For integer/None inputs, if classifier is True and ``y`` is either
        binary or multiclass, :class:`StratifiedKFold` is used. In all other
        cases, :class:`KFold` is used.
```
(here `classifier`  is a boolean that is True when the estimator is a classifier).  

According to the code of `check_cv()`, if `cv` is supplied as an integer (and so a cv method must be chosen), and the estimator is a classifier, and y is binary or multiclass, then the cv generator will indeed be a `StratifiedKFold`.
```
    cv = 5 if cv is None else cv
    if isinstance(cv, numbers.Integral):
        if (
            classifier
            and (y is not None)
            and (type_of_target(y, input_name="y") in ("binary", "multiclass"))
        ):
            return StratifiedKFold(cv)
        else:
            return KFold(cv)
```

Thank you!

### Suggest a potential alternative/fix

I suggest changing the phrasing "If the estimator is a classifier" to "If the estimator is not a classifier".

COMMENT:
Cheers.

Labels: Documentation
RFECV cross-validation generator (`cv`) parameter

### Describe the issue linked to the documentation

Hello,  
if I'm not mistaken, I think that the [documentation of RFECV](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFECV.html) about the `cv` parameter might be incorrect regarding the choice of StratifiedKFold or KFold when the estimator is a classifier. The docs read:

> For integer/None inputs, if y is binary or multiclass, [StratifiedKFold](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.html#sklearn.model_selection.StratifiedKFold) is used. If the estimator is a classifier or if y is neither binary nor multiclass, **[KFold](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html#sklearn.model_selection.KFold)** is used. 

Which matches the [_rfe.py](https://github.com/scikit-learn/scikit-learn/blob/156ef1b7fe9bc0ee5b281634cfd56b9c54e83277/sklearn/feature_selection/_rfe.py#L527) file.

I believe that the correct phrasing of the second sentence is that it's when the estimator **is not** a classifier, then KFold will be used. When a classifier is used (and y is binary or multiclass), it's possible to perform StratifiedKFold.

In fact, code-wise, the `cv` parameter is processed by [check_cv](https://github.com/scikit-learn/scikit-learn/blob/156ef1b7fe9bc0ee5b281634cfd56b9c54e83277/sklearn/model_selection/_split.py#L2634), which states,

```
        For integer/None inputs, if classifier is True and ``y`` is either
        binary or multiclass, :class:`StratifiedKFold` is used. In all other
        cases, :class:`KFold` is used.
```
(here `classifier`  is a boolean that is True when the estimator is a classifier).  

According to the code of `check_cv()`, if `cv` is supplied as an integer (and so a cv method must be chosen), and the estimator is a classifier, and y is binary or multiclass, then the cv generator will indeed be a `StratifiedKFold`.
```
    cv = 5 if cv is None else cv
    if isinstance(cv, numbers.Integral):
        if (
            classifier
            and (y is not None)
            and (type_of_target(y, input_name="y") in ("binary", "multiclass"))
        ):
            return StratifiedKFold(cv)
        else:
            return KFold(cv)
```

Thank you!

### Suggest a potential alternative/fix

I suggest changing the phrasing "If the estimator is a classifier" to "If the estimator is not a classifier".

COMMENT:
Is this closeable?

Labels: Documentation
RFECV cross-validation generator (`cv`) parameter

### Describe the issue linked to the documentation

Hello,  
if I'm not mistaken, I think that the [documentation of RFECV](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFECV.html) about the `cv` parameter might be incorrect regarding the choice of StratifiedKFold or KFold when the estimator is a classifier. The docs read:

> For integer/None inputs, if y is binary or multiclass, [StratifiedKFold](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.html#sklearn.model_selection.StratifiedKFold) is used. If the estimator is a classifier or if y is neither binary nor multiclass, **[KFold](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html#sklearn.model_selection.KFold)** is used. 

Which matches the [_rfe.py](https://github.com/scikit-learn/scikit-learn/blob/156ef1b7fe9bc0ee5b281634cfd56b9c54e83277/sklearn/feature_selection/_rfe.py#L527) file.

I believe that the correct phrasing of the second sentence is that it's when the estimator **is not** a classifier, then KFold will be used. When a classifier is used (and y is binary or multiclass), it's possible to perform StratifiedKFold.

In fact, code-wise, the `cv` parameter is processed by [check_cv](https://github.com/scikit-learn/scikit-learn/blob/156ef1b7fe9bc0ee5b281634cfd56b9c54e83277/sklearn/model_selection/_split.py#L2634), which states,

```
        For integer/None inputs, if classifier is True and ``y`` is either
        binary or multiclass, :class:`StratifiedKFold` is used. In all other
        cases, :class:`KFold` is used.
```
(here `classifier`  is a boolean that is True when the estimator is a classifier).  

According to the code of `check_cv()`, if `cv` is supplied as an integer (and so a cv method must be chosen), and the estimator is a classifier, and y is binary or multiclass, then the cv generator will indeed be a `StratifiedKFold`.
```
    cv = 5 if cv is None else cv
    if isinstance(cv, numbers.Integral):
        if (
            classifier
            and (y is not None)
            and (type_of_target(y, input_name="y") in ("binary", "multiclass"))
        ):
            return StratifiedKFold(cv)
        else:
            return KFold(cv)
```

Thank you!

### Suggest a potential alternative/fix

I suggest changing the phrasing "If the estimator is a classifier" to "If the estimator is not a classifier".

COMMENT:
Yes the typo was corrected in the above-mentioned merge, thank you!

Labels: Documentation
RFECV cross-validation generator (`cv`) parameter

### Describe the issue linked to the documentation

Hello,  
if I'm not mistaken, I think that the [documentation of RFECV](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFECV.html) about the `cv` parameter might be incorrect regarding the choice of StratifiedKFold or KFold when the estimator is a classifier. The docs read:

> For integer/None inputs, if y is binary or multiclass, [StratifiedKFold](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.html#sklearn.model_selection.StratifiedKFold) is used. If the estimator is a classifier or if y is neither binary nor multiclass, **[KFold](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html#sklearn.model_selection.KFold)** is used. 

Which matches the [_rfe.py](https://github.com/scikit-learn/scikit-learn/blob/156ef1b7fe9bc0ee5b281634cfd56b9c54e83277/sklearn/feature_selection/_rfe.py#L527) file.

I believe that the correct phrasing of the second sentence is that it's when the estimator **is not** a classifier, then KFold will be used. When a classifier is used (and y is binary or multiclass), it's possible to perform StratifiedKFold.

In fact, code-wise, the `cv` parameter is processed by [check_cv](https://github.com/scikit-learn/scikit-learn/blob/156ef1b7fe9bc0ee5b281634cfd56b9c54e83277/sklearn/model_selection/_split.py#L2634), which states,

```
        For integer/None inputs, if classifier is True and ``y`` is either
        binary or multiclass, :class:`StratifiedKFold` is used. In all other
        cases, :class:`KFold` is used.
```
(here `classifier`  is a boolean that is True when the estimator is a classifier).  

According to the code of `check_cv()`, if `cv` is supplied as an integer (and so a cv method must be chosen), and the estimator is a classifier, and y is binary or multiclass, then the cv generator will indeed be a `StratifiedKFold`.
```
    cv = 5 if cv is None else cv
    if isinstance(cv, numbers.Integral):
        if (
            classifier
            and (y is not None)
            and (type_of_target(y, input_name="y") in ("binary", "multiclass"))
        ):
            return StratifiedKFold(cv)
        else:
            return KFold(cv)
```

Thank you!

### Suggest a potential alternative/fix

I suggest changing the phrasing "If the estimator is a classifier" to "If the estimator is not a classifier".

COMMENT:
Alright good. You can close yourself you don't have to wait for a mod.

Labels: Documentation