Talip7/scikit-learn-issues-multilabel huggingface dataset:
VarianceThreshold does not state whether normalisation is required

### Describe the issue linked to the documentation

Is normalisation required?  If so, it would be good to state this in the docs:

https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.VarianceThreshold.html

The example applies the VarianceThreshold to unnormalised data.

### Suggest a potential alternative/fix

Describe whether normalisation should be required and if it is, update the example to have the normalisation step.

COMMENT:
> The example in the docstring is showcasing Use Case 1. I think we can add another example showcasing Use Case 2.

I created a small code snippet for this:

```
import sklearn
from sklearn.feature_selection import VarianceThreshold
from sklearn.preprocessing import Normalizer

X = [[0, 2, 0, 3], [0, 1, 4, 3], [0, 1, 1, 3]]

normalizer = Normalizer()
normalized_X = normalizer.fit_transform(X)
print(f"Normalized X:\n{normalized_X}\n")

thresholder = VarianceThreshold(threshold=0.1)
thresholded_X = thresholder.fit_transform(normalized_X)
print(f"Thresholded X:\n{thresholded_X}")
```

Output:

```
Normalized X:
[[0.         0.5547002  0.         0.83205029]
 [0.         0.19611614 0.78446454 0.58834841]
 [0.         0.30151134 0.30151134 0.90453403]]

Thresholded X:
[[0.        ]
 [0.78446454]
 [0.30151134]]
```

Should I create a PR to include this to the docs @thomasjpfan ?

Labels: Documentation, module:feature_selection
VarianceThreshold does not state whether normalisation is required

### Describe the issue linked to the documentation

Is normalisation required?  If so, it would be good to state this in the docs:

https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.VarianceThreshold.html

The example applies the VarianceThreshold to unnormalised data.

### Suggest a potential alternative/fix

Describe whether normalisation should be required and if it is, update the example to have the normalisation step.

COMMENT:
I was unclear about Case 2. In https://github.com/scikit-learn/scikit-learn/issues/23394#issuecomment-1132997413, I was thinking of normalizing the variance itself with the "Coefficient of variation", which is a feature request https://github.com/scikit-learn/scikit-learn/issues/15118 or the "quartile coefficient of dispersion".

Looking at it again, I think the current `VarianceThreshold` is only useful for removing constant features. In that case, we do not need to extend the docstring.

Labels: Documentation, module:feature_selection
VarianceThreshold does not state whether normalisation is required

### Describe the issue linked to the documentation

Is normalisation required?  If so, it would be good to state this in the docs:

https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.VarianceThreshold.html

The example applies the VarianceThreshold to unnormalised data.

### Suggest a potential alternative/fix

Describe whether normalisation should be required and if it is, update the example to have the normalisation step.

COMMENT:
Hi,

A simple question: When you perform a normalization, let's say a standardization, this make all std equal to 1. Which mean that all variances will be equal to 1. 
So no variance threshold !!!

Labels: Documentation, module:feature_selection
VarianceThreshold does not state whether normalisation is required

### Describe the issue linked to the documentation

Is normalisation required?  If so, it would be good to state this in the docs:

https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.VarianceThreshold.html

The example applies the VarianceThreshold to unnormalised data.

### Suggest a potential alternative/fix

Describe whether normalisation should be required and if it is, update the example to have the normalisation step.

COMMENT:
@fzlahlou Yeah, I am thinking also the same. If all features have the same variance, then either all will pass or not pass the threshold. I think `VarianceThreshold` is only meaningful for constant features.

Labels: Documentation, module:feature_selection
Perceptron.t_ appears off by 1

### Describe the bug

The [docs](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Perceptron.html) state that `Perceptron.t_` is the "[n]umber of weight updates performed during training" and that it should be the "[s]ame as `(n_iter_ * n_samples)`." However, the value returned is actually equal to `n_iter_ * n_samples + 1`.

I believe this is an off-by-one bug due to the initial value of 1.0 given to `t_` on [line 684](https://github.com/scikit-learn/scikit-learn/blob/32f9deaaf27c7ae56898222be9d820ba0fd1054f/sklearn/linear_model/_stochastic_gradient.py#L684) in the `_fit` method of `BaseSGDClassifier`. This value then gets added to in one of the later methods, for instance on [line 730](https://github.com/scikit-learn/scikit-learn/blob/32f9deaaf27c7ae56898222be9d820ba0fd1054f/sklearn/linear_model/_stochastic_gradient.py#L615) in the `_fit_binary` method: `self.t_ += n_iter_ * X.shape[0]`. If this is the issue, then [Line 615](https://github.com/scikit-learn/scikit-learn/blob/32f9deaaf27c7ae56898222be9d820ba0fd1054f/sklearn/linear_model/_stochastic_gradient.py#L615) in the `_partial_fit` method would also probably have to be adjusted.

Or, is this a docs issue?

I'd be willing to PR a fix for either reason (either change the default value or update the docs to reflect the `+ 1`), but I couldn't find a reason for why `t_` is initialized to 1.0.

### Steps/Code to Reproduce

```python
from sklearn.linear_model import Perceptron
import numpy as np

n_samples = 5
X = np.zeros((n_samples, 1))
y = np.array([x % 2 for x in range(n_samples)])

ppn = Perceptron().fit(X, y)
```

### Expected Results

```python
ppn.t_ == ppn.n_iter_ * n_samples
```

### Actual Results

```python
ppn.t_ == ppn.n_iter_ * n_samples + 1
```

### Versions

```shell
System:
    python: 3.10.4 | packaged by conda-forge | (main, Mar 24 2022, 17:38:57) [GCC 10.3.0]
executable: /home/doug/miniconda3/envs/cs434/bin/python
   machine: Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.31

Python dependencies:
      sklearn: 1.1.0
          pip: 22.1
   setuptools: 62.2.0
        numpy: 1.22.3
        scipy: 1.8.0
       Cython: None
       pandas: 1.4.2
   matplotlib: 3.5.2
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /home/doug/miniconda3/envs/cs434/lib/libopenblasp-r0.3.20.so
        version: 0.3.20
threading_layer: pthreads
   architecture: Haswell
    num_threads: 8

       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /home/doug/miniconda3/envs/cs434/lib/libgomp.so.1.0.0
        version: None
    num_threads: 8
```

COMMENT:
Would a PR updating the appropriate documentation section to:
```
t_ :  int
Number of weight updates performed during training. Same as (n_iter_ * n_samples + 1).
```
Be acceptable in this case?

Labels: Documentation, module:linear_model
Perceptron.t_ appears off by 1

### Describe the bug

The [docs](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Perceptron.html) state that `Perceptron.t_` is the "[n]umber of weight updates performed during training" and that it should be the "[s]ame as `(n_iter_ * n_samples)`." However, the value returned is actually equal to `n_iter_ * n_samples + 1`.

I believe this is an off-by-one bug due to the initial value of 1.0 given to `t_` on [line 684](https://github.com/scikit-learn/scikit-learn/blob/32f9deaaf27c7ae56898222be9d820ba0fd1054f/sklearn/linear_model/_stochastic_gradient.py#L684) in the `_fit` method of `BaseSGDClassifier`. This value then gets added to in one of the later methods, for instance on [line 730](https://github.com/scikit-learn/scikit-learn/blob/32f9deaaf27c7ae56898222be9d820ba0fd1054f/sklearn/linear_model/_stochastic_gradient.py#L615) in the `_fit_binary` method: `self.t_ += n_iter_ * X.shape[0]`. If this is the issue, then [Line 615](https://github.com/scikit-learn/scikit-learn/blob/32f9deaaf27c7ae56898222be9d820ba0fd1054f/sklearn/linear_model/_stochastic_gradient.py#L615) in the `_partial_fit` method would also probably have to be adjusted.

Or, is this a docs issue?

I'd be willing to PR a fix for either reason (either change the default value or update the docs to reflect the `+ 1`), but I couldn't find a reason for why `t_` is initialized to 1.0.

### Steps/Code to Reproduce

```python
from sklearn.linear_model import Perceptron
import numpy as np

n_samples = 5
X = np.zeros((n_samples, 1))
y = np.array([x % 2 for x in range(n_samples)])

ppn = Perceptron().fit(X, y)
```

### Expected Results

```python
ppn.t_ == ppn.n_iter_ * n_samples
```

### Actual Results

```python
ppn.t_ == ppn.n_iter_ * n_samples + 1
```

### Versions

```shell
System:
    python: 3.10.4 | packaged by conda-forge | (main, Mar 24 2022, 17:38:57) [GCC 10.3.0]
executable: /home/doug/miniconda3/envs/cs434/bin/python
   machine: Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.31

Python dependencies:
      sklearn: 1.1.0
          pip: 22.1
   setuptools: 62.2.0
        numpy: 1.22.3
        scipy: 1.8.0
       Cython: None
       pandas: 1.4.2
   matplotlib: 3.5.2
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /home/doug/miniconda3/envs/cs434/lib/libopenblasp-r0.3.20.so
        version: 0.3.20
threading_layer: pthreads
   architecture: Haswell
    num_threads: 8

       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /home/doug/miniconda3/envs/cs434/lib/libgomp.so.1.0.0
        version: None
    num_threads: 8
```

COMMENT:
> Be acceptable in this case?

I think fixing the documentation is acceptable in this case.

Labels: Documentation, module:linear_model
Unable to import joblib after update to 1.1.0

### Describe the bug

Hi community,

I was very excited after the get_features_names_out fixes in 1.1 and I wanted to incorporate changes in my training code according to that.

However now Im getting and error regarding joblib. Code to reproduce below:



### Steps/Code to Reproduce

```python
from azureml.core import Run, Datastore, Workspace
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import train_test_split, cross_validate
from sklearn.metrics import make_scorer, mean_tweedie_deviance
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.compose import make_column_selector as selector
from sklearn.compose import TransformedTargetRegressor
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.feature_selection import VarianceThreshold
from sklearn.impute import SimpleImputer
from sklearn.metrics import mean_absolute_error
from sklearn.metrics import roc_auc_score, confusion_matrix
from sklearn.metrics import ConfusionMatrixDisplay
from imblearn.over_sampling import SMOTE, RandomOverSampler
from imblearn.combine import SMOTEENN
from imblearn.under_sampling import RandomUnderSampler
from imblearn.pipeline import Pipeline
import xgboost as xgb
import joblib
import pandas as pd
import numpy as np
import argparse
```


### Expected Results

No error is thrown

### Actual Results

Traceback (most recent call last):
  File "main.py", line 9, in <module>
ImportError: cannot import name '_joblib_parallel_args' from 'sklearn.utils.fixes' (/azureml-envs/azureml_7e9749cc412d37eccb74a0bf3ca8abb2/lib/python3.8/site-packages/sklearn/utils/fixes.py)

### Versions

```shell
Conda environment:

channels:
  - anaconda
  - conda-forge
  - defaults
dependencies:
  - python=3.8.10
  - pip=20.1.1=py38_1
  - pip:
      - azureml-core==1.41.0
      - azureml-interpret==1.41.0
      - azureml-sdk==1.41.0
      - imbalanced-learn==0.9.0
      - numpy== 1.19.5
      - pandas==1.4.0
      - scikit-learn==1.1.0
      - scipy==1.5.2
      - seaborn==0.11.2
      - xgboost==1.4.2
name: test1.1
```

COMMENT:
Probably because imbalanced-learn has not been released yet, so it is not compatible with scikit-learn 1.1.0? See https://github.com/scikit-learn-contrib/imbalanced-learn/issues/894 for more details.

Labels: Bug, Needs Triage
Unable to import joblib after update to 1.1.0

### Describe the bug

Hi community,

I was very excited after the get_features_names_out fixes in 1.1 and I wanted to incorporate changes in my training code according to that.

However now Im getting and error regarding joblib. Code to reproduce below:



### Steps/Code to Reproduce

```python
from azureml.core import Run, Datastore, Workspace
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import train_test_split, cross_validate
from sklearn.metrics import make_scorer, mean_tweedie_deviance
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.compose import make_column_selector as selector
from sklearn.compose import TransformedTargetRegressor
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.feature_selection import VarianceThreshold
from sklearn.impute import SimpleImputer
from sklearn.metrics import mean_absolute_error
from sklearn.metrics import roc_auc_score, confusion_matrix
from sklearn.metrics import ConfusionMatrixDisplay
from imblearn.over_sampling import SMOTE, RandomOverSampler
from imblearn.combine import SMOTEENN
from imblearn.under_sampling import RandomUnderSampler
from imblearn.pipeline import Pipeline
import xgboost as xgb
import joblib
import pandas as pd
import numpy as np
import argparse
```


### Expected Results

No error is thrown

### Actual Results

Traceback (most recent call last):
  File "main.py", line 9, in <module>
ImportError: cannot import name '_joblib_parallel_args' from 'sklearn.utils.fixes' (/azureml-envs/azureml_7e9749cc412d37eccb74a0bf3ca8abb2/lib/python3.8/site-packages/sklearn/utils/fixes.py)

### Versions

```shell
Conda environment:

channels:
  - anaconda
  - conda-forge
  - defaults
dependencies:
  - python=3.8.10
  - pip=20.1.1=py38_1
  - pip:
      - azureml-core==1.41.0
      - azureml-interpret==1.41.0
      - azureml-sdk==1.41.0
      - imbalanced-learn==0.9.0
      - numpy== 1.19.5
      - pandas==1.4.0
      - scikit-learn==1.1.0
      - scipy==1.5.2
      - seaborn==0.11.2
      - xgboost==1.4.2
name: test1.1
```

COMMENT:
I am personally on it. The bug fix release will be available by tonight on PyPI and conda-forge and compatible with scikit-learn 1.0

Labels: Bug, Needs Triage
Unable to import joblib after update to 1.1.0

### Describe the bug

Hi community,

I was very excited after the get_features_names_out fixes in 1.1 and I wanted to incorporate changes in my training code according to that.

However now Im getting and error regarding joblib. Code to reproduce below:



### Steps/Code to Reproduce

```python
from azureml.core import Run, Datastore, Workspace
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import train_test_split, cross_validate
from sklearn.metrics import make_scorer, mean_tweedie_deviance
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.compose import make_column_selector as selector
from sklearn.compose import TransformedTargetRegressor
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.feature_selection import VarianceThreshold
from sklearn.impute import SimpleImputer
from sklearn.metrics import mean_absolute_error
from sklearn.metrics import roc_auc_score, confusion_matrix
from sklearn.metrics import ConfusionMatrixDisplay
from imblearn.over_sampling import SMOTE, RandomOverSampler
from imblearn.combine import SMOTEENN
from imblearn.under_sampling import RandomUnderSampler
from imblearn.pipeline import Pipeline
import xgboost as xgb
import joblib
import pandas as pd
import numpy as np
import argparse
```


### Expected Results

No error is thrown

### Actual Results

Traceback (most recent call last):
  File "main.py", line 9, in <module>
ImportError: cannot import name '_joblib_parallel_args' from 'sklearn.utils.fixes' (/azureml-envs/azureml_7e9749cc412d37eccb74a0bf3ca8abb2/lib/python3.8/site-packages/sklearn/utils/fixes.py)

### Versions

```shell
Conda environment:

channels:
  - anaconda
  - conda-forge
  - defaults
dependencies:
  - python=3.8.10
  - pip=20.1.1=py38_1
  - pip:
      - azureml-core==1.41.0
      - azureml-interpret==1.41.0
      - azureml-sdk==1.41.0
      - imbalanced-learn==0.9.0
      - numpy== 1.19.5
      - pandas==1.4.0
      - scikit-learn==1.1.0
      - scipy==1.5.2
      - seaborn==0.11.2
      - xgboost==1.4.2
name: test1.1
```

COMMENT:
Closing since the release is new release is available in PyPI and shortly in conda-forge

Labels: Bug, Needs Triage
CV integration for OOB-scoring

### Describe the workflow you want to enable

Out-of-Bag (OOB) scoring provides an estimate of the model generalizability for `RandomForest` without needing to refit the model several times as is demanded by k-fold cross validation (CV). Although `sklearn` provides a mechanism to obtain this estimate, it does *not* provide a mechanism to integrate it into the existing cross validation workflows. For example, we might have a `GridSearchCV` where we want to optimise hyperparameters for the forest, but only fitting *once* per parameter set. This in theory could be implemented using OOB error. 

As far as I can, the two parameters of interest here are `cv` and `scoring`, both inputs to all the CV-related classes, and ultimately to `cross_val_score()`. `scoring` can be implemented easily enough using a [custom scorer](https://scikit-learn.org/stable/modules/model_evaluation.html#implementing-your-own-scoring-object), since this has access to the final estimator and therefore the OOB error. What is problematic here is the `cv` argument, which *requires* that we split the dataset, and offers no alternative.

### Describe your proposed solution

* We add `sklearn.metrics.oob`, a scoring function that just returns the oob error on the trained classifier
* We add `sklearn.model_selection.IntegratedCV`, which is a cross validator that **does not split the data at all**. ie `IntegratedCV().split(X)` will return `X` unchanged

With the combination of these two entities, users will be able to perform OOB-based cross-validation

### Describe alternatives you've considered, if relevant

It is possible to apply general cross validation metrics to a `RandomForest`, such as k-folds. This is an alternative that already exists in sklearn today. However we are neglecting the significant (`k` times) speedup that could be obtained using OOB error.

### Additional context

This question is notably discussed in these threads:
* https://datascience.stackexchange.com/a/66238/83633
* https://datascience.stackexchange.com/questions/37393/scikitlearn-grid-search-random-forest-using-oob-as-metric
* https://datascience.stackexchange.com/questions/76304/gridsearchcv-with-random-forest-classifier

COMMENT:
Are there any papers that study using the RandomForest's OOB for cross validation?

Labels: New Feature, module:ensemble, Needs Decision - Include Feature
CV integration for OOB-scoring

### Describe the workflow you want to enable

Out-of-Bag (OOB) scoring provides an estimate of the model generalizability for `RandomForest` without needing to refit the model several times as is demanded by k-fold cross validation (CV). Although `sklearn` provides a mechanism to obtain this estimate, it does *not* provide a mechanism to integrate it into the existing cross validation workflows. For example, we might have a `GridSearchCV` where we want to optimise hyperparameters for the forest, but only fitting *once* per parameter set. This in theory could be implemented using OOB error. 

As far as I can, the two parameters of interest here are `cv` and `scoring`, both inputs to all the CV-related classes, and ultimately to `cross_val_score()`. `scoring` can be implemented easily enough using a [custom scorer](https://scikit-learn.org/stable/modules/model_evaluation.html#implementing-your-own-scoring-object), since this has access to the final estimator and therefore the OOB error. What is problematic here is the `cv` argument, which *requires* that we split the dataset, and offers no alternative.

### Describe your proposed solution

* We add `sklearn.metrics.oob`, a scoring function that just returns the oob error on the trained classifier
* We add `sklearn.model_selection.IntegratedCV`, which is a cross validator that **does not split the data at all**. ie `IntegratedCV().split(X)` will return `X` unchanged

With the combination of these two entities, users will be able to perform OOB-based cross-validation

### Describe alternatives you've considered, if relevant

It is possible to apply general cross validation metrics to a `RandomForest`, such as k-folds. This is an alternative that already exists in sklearn today. However we are neglecting the significant (`k` times) speedup that could be obtained using OOB error.

### Additional context

This question is notably discussed in these threads:
* https://datascience.stackexchange.com/a/66238/83633
* https://datascience.stackexchange.com/questions/37393/scikitlearn-grid-search-random-forest-using-oob-as-metric
* https://datascience.stackexchange.com/questions/76304/gridsearchcv-with-random-forest-classifier

COMMENT:
Yes, it's discussed quite extensively, e.g.

* https://doi.org/10.1023/A:1007519102914
* https://doi.org/10.1371%2Fjournal.pone.0201904
* https://dl.acm.org/doi/10.5555/315149.315316

I think the consensus is that it's not as accurate an estimator of generalization error as CV, but that doesn't mean it isn't useful (considering it's likely to be 5x faster or more)

Labels: New Feature, module:ensemble, Needs Decision - Include Feature