Talip7/scikit-learn-issues-multilabel huggingface dataset:
MinCovDet estimation of covariance with strong bias?

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/23161

<div type='discussions-op-text'>

<sup>Originally posted by **hongfei0224** April 20, 2022</sup>
I was playing with MinCovDet (Minimum Covariance Determinant) in sklearn:
[https://scikit-learn.org/stable/modules/generated/sklearn.covariance.MinCovDet.html#sklearn.covariance.MinCovDet](url)

I see the idea of MinCovDet is to make the covariance matrix estimation robust when there is outlier in the data.
To understand the behavior, I tried the multivariate normal data **without** outliers.

What I did:
- generate multivariate_normal samples with given covariance matrix (the true covariance matrix)
- use `np.cov` to estimate empirical covariance matrix
- use `MinCovDet` to estimate robust covariance matrix
- repeat above with same "true covariance matrix"
- check the distribution of empirical covariance matrix and robust covariance matrix

I was expecting both `np.cov` and `MinCovDet` estimation should be centered around the true covariance matrix, since there is no outlier in data.

but seems not the case.

![image](https://user-images.githubusercontent.com/51205365/164061364-1daa7614-5178-46ad-b3be-0ff2e3c38a42.png)
![image](https://user-images.githubusercontent.com/51205365/164061425-00155926-f523-45a3-b872-ed83660aba5e.png)

the code to replicate results

```python
import pandas as pd
import numpy as np
from sklearn.covariance import EmpiricalCovariance, MinCovDet
import matplotlib.pyplot as plt
from tqdm.autonotebook import tqdm

res = []
res2= []
a = 2
b = 5
covar = 3 

for i in tqdm(range(1000)):
    x=np.random.multivariate_normal(mean=[0,0],cov=[[a,covar],[covar,b]], size=250).T
    res.append(np.cov(x))
    mcd = MinCovDet().fit(x.T)
    res2.append(mcd.covariance_)

pd.Series([xx[0][0] for xx in res]).hist(bins=30, label='empirical',alpha=0.7)
pd.Series([xx[0][0] for xx in res2]).hist(bins=30, label='cvd',alpha=0.7)
plt.axvline(a, ls='--', color='black')
plt.title('variance')
plt.legend()
plt.show()

pd.Series([xx[0][1] for xx in res]).hist(bins=30, label='empirical',alpha=0.7)
pd.Series([xx[0][1] for xx in res2]).hist(bins=30, label='cvd',alpha=0.7)
plt.axvline(covar, ls='--', color='black')
plt.title('covariance')
plt.legend()
plt.show()
```

COMMENT:
We need to investigate the issue and go back to the original algorithm to understand if the bias is expected. It is indeed surprising.

Labels: module:covariance, Needs Investigation
MinCovDet estimation of covariance with strong bias?

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/23161

<div type='discussions-op-text'>

<sup>Originally posted by **hongfei0224** April 20, 2022</sup>
I was playing with MinCovDet (Minimum Covariance Determinant) in sklearn:
[https://scikit-learn.org/stable/modules/generated/sklearn.covariance.MinCovDet.html#sklearn.covariance.MinCovDet](url)

I see the idea of MinCovDet is to make the covariance matrix estimation robust when there is outlier in the data.
To understand the behavior, I tried the multivariate normal data **without** outliers.

What I did:
- generate multivariate_normal samples with given covariance matrix (the true covariance matrix)
- use `np.cov` to estimate empirical covariance matrix
- use `MinCovDet` to estimate robust covariance matrix
- repeat above with same "true covariance matrix"
- check the distribution of empirical covariance matrix and robust covariance matrix

I was expecting both `np.cov` and `MinCovDet` estimation should be centered around the true covariance matrix, since there is no outlier in data.

but seems not the case.

![image](https://user-images.githubusercontent.com/51205365/164061364-1daa7614-5178-46ad-b3be-0ff2e3c38a42.png)
![image](https://user-images.githubusercontent.com/51205365/164061425-00155926-f523-45a3-b872-ed83660aba5e.png)

the code to replicate results

```python
import pandas as pd
import numpy as np
from sklearn.covariance import EmpiricalCovariance, MinCovDet
import matplotlib.pyplot as plt
from tqdm.autonotebook import tqdm

res = []
res2= []
a = 2
b = 5
covar = 3 

for i in tqdm(range(1000)):
    x=np.random.multivariate_normal(mean=[0,0],cov=[[a,covar],[covar,b]], size=250).T
    res.append(np.cov(x))
    mcd = MinCovDet().fit(x.T)
    res2.append(mcd.covariance_)

pd.Series([xx[0][0] for xx in res]).hist(bins=30, label='empirical',alpha=0.7)
pd.Series([xx[0][0] for xx in res2]).hist(bins=30, label='cvd',alpha=0.7)
plt.axvline(a, ls='--', color='black')
plt.title('variance')
plt.legend()
plt.show()

pd.Series([xx[0][1] for xx in res]).hist(bins=30, label='empirical',alpha=0.7)
pd.Series([xx[0][1] for xx in res2]).hist(bins=30, label='cvd',alpha=0.7)
plt.axvline(covar, ls='--', color='black')
plt.title('covariance')
plt.legend()
plt.show()
```

COMMENT:
I am still running into this issue on sklearn `1.6.1`. @Encephala, was the point of the code in your comment that @hongfei0224 suggested fix of applying the same correction again is not correct, but rather that there is an additional correction missing?

Labels: module:covariance, Needs Investigation
MinCovDet estimation of covariance with strong bias?

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/23161

<div type='discussions-op-text'>

<sup>Originally posted by **hongfei0224** April 20, 2022</sup>
I was playing with MinCovDet (Minimum Covariance Determinant) in sklearn:
[https://scikit-learn.org/stable/modules/generated/sklearn.covariance.MinCovDet.html#sklearn.covariance.MinCovDet](url)

I see the idea of MinCovDet is to make the covariance matrix estimation robust when there is outlier in the data.
To understand the behavior, I tried the multivariate normal data **without** outliers.

What I did:
- generate multivariate_normal samples with given covariance matrix (the true covariance matrix)
- use `np.cov` to estimate empirical covariance matrix
- use `MinCovDet` to estimate robust covariance matrix
- repeat above with same "true covariance matrix"
- check the distribution of empirical covariance matrix and robust covariance matrix

I was expecting both `np.cov` and `MinCovDet` estimation should be centered around the true covariance matrix, since there is no outlier in data.

but seems not the case.

![image](https://user-images.githubusercontent.com/51205365/164061364-1daa7614-5178-46ad-b3be-0ff2e3c38a42.png)
![image](https://user-images.githubusercontent.com/51205365/164061425-00155926-f523-45a3-b872-ed83660aba5e.png)

the code to replicate results

```python
import pandas as pd
import numpy as np
from sklearn.covariance import EmpiricalCovariance, MinCovDet
import matplotlib.pyplot as plt
from tqdm.autonotebook import tqdm

res = []
res2= []
a = 2
b = 5
covar = 3 

for i in tqdm(range(1000)):
    x=np.random.multivariate_normal(mean=[0,0],cov=[[a,covar],[covar,b]], size=250).T
    res.append(np.cov(x))
    mcd = MinCovDet().fit(x.T)
    res2.append(mcd.covariance_)

pd.Series([xx[0][0] for xx in res]).hist(bins=30, label='empirical',alpha=0.7)
pd.Series([xx[0][0] for xx in res2]).hist(bins=30, label='cvd',alpha=0.7)
plt.axvline(a, ls='--', color='black')
plt.title('variance')
plt.legend()
plt.show()

pd.Series([xx[0][1] for xx in res]).hist(bins=30, label='empirical',alpha=0.7)
pd.Series([xx[0][1] for xx in res2]).hist(bins=30, label='cvd',alpha=0.7)
plt.axvline(covar, ls='--', color='black')
plt.title('covariance')
plt.legend()
plt.show()
```

COMMENT:
I think @hongfei0224 was on the right track that there are corrections missing. My code was a "naive" first try to just add another correction, but clearly it was wrong.

I think there is another correction to be applied, as I tried to explain above. My source was my lecturer's slides, but I think the best course of action would be to take a closer look at the implementation of MCD in R, which is unbiased under normality.

Labels: module:covariance, Needs Investigation
MinCovDet estimation of covariance with strong bias?

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/23161

<div type='discussions-op-text'>

<sup>Originally posted by **hongfei0224** April 20, 2022</sup>
I was playing with MinCovDet (Minimum Covariance Determinant) in sklearn:
[https://scikit-learn.org/stable/modules/generated/sklearn.covariance.MinCovDet.html#sklearn.covariance.MinCovDet](url)

I see the idea of MinCovDet is to make the covariance matrix estimation robust when there is outlier in the data.
To understand the behavior, I tried the multivariate normal data **without** outliers.

What I did:
- generate multivariate_normal samples with given covariance matrix (the true covariance matrix)
- use `np.cov` to estimate empirical covariance matrix
- use `MinCovDet` to estimate robust covariance matrix
- repeat above with same "true covariance matrix"
- check the distribution of empirical covariance matrix and robust covariance matrix

I was expecting both `np.cov` and `MinCovDet` estimation should be centered around the true covariance matrix, since there is no outlier in data.

but seems not the case.

![image](https://user-images.githubusercontent.com/51205365/164061364-1daa7614-5178-46ad-b3be-0ff2e3c38a42.png)
![image](https://user-images.githubusercontent.com/51205365/164061425-00155926-f523-45a3-b872-ed83660aba5e.png)

the code to replicate results

```python
import pandas as pd
import numpy as np
from sklearn.covariance import EmpiricalCovariance, MinCovDet
import matplotlib.pyplot as plt
from tqdm.autonotebook import tqdm

res = []
res2= []
a = 2
b = 5
covar = 3 

for i in tqdm(range(1000)):
    x=np.random.multivariate_normal(mean=[0,0],cov=[[a,covar],[covar,b]], size=250).T
    res.append(np.cov(x))
    mcd = MinCovDet().fit(x.T)
    res2.append(mcd.covariance_)

pd.Series([xx[0][0] for xx in res]).hist(bins=30, label='empirical',alpha=0.7)
pd.Series([xx[0][0] for xx in res2]).hist(bins=30, label='cvd',alpha=0.7)
plt.axvline(a, ls='--', color='black')
plt.title('variance')
plt.legend()
plt.show()

pd.Series([xx[0][1] for xx in res]).hist(bins=30, label='empirical',alpha=0.7)
pd.Series([xx[0][1] for xx in res2]).hist(bins=30, label='cvd',alpha=0.7)
plt.axvline(covar, ls='--', color='black')
plt.title('covariance')
plt.legend()
plt.show()
```

COMMENT:
I went ahead and submitted a PR with the fix proposed above https://github.com/scikit-learn/scikit-learn/pull/32117

@thomasjpfan @glemaitre let me know if there is anything I can do to help with making the PR review for this fix smoother.

Labels: module:covariance, Needs Investigation
Compare BIRCH and MiniBatchKMeans - unused variable

### Describe the issue linked to the documentation

In the code example of [plot_birch_vs_minibatchkmeans](https://scikit-learn.org/stable/auto_examples/cluster/plot_birch_vs_minibatchkmeans.html) `time_ = time() - t` is defined but not used.

### Suggest a potential alternative/fix

Remove the line.

COMMENT:
Thanks for the report @chritter. Would you like to submit a PR to fix this ?

Labels: Documentation
Compare BIRCH and MiniBatchKMeans - unused variable

### Describe the issue linked to the documentation

In the code example of [plot_birch_vs_minibatchkmeans](https://scikit-learn.org/stable/auto_examples/cluster/plot_birch_vs_minibatchkmeans.html) `time_ = time() - t` is defined but not used.

### Suggest a potential alternative/fix

Remove the line.

COMMENT:
Yes, I will do it. Thanks.

Labels: Documentation
The score evaluation with "roc_auc" is not the same in  GridSearchCV() and  roc_auc_score() functon

### Describe the bug

The score evaluation results in GridSearchCV () with refit=roc_auc not eqaul to the score caculated from roc_auc_score() functon.

### Steps/Code to Reproduce

import numpy as np
from matplotlib import pyplot as plt
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_hastie_10_2
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import make_scorer
from sklearn.metrics import accuracy_score, roc_auc_score,r2_score
from sklearn.tree import DecisionTreeClassifier
X, y = make_hastie_10_2(n_samples=8000, random_state=42)

param_test1 ={'n_estimators':range(10,71,10)}
gsearch1= GridSearchCV(estimator =RandomForestClassifier(min_samples_split=100,
                                 min_samples_leaf=20,max_depth=8,max_features='sqrt',random_state=10),
                       param_grid =param_test1,scoring='roc_auc',cv=5, refit='roc_auc')
gsearch1.fit(X,y)
print(gsearch1.score(X,y))
print(roc_auc_score(gsearch1.predict(X),y))

### Expected Results

the score in GridSearchCV with refit=roc_auc should eqaul to the score caculated from roc_auc_score() functon.

### Actual Results

print(gsearch1.score(X,y))
0.9369802905759941
print(roc_auc_score(gsearch1.predict(X),y))
0.8463063607085346


### Versions

```shell
1.0.2
```

COMMENT:
Hi @jianzhongchen1, the reason for the discrepancy is due to the incorrect calculation of auc score. You need to use `predict_proba` instead of `predict` to get probability output for correct auc calculation

```python
print(roc_auc_score(y, gsearch1.predict_proba(X)[:, 1])).  #0.9369802905759941
```

Labels: Question
The score evaluation with "roc_auc" is not the same in  GridSearchCV() and  roc_auc_score() functon

### Describe the bug

The score evaluation results in GridSearchCV () with refit=roc_auc not eqaul to the score caculated from roc_auc_score() functon.

### Steps/Code to Reproduce

import numpy as np
from matplotlib import pyplot as plt
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_hastie_10_2
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import make_scorer
from sklearn.metrics import accuracy_score, roc_auc_score,r2_score
from sklearn.tree import DecisionTreeClassifier
X, y = make_hastie_10_2(n_samples=8000, random_state=42)

param_test1 ={'n_estimators':range(10,71,10)}
gsearch1= GridSearchCV(estimator =RandomForestClassifier(min_samples_split=100,
                                 min_samples_leaf=20,max_depth=8,max_features='sqrt',random_state=10),
                       param_grid =param_test1,scoring='roc_auc',cv=5, refit='roc_auc')
gsearch1.fit(X,y)
print(gsearch1.score(X,y))
print(roc_auc_score(gsearch1.predict(X),y))

### Expected Results

the score in GridSearchCV with refit=roc_auc should eqaul to the score caculated from roc_auc_score() functon.

### Actual Results

print(gsearch1.score(X,y))
0.9369802905759941
print(roc_auc_score(gsearch1.predict(X),y))
0.8463063607085346


### Versions

```shell
1.0.2
```

COMMENT:
Thanks @MaxwellLZH, this seems like the right answer to this question, closing this one.

@jianzhongchen1, you can read more about ROC-AUC in https://scikit-learn.org/stable/modules/model_evaluation.html#binary-case.

Labels: Question
The score evaluation with "roc_auc" is not the same in  GridSearchCV() and  roc_auc_score() functon

### Describe the bug

The score evaluation results in GridSearchCV () with refit=roc_auc not eqaul to the score caculated from roc_auc_score() functon.

### Steps/Code to Reproduce

import numpy as np
from matplotlib import pyplot as plt
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_hastie_10_2
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import make_scorer
from sklearn.metrics import accuracy_score, roc_auc_score,r2_score
from sklearn.tree import DecisionTreeClassifier
X, y = make_hastie_10_2(n_samples=8000, random_state=42)

param_test1 ={'n_estimators':range(10,71,10)}
gsearch1= GridSearchCV(estimator =RandomForestClassifier(min_samples_split=100,
                                 min_samples_leaf=20,max_depth=8,max_features='sqrt',random_state=10),
                       param_grid =param_test1,scoring='roc_auc',cv=5, refit='roc_auc')
gsearch1.fit(X,y)
print(gsearch1.score(X,y))
print(roc_auc_score(gsearch1.predict(X),y))

### Expected Results

the score in GridSearchCV with refit=roc_auc should eqaul to the score caculated from roc_auc_score() functon.

### Actual Results

print(gsearch1.score(X,y))
0.9369802905759941
print(roc_auc_score(gsearch1.predict(X),y))
0.8463063607085346


### Versions

```shell
1.0.2
```

COMMENT:
Thank you.

Labels: Question
Getting ETA for model training

### Describe the workflow you want to enable

Getting an estimate time for a model to train or an estimated time remaining before the training is completed.

### Describe your proposed solution

Understand computational resources to get an estimate on time to train. OR
Use the already elapsed time to computed estimated time remaining. For example. if 10% training gets completed in 60 sec, ETA for 100% becomes 600 sec or similar. 



### Describe alternatives you've considered, if relevant

Tried scitime but thats a very invasive method. Would prefer a method that is more tightly coupled with sklearn functionality.

### Additional context

_No response_

COMMENT:
This is a recurrent feature request that will be resolved by the ongoing work on a callback API (https://github.com/scikit-learn/scikit-learn/pull/22000)

I'm closing this issue because it is a duplicate of https://github.com/scikit-learn/scikit-learn/issues/10973 and https://github.com/scikit-learn/scikit-learn/issues/78.

Labels: New Feature, Needs Triage
IsotonicRegression returns Null on normal input

### Describe the bug

calibration model IsotonicRegression returns null on scalar numeric input.

```
calibration_model.predict([[0.95803388], [0.313388]])
Out[29]: array([nan,  1.])

```

### Steps/Code to Reproduce

```
from sklearn.calibration import IsotonicRegression
m = IsotonicRegression()
m.fit([0, 0.1, 0.3, 0.4], [0, 0, 1, 1]).predict([0.5])
```

### Expected Results

array([1])

### Actual Results

array([nan])

### Versions

```shell
System:
    python: 3.9.12 (main, Mar 26 2022, 15:51:15)  [Clang 13.1.6 (clang-1316.0.21.2)]
executable: /Users/hanan/private/buzzer/lead-routing-poc/.venv/bin/python
   machine: macOS-12.3.1-x86_64-i386-64bit
Python dependencies:
          pip: 21.2.3
   setuptools: 57.4.0
      sklearn: 1.0.2
        numpy: 1.22.3
        scipy: 1.8.0
       Cython: None
       pandas: 1.4.1
   matplotlib: 3.5.1
       joblib: 1.1.0
threadpoolctl: 3.1.0
Built with OpenMP: True
```

COMMENT:
By default, [IsotonicRegression](https://scikit-learn.org/stable/modules/generated/sklearn.isotonic.IsotonicRegression.html#sklearn.isotonic.IsotonicRegression) will use `np.nan` for values that are out of bounds of the training data. You can set the `out_of_bounds` parameter to change this behavior:

```python
from sklearn.calibration import IsotonicRegression
m = IsotonicRegression(out_of_bounds='clip')

m.fit([0, 0.1, 0.3, 0.4], [0, 0, 1, 1]).predict([0.5])
# array([1.])
```

Labels: Bug, module:isotonic
Add PAV algorithm for calibration_curve/reliability diagrams

### Describe the workflow you want to enable

```python
import numpy as np
from sklearn.calibration import calibration_curve

y_true = np.array([0, 0, 0, 0, 1, 1, 1, 1, 1])
y_pred = np.array([0.1, 0.2, 0.3, 0.4, 0.65, 0.7, 0.8, 0.9,  1.])
prob_true, prob_pred = calibration_curve(y_true, y_pred, strategy="pav")
```

### Describe your proposed solution

Add the strategy **PAV** as [1] and [2] to `calibration_curve`, there called **CORP**. This is basically applying isotonic regression as binning strategy, which we already have in scikit-learn.

[1] Dimitriadis, T., Gneiting, T., & Jordan, A.I. (2021). Stable reliability diagrams for probabilistic classifiers. Proceedings of the National Academy of Sciences of the United States of America, 118. https://doi.org/10.1073/pnas.2016191118
[2] https://cran.r-project.org/package=reliabilitydiag

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Given the recency of the paper, it has clearly not enough citations (yet). But I have the impression that this is a good strategy for a reliability diagrams with good theoretical and practical properties.

To my knowledge, this strategy is nowhere available in the Python universe as of now.

COMMENT:
@aijordan For your information.

Labels: New Feature, module:calibration, Needs Decision - Include Feature
Add PAV algorithm for calibration_curve/reliability diagrams

### Describe the workflow you want to enable

```python
import numpy as np
from sklearn.calibration import calibration_curve

y_true = np.array([0, 0, 0, 0, 1, 1, 1, 1, 1])
y_pred = np.array([0.1, 0.2, 0.3, 0.4, 0.65, 0.7, 0.8, 0.9,  1.])
prob_true, prob_pred = calibration_curve(y_true, y_pred, strategy="pav")
```

### Describe your proposed solution

Add the strategy **PAV** as [1] and [2] to `calibration_curve`, there called **CORP**. This is basically applying isotonic regression as binning strategy, which we already have in scikit-learn.

[1] Dimitriadis, T., Gneiting, T., & Jordan, A.I. (2021). Stable reliability diagrams for probabilistic classifiers. Proceedings of the National Academy of Sciences of the United States of America, 118. https://doi.org/10.1073/pnas.2016191118
[2] https://cran.r-project.org/package=reliabilitydiag

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Given the recency of the paper, it has clearly not enough citations (yet). But I have the impression that this is a good strategy for a reliability diagrams with good theoretical and practical properties.

To my knowledge, this strategy is nowhere available in the Python universe as of now.

COMMENT:
It's possible that using Centered Isotonic Regression (#21454) would make the reliability diagram look even better but might break the theoretical results of the paper you linked above.

Labels: New Feature, module:calibration, Needs Decision - Include Feature
Add PAV algorithm for calibration_curve/reliability diagrams

### Describe the workflow you want to enable

```python
import numpy as np
from sklearn.calibration import calibration_curve

y_true = np.array([0, 0, 0, 0, 1, 1, 1, 1, 1])
y_pred = np.array([0.1, 0.2, 0.3, 0.4, 0.65, 0.7, 0.8, 0.9,  1.])
prob_true, prob_pred = calibration_curve(y_true, y_pred, strategy="pav")
```

### Describe your proposed solution

Add the strategy **PAV** as [1] and [2] to `calibration_curve`, there called **CORP**. This is basically applying isotonic regression as binning strategy, which we already have in scikit-learn.

[1] Dimitriadis, T., Gneiting, T., & Jordan, A.I. (2021). Stable reliability diagrams for probabilistic classifiers. Proceedings of the National Academy of Sciences of the United States of America, 118. https://doi.org/10.1073/pnas.2016191118
[2] https://cran.r-project.org/package=reliabilitydiag

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Given the recency of the paper, it has clearly not enough citations (yet). But I have the impression that this is a good strategy for a reliability diagrams with good theoretical and practical properties.

To my knowledge, this strategy is nowhere available in the Python universe as of now.

COMMENT:
@ogrisel As pointed out in #21454, centered isotonic regression seems to be an invalid option because it is not calibrated itself which is a crucial property of (standard) isotonic regression, in particular important for assessing (auto-) calibration of a model as is done in a reliability diagram.

Labels: New Feature, module:calibration, Needs Decision - Include Feature