Talip7/scikit-learn-issues-multilabel huggingface dataset:
Add PAV algorithm for calibration_curve/reliability diagrams

### Describe the workflow you want to enable

```python
import numpy as np
from sklearn.calibration import calibration_curve

y_true = np.array([0, 0, 0, 0, 1, 1, 1, 1, 1])
y_pred = np.array([0.1, 0.2, 0.3, 0.4, 0.65, 0.7, 0.8, 0.9,  1.])
prob_true, prob_pred = calibration_curve(y_true, y_pred, strategy="pav")
```

### Describe your proposed solution

Add the strategy **PAV** as [1] and [2] to `calibration_curve`, there called **CORP**. This is basically applying isotonic regression as binning strategy, which we already have in scikit-learn.

[1] Dimitriadis, T., Gneiting, T., & Jordan, A.I. (2021). Stable reliability diagrams for probabilistic classifiers. Proceedings of the National Academy of Sciences of the United States of America, 118. https://doi.org/10.1073/pnas.2016191118
[2] https://cran.r-project.org/package=reliabilitydiag

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Given the recency of the paper, it has clearly not enough citations (yet). But I have the impression that this is a good strategy for a reliability diagrams with good theoretical and practical properties.

To my knowledge, this strategy is nowhere available in the Python universe as of now.

COMMENT:
Posted by @ogrisel in https://github.com/scikit-learn/scikit-learn/issues/23767#issuecomment-1848942730

> My main concern with the CORP reliability diagrams is that they are very square-looking on finite size test sets, see for instance:
>
>![image](https://github.com/scikit-learn/scikit-learn/assets/89061/4e096a6d-b666-43a2-95ef-804320f8e0c4)
>
>Those diagrams would probably qualitatively look very different in the asymptotic regime of large test sets.
>
>To avoid this finite test sample artifact, we might also what to consider methods such as implemented in:
>
>https://github.com/apple/ml-calibration
>
>Smooth ECE: Principled Reliability Diagrams via Kernel Smoothing
Jarosław Błasiok, Preetum Nakkiran
https://arxiv.org/abs/2309.12236
>
>![image](https://github.com/scikit-learn/scikit-learn/assets/89061/b67bf731-e0e8-4b01-bc92-54290b67dc39)

Labels: New Feature, module:calibration, Needs Decision - Include Feature
Add PAV algorithm for calibration_curve/reliability diagrams

### Describe the workflow you want to enable

```python
import numpy as np
from sklearn.calibration import calibration_curve

y_true = np.array([0, 0, 0, 0, 1, 1, 1, 1, 1])
y_pred = np.array([0.1, 0.2, 0.3, 0.4, 0.65, 0.7, 0.8, 0.9,  1.])
prob_true, prob_pred = calibration_curve(y_true, y_pred, strategy="pav")
```

### Describe your proposed solution

Add the strategy **PAV** as [1] and [2] to `calibration_curve`, there called **CORP**. This is basically applying isotonic regression as binning strategy, which we already have in scikit-learn.

[1] Dimitriadis, T., Gneiting, T., & Jordan, A.I. (2021). Stable reliability diagrams for probabilistic classifiers. Proceedings of the National Academy of Sciences of the United States of America, 118. https://doi.org/10.1073/pnas.2016191118
[2] https://cran.r-project.org/package=reliabilitydiag

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Given the recency of the paper, it has clearly not enough citations (yet). But I have the impression that this is a good strategy for a reliability diagrams with good theoretical and practical properties.

To my knowledge, this strategy is nowhere available in the Python universe as of now.

COMMENT:
@ogrisel 
> My main concern with the CORP reliability diagrams is that they are very square-looking on finite size

As [Prof Simon Woods says](https://www.maths.ed.ac.uk/~swood34/):
> Statistics is the honest interpretation of data

The reliability diagram is a statistical diagnostics/verification tool and not something that needs to be pleasant for the eye, but easy to interpret.
The zigzag is a mere consequence of the underlying (small sample) uncertainty in the estimation of $E[Y|prediction]$.

BTW, I never understood why the PAV (=CORP) approach is good enough for calibration of classifiers (modifies actual model predictions), but not good enough in a reliability diagram (diagnostics) - within scikit-learn :raised_eyebrow:

Labels: New Feature, module:calibration, Needs Decision - Include Feature
Add PAV algorithm for calibration_curve/reliability diagrams

### Describe the workflow you want to enable

```python
import numpy as np
from sklearn.calibration import calibration_curve

y_true = np.array([0, 0, 0, 0, 1, 1, 1, 1, 1])
y_pred = np.array([0.1, 0.2, 0.3, 0.4, 0.65, 0.7, 0.8, 0.9,  1.])
prob_true, prob_pred = calibration_curve(y_true, y_pred, strategy="pav")
```

### Describe your proposed solution

Add the strategy **PAV** as [1] and [2] to `calibration_curve`, there called **CORP**. This is basically applying isotonic regression as binning strategy, which we already have in scikit-learn.

[1] Dimitriadis, T., Gneiting, T., & Jordan, A.I. (2021). Stable reliability diagrams for probabilistic classifiers. Proceedings of the National Academy of Sciences of the United States of America, 118. https://doi.org/10.1073/pnas.2016191118
[2] https://cran.r-project.org/package=reliabilitydiag

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Given the recency of the paper, it has clearly not enough citations (yet). But I have the impression that this is a good strategy for a reliability diagrams with good theoretical and practical properties.

To my knowledge, this strategy is nowhere available in the Python universe as of now.

COMMENT:
> The reliability diagram is a statistical diagnostics/verification tool and not something that needs to be pleasant for the eye, but easy to interpret.

It's not a matter of being pleasing to the eye but a matter about being misleading about the shape of the asymptotic curve. The asymptotic curve will be smooth most of the time, and the CORP finite sample size estimate can induce the reader into thinking otherwise, which I find misleading and a potential source of confusion for our users.

> BTW, I never understood why the PAV (=CORP) approach is good enough for calibration of classifiers (modifies actual model predictions), but not good enough in a reliability diagram (diagnostics) - within scikit-learn

I actually have the exact same concern with isotonic regression as a post hoc calibrator. I would much rather use the centered isotonic regression as post hoc classifier which is mostly strictly monotonic (and as a result would not introduce an unexpected change to pure ranking metrics such as ROC AUC / Gini index) and converges to the same solution as the step-wise constant calibrator in the large sample size.

Labels: New Feature, module:calibration, Needs Decision - Include Feature
Add PAV algorithm for calibration_curve/reliability diagrams

### Describe the workflow you want to enable

```python
import numpy as np
from sklearn.calibration import calibration_curve

y_true = np.array([0, 0, 0, 0, 1, 1, 1, 1, 1])
y_pred = np.array([0.1, 0.2, 0.3, 0.4, 0.65, 0.7, 0.8, 0.9,  1.])
prob_true, prob_pred = calibration_curve(y_true, y_pred, strategy="pav")
```

### Describe your proposed solution

Add the strategy **PAV** as [1] and [2] to `calibration_curve`, there called **CORP**. This is basically applying isotonic regression as binning strategy, which we already have in scikit-learn.

[1] Dimitriadis, T., Gneiting, T., & Jordan, A.I. (2021). Stable reliability diagrams for probabilistic classifiers. Proceedings of the National Academy of Sciences of the United States of America, 118. https://doi.org/10.1073/pnas.2016191118
[2] https://cran.r-project.org/package=reliabilitydiag

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Given the recency of the paper, it has clearly not enough citations (yet). But I have the impression that this is a good strategy for a reliability diagrams with good theoretical and practical properties.

To my knowledge, this strategy is nowhere available in the Python universe as of now.

COMMENT:
> The asymptotic curve will be smooth most of the time

That‘s not correct. For instance, tree based models or GLMs with categorical features are not smooth in the predictions.

Labels: New Feature, module:calibration, Needs Decision - Include Feature
Add PAV algorithm for calibration_curve/reliability diagrams

### Describe the workflow you want to enable

```python
import numpy as np
from sklearn.calibration import calibration_curve

y_true = np.array([0, 0, 0, 0, 1, 1, 1, 1, 1])
y_pred = np.array([0.1, 0.2, 0.3, 0.4, 0.65, 0.7, 0.8, 0.9,  1.])
prob_true, prob_pred = calibration_curve(y_true, y_pred, strategy="pav")
```

### Describe your proposed solution

Add the strategy **PAV** as [1] and [2] to `calibration_curve`, there called **CORP**. This is basically applying isotonic regression as binning strategy, which we already have in scikit-learn.

[1] Dimitriadis, T., Gneiting, T., & Jordan, A.I. (2021). Stable reliability diagrams for probabilistic classifiers. Proceedings of the National Academy of Sciences of the United States of America, 118. https://doi.org/10.1073/pnas.2016191118
[2] https://cran.r-project.org/package=reliabilitydiag

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Given the recency of the paper, it has clearly not enough citations (yet). But I have the impression that this is a good strategy for a reliability diagrams with good theoretical and practical properties.

To my knowledge, this strategy is nowhere available in the Python universe as of now.

COMMENT:
> That‘s not correct. For instance, tree based models or GLMs with categorical features are not smooth in the predictions.

Indeed that might be the case. Although depending on the size of the training set, I suspect that they are still be much smoother than what the CORP reliability diagram suggests. To settle this debate we will need some experiments with a few large datasets we we can subsample both the training set and the test set used to estimate the reliability curve and compare the small test sample CORP/smoothed reliability curves to the CORP curve on the full test set.

We could also have a reliability diagram with a user settable option to decide what strategy they want to use (fixed binning as we do now, CORP induced bin edges and some smooth estimate). Still comparing the methods on a few canonical datasets would help us make informed recommendations in the docstring of that parameter.

Labels: New Feature, module:calibration, Needs Decision - Include Feature
Macro Average F1 Score Computation Bug

### Describe the bug

The current algorithm computes labels with no support:

### Steps/Code to Reproduce

```
           precision    recall  f1-score   support

     archive       0.00      0.00      0.00         0
        high       0.35      0.34      0.34      1400
         low       0.87      0.89      0.88     29585
      medium       0.82      0.80      0.81     20102
      module       0.00      0.00      0.00         0
     project       0.00      0.00      0.00         0

    accuracy                           0.84     51087
   macro avg       0.34      0.34      0.34     51087
weighted avg       0.84      0.84      0.84     51087
```

### Expected Results

It should ignore labels with 0 supports(these labels are from the model predictions, which were out-of-distribution) and output (0.35+0.87+0.82)/3=0.68 for the Precision Macro Average. 

Consider this algorithm: https://stackoverflow.com/a/64863315/4843407

### Actual Results

It includes the labels with no support (no such label in the actual test data).

### Versions

```shell
System:
    python: 3.9.7 (default, Sep 16 2021, 13:09:58)  [GCC 7.5.0]
executable: /home/zing/anaconda3/bin/python
   machine: Linux-5.13.0-30-generic-x86_64-with-glibc2.31

Python dependencies:
          pip: 21.2.4
   setuptools: 58.0.4
      sklearn: 0.24.2
        numpy: 1.20.3
        scipy: 1.7.1
       Cython: 0.29.24
       pandas: 1.3.4
   matplotlib: 3.4.3
       joblib: 1.1.0
threadpoolctl: 2.2.0

Built with OpenMP: True
```
```

COMMENT:
For future reference, it's best to have a [minimal reproducer](https://scikit-learn.org/dev/developers/minimal_reproducer.html) for us to debug the issue.

In this case, here is a minimal reproducer:

```python
import numpy as np
from sklearn.metrics import precision_score

y_true = np.asarray([0, 0, 1, 1])
y_pred = np.asarray([0, 2, 1, 1])

precision_score(y_true, y_pred, average="macro")
# 0.6666666
```

I think this as the expected behavior when class 2 is a valid class but it so happens that a subset of the data does not contain class 2. For example:

```python
log_reg = LogisticRegression().fit(X, [0, ..., 1, ..., 2])

precision_score([0, 0, 1, 1], log_reg.predict(...), average="macro")
```

In the above case, I would want class 2 to be a part of the macro averaging.

@jnothman What do you think?

Labels: Bug, module:metrics
Macro Average F1 Score Computation Bug

### Describe the bug

The current algorithm computes labels with no support:

### Steps/Code to Reproduce

```
           precision    recall  f1-score   support

     archive       0.00      0.00      0.00         0
        high       0.35      0.34      0.34      1400
         low       0.87      0.89      0.88     29585
      medium       0.82      0.80      0.81     20102
      module       0.00      0.00      0.00         0
     project       0.00      0.00      0.00         0

    accuracy                           0.84     51087
   macro avg       0.34      0.34      0.34     51087
weighted avg       0.84      0.84      0.84     51087
```

### Expected Results

It should ignore labels with 0 supports(these labels are from the model predictions, which were out-of-distribution) and output (0.35+0.87+0.82)/3=0.68 for the Precision Macro Average. 

Consider this algorithm: https://stackoverflow.com/a/64863315/4843407

### Actual Results

It includes the labels with no support (no such label in the actual test data).

### Versions

```shell
System:
    python: 3.9.7 (default, Sep 16 2021, 13:09:58)  [GCC 7.5.0]
executable: /home/zing/anaconda3/bin/python
   machine: Linux-5.13.0-30-generic-x86_64-with-glibc2.31

Python dependencies:
          pip: 21.2.4
   setuptools: 58.0.4
      sklearn: 0.24.2
        numpy: 1.20.3
        scipy: 1.7.1
       Cython: 0.29.24
       pandas: 1.3.4
   matplotlib: 3.4.3
       joblib: 1.1.0
threadpoolctl: 2.2.0

Built with OpenMP: True
```
```

COMMENT:
> For future reference, it's best to have a [minimal reproducer](https://scikit-learn.org/dev/developers/minimal_reproducer.html) for us to debug the issue.
> 
> In this case, here is a minimal reproducer:
> 
> ```python
> import numpy as np
> from sklearn.metrics import precision_score
> 
> y_true = np.asarray([0, 0, 1, 1])
> y_pred = np.asarray([0, 2, 1, 1])
> 
> precision_score(y_true, y_pred, average="macro")
> # 0.6666666
> ```
> 
> I think this as the expected behavior when class 2 is a valid class but it so happens that a subset of the data does not contain class 2. For example:
> 
> ```python
> log_reg = LogisticRegression().fit(X, [0, ..., 1, ..., 2])
> 
> precision_score([0, 0, 1, 1], log_reg.predict(...), average="macro")
> ```
> 
> In the above case, I would want class 2 to be a part of the macro averaging.
> 
> @jnothman What do you think?

You should not assume predicted “class 2” is a valid class. It should count np.unique(actual), not no.unique(prediction)

Labels: Bug, module:metrics
Macro Average F1 Score Computation Bug

### Describe the bug

The current algorithm computes labels with no support:

### Steps/Code to Reproduce

```
           precision    recall  f1-score   support

     archive       0.00      0.00      0.00         0
        high       0.35      0.34      0.34      1400
         low       0.87      0.89      0.88     29585
      medium       0.82      0.80      0.81     20102
      module       0.00      0.00      0.00         0
     project       0.00      0.00      0.00         0

    accuracy                           0.84     51087
   macro avg       0.34      0.34      0.34     51087
weighted avg       0.84      0.84      0.84     51087
```

### Expected Results

It should ignore labels with 0 supports(these labels are from the model predictions, which were out-of-distribution) and output (0.35+0.87+0.82)/3=0.68 for the Precision Macro Average. 

Consider this algorithm: https://stackoverflow.com/a/64863315/4843407

### Actual Results

It includes the labels with no support (no such label in the actual test data).

### Versions

```shell
System:
    python: 3.9.7 (default, Sep 16 2021, 13:09:58)  [GCC 7.5.0]
executable: /home/zing/anaconda3/bin/python
   machine: Linux-5.13.0-30-generic-x86_64-with-glibc2.31

Python dependencies:
          pip: 21.2.4
   setuptools: 58.0.4
      sklearn: 0.24.2
        numpy: 1.20.3
        scipy: 1.7.1
       Cython: 0.29.24
       pandas: 1.3.4
   matplotlib: 3.4.3
       joblib: 1.1.0
threadpoolctl: 2.2.0

Built with OpenMP: True
```
```

COMMENT:
> It should count np.unique(actual), not no.unique(prediction)

I think most of the scikit-learn metrics uses the union of the classes in `y_true` and `y_pred` as labels. If you want to only use the labels in `y_true`, you can specific them with the `labels` parameter:

```python
import numpy as np
from sklearn.metrics import precision_score

y_true = np.asarray([0, 0, 1, 1])
y_pred = np.asarray([0, 2, 1, 1])

precision_score(y_true, y_pred, average="macro", labels=np.unique(y_true))
# 1.0
```

This works with `classification_report` too:

```python
print(classification_report(y_true, y_pred, labels=np.unique(y_true)))
```

which results in:

```
              precision    recall  f1-score   support

           0       1.00      0.50      0.67         2
           1       1.00      1.00      1.00         2

   micro avg       1.00      0.75      0.86         4
   macro avg       1.00      0.75      0.83         4
weighted avg       1.00      0.75      0.83         4
```

**Edit**: Given that `labels` can be used to get the desired behavior, I prefer not to change the behavior of `precision_score` to ignore the classes because of backward compatibility.

Labels: Bug, module:metrics
Macro Average F1 Score Computation Bug

### Describe the bug

The current algorithm computes labels with no support:

### Steps/Code to Reproduce

```
           precision    recall  f1-score   support

     archive       0.00      0.00      0.00         0
        high       0.35      0.34      0.34      1400
         low       0.87      0.89      0.88     29585
      medium       0.82      0.80      0.81     20102
      module       0.00      0.00      0.00         0
     project       0.00      0.00      0.00         0

    accuracy                           0.84     51087
   macro avg       0.34      0.34      0.34     51087
weighted avg       0.84      0.84      0.84     51087
```

### Expected Results

It should ignore labels with 0 supports(these labels are from the model predictions, which were out-of-distribution) and output (0.35+0.87+0.82)/3=0.68 for the Precision Macro Average. 

Consider this algorithm: https://stackoverflow.com/a/64863315/4843407

### Actual Results

It includes the labels with no support (no such label in the actual test data).

### Versions

```shell
System:
    python: 3.9.7 (default, Sep 16 2021, 13:09:58)  [GCC 7.5.0]
executable: /home/zing/anaconda3/bin/python
   machine: Linux-5.13.0-30-generic-x86_64-with-glibc2.31

Python dependencies:
          pip: 21.2.4
   setuptools: 58.0.4
      sklearn: 0.24.2
        numpy: 1.20.3
        scipy: 1.7.1
       Cython: 0.29.24
       pandas: 1.3.4
   matplotlib: 3.4.3
       joblib: 1.1.0
threadpoolctl: 2.2.0

Built with OpenMP: True
```
```

COMMENT:
I see. Thanks for the info. I will specify label parameter then.

Labels: Bug, module:metrics
GridSearchCV scoring behavior on weighted metrics, such as Balanced Accuracy

### Describe the workflow you want to enable

As a user who specify weighted metrics/scoring such as balanced accuracy, I would like the scoring `GridSearchCV` uses for optimising and `best_score_` to be weighted as well. Based on current documentation, I think it is currently simply a mean, so I do not aware if this has been tackled/asked before.

I am thinking about this since my data is imbalanced, and saw [this](https://stackoverflow.com/questions/47572191/scikit-learn-scoring-in-gridsearchcv) post in SO too.

The mean seems to work fine when `StratifiedKFold` with minority class is multiple of `K`. If not (where the number minority class in each fold can be different) the scoring could be different with the 'combined scoring'. However, this is not as severe as doing this in `LeaveOneOut`.

### Describe your proposed solution

When the user specify built-in weighted scoring, record the validation results for all folds and calculate scoring from combined validation result.

This can be default behavior or an option to turn this on/off. However, I couldn't see why people who specify weighted metrics want to turn this off.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Hi @thomasjpfan , apology for bringing up an old issue. I'm just wondering if this has been discussed/tackled? Thanks!

Labels: New Feature, module:model_selection, Needs Decision - Include Feature
GridSearchCV scoring behavior on weighted metrics, such as Balanced Accuracy

### Describe the workflow you want to enable

As a user who specify weighted metrics/scoring such as balanced accuracy, I would like the scoring `GridSearchCV` uses for optimising and `best_score_` to be weighted as well. Based on current documentation, I think it is currently simply a mean, so I do not aware if this has been tackled/asked before.

I am thinking about this since my data is imbalanced, and saw [this](https://stackoverflow.com/questions/47572191/scikit-learn-scoring-in-gridsearchcv) post in SO too.

The mean seems to work fine when `StratifiedKFold` with minority class is multiple of `K`. If not (where the number minority class in each fold can be different) the scoring could be different with the 'combined scoring'. However, this is not as severe as doing this in `LeaveOneOut`.

### Describe your proposed solution

When the user specify built-in weighted scoring, record the validation results for all folds and calculate scoring from combined validation result.

This can be default behavior or an option to turn this on/off. However, I couldn't see why people who specify weighted metrics want to turn this off.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
This will be fixed with metadata routing, which we're working on.

Labels: New Feature, module:model_selection, Needs Decision - Include Feature