Talip7/scikit-learn-issues-multilabel huggingface dataset:
Enhance calibration plots

### Describe the workflow you want to enable

I would like to enhance calibration plots by adding these new aspects:
1. Plot bins as vertical lines in the background.
2. Equal aspect on both axis to have a square figure and enable easier visual comparisons.
3. Changing xlabel from "Mean predicted probability" to "Mean confidence scores" or "Confidence scores" or similar, to avoid confusion between scores output by classifiers and true posterior probabilities.
4. Integrate a visualization of the density to see where the mass is on the calibration curve. This is important as a calibration curve can appear very bad in some part whereas it can be due to a low number of samples in this part, and actually have a good Expected Calibration Error. Several choices are possible as described in the proposed solution section.
5. Edit: add support for multiclass calibration plot by either looking at the class-wise calibration (considering one of the classes as the positive one) or the top-label problem (binarizing the multiclass problem with the prediction taken as the argmax of the confidences, a positive label iff the prediction is right and an associated confidence given by the maximum of the confidences). These are 2 of the 3 definitions of calibration in the multiclass setting.

### Describe your proposed solution

To address the previous points, I propose to:
1. Update `CalibrationDisplay`.
2. Idem
3. Idem
4. Several choices are possible here. Either adding an histogram on each edge as in the figure below. Or adding a single histogram on the confidence scores axis (either on the edge or integrated in the background of the calibration plot) since the density on the "fraction of positive" axis is less informative as the one on the confidence scores axis. This last option might be better. Also, this feature might be restricted to a single calibration curve only as it might become illegible with several classifiers.
5. Edit: This could be done by adapting the `_get_response` to allow y_pred.shape[1] > 2 and either choosing one of the K class as the positive one, or the top-label one.

Should this change be approved, I would be willing to work on it.

Example with histograms on the edges:
<img src="https://user-images.githubusercontent.com/33580936/174766358-56fc2afa-5452-4318-b0cb-18caf1b2a004.jpg" alt="drawing" width="400"/>

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Thanks @aperezlebel, this looks great.

To handle the case where we want to plot several estimators on the same `ax` we could instead introduce a new `from_estimators` method.

/cc @glemaitre and @lucyleeow who might also be interested in this discussion.

Labels: New Feature, module:calibration
Enhance calibration plots

### Describe the workflow you want to enable

I would like to enhance calibration plots by adding these new aspects:
1. Plot bins as vertical lines in the background.
2. Equal aspect on both axis to have a square figure and enable easier visual comparisons.
3. Changing xlabel from "Mean predicted probability" to "Mean confidence scores" or "Confidence scores" or similar, to avoid confusion between scores output by classifiers and true posterior probabilities.
4. Integrate a visualization of the density to see where the mass is on the calibration curve. This is important as a calibration curve can appear very bad in some part whereas it can be due to a low number of samples in this part, and actually have a good Expected Calibration Error. Several choices are possible as described in the proposed solution section.
5. Edit: add support for multiclass calibration plot by either looking at the class-wise calibration (considering one of the classes as the positive one) or the top-label problem (binarizing the multiclass problem with the prediction taken as the argmax of the confidences, a positive label iff the prediction is right and an associated confidence given by the maximum of the confidences). These are 2 of the 3 definitions of calibration in the multiclass setting.

### Describe your proposed solution

To address the previous points, I propose to:
1. Update `CalibrationDisplay`.
2. Idem
3. Idem
4. Several choices are possible here. Either adding an histogram on each edge as in the figure below. Or adding a single histogram on the confidence scores axis (either on the edge or integrated in the background of the calibration plot) since the density on the "fraction of positive" axis is less informative as the one on the confidence scores axis. This last option might be better. Also, this feature might be restricted to a single calibration curve only as it might become illegible with several classifiers.
5. Edit: This could be done by adapting the `_get_response` to allow y_pred.shape[1] > 2 and either choosing one of the K class as the positive one, or the top-label one.

Should this change be approved, I would be willing to work on it.

Example with histograms on the edges:
<img src="https://user-images.githubusercontent.com/33580936/174766358-56fc2afa-5452-4318-b0cb-18caf1b2a004.jpg" alt="drawing" width="400"/>

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
For quantile binning we could keep an histogram but using a uniform binning just for the histogram (it would not align with the quantile bin thresholds).

Labels: New Feature, module:calibration
Enhance calibration plots

### Describe the workflow you want to enable

I would like to enhance calibration plots by adding these new aspects:
1. Plot bins as vertical lines in the background.
2. Equal aspect on both axis to have a square figure and enable easier visual comparisons.
3. Changing xlabel from "Mean predicted probability" to "Mean confidence scores" or "Confidence scores" or similar, to avoid confusion between scores output by classifiers and true posterior probabilities.
4. Integrate a visualization of the density to see where the mass is on the calibration curve. This is important as a calibration curve can appear very bad in some part whereas it can be due to a low number of samples in this part, and actually have a good Expected Calibration Error. Several choices are possible as described in the proposed solution section.
5. Edit: add support for multiclass calibration plot by either looking at the class-wise calibration (considering one of the classes as the positive one) or the top-label problem (binarizing the multiclass problem with the prediction taken as the argmax of the confidences, a positive label iff the prediction is right and an associated confidence given by the maximum of the confidences). These are 2 of the 3 definitions of calibration in the multiclass setting.

### Describe your proposed solution

To address the previous points, I propose to:
1. Update `CalibrationDisplay`.
2. Idem
3. Idem
4. Several choices are possible here. Either adding an histogram on each edge as in the figure below. Or adding a single histogram on the confidence scores axis (either on the edge or integrated in the background of the calibration plot) since the density on the "fraction of positive" axis is less informative as the one on the confidence scores axis. This last option might be better. Also, this feature might be restricted to a single calibration curve only as it might become illegible with several classifiers.
5. Edit: This could be done by adapting the `_get_response` to allow y_pred.shape[1] > 2 and either choosing one of the K class as the positive one, or the top-label one.

Should this change be approved, I would be willing to work on it.

Example with histograms on the edges:
<img src="https://user-images.githubusercontent.com/33580936/174766358-56fc2afa-5452-4318-b0cb-18caf1b2a004.jpg" alt="drawing" width="400"/>

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Good ideas, thanks! Working on it.

Labels: New Feature, module:calibration
Enhance calibration plots

### Describe the workflow you want to enable

I would like to enhance calibration plots by adding these new aspects:
1. Plot bins as vertical lines in the background.
2. Equal aspect on both axis to have a square figure and enable easier visual comparisons.
3. Changing xlabel from "Mean predicted probability" to "Mean confidence scores" or "Confidence scores" or similar, to avoid confusion between scores output by classifiers and true posterior probabilities.
4. Integrate a visualization of the density to see where the mass is on the calibration curve. This is important as a calibration curve can appear very bad in some part whereas it can be due to a low number of samples in this part, and actually have a good Expected Calibration Error. Several choices are possible as described in the proposed solution section.
5. Edit: add support for multiclass calibration plot by either looking at the class-wise calibration (considering one of the classes as the positive one) or the top-label problem (binarizing the multiclass problem with the prediction taken as the argmax of the confidences, a positive label iff the prediction is right and an associated confidence given by the maximum of the confidences). These are 2 of the 3 definitions of calibration in the multiclass setting.

### Describe your proposed solution

To address the previous points, I propose to:
1. Update `CalibrationDisplay`.
2. Idem
3. Idem
4. Several choices are possible here. Either adding an histogram on each edge as in the figure below. Or adding a single histogram on the confidence scores axis (either on the edge or integrated in the background of the calibration plot) since the density on the "fraction of positive" axis is less informative as the one on the confidence scores axis. This last option might be better. Also, this feature might be restricted to a single calibration curve only as it might become illegible with several classifiers.
5. Edit: This could be done by adapting the `_get_response` to allow y_pred.shape[1] > 2 and either choosing one of the K class as the positive one, or the top-label one.

Should this change be approved, I would be willing to work on it.

Example with histograms on the edges:
<img src="https://user-images.githubusercontent.com/33580936/174766358-56fc2afa-5452-4318-b0cb-18caf1b2a004.jpg" alt="drawing" width="400"/>

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I implemented a `from_estimators` method and this is how it looks like:
<img src="https://user-images.githubusercontent.com/33580936/175604227-6e83c01a-9635-46bf-8e4b-45f284055ff5.png" alt="drawing" width="400"/>

With quantile binning (bins are disabled and grid is plotted instead):
<img src="https://user-images.githubusercontent.com/33580936/175604454-621a8cc4-ac61-4287-876d-d266b310a97a.png" alt="drawing" width="400"/>

Labels: New Feature, module:calibration
Enhance calibration plots

### Describe the workflow you want to enable

I would like to enhance calibration plots by adding these new aspects:
1. Plot bins as vertical lines in the background.
2. Equal aspect on both axis to have a square figure and enable easier visual comparisons.
3. Changing xlabel from "Mean predicted probability" to "Mean confidence scores" or "Confidence scores" or similar, to avoid confusion between scores output by classifiers and true posterior probabilities.
4. Integrate a visualization of the density to see where the mass is on the calibration curve. This is important as a calibration curve can appear very bad in some part whereas it can be due to a low number of samples in this part, and actually have a good Expected Calibration Error. Several choices are possible as described in the proposed solution section.
5. Edit: add support for multiclass calibration plot by either looking at the class-wise calibration (considering one of the classes as the positive one) or the top-label problem (binarizing the multiclass problem with the prediction taken as the argmax of the confidences, a positive label iff the prediction is right and an associated confidence given by the maximum of the confidences). These are 2 of the 3 definitions of calibration in the multiclass setting.

### Describe your proposed solution

To address the previous points, I propose to:
1. Update `CalibrationDisplay`.
2. Idem
3. Idem
4. Several choices are possible here. Either adding an histogram on each edge as in the figure below. Or adding a single histogram on the confidence scores axis (either on the edge or integrated in the background of the calibration plot) since the density on the "fraction of positive" axis is less informative as the one on the confidence scores axis. This last option might be better. Also, this feature might be restricted to a single calibration curve only as it might become illegible with several classifiers.
5. Edit: This could be done by adapting the `_get_response` to allow y_pred.shape[1] > 2 and either choosing one of the K class as the positive one, or the top-label one.

Should this change be approved, I would be willing to work on it.

Example with histograms on the edges:
<img src="https://user-images.githubusercontent.com/33580936/174766358-56fc2afa-5452-4318-b0cb-18caf1b2a004.jpg" alt="drawing" width="400"/>

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
For the multiple estimator case, I find the histograms plotted on top of each other, harder to interpret.

In the compare calibration example on [stable](https://scikit-learn.org/stable/auto_examples/calibration/plot_compare_calibration.html#calibration-curves), the histograms were plotted separately next to each other. I think this was an improvement over the same example in [0.24](https://scikit-learn.org/0.24/auto_examples/calibration/plot_compare_calibration.html#comparison-of-calibration-of-classifiers) where the histogram was plotted on top of each other.

Labels: New Feature, module:calibration
Enhance calibration plots

### Describe the workflow you want to enable

I would like to enhance calibration plots by adding these new aspects:
1. Plot bins as vertical lines in the background.
2. Equal aspect on both axis to have a square figure and enable easier visual comparisons.
3. Changing xlabel from "Mean predicted probability" to "Mean confidence scores" or "Confidence scores" or similar, to avoid confusion between scores output by classifiers and true posterior probabilities.
4. Integrate a visualization of the density to see where the mass is on the calibration curve. This is important as a calibration curve can appear very bad in some part whereas it can be due to a low number of samples in this part, and actually have a good Expected Calibration Error. Several choices are possible as described in the proposed solution section.
5. Edit: add support for multiclass calibration plot by either looking at the class-wise calibration (considering one of the classes as the positive one) or the top-label problem (binarizing the multiclass problem with the prediction taken as the argmax of the confidences, a positive label iff the prediction is right and an associated confidence given by the maximum of the confidences). These are 2 of the 3 definitions of calibration in the multiclass setting.

### Describe your proposed solution

To address the previous points, I propose to:
1. Update `CalibrationDisplay`.
2. Idem
3. Idem
4. Several choices are possible here. Either adding an histogram on each edge as in the figure below. Or adding a single histogram on the confidence scores axis (either on the edge or integrated in the background of the calibration plot) since the density on the "fraction of positive" axis is less informative as the one on the confidence scores axis. This last option might be better. Also, this feature might be restricted to a single calibration curve only as it might become illegible with several classifiers.
5. Edit: This could be done by adapting the `_get_response` to allow y_pred.shape[1] > 2 and either choosing one of the K class as the positive one, or the top-label one.

Should this change be approved, I would be willing to work on it.

Example with histograms on the edges:
<img src="https://user-images.githubusercontent.com/33580936/174766358-56fc2afa-5452-4318-b0cb-18caf1b2a004.jpg" alt="drawing" width="400"/>

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
For the figure containing 4 axes, I am wondering if adding the ratio number of samples in the bin (or absolute) would be helpful as well since we don't have the y-axis for the histogram.
I know that recent matplotlib should have a `plt.bar_label` that is handy for this.

Labels: New Feature, module:calibration
Enhance calibration plots

### Describe the workflow you want to enable

I would like to enhance calibration plots by adding these new aspects:
1. Plot bins as vertical lines in the background.
2. Equal aspect on both axis to have a square figure and enable easier visual comparisons.
3. Changing xlabel from "Mean predicted probability" to "Mean confidence scores" or "Confidence scores" or similar, to avoid confusion between scores output by classifiers and true posterior probabilities.
4. Integrate a visualization of the density to see where the mass is on the calibration curve. This is important as a calibration curve can appear very bad in some part whereas it can be due to a low number of samples in this part, and actually have a good Expected Calibration Error. Several choices are possible as described in the proposed solution section.
5. Edit: add support for multiclass calibration plot by either looking at the class-wise calibration (considering one of the classes as the positive one) or the top-label problem (binarizing the multiclass problem with the prediction taken as the argmax of the confidences, a positive label iff the prediction is right and an associated confidence given by the maximum of the confidences). These are 2 of the 3 definitions of calibration in the multiclass setting.

### Describe your proposed solution

To address the previous points, I propose to:
1. Update `CalibrationDisplay`.
2. Idem
3. Idem
4. Several choices are possible here. Either adding an histogram on each edge as in the figure below. Or adding a single histogram on the confidence scores axis (either on the edge or integrated in the background of the calibration plot) since the density on the "fraction of positive" axis is less informative as the one on the confidence scores axis. This last option might be better. Also, this feature might be restricted to a single calibration curve only as it might become illegible with several classifiers.
5. Edit: This could be done by adapting the `_get_response` to allow y_pred.shape[1] > 2 and either choosing one of the K class as the positive one, or the top-label one.

Should this change be approved, I would be willing to work on it.

Example with histograms on the edges:
<img src="https://user-images.githubusercontent.com/33580936/174766358-56fc2afa-5452-4318-b0cb-18caf1b2a004.jpg" alt="drawing" width="400"/>

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I like this suggestion.

Labels: New Feature, module:calibration
Enhance calibration plots

### Describe the workflow you want to enable

I would like to enhance calibration plots by adding these new aspects:
1. Plot bins as vertical lines in the background.
2. Equal aspect on both axis to have a square figure and enable easier visual comparisons.
3. Changing xlabel from "Mean predicted probability" to "Mean confidence scores" or "Confidence scores" or similar, to avoid confusion between scores output by classifiers and true posterior probabilities.
4. Integrate a visualization of the density to see where the mass is on the calibration curve. This is important as a calibration curve can appear very bad in some part whereas it can be due to a low number of samples in this part, and actually have a good Expected Calibration Error. Several choices are possible as described in the proposed solution section.
5. Edit: add support for multiclass calibration plot by either looking at the class-wise calibration (considering one of the classes as the positive one) or the top-label problem (binarizing the multiclass problem with the prediction taken as the argmax of the confidences, a positive label iff the prediction is right and an associated confidence given by the maximum of the confidences). These are 2 of the 3 definitions of calibration in the multiclass setting.

### Describe your proposed solution

To address the previous points, I propose to:
1. Update `CalibrationDisplay`.
2. Idem
3. Idem
4. Several choices are possible here. Either adding an histogram on each edge as in the figure below. Or adding a single histogram on the confidence scores axis (either on the edge or integrated in the background of the calibration plot) since the density on the "fraction of positive" axis is less informative as the one on the confidence scores axis. This last option might be better. Also, this feature might be restricted to a single calibration curve only as it might become illegible with several classifiers.
5. Edit: This could be done by adapting the `_get_response` to allow y_pred.shape[1] > 2 and either choosing one of the K class as the positive one, or the top-label one.

Should this change be approved, I would be willing to work on it.

Example with histograms on the edges:
<img src="https://user-images.githubusercontent.com/33580936/174766358-56fc2afa-5452-4318-b0cb-18caf1b2a004.jpg" alt="drawing" width="400"/>

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
> I am wondering if adding the ratio number of samples in the bin (or absolute) would be helpful

I implemented the feature with bin ratios. It can be enabled with `bin_label=True`:

![test_calibration_display_example](https://user-images.githubusercontent.com/33580936/176736333-d2e8879e-a4c2-45ad-a406-4c87b1201b74.png)

Labels: New Feature, module:calibration