Talip7/scikit-learn-issues-multilabel huggingface dataset:
Add sankey style confusion matrix visualization

### Describe the workflow you want to enable

Confusion matrices can be displayed as a colored matrix using the [ConfusionMatrixDisplay](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.ConfusionMatrixDisplay.html#sklearn.metrics.ConfusionMatrixDisplay) class.

![confusion_matrix](https://scikit-learn.org/stable/_images/sphx_glr_plot_label_propagation_digits_001.png)

However color scaling is hard to interpret, and spatial cues are easier to help interpret quantities than color variations. 
The number represented in each box represent absolute ones, while one may be interested in either row based, column based, or complete matrix sum based normalization. 
Plus it gets even harder to read for multiclass classification.  

### Describe your proposed solution

I would propose introducing sankey like plots to visualize the confusion matrix data, using a `ConfusionMatrixSankeyDisplay` class.
The qualitative information is displayed by different colors (easy to interpret), while actual amounts in each cell are represented by the size of the flows (easier to interpret quantitatively than colorscale variations).
On the left size one can see the number of occurrence of each label in the ground truth data (confusion matrix row marginals). On the right side the number of occurrence of each label in the predictions (confusion matrix column marginals). Each flow represents both row-normalized (left side) and column-normalized (right side) at the same time, and could be labeled with the actual absolute number of examples in each confusion matrix cell.
Interpretation is straightforward even in the multiclass case.

There exist a matplotlib based implementation doing almost exactly this in the small [pySankeyBeta](https://github.com/Pierre-Sassoulas/pySankey) package. Here is a sample from its readme:
![sankey](https://github.com/Pierre-Sassoulas/pySankey/raw/main/.github/img/fruits.png)

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
> While I really like good visualisations, I am -1 on this one. With the plot library of your choice, it should be straightforward to produce a Sankey diagram with the result of `confusion_matrix(..)`.

While I agree in theory, the reality seems a bit different:

1. Though it should, it is not that straightforward to produce a sankey plot for a confusion matrix based on matplotlib (on which scikit has a hard dependency)
2. Scikit already proposes a visualization for confusion matrices, but a hard to read one. Still, this visualization is used a lot, simply due to Scikit's wide audience. In that sense pushing sankey plots into scikit would help promote better visualization and transparency in ML in general.

Labels: New Feature, Needs Decision - Include Feature
Add sankey style confusion matrix visualization

### Describe the workflow you want to enable

Confusion matrices can be displayed as a colored matrix using the [ConfusionMatrixDisplay](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.ConfusionMatrixDisplay.html#sklearn.metrics.ConfusionMatrixDisplay) class.

![confusion_matrix](https://scikit-learn.org/stable/_images/sphx_glr_plot_label_propagation_digits_001.png)

However color scaling is hard to interpret, and spatial cues are easier to help interpret quantities than color variations. 
The number represented in each box represent absolute ones, while one may be interested in either row based, column based, or complete matrix sum based normalization. 
Plus it gets even harder to read for multiclass classification.  

### Describe your proposed solution

I would propose introducing sankey like plots to visualize the confusion matrix data, using a `ConfusionMatrixSankeyDisplay` class.
The qualitative information is displayed by different colors (easy to interpret), while actual amounts in each cell are represented by the size of the flows (easier to interpret quantitatively than colorscale variations).
On the left size one can see the number of occurrence of each label in the ground truth data (confusion matrix row marginals). On the right side the number of occurrence of each label in the predictions (confusion matrix column marginals). Each flow represents both row-normalized (left side) and column-normalized (right side) at the same time, and could be labeled with the actual absolute number of examples in each confusion matrix cell.
Interpretation is straightforward even in the multiclass case.

There exist a matplotlib based implementation doing almost exactly this in the small [pySankeyBeta](https://github.com/Pierre-Sassoulas/pySankey) package. Here is a sample from its readme:
![sankey](https://github.com/Pierre-Sassoulas/pySankey/raw/main/.github/img/fruits.png)

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
> Though it should, it is not that straightforward to produce a sankey plot for a confusion matrix based on matplotlib (on which scikit has a hard dependency)

Scikit-learn is not a plotting library/backend. If you miss a feature in matplotlib, better contribute there. Also, scikit-learn does not depend on matplotlib. The only hard dependencies are: https://github.com/scikit-learn/scikit-learn/blob/7d0cbaf20f2a517139cad35540da72858fcb14ab/pyproject.toml#L10-L15

> ..this visualization is used a lot, simply due to Scikit's wide audience. In that sense pushing sankey plots into scikit would help promote better visualization..

You have a good point. If only there were a good sankey implementation in matplotlib..(not meant to be sarcastic)

Labels: New Feature, Needs Decision - Include Feature
Add sankey style confusion matrix visualization

### Describe the workflow you want to enable

Confusion matrices can be displayed as a colored matrix using the [ConfusionMatrixDisplay](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.ConfusionMatrixDisplay.html#sklearn.metrics.ConfusionMatrixDisplay) class.

![confusion_matrix](https://scikit-learn.org/stable/_images/sphx_glr_plot_label_propagation_digits_001.png)

However color scaling is hard to interpret, and spatial cues are easier to help interpret quantities than color variations. 
The number represented in each box represent absolute ones, while one may be interested in either row based, column based, or complete matrix sum based normalization. 
Plus it gets even harder to read for multiclass classification.  

### Describe your proposed solution

I would propose introducing sankey like plots to visualize the confusion matrix data, using a `ConfusionMatrixSankeyDisplay` class.
The qualitative information is displayed by different colors (easy to interpret), while actual amounts in each cell are represented by the size of the flows (easier to interpret quantitatively than colorscale variations).
On the left size one can see the number of occurrence of each label in the ground truth data (confusion matrix row marginals). On the right side the number of occurrence of each label in the predictions (confusion matrix column marginals). Each flow represents both row-normalized (left side) and column-normalized (right side) at the same time, and could be labeled with the actual absolute number of examples in each confusion matrix cell.
Interpretation is straightforward even in the multiclass case.

There exist a matplotlib based implementation doing almost exactly this in the small [pySankeyBeta](https://github.com/Pierre-Sassoulas/pySankey) package. Here is a sample from its readme:
![sankey](https://github.com/Pierre-Sassoulas/pySankey/raw/main/.github/img/fruits.png)

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Sorry for this erroneous claim on dependency I should have checked the pyproject.toml, thanks for pointing it out!
The only difference between matplotlib and plotly dependency-wise is that matplolib is already included in the `tests` dependency group while plotly is not.

Labels: New Feature, Needs Decision - Include Feature
Add sankey style confusion matrix visualization

### Describe the workflow you want to enable

Confusion matrices can be displayed as a colored matrix using the [ConfusionMatrixDisplay](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.ConfusionMatrixDisplay.html#sklearn.metrics.ConfusionMatrixDisplay) class.

![confusion_matrix](https://scikit-learn.org/stable/_images/sphx_glr_plot_label_propagation_digits_001.png)

However color scaling is hard to interpret, and spatial cues are easier to help interpret quantities than color variations. 
The number represented in each box represent absolute ones, while one may be interested in either row based, column based, or complete matrix sum based normalization. 
Plus it gets even harder to read for multiclass classification.  

### Describe your proposed solution

I would propose introducing sankey like plots to visualize the confusion matrix data, using a `ConfusionMatrixSankeyDisplay` class.
The qualitative information is displayed by different colors (easy to interpret), while actual amounts in each cell are represented by the size of the flows (easier to interpret quantitatively than colorscale variations).
On the left size one can see the number of occurrence of each label in the ground truth data (confusion matrix row marginals). On the right side the number of occurrence of each label in the predictions (confusion matrix column marginals). Each flow represents both row-normalized (left side) and column-normalized (right side) at the same time, and could be labeled with the actual absolute number of examples in each confusion matrix cell.
Interpretation is straightforward even in the multiclass case.

There exist a matplotlib based implementation doing almost exactly this in the small [pySankeyBeta](https://github.com/Pierre-Sassoulas/pySankey) package. Here is a sample from its readme:
![sankey](https://github.com/Pierre-Sassoulas/pySankey/raw/main/.github/img/fruits.png)

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
https://github.com/matplotlib/matplotlib/issues/28227 may be linked.

Labels: New Feature, Needs Decision - Include Feature
Add sankey style confusion matrix visualization

### Describe the workflow you want to enable

Confusion matrices can be displayed as a colored matrix using the [ConfusionMatrixDisplay](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.ConfusionMatrixDisplay.html#sklearn.metrics.ConfusionMatrixDisplay) class.

![confusion_matrix](https://scikit-learn.org/stable/_images/sphx_glr_plot_label_propagation_digits_001.png)

However color scaling is hard to interpret, and spatial cues are easier to help interpret quantities than color variations. 
The number represented in each box represent absolute ones, while one may be interested in either row based, column based, or complete matrix sum based normalization. 
Plus it gets even harder to read for multiclass classification.  

### Describe your proposed solution

I would propose introducing sankey like plots to visualize the confusion matrix data, using a `ConfusionMatrixSankeyDisplay` class.
The qualitative information is displayed by different colors (easy to interpret), while actual amounts in each cell are represented by the size of the flows (easier to interpret quantitatively than colorscale variations).
On the left size one can see the number of occurrence of each label in the ground truth data (confusion matrix row marginals). On the right side the number of occurrence of each label in the predictions (confusion matrix column marginals). Each flow represents both row-normalized (left side) and column-normalized (right side) at the same time, and could be labeled with the actual absolute number of examples in each confusion matrix cell.
Interpretation is straightforward even in the multiclass case.

There exist a matplotlib based implementation doing almost exactly this in the small [pySankeyBeta](https://github.com/Pierre-Sassoulas/pySankey) package. Here is a sample from its readme:
![sankey](https://github.com/Pierre-Sassoulas/pySankey/raw/main/.github/img/fruits.png)

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
> it should be straightforward to produce a Sankey diagram with the result of confusion_matrix(..)

Easier than is the case with our other display objects? E.g., `RocCurveDisplay` using `roc_curve` ? Admittedly the addition of axis labels and chance line are probably the 'difficult' parts of producing that curve.

Labels: New Feature, Needs Decision - Include Feature
Add sankey style confusion matrix visualization

### Describe the workflow you want to enable

Confusion matrices can be displayed as a colored matrix using the [ConfusionMatrixDisplay](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.ConfusionMatrixDisplay.html#sklearn.metrics.ConfusionMatrixDisplay) class.

![confusion_matrix](https://scikit-learn.org/stable/_images/sphx_glr_plot_label_propagation_digits_001.png)

However color scaling is hard to interpret, and spatial cues are easier to help interpret quantities than color variations. 
The number represented in each box represent absolute ones, while one may be interested in either row based, column based, or complete matrix sum based normalization. 
Plus it gets even harder to read for multiclass classification.  

### Describe your proposed solution

I would propose introducing sankey like plots to visualize the confusion matrix data, using a `ConfusionMatrixSankeyDisplay` class.
The qualitative information is displayed by different colors (easy to interpret), while actual amounts in each cell are represented by the size of the flows (easier to interpret quantitatively than colorscale variations).
On the left size one can see the number of occurrence of each label in the ground truth data (confusion matrix row marginals). On the right side the number of occurrence of each label in the predictions (confusion matrix column marginals). Each flow represents both row-normalized (left side) and column-normalized (right side) at the same time, and could be labeled with the actual absolute number of examples in each confusion matrix cell.
Interpretation is straightforward even in the multiclass case.

There exist a matplotlib based implementation doing almost exactly this in the small [pySankeyBeta](https://github.com/Pierre-Sassoulas/pySankey) package. Here is a sample from its readme:
![sankey](https://github.com/Pierre-Sassoulas/pySankey/raw/main/.github/img/fruits.png)

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
As @betatim said
> I think it would be nice to have this as part of the visualisations in scikit-learn.

I agree with that. I am just hesitant to have a full sankey-diagram implementation in scikit-learn. As I said, the plotting library should provide that, not scikit-learn.
If someone wants to give it a shot, maybe using matplotlibs sankey functionality, go for it.

Labels: New Feature, Needs Decision - Include Feature
Duplicate/incomplete dependency information

### Describe the issue linked to the documentation

Scikit-learn dependecies are described in two places:
* [**Installing scikit-learn**](https://scikit-learn.org/stable/install.html)
  https://scikit-learn.org/stable/install.html#installing-the-latest-release
* [**Installing the development version of scikit-learn**](https://scikit-learn.org/stable/developers/advanced_installation.html)
  https://scikit-learn.org/stable/developers/advanced_installation.html#dependencies


### Suggest a potential alternative/fix

I suggest removing the incomplete [Dependencies](https://scikit-learn.org/stable/developers/advanced_installation.html#dependencies) section from _[Installing the development version of scikit-learn](https://scikit-learn.org/stable/developers/advanced_installation.html)_ and referring to the extensive dependency list under _[Installing scikit-learn](https://scikit-learn.org/stable/install.html)_.

COMMENT:
To me both serve different purposes. The second ones are the minimum dependencies to be able to build scikit-learn from source. I think it's worth keeping it simple like that rather than adding a redirection which then requires to find the useful information.

Labels: Documentation
Duplicate/incomplete dependency information

### Describe the issue linked to the documentation

Scikit-learn dependecies are described in two places:
* [**Installing scikit-learn**](https://scikit-learn.org/stable/install.html)
  https://scikit-learn.org/stable/install.html#installing-the-latest-release
* [**Installing the development version of scikit-learn**](https://scikit-learn.org/stable/developers/advanced_installation.html)
  https://scikit-learn.org/stable/developers/advanced_installation.html#dependencies


### Suggest a potential alternative/fix

I suggest removing the incomplete [Dependencies](https://scikit-learn.org/stable/developers/advanced_installation.html#dependencies) section from _[Installing the development version of scikit-learn](https://scikit-learn.org/stable/developers/advanced_installation.html)_ and referring to the extensive dependency list under _[Installing scikit-learn](https://scikit-learn.org/stable/install.html)_.

COMMENT:
The first one also lists the minimum dependencies to be able to build scikit-learn from source (with the additional distinction between **build** and **install**):

| Dependency    | Minimum Version | Purpose        |
|---------------|-----------------|----------------|
| numpy         | 1.19.5          | build, install |
| scipy         | 1.6.0           | build, install |
| joblib        | 1.2.0           | install        |
| threadpoolctl | 3.1.0           | install        |
| cython        | 3.0.10          | build          |
| meson-python  | 0.16.0          | build          |

The second one is incomplete and inconsistent:
* the [Dependencies](https://scikit-learn.org/stable/developers/advanced_installation.html#dependencies) section lacks meson-python:
  > ### Runtime dependencies
  > Scikit-learn requires the following dependencies both at build time and at runtime:
  > * Python (>= 3.8),
  > * NumPy (>= 1.19.5),
  > * SciPy (>= 1.6.0),
  > * Joblib (>= 1.2.0),
  > * threadpoolctl (>= 3.1.0).
  > ### Build dependencies
  > Building Scikit-learn also requires:
  > * Cython >= 3.0.10
  > * A C/C++ compiler and a matching [OpenMP](https://en.wikipedia.org/wiki/OpenMP) runtime library. See the [platform system specific instructions](https://scikit-learn.org/stable/developers/advanced_installation.html#platform-specific-instructions) for more details. 
* the [Building from source](https://scikit-learn.org/stable/developers/advanced_installation.html#building-from-source) section refers to Ninja which is absent from other sources:
  > If you installed Python with conda, we recommend to create a dedicated [conda environment](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html) with all the build dependencies of scikit-learn (namely [NumPy](https://numpy.org/), [SciPy](https://www.scipy.org/), [Cython](https://cython.org/), [meson-python](https://mesonbuild.com/meson-python) and [Ninja](https://ninja-build.org/)):

I feel it's better to a have a single correct source than multiple incomplete and inconsistent sources.

Is Ninja part of the minimum dependencies?

Labels: Documentation
Duplicate/incomplete dependency information

### Describe the issue linked to the documentation

Scikit-learn dependecies are described in two places:
* [**Installing scikit-learn**](https://scikit-learn.org/stable/install.html)
  https://scikit-learn.org/stable/install.html#installing-the-latest-release
* [**Installing the development version of scikit-learn**](https://scikit-learn.org/stable/developers/advanced_installation.html)
  https://scikit-learn.org/stable/developers/advanced_installation.html#dependencies


### Suggest a potential alternative/fix

I suggest removing the incomplete [Dependencies](https://scikit-learn.org/stable/developers/advanced_installation.html#dependencies) section from _[Installing the development version of scikit-learn](https://scikit-learn.org/stable/developers/advanced_installation.html)_ and referring to the extensive dependency list under _[Installing scikit-learn](https://scikit-learn.org/stable/install.html)_.

COMMENT:
It's true that it is incomplete. But I'd really like to avoid a redirection. Having all the info in one page is a real quality of life benefit. Maybe there's a way to have a single source but we directly include the list of deps and avoid the redirection. And ideally only the build/install deps

Labels: Documentation
Duplicate/incomplete dependency information

### Describe the issue linked to the documentation

Scikit-learn dependecies are described in two places:
* [**Installing scikit-learn**](https://scikit-learn.org/stable/install.html)
  https://scikit-learn.org/stable/install.html#installing-the-latest-release
* [**Installing the development version of scikit-learn**](https://scikit-learn.org/stable/developers/advanced_installation.html)
  https://scikit-learn.org/stable/developers/advanced_installation.html#dependencies


### Suggest a potential alternative/fix

I suggest removing the incomplete [Dependencies](https://scikit-learn.org/stable/developers/advanced_installation.html#dependencies) section from _[Installing the development version of scikit-learn](https://scikit-learn.org/stable/developers/advanced_installation.html)_ and referring to the extensive dependency list under _[Installing scikit-learn](https://scikit-learn.org/stable/install.html)_.

COMMENT:
> Is Ninja part of the minimum dependencies?

That's not clear to me as well :) @lesteve probably knows better the role of ninja in our build system

Labels: Documentation