Talip7/scikit-learn-issues-multilabel huggingface dataset:
Enhance calibration plots

### Describe the workflow you want to enable

I would like to enhance calibration plots by adding these new aspects:
1. Plot bins as vertical lines in the background.
2. Equal aspect on both axis to have a square figure and enable easier visual comparisons.
3. Changing xlabel from "Mean predicted probability" to "Mean confidence scores" or "Confidence scores" or similar, to avoid confusion between scores output by classifiers and true posterior probabilities.
4. Integrate a visualization of the density to see where the mass is on the calibration curve. This is important as a calibration curve can appear very bad in some part whereas it can be due to a low number of samples in this part, and actually have a good Expected Calibration Error. Several choices are possible as described in the proposed solution section.
5. Edit: add support for multiclass calibration plot by either looking at the class-wise calibration (considering one of the classes as the positive one) or the top-label problem (binarizing the multiclass problem with the prediction taken as the argmax of the confidences, a positive label iff the prediction is right and an associated confidence given by the maximum of the confidences). These are 2 of the 3 definitions of calibration in the multiclass setting.

### Describe your proposed solution

To address the previous points, I propose to:
1. Update `CalibrationDisplay`.
2. Idem
3. Idem
4. Several choices are possible here. Either adding an histogram on each edge as in the figure below. Or adding a single histogram on the confidence scores axis (either on the edge or integrated in the background of the calibration plot) since the density on the "fraction of positive" axis is less informative as the one on the confidence scores axis. This last option might be better. Also, this feature might be restricted to a single calibration curve only as it might become illegible with several classifiers.
5. Edit: This could be done by adapting the `_get_response` to allow y_pred.shape[1] > 2 and either choosing one of the K class as the positive one, or the top-label one.

Should this change be approved, I would be willing to work on it.

Example with histograms on the edges:
<img src="https://user-images.githubusercontent.com/33580936/174766358-56fc2afa-5452-4318-b0cb-18caf1b2a004.jpg" alt="drawing" width="400"/>

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
> > I am wondering if adding the ratio number of samples in the bin (or absolute) would be helpful
> 
> I implemented the feature with bin ratios. It can be enabled with `bin_label=True`:
> 
> ![test_calibration_display_example](https://user-images.githubusercontent.com/33580936/176736333-d2e8879e-a4c2-45ad-a406-4c87b1201b74.png)

could we get the code, please?

Labels: New Feature, module:calibration
Enhance calibration plots

### Describe the workflow you want to enable

I would like to enhance calibration plots by adding these new aspects:
1. Plot bins as vertical lines in the background.
2. Equal aspect on both axis to have a square figure and enable easier visual comparisons.
3. Changing xlabel from "Mean predicted probability" to "Mean confidence scores" or "Confidence scores" or similar, to avoid confusion between scores output by classifiers and true posterior probabilities.
4. Integrate a visualization of the density to see where the mass is on the calibration curve. This is important as a calibration curve can appear very bad in some part whereas it can be due to a low number of samples in this part, and actually have a good Expected Calibration Error. Several choices are possible as described in the proposed solution section.
5. Edit: add support for multiclass calibration plot by either looking at the class-wise calibration (considering one of the classes as the positive one) or the top-label problem (binarizing the multiclass problem with the prediction taken as the argmax of the confidences, a positive label iff the prediction is right and an associated confidence given by the maximum of the confidences). These are 2 of the 3 definitions of calibration in the multiclass setting.

### Describe your proposed solution

To address the previous points, I propose to:
1. Update `CalibrationDisplay`.
2. Idem
3. Idem
4. Several choices are possible here. Either adding an histogram on each edge as in the figure below. Or adding a single histogram on the confidence scores axis (either on the edge or integrated in the background of the calibration plot) since the density on the "fraction of positive" axis is less informative as the one on the confidence scores axis. This last option might be better. Also, this feature might be restricted to a single calibration curve only as it might become illegible with several classifiers.
5. Edit: This could be done by adapting the `_get_response` to allow y_pred.shape[1] > 2 and either choosing one of the K class as the positive one, or the top-label one.

Should this change be approved, I would be willing to work on it.

Example with histograms on the edges:
<img src="https://user-images.githubusercontent.com/33580936/174766358-56fc2afa-5452-4318-b0cb-18caf1b2a004.jpg" alt="drawing" width="400"/>

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
> could we get the code, please?

Yes, it is in PR #23748. Note that it is still a work in progress at the moment.

Labels: New Feature, module:calibration
TSNE is broken in pyodide

While testing `pyscript` and thus `pyodide`, it seems that `TSNE` is broken.
The reason is that we are specifically using 32 bits internally that do not work with `pyodide`.
I think that we should make sure that `TSNE` preserve dtype in 64 bits and 32 bits without silently reducing the numerical precision.

The current workaround is to use `method="exact"` and externalize the initialization and pass a 64 bits NumPy array. Solely this case preserves the computation in 64 bits.

COMMENT:
The behaviour I am seeing while trying on https://jupyter.org/try-jupyter/lab/ is that `.fit` never completes:

![image](https://user-images.githubusercontent.com/1680079/174829710-9e0411b5-4ee3-415c-8f44-90714a5c9dd6.png)

The snippet I used:
```py
import numpy as np
from sklearn.manifold import TSNE

X = np.array([[0, 0, 0], [0, 1, 1], [1, 0, 1], [1, 1, 1]])
tsne = TSNE(n_components=2, learning_rate='auto', init='random')
tsne.fit(X)
```

I am curious if you find more details about what is the underlying issue with 32bit float support in pyodide, I googled a bit but did not find anything useful ...

Labels: Bug
TSNE is broken in pyodide

While testing `pyscript` and thus `pyodide`, it seems that `TSNE` is broken.
The reason is that we are specifically using 32 bits internally that do not work with `pyodide`.
I think that we should make sure that `TSNE` preserve dtype in 64 bits and 32 bits without silently reducing the numerical precision.

The current workaround is to use `method="exact"` and externalize the initialization and pass a 64 bits NumPy array. Solely this case preserves the computation in 64 bits.

COMMENT:
And while trying debugging, we checked the following:

https://pyodide.org/en/latest/development/debugging.html

Labels: Bug
TSNE is broken in pyodide

While testing `pyscript` and thus `pyodide`, it seems that `TSNE` is broken.
The reason is that we are specifically using 32 bits internally that do not work with `pyodide`.
I think that we should make sure that `TSNE` preserve dtype in 64 bits and 32 bits without silently reducing the numerical precision.

The current workaround is to use `method="exact"` and externalize the initialization and pass a 64 bits NumPy array. Solely this case preserves the computation in 64 bits.

COMMENT:
@rth do you mind recalling if there is any issue that you are aware of regarding 32 bits operation in `pyodide`?

Labels: Bug
TSNE is broken in pyodide

While testing `pyscript` and thus `pyodide`, it seems that `TSNE` is broken.
The reason is that we are specifically using 32 bits internally that do not work with `pyodide`.
I think that we should make sure that `TSNE` preserve dtype in 64 bits and 32 bits without silently reducing the numerical precision.

The current workaround is to use `method="exact"` and externalize the initialization and pass a 64 bits NumPy array. Solely this case preserves the computation in 64 bits.

COMMENT:
@lesteve the work around for the example above that preserve the dtype in 64 bits would be:

```python
import numpy as np
from sklearn.manifold import TSNE
X = np.array([[0, 0, 0], [0, 1, 1], [1, 0, 1], [1, 1, 1]])

n_components = 2
init = np.random.randn(X.shape[0], 2)
X_embedded = TSNE(
    n_components=n_components, learning_rate='auto', init=init, method="exact"
).fit_transform(X)
```

where not internal conversion will happen.

Labels: Bug
TSNE is broken in pyodide

While testing `pyscript` and thus `pyodide`, it seems that `TSNE` is broken.
The reason is that we are specifically using 32 bits internally that do not work with `pyodide`.
I think that we should make sure that `TSNE` preserve dtype in 64 bits and 32 bits without silently reducing the numerical precision.

The current workaround is to use `method="exact"` and externalize the initialization and pass a 64 bits NumPy array. Solely this case preserves the computation in 64 bits.

COMMENT:
It seems like I can reproduce the issue by calling `scipy.linalg.norm` on float32 numpy arrays:

```py
import numpy as np
arr = np.array([1, 2, 3], dtype=np.float32)

import scipy.linalg
scipy.linalg.norm(arr)
```

Labels: Bug
TSNE is broken in pyodide

While testing `pyscript` and thus `pyodide`, it seems that `TSNE` is broken.
The reason is that we are specifically using 32 bits internally that do not work with `pyodide`.
I think that we should make sure that `TSNE` preserve dtype in 64 bits and 32 bits without silently reducing the numerical precision.

The current workaround is to use `method="exact"` and externalize the initialization and pass a 64 bits NumPy array. Solely this case preserves the computation in 64 bits.

COMMENT:
I confirm I can reproduce the `RuntimeError: indirect call signature mismatch` in the javascript console of firefox when running [this code snippet](https://github.com/scikit-learn/scikit-learn/issues/23707#issuecomment-1161961037).

```
File "/lib/python3.10/site-packages/scipy/linalg/_misc.py", line 154 in norm
```

The version of scipy on this jupyterlite environment is 1.8.0 which means that this happens here:

https://github.com/scipy/scipy/blob/b5d8bab88af61d61de09641243848df63380a67f/scipy/linalg/_misc.py#L154

where nrm2 is defined using the blas runtime wrapper of scipy:

```python
            nrm2 = get_blas_funcs('nrm2', dtype=a.dtype, ilp64='preferred')
```

Labels: Bug
TSNE is broken in pyodide

While testing `pyscript` and thus `pyodide`, it seems that `TSNE` is broken.
The reason is that we are specifically using 32 bits internally that do not work with `pyodide`.
I think that we should make sure that `TSNE` preserve dtype in 64 bits and 32 bits without silently reducing the numerical precision.

The current workaround is to use `method="exact"` and externalize the initialization and pass a 64 bits NumPy array. Solely this case preserves the computation in 64 bits.

COMMENT:
I have tried to change the value of the ilp64 argument and I get the same error.

```python
import numpy as np
from scipy.linalg.blas import get_blas_funcs

arr = np.array([1, 2, 3], dtype=np.float32)
get_blas_funcs('nrm2', dtype=arr.dtype, ilp64=False)(arr)
```

Labels: Bug
TSNE is broken in pyodide

While testing `pyscript` and thus `pyodide`, it seems that `TSNE` is broken.
The reason is that we are specifically using 32 bits internally that do not work with `pyodide`.
I think that we should make sure that `TSNE` preserve dtype in 64 bits and 32 bits without silently reducing the numerical precision.

The current workaround is to use `method="exact"` and externalize the initialization and pass a 64 bits NumPy array. Solely this case preserves the computation in 64 bits.

COMMENT:
If I set `ilp64=True` explicitly, then I get a regular Python exception instead of a lower-level WASM crash in the js console:

```python
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Input In [4], in <cell line: 5>()
      2 from scipy.linalg.blas import get_blas_funcs
      4 arr = np.array([1, 2, 3], dtype=np.float32)
----> 5 get_blas_funcs('nrm2', dtype=arr.dtype, ilp64=True)(arr)

File /lib/python3.10/site-packages/scipy/linalg/blas.py:400, in _memoize_get_funcs.<locals>.getter(names, arrays, dtype, ilp64)
    397 if value is not None:
    398     return value
--> 400 value = func(names, arrays, dtype, ilp64)
    402 if key is not None:
    403     memo[key] = value

File /lib/python3.10/site-packages/scipy/linalg/blas.py:478, in get_blas_funcs(names, arrays, dtype, ilp64)
    476 else:
    477     if not HAS_ILP64:
--> 478         raise RuntimeError("BLAS ILP64 routine requested, but Scipy "
    479                            "compiled only with 32-bit BLAS")
    480     return _get_funcs(names, arrays, dtype,
    481                       "BLAS", _fblas_64, None, "fblas_64", None,
    482                       _blas_alias, ilp64=True)

RuntimeError: BLAS ILP64 routine requested, but Scipy compiled only with 32-bit BLAS
```

Labels: Bug
TSNE is broken in pyodide

While testing `pyscript` and thus `pyodide`, it seems that `TSNE` is broken.
The reason is that we are specifically using 32 bits internally that do not work with `pyodide`.
I think that we should make sure that `TSNE` preserve dtype in 64 bits and 32 bits without silently reducing the numerical precision.

The current workaround is to use `method="exact"` and externalize the initialization and pass a 64 bits NumPy array. Solely this case preserves the computation in 64 bits.

COMMENT:
I reported the minimal reproducer upstream:

- https://github.com/pyodide/pyodide/issues/2763

Labels: Bug
TSNE is broken in pyodide

While testing `pyscript` and thus `pyodide`, it seems that `TSNE` is broken.
The reason is that we are specifically using 32 bits internally that do not work with `pyodide`.
I think that we should make sure that `TSNE` preserve dtype in 64 bits and 32 bits without silently reducing the numerical precision.

The current workaround is to use `method="exact"` and externalize the initialization and pass a 64 bits NumPy array. Solely this case preserves the computation in 64 bits.

COMMENT:
This is fixed in the main branch of pyodide (built with the latest versions of numpy and emscripten): https://pyodide.org/en/latest/console.html

Labels: Bug
TSNE is broken in pyodide

While testing `pyscript` and thus `pyodide`, it seems that `TSNE` is broken.
The reason is that we are specifically using 32 bits internally that do not work with `pyodide`.
I think that we should make sure that `TSNE` preserve dtype in 64 bits and 32 bits without silently reducing the numerical precision.

The current workaround is to use `method="exact"` and externalize the initialization and pass a 64 bits NumPy array. Solely this case preserves the computation in 64 bits.

COMMENT:
Yes, the Scipy Pyodide port is still a bit of hack with [lots of patches](https://github.com/pyodide/pyodide/blob/main/packages/scipy/meta.yaml), function signature fixes, and until there is no Fortran compiler with WASM support,  code rewrites [before](https://github.com/pyodide/pyodide/blob/main/pyodide-build/pyodide_build/_f2c_fixes.py#L11) and [after](https://github.com/pyodide/pyodide/blob/main/pyodide-build/pyodide_build/_f2c_fixes.py#L197) f2c is run. So it's a bit horrible, but works to some extent, and a good fraction of the test suite passes https://github.com/pyodide/pyodide/issues/2727#issuecomment-1157866792 (since that comment scipy.sparse tests are also now passing).

For scikit-learn since we are using only a subset of scipy, the situation might not be too bad (but still needs improvement). If anyone wants to play with it, you can run the test suite in the above link with, e.g.
```py
>>> import micropip
>>> await micropip.install(['scikit-learn', 'scikit-learn-tests'])  # tests are unvendored into a separate package
>>> import pytest
>>> pytest.main(['--pyargs', 'sklearn.preprocessing'])   # or some other folder
```
For instance, in the above example all tests pass,
```
================= 949 passed, 77 skipped, 9 warnings in 22.73s =================
```
but well there is little scipy & linalg in pre-processing. It's probably better to run sub-module by sub-module as it takes a while, there is no intermediary output shown until all tests are run, and some code paths in scipy can still crash the interpreter (producing a fatal error) in which case you will not see even tests that were successful.

> do you mind recalling if there is any issue that you are aware of regarding 32 bits operation in pyodide?

Not particularly, the whole architecture is 32bit, it's just that some code paths in scipy are used & tested less than others.

Labels: Bug
TSNE is broken in pyodide

While testing `pyscript` and thus `pyodide`, it seems that `TSNE` is broken.
The reason is that we are specifically using 32 bits internally that do not work with `pyodide`.
I think that we should make sure that `TSNE` preserve dtype in 64 bits and 32 bits without silently reducing the numerical precision.

The current workaround is to use `method="exact"` and externalize the initialization and pass a 64 bits NumPy array. Solely this case preserves the computation in 64 bits.

COMMENT:
Hello,

I would like to make a comment concerning this issue.

So i tried to reproduce the error:
```bash
RuntimeError: indirect call signature mismatch
```
But when i try importing TSNE and perform the calculation on the terminal of [Pyodide](https://pyodide.org/en/stable/console.html), i did not see any error produced. So i decide to make a small code snippet:

```html
<!doctype html>
<html>
  <head>
      <script src="https://cdn.jsdelivr.net/pyodide/v0.22.0/full/pyodide.js"></script>
  </head>
  <body>
    Code Snippet to test TSNE in Pyodide <br>
    Open your browser console to see Pyodide output
    <script type="module">
      async function main(){
        let pyodide = await loadPyodide();
        console.log(pyodide.runPython(`
            import sys
            sys.version
        `));
        await pyodide.loadPackage("scikit-learn");
        pyodide.runPython(`
            import sklearn
            import scipy
            from sklearn.manifold import TSNE
            from sklearn.datasets import load_digits

            print(f"Version de Sklearn: {sklearn.__version__}")
            print(f"Version de Scipy: {scipy.__version__}")

            data = load_digits()
            X = data['data']
            y = data['target']

            tsne = TSNE(n_components=2, n_iter=500, verbose=0)
            X_2d = tsne.fit_transform(X)

            `);
        console.log(pyodide.globals.get("X_2d").toString())
      }
      main();
    </script>
  </body>
</html>
```

And the output on the console is:
![console_output](https://user-images.githubusercontent.com/110120278/211446058-b45d7a41-7c6e-4579-8b9b-e9e933173e96.png)

The result is in 32-bits as well ! And Sklearn on Pyodide is also the latest version (1.2.0). We can now perform T-SNE without preserving 64-bits computation.
If i misunderstand something, please let me know.
Thank you for reading.

Labels: Bug
TSNE is broken in pyodide

While testing `pyscript` and thus `pyodide`, it seems that `TSNE` is broken.
The reason is that we are specifically using 32 bits internally that do not work with `pyodide`.
I think that we should make sure that `TSNE` preserve dtype in 64 bits and 32 bits without silently reducing the numerical precision.

The current workaround is to use `method="exact"` and externalize the initialization and pass a 64 bits NumPy array. Solely this case preserves the computation in 64 bits.

COMMENT:
This issue is closed indeed and has been fixed in Pyodide in https://github.com/pyodide/pyodide/pull/2728 so indeed there should not be any issue any more.

Labels: Bug
Remove sklearn.utils.fixes.loguniform for 1.3 release

`scipy.stats.loguniform` is a thing since scipy 1.4, see 
https://github.com/scipy/scipy/pull/10815.

According to https://github.com/scikit-learn/scikit-learn/pull/23637#issuecomment-1157337426, the minimum Scipy release will be 1.6 when scikit-learn 1.2 is released.

That means `sklearn.utils.fixes.loguniform` can be removed, adding this issue so that we remember to do it when 1.2 release is close.

COMMENT:
I am adding the "help wanted" tag.

Labels: module:utils
Remove sklearn.utils.fixes.loguniform for 1.3 release

`scipy.stats.loguniform` is a thing since scipy 1.4, see 
https://github.com/scipy/scipy/pull/10815.

According to https://github.com/scikit-learn/scikit-learn/pull/23637#issuecomment-1157337426, the minimum Scipy release will be 1.6 when scikit-learn 1.2 is released.

That means `sklearn.utils.fixes.loguniform` can be removed, adding this issue so that we remember to do it when 1.2 release is close.

COMMENT:
/take

Labels: module:utils
Remove sklearn.utils.fixes.loguniform for 1.3 release

`scipy.stats.loguniform` is a thing since scipy 1.4, see 
https://github.com/scipy/scipy/pull/10815.

According to https://github.com/scikit-learn/scikit-learn/pull/23637#issuecomment-1157337426, the minimum Scipy release will be 1.6 when scikit-learn 1.2 is released.

That means `sklearn.utils.fixes.loguniform` can be removed, adding this issue so that we remember to do it when 1.2 release is close.

COMMENT:
@brgopalakrishnan sorry there is already a PR on this one, feel free to look for another issue to work on!

I guess we should have removed the help wanted label ...

Labels: module:utils
Remove sklearn.utils.fixes.loguniform for 1.3 release

`scipy.stats.loguniform` is a thing since scipy 1.4, see 
https://github.com/scipy/scipy/pull/10815.

According to https://github.com/scikit-learn/scikit-learn/pull/23637#issuecomment-1157337426, the minimum Scipy release will be 1.6 when scikit-learn 1.2 is released.

That means `sklearn.utils.fixes.loguniform` can be removed, adding this issue so that we remember to do it when 1.2 release is close.

COMMENT:
@lesteve  Thank you for the update. Just curious, I see only the same in grid_search.rst line number 152.  Is that the one supposed to be removed.?

Labels: module:utils
Remove sklearn.utils.fixes.loguniform for 1.3 release

`scipy.stats.loguniform` is a thing since scipy 1.4, see 
https://github.com/scipy/scipy/pull/10815.

According to https://github.com/scikit-learn/scikit-learn/pull/23637#issuecomment-1157337426, the minimum Scipy release will be 1.6 when scikit-learn 1.2 is released.

That means `sklearn.utils.fixes.loguniform` can be removed, adding this issue so that we remember to do it when 1.2 release is close.

COMMENT:
We finally decided to be conservative and keep scipy min version = 1.3.3 for one more version (reasons explained in #24650), so we need to keep the backport in the mean time unfortunately. I'm moving it to the 1.3 milestone

Labels: module:utils