Talip7/scikit-learn-issues-multilabel huggingface dataset:
Revisit the "chance level" for the different displays

@e-pet commented on different PRs & issues some interesting fact. I take the opportunity to consolidate some of those comments here.

First, we use the term "chance" that is ambiguous depending of the displays. The term "baseline" would probably be better. In addition, I checked and I think we should make an extra effort on the definition of the baseline for each of the type of plot: for ROC curve, the baseline is "a random classifier assigning the positive class with probability p and the negative class with probability 1 − p" [1] while for the PR curve, the baseline is derived from the "always-positive classifier" where any recall or precision under π should be discarded [1].

It leads to a second where in the PR curve, we plot the horizontal line derived from the always-positive classifier but we don't discard when recall < π. In this case, as mentioned by @e-pet, it might make sense to show the hyperbolic line of the always-positive classifier instead (cf. Fig. 2 in [1]).

@e-pet feel free to add any other points that you wanted to discuss. Here, I wanted to focus on the one that looks critical and could be addressed.

[1] [Flach, P., & Kull, M. (2015). Precision-recall-gain curves: PR analysis done right. Advances in neural information processing systems, 28.](https://papers.nips.cc/paper_files/paper/2015/file/33e8075e9970de0cfea955afd4644bb2-Paper.pdf)

COMMENT:
Going forward, the first things we need to decide on would be (1) how we provide the user with a non-informative/baseline/chance-level plot and (2) what to call it.

So, maybe we can add something like `plot_baseline=False` param to `Display.plot()`, that will plot a baseline (or several?) in an unremarkable colour (light grey?) when set to `True`. This preserves backwards compatibility (because the default behaviour of the display is unchanged) and if we want to change this some time in the future, we can.

Regarding the name it seems to consolidate around "Non-informative", which could also be part of a param description. I personally think that "baseline" alone can be misinterpreted (it's unclear that it stems from random data, rather than depending on the provided data trained in some model) and it should show better in the param name. Such as `plot_chance_level`, `plot_random_baseline` or `plot_non_informative_baseline` 😅. In any case, I strongly agree on telling users on how the baseline(s) would be constructed in the docs.

How the non-informative baseline would be calculated and whether there should be many, can be discussed per Display. Once we have agreed on (1) and (2), this issue could become a meta-issue and we open sub-issues per display.

Labels: Documentation, API
NuSVC argument `class_weight` is not used

### Describe the bug

Like `SVC`, the class `NuSVC` takes argument `class_weight`. However, it looks like this argument is not used. After a quick look at the libsvm C code within sklearn as well as [libsvm's original documentation](https://www.csie.ntu.edu.tw/~cjlin/libsvm/), this seems to be expected: "`wi` set the parameter C of class i to weight*C, for C-SVC". I suggest that this argument should be removed from `NuSVC`'s constructor and from the documentation.

### Steps/Code to Reproduce

```python
from sklearn.svm import SVC, NuSVC

X = [[1., 2, 3], [0, 5, 2]]
y = [-1, 1]

NuSVC(verbose=True).fit(X, y).dual_coef_

optimization finished, #iter = 0
C = 2.587063
obj = 1.293532, rho = 0.000000
nSV = 2, nBSV = 0
Total nSV = 2
Out: [LibSVM]array([[-1.29353162,  1.29353162]])

SVC(C=2.587063, verbose=True).fit(X, y).dual_coef_

optimization finished, #iter = 1
obj = -1.293532, rho = 0.000000
nSV = 2, nBSV = 0
Total nSV = 2
Out: [LibSVM]array([[-1.29353162,  1.29353162]])

NuSVC(class_weight={-1:1.5, 1:.2}, verbose=True).fit(X, y).dual_coef_

optimization finished, #iter = 0
C = 2.587063
obj = 1.293532, rho = 0.000000
nSV = 2, nBSV = 0
Total nSV = 2
Out: [LibSVM]array([[-1.29353162,  1.29353162]])

SVC(C=2.587063, class_weight={-1:1.5, 1:.2}, verbose=True).fit(X, y).dual_coef_

optimization finished, #iter = 1
obj = -0.827860, rho = -0.600000
nSV = 2, nBSV = 1
Total nSV = 2
Out: [LibSVM]array([[-0.5174126,  0.5174126]])


NuSVC(class_weight={-1:0, 1:0}).fit(X, y).dual_coef_
Out: array([[-1.29353162,  1.29353162]])

SVC(class_weight={-1:0, 1:0}).fit(X, y).dual_coef_
Out: array([], shape=(1, 0), dtype=float64)
```


### Expected Results

As in the case of no `class_weight`, `NuSVM` should give the same `dual_coef_` as an `SVC` with the same `C`.
Also `class_weight={-1:0, 1:0}` should give the "empty" result.

### Actual Results

In all cases above `NuSVM` with class weight behaves exactly as when no weights are given.

### Versions

```shell
System:
    python: 3.9.16 | packaged by conda-forge | (main, Feb  1 2023, 21:39:03)  [GCC 11.3.0]
executable: .../bin/python3.9
   machine: Linux-6.8.0-48-generic-x86_64-with-glibc2.39

Python dependencies:
      sklearn: 1.5.2
          pip: 23.0.1
   setuptools: 67.6.0
        numpy: 2.0.2
        scipy: 1.13.1
       Cython: None
       pandas: None
   matplotlib: 3.9.2
       joblib: 1.4.2
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: blis
    num_threads: 1
         prefix: libblis
       filepath: .../lib/libblis.so.4.0.0
        version: 0.9.0
threading_layer: pthreads
   architecture: skx

       user_api: openmp
   internal_api: openmp
    num_threads: 8
         prefix: libgomp
       filepath: .../lib/libgomp.so.1.0.0
        version: None
```

COMMENT:
If it's okay with you, I can edit the code and of the doc

Labels: Bug, Needs Investigation
NuSVC argument `class_weight` is not used

### Describe the bug

Like `SVC`, the class `NuSVC` takes argument `class_weight`. However, it looks like this argument is not used. After a quick look at the libsvm C code within sklearn as well as [libsvm's original documentation](https://www.csie.ntu.edu.tw/~cjlin/libsvm/), this seems to be expected: "`wi` set the parameter C of class i to weight*C, for C-SVC". I suggest that this argument should be removed from `NuSVC`'s constructor and from the documentation.

### Steps/Code to Reproduce

```python
from sklearn.svm import SVC, NuSVC

X = [[1., 2, 3], [0, 5, 2]]
y = [-1, 1]

NuSVC(verbose=True).fit(X, y).dual_coef_

optimization finished, #iter = 0
C = 2.587063
obj = 1.293532, rho = 0.000000
nSV = 2, nBSV = 0
Total nSV = 2
Out: [LibSVM]array([[-1.29353162,  1.29353162]])

SVC(C=2.587063, verbose=True).fit(X, y).dual_coef_

optimization finished, #iter = 1
obj = -1.293532, rho = 0.000000
nSV = 2, nBSV = 0
Total nSV = 2
Out: [LibSVM]array([[-1.29353162,  1.29353162]])

NuSVC(class_weight={-1:1.5, 1:.2}, verbose=True).fit(X, y).dual_coef_

optimization finished, #iter = 0
C = 2.587063
obj = 1.293532, rho = 0.000000
nSV = 2, nBSV = 0
Total nSV = 2
Out: [LibSVM]array([[-1.29353162,  1.29353162]])

SVC(C=2.587063, class_weight={-1:1.5, 1:.2}, verbose=True).fit(X, y).dual_coef_

optimization finished, #iter = 1
obj = -0.827860, rho = -0.600000
nSV = 2, nBSV = 1
Total nSV = 2
Out: [LibSVM]array([[-0.5174126,  0.5174126]])


NuSVC(class_weight={-1:0, 1:0}).fit(X, y).dual_coef_
Out: array([[-1.29353162,  1.29353162]])

SVC(class_weight={-1:0, 1:0}).fit(X, y).dual_coef_
Out: array([], shape=(1, 0), dtype=float64)
```


### Expected Results

As in the case of no `class_weight`, `NuSVM` should give the same `dual_coef_` as an `SVC` with the same `C`.
Also `class_weight={-1:0, 1:0}` should give the "empty" result.

### Actual Results

In all cases above `NuSVM` with class weight behaves exactly as when no weights are given.

### Versions

```shell
System:
    python: 3.9.16 | packaged by conda-forge | (main, Feb  1 2023, 21:39:03)  [GCC 11.3.0]
executable: .../bin/python3.9
   machine: Linux-6.8.0-48-generic-x86_64-with-glibc2.39

Python dependencies:
      sklearn: 1.5.2
          pip: 23.0.1
   setuptools: 67.6.0
        numpy: 2.0.2
        scipy: 1.13.1
       Cython: None
       pandas: None
   matplotlib: 3.9.2
       joblib: 1.4.2
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: blis
    num_threads: 1
         prefix: libblis
       filepath: .../lib/libblis.so.4.0.0
        version: 0.9.0
threading_layer: pthreads
   architecture: skx

       user_api: openmp
   internal_api: openmp
    num_threads: 8
         prefix: libgomp
       filepath: .../lib/libgomp.so.1.0.0
        version: None
```

COMMENT:
Absolutely. Thanks!

Labels: Bug, Needs Investigation
NuSVC argument `class_weight` is not used

### Describe the bug

Like `SVC`, the class `NuSVC` takes argument `class_weight`. However, it looks like this argument is not used. After a quick look at the libsvm C code within sklearn as well as [libsvm's original documentation](https://www.csie.ntu.edu.tw/~cjlin/libsvm/), this seems to be expected: "`wi` set the parameter C of class i to weight*C, for C-SVC". I suggest that this argument should be removed from `NuSVC`'s constructor and from the documentation.

### Steps/Code to Reproduce

```python
from sklearn.svm import SVC, NuSVC

X = [[1., 2, 3], [0, 5, 2]]
y = [-1, 1]

NuSVC(verbose=True).fit(X, y).dual_coef_

optimization finished, #iter = 0
C = 2.587063
obj = 1.293532, rho = 0.000000
nSV = 2, nBSV = 0
Total nSV = 2
Out: [LibSVM]array([[-1.29353162,  1.29353162]])

SVC(C=2.587063, verbose=True).fit(X, y).dual_coef_

optimization finished, #iter = 1
obj = -1.293532, rho = 0.000000
nSV = 2, nBSV = 0
Total nSV = 2
Out: [LibSVM]array([[-1.29353162,  1.29353162]])

NuSVC(class_weight={-1:1.5, 1:.2}, verbose=True).fit(X, y).dual_coef_

optimization finished, #iter = 0
C = 2.587063
obj = 1.293532, rho = 0.000000
nSV = 2, nBSV = 0
Total nSV = 2
Out: [LibSVM]array([[-1.29353162,  1.29353162]])

SVC(C=2.587063, class_weight={-1:1.5, 1:.2}, verbose=True).fit(X, y).dual_coef_

optimization finished, #iter = 1
obj = -0.827860, rho = -0.600000
nSV = 2, nBSV = 1
Total nSV = 2
Out: [LibSVM]array([[-0.5174126,  0.5174126]])


NuSVC(class_weight={-1:0, 1:0}).fit(X, y).dual_coef_
Out: array([[-1.29353162,  1.29353162]])

SVC(class_weight={-1:0, 1:0}).fit(X, y).dual_coef_
Out: array([], shape=(1, 0), dtype=float64)
```


### Expected Results

As in the case of no `class_weight`, `NuSVM` should give the same `dual_coef_` as an `SVC` with the same `C`.
Also `class_weight={-1:0, 1:0}` should give the "empty" result.

### Actual Results

In all cases above `NuSVM` with class weight behaves exactly as when no weights are given.

### Versions

```shell
System:
    python: 3.9.16 | packaged by conda-forge | (main, Feb  1 2023, 21:39:03)  [GCC 11.3.0]
executable: .../bin/python3.9
   machine: Linux-6.8.0-48-generic-x86_64-with-glibc2.39

Python dependencies:
      sklearn: 1.5.2
          pip: 23.0.1
   setuptools: 67.6.0
        numpy: 2.0.2
        scipy: 1.13.1
       Cython: None
       pandas: None
   matplotlib: 3.9.2
       joblib: 1.4.2
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: blis
    num_threads: 1
         prefix: libblis
       filepath: .../lib/libblis.so.4.0.0
        version: 0.9.0
threading_layer: pthreads
   architecture: skx

       user_api: openmp
   internal_api: openmp
    num_threads: 8
         prefix: libgomp
       filepath: .../lib/libgomp.so.1.0.0
        version: None
```

COMMENT:
> "Maybe we can deprecate this parameter for now (as it never had any effect)."

Ok, perfect, who should make this decision? Do I make a new PR and delete the previous one?

@ogrisel

Labels: Bug, Needs Investigation
Classification report, digits variable

### Describe the workflow you want to enable

Hi, as of right now the digits variable which limits how many numbers are shown after the decimal point does not apply to the support column for the classification report. Support normally does not have decimals, but that happens when we apply sample weights to the report.

### Describe your proposed solution

apply decimal variable to support column as well as the metric columns

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I paste here a small reproducer to show the limitation

```python
from sklearn.metrics import classification_report
y_true = [0, 1, 2, 2, 2]
y_pred = [0, 0, 2, 2, 1]
sample_weight = [0.533, 2.333, 1.333, 2.5, 1]
target_names = ['class 0', 'class 1', 'class 2']

print(classification_report(
    y_true,
    y_pred,
    sample_weight=sample_weight,
    target_names=target_names,
    digits=2,
))
```

```
              precision    recall  f1-score   support

     class 0       0.19      1.00      0.31     0.533
     class 1       0.00      0.00      0.00     2.333
     class 2       1.00      0.79      0.88     4.833

    accuracy                           0.57     7.699
   macro avg       0.40      0.60      0.40     7.699
weighted avg       0.64      0.57      0.58     7.699
```

So indeed, it would make sense to extend the support of `digits` to this statistics as well

Labels: Enhancement
Classification report, digits variable

### Describe the workflow you want to enable

Hi, as of right now the digits variable which limits how many numbers are shown after the decimal point does not apply to the support column for the classification report. Support normally does not have decimals, but that happens when we apply sample weights to the report.

### Describe your proposed solution

apply decimal variable to support column as well as the metric columns

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I am new to this, could you also help me with how I can make sure these changes reflect in my code? Do I only need to do a pip update?

Labels: Enhancement
Classification report, digits variable

### Describe the workflow you want to enable

Hi, as of right now the digits variable which limits how many numbers are shown after the decimal point does not apply to the support column for the classification report. Support normally does not have decimals, but that happens when we apply sample weights to the report.

### Describe your proposed solution

apply decimal variable to support column as well as the metric columns

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
First, we need to implement the feature via a pull-request. I already see that #30331 is going that road. Once the feature implemented and reviewed, we are going to merge it in the `main` branch. Then, at the next release of scikit-learn, this changes will be included in it and available via the PyPI and conda-force releases. Note that we have 2 releases per year.

Labels: Enhancement
Classification report, digits variable

### Describe the workflow you want to enable

Hi, as of right now the digits variable which limits how many numbers are shown after the decimal point does not apply to the support column for the classification report. Support normally does not have decimals, but that happens when we apply sample weights to the report.

### Describe your proposed solution

apply decimal variable to support column as well as the metric columns

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Hello, @glemaitre. The pull request #30331 is currently ready for review, as it has passed all checks. Could a core developer please review the PR?

Labels: Enhancement
DOC Example on model selection for Gaussian Mixture Models

### Describe the issue linked to the documentation

We have an example that illustrates how to use the BIC score to tune the number of components and the type of covariance matrix parametrization here:

https://scikit-learn.org/stable/auto_examples/mixture/plot_gmm_selection.html

However, the BIC score is not meant to be computed in a CV loop, but instead directly on the training set. So we should not use it with a `GridSearchCV` call. Indeed, the BIC score already penalizes the number of parameters depending on the number of data-points in the training set.

Instead, we should call the `GridSearchCV` on the default `.score` method of the GMM estimator, which computes the log-likelihood and is a perfectly fine metric to select the best model on held out data in a CV loop.

Note that we can keep computing the BIC score for all the hparam combinations but we should either do it in a single for loop (without train-test split), e.g.:

```python
from sklearn.model_selection import ParameterGrid
from sklearn.mixture import GaussianMixture
import pandas as pd
import numpy as np


n_samples = 500
rng = np.random.default_rng(0)
C = np.array([[0.0, -0.1], [1.7, 0.4]])
component_1 = rng.normal(size=(n_samples, 2)) @ C  # general
component_2 = 0.7 * rng.normal(size=(n_samples, 2)) + np.array([-4, 1])  # spherical
X = np.concatenate([component_1, component_2])

param_grid = {
    "n_components": np.arange(1, 7),
    "covariance_type": ["full", "tied", "diag", "spherical"],
}

bic_evaluations = []
for params in ParameterGrid(param_grid):
    bic_value = GaussianMixture(**params).fit(X).bic(X)
    bic_evaluations.append({**params, "BIC": bic_value})

bic_evaluations = pd.DataFrame(bic_evaluations).sort_values("BIC", ascending=True)
bic_evaluations.head()
```

So in summary I would recommend to:

- update the existing `GridSearchCV` code to use the `scoring=None` default that would use the built-in log-likelihood based model evaluation (averaged on the test sets of the CV loop);
- add a new section for BIC-based model selection with the for loop I proposed above as a computationally cheaper alternative to CV-based model selection with the log-likelihood.

We can then check that the two methods select the same model: 2 components with "full" (non-spherical) parametrization of the covariance matrix.

COMMENT:
I think the things you asked for should be implemented because these changes ensure correct usage of model selection methods with GaussianMixture.

I tried the recommendations you mentioned, the result- 

![Screenshot 2024-11-22 at 1 36 00 AM](https://github.com/user-attachments/assets/42e229d6-186c-4624-b5a7-88201d6b65b4)

Labels: Documentation
DOC Example on model selection for Gaussian Mixture Models

### Describe the issue linked to the documentation

We have an example that illustrates how to use the BIC score to tune the number of components and the type of covariance matrix parametrization here:

https://scikit-learn.org/stable/auto_examples/mixture/plot_gmm_selection.html

However, the BIC score is not meant to be computed in a CV loop, but instead directly on the training set. So we should not use it with a `GridSearchCV` call. Indeed, the BIC score already penalizes the number of parameters depending on the number of data-points in the training set.

Instead, we should call the `GridSearchCV` on the default `.score` method of the GMM estimator, which computes the log-likelihood and is a perfectly fine metric to select the best model on held out data in a CV loop.

Note that we can keep computing the BIC score for all the hparam combinations but we should either do it in a single for loop (without train-test split), e.g.:

```python
from sklearn.model_selection import ParameterGrid
from sklearn.mixture import GaussianMixture
import pandas as pd
import numpy as np


n_samples = 500
rng = np.random.default_rng(0)
C = np.array([[0.0, -0.1], [1.7, 0.4]])
component_1 = rng.normal(size=(n_samples, 2)) @ C  # general
component_2 = 0.7 * rng.normal(size=(n_samples, 2)) + np.array([-4, 1])  # spherical
X = np.concatenate([component_1, component_2])

param_grid = {
    "n_components": np.arange(1, 7),
    "covariance_type": ["full", "tied", "diag", "spherical"],
}

bic_evaluations = []
for params in ParameterGrid(param_grid):
    bic_value = GaussianMixture(**params).fit(X).bic(X)
    bic_evaluations.append({**params, "BIC": bic_value})

bic_evaluations = pd.DataFrame(bic_evaluations).sort_values("BIC", ascending=True)
bic_evaluations.head()
```

So in summary I would recommend to:

- update the existing `GridSearchCV` code to use the `scoring=None` default that would use the built-in log-likelihood based model evaluation (averaged on the test sets of the CV loop);
- add a new section for BIC-based model selection with the for loop I proposed above as a computationally cheaper alternative to CV-based model selection with the log-likelihood.

We can then check that the two methods select the same model: 2 components with "full" (non-spherical) parametrization of the covariance matrix.

COMMENT:
@glemaitre review my comment and let me know if the changes are to be made?

Labels: Documentation
DOC Example on model selection for Gaussian Mixture Models

### Describe the issue linked to the documentation

We have an example that illustrates how to use the BIC score to tune the number of components and the type of covariance matrix parametrization here:

https://scikit-learn.org/stable/auto_examples/mixture/plot_gmm_selection.html

However, the BIC score is not meant to be computed in a CV loop, but instead directly on the training set. So we should not use it with a `GridSearchCV` call. Indeed, the BIC score already penalizes the number of parameters depending on the number of data-points in the training set.

Instead, we should call the `GridSearchCV` on the default `.score` method of the GMM estimator, which computes the log-likelihood and is a perfectly fine metric to select the best model on held out data in a CV loop.

Note that we can keep computing the BIC score for all the hparam combinations but we should either do it in a single for loop (without train-test split), e.g.:

```python
from sklearn.model_selection import ParameterGrid
from sklearn.mixture import GaussianMixture
import pandas as pd
import numpy as np


n_samples = 500
rng = np.random.default_rng(0)
C = np.array([[0.0, -0.1], [1.7, 0.4]])
component_1 = rng.normal(size=(n_samples, 2)) @ C  # general
component_2 = 0.7 * rng.normal(size=(n_samples, 2)) + np.array([-4, 1])  # spherical
X = np.concatenate([component_1, component_2])

param_grid = {
    "n_components": np.arange(1, 7),
    "covariance_type": ["full", "tied", "diag", "spherical"],
}

bic_evaluations = []
for params in ParameterGrid(param_grid):
    bic_value = GaussianMixture(**params).fit(X).bic(X)
    bic_evaluations.append({**params, "BIC": bic_value})

bic_evaluations = pd.DataFrame(bic_evaluations).sort_values("BIC", ascending=True)
bic_evaluations.head()
```

So in summary I would recommend to:

- update the existing `GridSearchCV` code to use the `scoring=None` default that would use the built-in log-likelihood based model evaluation (averaged on the test sets of the CV loop);
- add a new section for BIC-based model selection with the for loop I proposed above as a computationally cheaper alternative to CV-based model selection with the log-likelihood.

We can then check that the two methods select the same model: 2 components with "full" (non-spherical) parametrization of the covariance matrix.

COMMENT:
@Yuvraj-Pradhan-27: your original comment feels like the output of an LLM paraphrasing the description I wrote in the issue. If that's the case, please refrain from posting such redundant and verbose comments in the future. If that's not the case, then I am sorry ;)

I see that you opened two PRs:

- #30326
- #30329

I closed the first one because it felt subsumed by #30329.

When opening PRs, please refer to the issue they are related to in the description to make it easier to discover them and avoid having several people concurrently working on the same issue.

Labels: Documentation