Talip7/scikit-learn-issues-multilabel huggingface dataset:
Visual improvements for ROC and precision-recall plots

### Describe the workflow you want to enable

Hello,

I would like to suggest the following improvements:

- [x] (1) Add x and y axis limit: [0, 1], in sklearn axes currently start at ~-0.1 (#26366)
- [x] (2) Modify the plotting frame: either remove the top and right lines to see the curve better when values are close to 1, or plot the frame with a dotted line (#26367)
- [x] (3). Fix aspect ratio to squared, since the two axes are the same scale. (#26366)
- [x] (4). Add title (not planned)

### Describe your proposed solution

Hello,

While working on ROC and precision-recall plots, I came up with the following suggestions:

1. Add x and y axis limit: [0, 1], in sklearn axes currently start at ~-0.1
2. Modify the plotting frame: either remove the top and right lines to see the curve better when values are close to 1, or plot the frame with a dotted line
3. Fix aspect ratio to squared, since the two axes are the same scale.
4. Add title

```
        ax.set_xlim((-0.01, 1.01))
        ax.set_ylim((-0.01, 1.01))
        for s in ['right', 'left', 'top', 'bottom']:
            ax.spines[s].set_visible(False)
        ax.axvline(1, color='black', linewidth=0.5, ls='dotted')
        ax.axhline(1, color='black', linewidth=0.5, ls='dotted')
        ax.axvline(0, color='black', linewidth=0.5, ls='dotted')
        ax.axhline(0, color='black', linewidth=0.5, ls='dotted')
        ax.set_aspect(1)
        ax.set_title(f'Precision-Recall Curve')
```

### Example 1 Edge case
#### PR
Before:
![image](https://user-images.githubusercontent.com/1184396/226746221-40b40d56-74a4-49c1-82aa-4a3f3366abe2.png)

After:
![image](https://user-images.githubusercontent.com/1184396/226746245-3e0a0ffd-28f0-4851-8a41-04629750bd2c.png)

#### ROC
Before:
![image](https://user-images.githubusercontent.com/1184396/226746270-ecc2a1ce-48dc-4682-8acd-86a4d06dfcf0.png)

After:
![image](https://user-images.githubusercontent.com/1184396/226746289-8ba8c906-f8f8-4d74-9269-b12c16d339fa.png)

### Example 2 - Edge case

#### PR
Before:
![image](https://user-images.githubusercontent.com/1184396/226745897-de69cf76-f520-499f-b8b6-4e9ede9e7ba2.png)

After:
![image](https://user-images.githubusercontent.com/1184396/226745936-5fc34fff-eff3-4a70-8b99-b7c731984723.png)

#### ROC
Before:
![image](https://user-images.githubusercontent.com/1184396/226745998-d1788d4a-fd0a-4293-bef5-170a71c0a098.png)

After:
![image](https://user-images.githubusercontent.com/1184396/226746031-e93dff40-2487-4776-a5b8-baeef9a36e5d.png)



### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Also the last vertical segment of the PR curve seems to be missing with your implementation of Davis & Goadrich.

I think I would be in favor of adding an explicit option to chose among linear and constant piecewise interpolation. I am not sure if the extra complexity of the Davis & Goadrich method is justified though.

Labels: Enhancement
Visual improvements for ROC and precision-recall plots

### Describe the workflow you want to enable

Hello,

I would like to suggest the following improvements:

- [x] (1) Add x and y axis limit: [0, 1], in sklearn axes currently start at ~-0.1 (#26366)
- [x] (2) Modify the plotting frame: either remove the top and right lines to see the curve better when values are close to 1, or plot the frame with a dotted line (#26367)
- [x] (3). Fix aspect ratio to squared, since the two axes are the same scale. (#26366)
- [x] (4). Add title (not planned)

### Describe your proposed solution

Hello,

While working on ROC and precision-recall plots, I came up with the following suggestions:

1. Add x and y axis limit: [0, 1], in sklearn axes currently start at ~-0.1
2. Modify the plotting frame: either remove the top and right lines to see the curve better when values are close to 1, or plot the frame with a dotted line
3. Fix aspect ratio to squared, since the two axes are the same scale.
4. Add title

```
        ax.set_xlim((-0.01, 1.01))
        ax.set_ylim((-0.01, 1.01))
        for s in ['right', 'left', 'top', 'bottom']:
            ax.spines[s].set_visible(False)
        ax.axvline(1, color='black', linewidth=0.5, ls='dotted')
        ax.axhline(1, color='black', linewidth=0.5, ls='dotted')
        ax.axvline(0, color='black', linewidth=0.5, ls='dotted')
        ax.axhline(0, color='black', linewidth=0.5, ls='dotted')
        ax.set_aspect(1)
        ax.set_title(f'Precision-Recall Curve')
```

### Example 1 Edge case
#### PR
Before:
![image](https://user-images.githubusercontent.com/1184396/226746221-40b40d56-74a4-49c1-82aa-4a3f3366abe2.png)

After:
![image](https://user-images.githubusercontent.com/1184396/226746245-3e0a0ffd-28f0-4851-8a41-04629750bd2c.png)

#### ROC
Before:
![image](https://user-images.githubusercontent.com/1184396/226746270-ecc2a1ce-48dc-4682-8acd-86a4d06dfcf0.png)

After:
![image](https://user-images.githubusercontent.com/1184396/226746289-8ba8c906-f8f8-4d74-9269-b12c16d339fa.png)

### Example 2 - Edge case

#### PR
Before:
![image](https://user-images.githubusercontent.com/1184396/226745897-de69cf76-f520-499f-b8b6-4e9ede9e7ba2.png)

After:
![image](https://user-images.githubusercontent.com/1184396/226745936-5fc34fff-eff3-4a70-8b99-b7c731984723.png)

#### ROC
Before:
![image](https://user-images.githubusercontent.com/1184396/226745998-d1788d4a-fd0a-4293-bef5-170a71c0a098.png)

After:
![image](https://user-images.githubusercontent.com/1184396/226746031-e93dff40-2487-4776-a5b8-baeef9a36e5d.png)



### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
+1 on @kno10's comment: PR analysis is often specifically suggested for the strongly class-imbalanced case, where there may be very few positives, and the choice of the interpolation scheme makes a _huge_ difference in terms of the resulting AUPRC.

Labels: Enhancement
average_precision_score() and roc_auc_score() require a full list of arguments

### Describe the bug

These two functions, namely average_precision_score() and roc_auc_score() require a full list of arguments. Otherwise, it reports errors!

### Steps/Code to Reproduce
```
import numpy as np
from sklearn.metrics import average_precision_score
y_true = np.array([0, 0, 1, 1])
y_scores = np.array([0.1, 0.4, 0.35, 0.8])
average_precision_score(y_true, y_scores)
```
### Expected Results
```
0.83...
```
### Actual Results
```
Traceback (most recent call last):

  File "<ipython-input-211-0b5f443434a6>", line 5, in <module>
    average_precision_score(y_true, y_scores)

TypeError: average_precision_score() missing 3 required keyword-only arguments: 'average', 'pos_label', and 'sample_weight'
```
### Versions

```shell
System:
    python: 3.8.5 (default, Sep  4 2020, 02:22:02)  [Clang 10.0.0 ]
executable: /opt/anaconda3/bin/python
   machine: macOS-10.16-x86_64-i386-64bit

Python dependencies:
      sklearn: 0.22
          pip: 20.2.4
   setuptools: 50.3.1.post20201107
        numpy: 1.22.4
        scipy: 1.10.1
       Cython: 0.29.21
       pandas: 1.3.4
   matplotlib: 3.5.0
       joblib: 0.17.0
threadpoolctl: 2.1.0

Built with OpenMP: True
```

COMMENT:
Keyword-only arguments were not specified in 0.22, here is the code from the 0.22.X branch:

https://github.com/scikit-learn/scikit-learn/blob/95d4f0841d57e8b5f6b2a570312e9d832e69debc/sklearn/metrics/_ranking.py#L104-L105

On `main`, the code snippet works for me. I also tried it on scikit-learn 0.24 on [Google Colab](https://colab.research.google.com/drive/182qTZDI8FQNmnuUX1TSCXPQOJ4XqUeFA#scrollTo=VMhY1qbWqgXM) and it also works. I suspect the issue is the installation Python. Can you try creating a new environment and installing the latest version of scikit-learn?

Labels: Bug
average_precision_score() and roc_auc_score() require a full list of arguments

### Describe the bug

These two functions, namely average_precision_score() and roc_auc_score() require a full list of arguments. Otherwise, it reports errors!

### Steps/Code to Reproduce
```
import numpy as np
from sklearn.metrics import average_precision_score
y_true = np.array([0, 0, 1, 1])
y_scores = np.array([0.1, 0.4, 0.35, 0.8])
average_precision_score(y_true, y_scores)
```
### Expected Results
```
0.83...
```
### Actual Results
```
Traceback (most recent call last):

  File "<ipython-input-211-0b5f443434a6>", line 5, in <module>
    average_precision_score(y_true, y_scores)

TypeError: average_precision_score() missing 3 required keyword-only arguments: 'average', 'pos_label', and 'sample_weight'
```
### Versions

```shell
System:
    python: 3.8.5 (default, Sep  4 2020, 02:22:02)  [Clang 10.0.0 ]
executable: /opt/anaconda3/bin/python
   machine: macOS-10.16-x86_64-i386-64bit

Python dependencies:
      sklearn: 0.22
          pip: 20.2.4
   setuptools: 50.3.1.post20201107
        numpy: 1.22.4
        scipy: 1.10.1
       Cython: 0.29.21
       pandas: 1.3.4
   matplotlib: 3.5.0
       joblib: 0.17.0
threadpoolctl: 2.1.0

Built with OpenMP: True
```

COMMENT:
Closing due to inactivity, @ZhongLIFR please comment if you're still having this issue.

Labels: Bug
Use sample_weight when validating LogisticRegressionCV

Metadata Routing https://github.com/scikit-learn/scikit-learn/pull/24027 will add much needed support for taking into account sample_weight when cross-validating. However, the current implementation of LogisticRegressionCV doesn't seem to be taking advantage of this:

https://github.com/scikit-learn/scikit-learn/blob/63e364aec8d5e9f39a9d511b0bc2101cd075dc39/sklearn/linear_model/_logistic.py#L778

Therefore, the scores used for choosing the correct hyperparameter will still be misleading even when the tools for solving this become available.

COMMENT:
During the triaging meeting, we decided that this a bug and that `sample_weights` should be passed into the scoring. There is SLEP006 work moving in this direction: https://github.com/scikit-learn/scikit-learn/pull/24498

But I can see passing in the sample weights as a fix before https://github.com/scikit-learn/scikit-learn/pull/24498 is completed.

Labels: Bug, module:linear_model
Use sample_weight when validating LogisticRegressionCV

Metadata Routing https://github.com/scikit-learn/scikit-learn/pull/24027 will add much needed support for taking into account sample_weight when cross-validating. However, the current implementation of LogisticRegressionCV doesn't seem to be taking advantage of this:

https://github.com/scikit-learn/scikit-learn/blob/63e364aec8d5e9f39a9d511b0bc2101cd075dc39/sklearn/linear_model/_logistic.py#L778

Therefore, the scores used for choosing the correct hyperparameter will still be misleading even when the tools for solving this become available.

COMMENT:
I don't think this is a _bug_ per say that we can/should fix in the current state of affairs. The result of `LogisticRegressionCV(...).fit(...)` should be more or less the same as `GridSearchCV(LogisticRegression(), ...).fit(...)`, and we don't have a way of passing `sample_weight` to the scorer in `*SearchCV` classes w/o SLEP006.

Labels: Bug, module:linear_model
Use sample_weight when validating LogisticRegressionCV

Metadata Routing https://github.com/scikit-learn/scikit-learn/pull/24027 will add much needed support for taking into account sample_weight when cross-validating. However, the current implementation of LogisticRegressionCV doesn't seem to be taking advantage of this:

https://github.com/scikit-learn/scikit-learn/blob/63e364aec8d5e9f39a9d511b0bc2101cd075dc39/sklearn/linear_model/_logistic.py#L778

Therefore, the scores used for choosing the correct hyperparameter will still be misleading even when the tools for solving this become available.

COMMENT:
As a user, for me it is also clear that it is a bug and I would appreciate it if it could be fixed sooner. However, I understand that this is part of a much deeper issue with lots of factors to consider. Knowing that there is work moving in that direction #24498 as part of SLEP006 is comforting enough.

Labels: Bug, module:linear_model
Use sample_weight when validating LogisticRegressionCV

Metadata Routing https://github.com/scikit-learn/scikit-learn/pull/24027 will add much needed support for taking into account sample_weight when cross-validating. However, the current implementation of LogisticRegressionCV doesn't seem to be taking advantage of this:

https://github.com/scikit-learn/scikit-learn/blob/63e364aec8d5e9f39a9d511b0bc2101cd075dc39/sklearn/linear_model/_logistic.py#L778

Therefore, the scores used for choosing the correct hyperparameter will still be misleading even when the tools for solving this become available.

COMMENT:
I have the feeling that both `LogisticRegressionCV` and `GridSearchCV` have the same bug: I think that by default scikit-learn should always score models with nested CV with the weights passed to the outer fit.

Labels: Bug, module:linear_model
Use sample_weight when validating LogisticRegressionCV

Metadata Routing https://github.com/scikit-learn/scikit-learn/pull/24027 will add much needed support for taking into account sample_weight when cross-validating. However, the current implementation of LogisticRegressionCV doesn't seem to be taking advantage of this:

https://github.com/scikit-learn/scikit-learn/blob/63e364aec8d5e9f39a9d511b0bc2101cd075dc39/sklearn/linear_model/_logistic.py#L778

Therefore, the scores used for choosing the correct hyperparameter will still be misleading even when the tools for solving this become available.

COMMENT:
only pass `sample_weight`? or any metadata passed which somehow matches the name present in the signature of a scorer? also, this would be a breaking change. People know scorers don't get the metadata in a gridsearch and have hacks around it. Suddenly passing things around would break things quite a bit. ref: https://stackoverflow.com/questions/49581104/sklearn-gridsearchcv-not-using-sample-weight-in-score-function/49598597#49598597

Labels: Bug, module:linear_model
Use sample_weight when validating LogisticRegressionCV

Metadata Routing https://github.com/scikit-learn/scikit-learn/pull/24027 will add much needed support for taking into account sample_weight when cross-validating. However, the current implementation of LogisticRegressionCV doesn't seem to be taking advantage of this:

https://github.com/scikit-learn/scikit-learn/blob/63e364aec8d5e9f39a9d511b0bc2101cd075dc39/sklearn/linear_model/_logistic.py#L778

Therefore, the scores used for choosing the correct hyperparameter will still be misleading even when the tools for solving this become available.

COMMENT:
We could special case `sample_weight`. For the other future metadata, I don't know ahead of time what is the good practice, and for those, maybe explicit routing is better.

Fixing only the `sample_weight` case would be a bug fix, not a breaking change.

Labels: Bug, module:linear_model
Use sample_weight when validating LogisticRegressionCV

Metadata Routing https://github.com/scikit-learn/scikit-learn/pull/24027 will add much needed support for taking into account sample_weight when cross-validating. However, the current implementation of LogisticRegressionCV doesn't seem to be taking advantage of this:

https://github.com/scikit-learn/scikit-learn/blob/63e364aec8d5e9f39a9d511b0bc2101cd075dc39/sklearn/linear_model/_logistic.py#L778

Therefore, the scores used for choosing the correct hyperparameter will still be misleading even when the tools for solving this become available.

COMMENT:
> only pass `sample_weight`? or any metadata passed which somehow matches the name present in the signature of a scorer? also, this would be a breaking change. People know scorers don't get the metadata in a gridsearch and have hacks around it. Suddenly passing things around would break things quite a bit. ref: https://stackoverflow.com/questions/49581104/sklearn-gridsearchcv-not-using-sample-weight-in-score-function/49598597#49598597

That Stack Overflow discussion reminded me of [this joke](https://github.com/scikit-learn/scikit-learn/issues/4632#issuecomment-854145137).

Labels: Bug, module:linear_model
Use sample_weight when validating LogisticRegressionCV

Metadata Routing https://github.com/scikit-learn/scikit-learn/pull/24027 will add much needed support for taking into account sample_weight when cross-validating. However, the current implementation of LogisticRegressionCV doesn't seem to be taking advantage of this:

https://github.com/scikit-learn/scikit-learn/blob/63e364aec8d5e9f39a9d511b0bc2101cd075dc39/sklearn/linear_model/_logistic.py#L778

Therefore, the scores used for choosing the correct hyperparameter will still be misleading even when the tools for solving this become available.

COMMENT:
so what happens if the user passes `sample_weight`, but neither scorer or the sub-estimator accept it?

what happens when they accept it but through `**kwargs`?

what if a scorer doesn't support sample weight in version x, but does it in version x+1?

what if the sub-estimator doesn't support sample weights, but scorer does?

we have thought about these in the context of slep6, and I don't think the proposal here to pass it always around if we can (which is not well defined), is a good solution really.

Labels: Bug, module:linear_model
Use sample_weight when validating LogisticRegressionCV

Metadata Routing https://github.com/scikit-learn/scikit-learn/pull/24027 will add much needed support for taking into account sample_weight when cross-validating. However, the current implementation of LogisticRegressionCV doesn't seem to be taking advantage of this:

https://github.com/scikit-learn/scikit-learn/blob/63e364aec8d5e9f39a9d511b0bc2101cd075dc39/sklearn/linear_model/_logistic.py#L778

Therefore, the scores used for choosing the correct hyperparameter will still be misleading even when the tools for solving this become available.

COMMENT:
> so what happens if the user passes sample_weight, but neither scorer or the sub-estimator accept it?

we could warn first and raise in a later release.

> what happens when they accept it but through **kwargs?

best effort we pass them and hope for the best :)

> what if a scorer doesn't support sample weight in version x, but does it in version x+1?

warn in old version and then work correctly in new version

> what if the sub-estimator doesn't support sample weights, but scorer does?

raise an error: why would passing sample weight to fit would make sense in this case?

> we have thought about these in the context of slep6, and I don't think the proposal here to pass it always around if we can (which is not well defined), is a good solution really.

What I described above would be the way we should aim for (I think). If we implement this via SLEP6, this should be the default behavior, without explicit requests. If the users configure explicit routing, they can override the default behavior.

Labels: Bug, module:linear_model
Use sample_weight when validating LogisticRegressionCV

Metadata Routing https://github.com/scikit-learn/scikit-learn/pull/24027 will add much needed support for taking into account sample_weight when cross-validating. However, the current implementation of LogisticRegressionCV doesn't seem to be taking advantage of this:

https://github.com/scikit-learn/scikit-learn/blob/63e364aec8d5e9f39a9d511b0bc2101cd075dc39/sklearn/linear_model/_logistic.py#L778

Therefore, the scores used for choosing the correct hyperparameter will still be misleading even when the tools for solving this become available.

COMMENT:
Considering the complications of fixing this w/o SLEP006, is the resolution to fix it in the context of the slep, or to fix it regardless?

Labels: Bug, module:linear_model
Use sample_weight when validating LogisticRegressionCV

Metadata Routing https://github.com/scikit-learn/scikit-learn/pull/24027 will add much needed support for taking into account sample_weight when cross-validating. However, the current implementation of LogisticRegressionCV doesn't seem to be taking advantage of this:

https://github.com/scikit-learn/scikit-learn/blob/63e364aec8d5e9f39a9d511b0bc2101cd075dc39/sklearn/linear_model/_logistic.py#L778

Therefore, the scores used for choosing the correct hyperparameter will still be misleading even when the tools for solving this become available.

COMMENT:
We add a drafting meeting on SLEP6 integration yesterday and we by using the new routing mechanism we will have a way forward to implement the good routing policy for this case with a global setting (that might be the default when SLEP6 is enabled, to be decided yet). See the working proposal here in #26050.

Labels: Bug, module:linear_model
Use sample_weight when validating LogisticRegressionCV

Metadata Routing https://github.com/scikit-learn/scikit-learn/pull/24027 will add much needed support for taking into account sample_weight when cross-validating. However, the current implementation of LogisticRegressionCV doesn't seem to be taking advantage of this:

https://github.com/scikit-learn/scikit-learn/blob/63e364aec8d5e9f39a9d511b0bc2101cd075dc39/sklearn/linear_model/_logistic.py#L778

Therefore, the scores used for choosing the correct hyperparameter will still be misleading even when the tools for solving this become available.

COMMENT:
> Considering the complications of fixing this w/o SLEP006, is the resolution to fix it in the context of the slep, or to fix it regardless?

For `LogisticRegressionCV`, it's probably quite easy to fix it without SLEP6. For `GridSearchCV` and other similar meta-estimators, I don't know.

Labels: Bug, module:linear_model
make_classification: repeated features are not chosen uniformly at random

I have just noticed that this piece of code in `make_classification` which selects features to repeat does not do so uniformly at random:
https://github.com/scikit-learn/scikit-learn/blob/9aaed498795f68e5956ea762fef9c440ca9eb239/sklearn/datasets/_samples_generator.py#L255-L259
It selects the first and last feature with half the probability of the other features, as `(n-1)*...+0.5` ranges from 0.5 to n-0.5, so 0 and n-1 only have half the probability of being selected compared to 1, 2, ..., n-2.  A corrected version would read:
```python
        indices = (n * generator.uniform(size=n_repeated)).astype(np.intp)
```
Making this correction, though, would cause `make_classification` to probably produce different output if `n_repeated > 0`, so I do not know how to fix at this point; I haven't checked whether `scikit-learn` makes any guarantees about reproducibility between versions.

COMMENT:
It took me a while to figure out what we want to do here and why it works/doesn't work.

Now I'm thinking: wouldn't it be easier to use something like `generator.randint(0, n, size=n_repeated)`. It somehow expresses "pick some random ints we can use as indices" much more clearly for me.

I think making this fix would be Ok, even if it changes the output.

What do you think? And do you want to work on a Pull Request?

Labels: Needs Triage