Talip7/scikit-learn-issues-multilabel huggingface dataset:
Add "scoring" argument to estimator's ``score`` method

### Describe the workflow you want to enable

I want to enable non-accuracy metrics to ``estimator.score``, and ultimately deprecate the default values of ``accuracy`` and ``r2``. I would call it ``scoring`` though it's a bit redundant but consistent.
That would allow us to get rid of the default scoring methods, which are objectively bad and misleading, and it would require the minimum amount of code changes for anyone. 

### Describe your proposed solution

Replace

```python
est.score(X, y)
```
with
```python
est.score(X, y, scoring="accuracy")
```
or rather
```python
est.score(X, y, scoring="recall_macro")
```
(or ``r2`` for regression).

### Describe alternatives you've considered, if relevant

- Keep current status, which is bad (both ``accuracy`` and ``r2`` are bad)
- Remove ``scoring`` method.

I can't think of any other alternatives tbh.

### Additional context

I think in theory this requires a slep, as it's changing shared API, right?

COMMENT:
Thanks for the work you've done so far. The goal of this comment
is to set expectations.

Deciding on new features or substantial changes is a lengthy
process. It frequently happens that no maintainer is available
to take on this task right now.

Please do not create a Pull Request before a decision has been
made regarding the proposed work. Making this decision can
often take a significant amount of time and effort.

Labels: New Feature, API, Needs Decision, RFC, module:metrics
Add "scoring" argument to estimator's ``score`` method

### Describe the workflow you want to enable

I want to enable non-accuracy metrics to ``estimator.score``, and ultimately deprecate the default values of ``accuracy`` and ``r2``. I would call it ``scoring`` though it's a bit redundant but consistent.
That would allow us to get rid of the default scoring methods, which are objectively bad and misleading, and it would require the minimum amount of code changes for anyone. 

### Describe your proposed solution

Replace

```python
est.score(X, y)
```
with
```python
est.score(X, y, scoring="accuracy")
```
or rather
```python
est.score(X, y, scoring="recall_macro")
```
(or ``r2`` for regression).

### Describe alternatives you've considered, if relevant

- Keep current status, which is bad (both ``accuracy`` and ``r2`` are bad)
- Remove ``scoring`` method.

I can't think of any other alternatives tbh.

### Additional context

I think in theory this requires a slep, as it's changing shared API, right?

COMMENT:
Added the "needs decision" to see if this helps reduce the need to repeat ourselves about "can I make a PR for this?"

I think I prefer option (1). Though I think we should have a reasonable default value for it, which might be different from the current default for `score()`. And leave the door open for enabling the idea from https://github.com/scikit-learn/scikit-learn/issues/28995#issuecomment-2122718356 - but we don't have to solve that now as well, we just need to leave enough room for doing it in the future.

Labels: New Feature, API, Needs Decision, RFC, module:metrics
Add "scoring" argument to estimator's ``score`` method

### Describe the workflow you want to enable

I want to enable non-accuracy metrics to ``estimator.score``, and ultimately deprecate the default values of ``accuracy`` and ``r2``. I would call it ``scoring`` though it's a bit redundant but consistent.
That would allow us to get rid of the default scoring methods, which are objectively bad and misleading, and it would require the minimum amount of code changes for anyone. 

### Describe your proposed solution

Replace

```python
est.score(X, y)
```
with
```python
est.score(X, y, scoring="accuracy")
```
or rather
```python
est.score(X, y, scoring="recall_macro")
```
(or ``r2`` for regression).

### Describe alternatives you've considered, if relevant

- Keep current status, which is bad (both ``accuracy`` and ``r2`` are bad)
- Remove ``scoring`` method.

I can't think of any other alternatives tbh.

### Additional context

I think in theory this requires a slep, as it's changing shared API, right?

COMMENT:
I opened SLEP 25 for it: https://github.com/scikit-learn/enhancement_proposals/pull/96.

Labels: New Feature, API, Needs Decision, RFC, module:metrics
What about negative coefficients / feature weights?

### Describe the issue linked to the documentation

https://scikit-learn.org/stable/auto_examples/text/plot_document_classification_20newsgroups.html#model-without-metadata-stripping

In this example, in the code for the function `plot_feature_effects` it sorts the weights and takes the top 5, but shouldn't it first absolute the coefficients since they can be negative too and a negative weight is important?

### Suggest a potential alternative/fix

Absolute the weights first

COMMENT:
Hey @fpt-ian ,
It is absolutely correct to say that negative coef_ will create wrong interpretations.
Please confirm do you want follow the only method to absolute the coef and compute the average, because in this approach we are gaining the right coef_ but losing its direction.

So, better we provide color coding them, after absoluting the feature coefficients and plot the colors in the graph.

If we don't want to be dependent on feature, we can use random forest feature importance but would take lot of computation on the other hand.

Therefore, please suggest which option you would want to go with...

Labels: Documentation, Needs Triage
What about negative coefficients / feature weights?

### Describe the issue linked to the documentation

https://scikit-learn.org/stable/auto_examples/text/plot_document_classification_20newsgroups.html#model-without-metadata-stripping

In this example, in the code for the function `plot_feature_effects` it sorts the weights and takes the top 5, but shouldn't it first absolute the coefficients since they can be negative too and a negative weight is important?

### Suggest a potential alternative/fix

Absolute the weights first

COMMENT:
![image](https://github.com/scikit-learn/scikit-learn/assets/148944562/d4f5f0ed-4b3e-4b33-a46f-c6c5ef596c19)
Looking at this section of the documentation, I can see that some words are prominent negative weights.

My understand is that negative weights mean that feature _reduces_ the probability of that class being chosen as the output. 
Or, in RidgeClassification are the weights simply heavy (negative or positive) if they improve the the probability of that class being chosen as the output.
Just want to clarify my understanding.

Labels: Documentation, Needs Triage
What about negative coefficients / feature weights?

### Describe the issue linked to the documentation

https://scikit-learn.org/stable/auto_examples/text/plot_document_classification_20newsgroups.html#model-without-metadata-stripping

In this example, in the code for the function `plot_feature_effects` it sorts the weights and takes the top 5, but shouldn't it first absolute the coefficients since they can be negative too and a negative weight is important?

### Suggest a potential alternative/fix

Absolute the weights first

COMMENT:
> In this example, in the code for the function plot_feature_effects it sorts the weights and takes the top 5, but shouldn't it first absolute the coefficients since they can be negative too and a negative weight is important?

Negative weights can be important (as seen on the plot), however, it is also interesting to see what words/tokens are most positively associated with a given class. Based on the plot it seems that the negative weights are mostly useful to contrast even more positive weights for the same token w.r.t. competing classes: for each word, there is at least one class with a large positive weights and the negative weights for the other classes are often smaller.

> My understand is that negative weights mean that feature reduces the probability of that class being chosen as the output.

This is correct but I don't see how the way the example is written would lead to think otherwise.

I think the example is good the way it is but let me cc @ArturoAmorQ.

Labels: Documentation, Needs Triage
What about negative coefficients / feature weights?

### Describe the issue linked to the documentation

https://scikit-learn.org/stable/auto_examples/text/plot_document_classification_20newsgroups.html#model-without-metadata-stripping

In this example, in the code for the function `plot_feature_effects` it sorts the weights and takes the top 5, but shouldn't it first absolute the coefficients since they can be negative too and a negative weight is important?

### Suggest a potential alternative/fix

Absolute the weights first

COMMENT:
I would argue that examining only the positive coefficients to determine the predictiveness of a token is often sufficient.

In a binary classification, a positive weight for a feature indicates its predictive value for a particular class, whereas a negative weight suggests its predictiveness for the opposite class. Thus, the information provided by the negative weights is already captured by the positive weights in such settings.

In multiclass scenarios, focusing on positive weights is more informative. This is because a positive association between a feature and a specific class directly indicates relevance, while a negative weight merely suggests non-relevance to that particular class without specifying which of the other classes it might correspond to.

Take the word "space" from the plot in https://github.com/scikit-learn/scikit-learn/issues/28985#issuecomment-2103689458, for instance. Imagine you only know it has a large negative value for class "computer graphics". Then you don't have any implication on the weights for the other classes. Not even their sign.

As the text of the [example](https://scikit-learn.org/stable/auto_examples/text/plot_document_classification_20newsgroups.html) reads, the only affirmation we can make is that:

> [...] the most predictive words are often strongly positively associated with a single class and negatively associated with all the other classes.

Labels: Documentation, Needs Triage
What about negative coefficients / feature weights?

### Describe the issue linked to the documentation

https://scikit-learn.org/stable/auto_examples/text/plot_document_classification_20newsgroups.html#model-without-metadata-stripping

In this example, in the code for the function `plot_feature_effects` it sorts the weights and takes the top 5, but shouldn't it first absolute the coefficients since they can be negative too and a negative weight is important?

### Suggest a potential alternative/fix

Absolute the weights first

COMMENT:
> I think the example is good the way it is.

In conclusion, I agree it is.

Labels: Documentation, Needs Triage
What about negative coefficients / feature weights?

### Describe the issue linked to the documentation

https://scikit-learn.org/stable/auto_examples/text/plot_document_classification_20newsgroups.html#model-without-metadata-stripping

In this example, in the code for the function `plot_feature_effects` it sorts the weights and takes the top 5, but shouldn't it first absolute the coefficients since they can be negative too and a negative weight is important?

### Suggest a potential alternative/fix

Absolute the weights first

COMMENT:
I still didn't get it...plz clarify while the model learns weights, can we conclude that negative weights depict a weak relationship because that's what we are plotting?

There are two interpretations of negative_coef:

1. They are shown as having weak relationships with a particular class
2. They are shown as strong inverse correlated relationship

Hey @ArturoAmorQ , Please confirm what negative coefficients actually signifying

Labels: Documentation, Needs Triage
What about negative coefficients / feature weights?

### Describe the issue linked to the documentation

https://scikit-learn.org/stable/auto_examples/text/plot_document_classification_20newsgroups.html#model-without-metadata-stripping

In this example, in the code for the function `plot_feature_effects` it sorts the weights and takes the top 5, but shouldn't it first absolute the coefficients since they can be negative too and a negative weight is important?

### Suggest a potential alternative/fix

Absolute the weights first

COMMENT:
Hey, @fpt-ian I think, the plot_feature_effects function in the example is designed to visualize the impact of changing a feature's value on the classification decision. By keeping the coefficients as they are (positive or negative), the function can accurately represent how increasing or decreasing the value of a feature affects the likelihood of a document being classified into a particular category.

If you were to take the absolute value of the coefficients, you would lose this directional information and the resulting visualization might not accurately represent the impact of each feature on the classification decision.

Labels: Documentation, Needs Triage
What about negative coefficients / feature weights?

### Describe the issue linked to the documentation

https://scikit-learn.org/stable/auto_examples/text/plot_document_classification_20newsgroups.html#model-without-metadata-stripping

In this example, in the code for the function `plot_feature_effects` it sorts the weights and takes the top 5, but shouldn't it first absolute the coefficients since they can be negative too and a negative weight is important?

### Suggest a potential alternative/fix

Absolute the weights first

COMMENT:
Hey @PragyanTiwari 

As positive coefficients indicate that the presence of that word increases the likelihood of a document belonging to a certain class, while negative coefficients suggest the opposite.

So, I think negative coefficients would also mean that when certain words are present, it makes the model less likely to think the document belongs to a certain category. It's like these words are saying, "No, this document doesn't fit here."

Labels: Documentation, Needs Triage
What about negative coefficients / feature weights?

### Describe the issue linked to the documentation

https://scikit-learn.org/stable/auto_examples/text/plot_document_classification_20newsgroups.html#model-without-metadata-stripping

In this example, in the code for the function `plot_feature_effects` it sorts the weights and takes the top 5, but shouldn't it first absolute the coefficients since they can be negative too and a negative weight is important?

### Suggest a potential alternative/fix

Absolute the weights first

COMMENT:
Hi everyone, thanks for the discussion. I understand better now. I also found this document specifically on interpreting linear models https://scikit-learn.org/stable/auto_examples/inspection/plot_linear_model_coefficient_interpretation.html

I am OK with the positive weights being the main focus on the above, but I guess secondarily I want to check, are the weights between classes normalized? In the document linked above it talks about using StandardScaler to ensure weights of the coefficient are comparable, but in the sample (of this issue) the it does not do any scaling, and thus the results are not really comparable in terms of weights. It this taken care of by the 
`# learned coefficients weighted by frequency of appearance` section?

If so then I have no more concerns.

Labels: Documentation, Needs Triage
What about negative coefficients / feature weights?

### Describe the issue linked to the documentation

https://scikit-learn.org/stable/auto_examples/text/plot_document_classification_20newsgroups.html#model-without-metadata-stripping

In this example, in the code for the function `plot_feature_effects` it sorts the weights and takes the top 5, but shouldn't it first absolute the coefficients since they can be negative too and a negative weight is important?

### Suggest a potential alternative/fix

Absolute the weights first

COMMENT:
> It this taken care of by the `# learned coefficients weighted by frequency of appearance` section?

Indeed, it is taken into account by multiplying by the mean appearance of the terms. Here, we don't do any scaling in terms of variance because every feature was originally on the same unit (word counts).

I therefore close the issue.

Labels: Documentation, Needs Triage
Saving and loading calibratedclassifierCV model (ensemble)

### Describe the bug

Unable to load the saved calibratedclassifierCV model to a pickle file (.pkl) trained with cv=n
as that is a list of models

### Steps/Code to Reproduce

calibratedclassifier.dump('model.pkl')
model= pickle.load('model.pkl')

### Expected Results

Expected results - model object loaded

### Actual Results

Attribute error: _CalibratedClassifier has no attribute 'estimator'.

### Versions

```shell
1.1.1
```

COMMENT:
@ankitaash, could you share complete code. I dont think so calibratedclassifier was saved using dump method. Here is a working code for me:

```
import pickle

filename = 'calibrated_classifier.pkl'
pickle.dump(calibrated_clf, open(filename, 'wb'))

loaded_classifier = pickle.load(open(filename, 'rb')) 
```

Labels: Bug, Needs Triage
Saving and loading calibratedclassifierCV model (ensemble)

### Describe the bug

Unable to load the saved calibratedclassifierCV model to a pickle file (.pkl) trained with cv=n
as that is a list of models

### Steps/Code to Reproduce

calibratedclassifier.dump('model.pkl')
model= pickle.load('model.pkl')

### Expected Results

Expected results - model object loaded

### Actual Results

Attribute error: _CalibratedClassifier has no attribute 'estimator'.

### Versions

```shell
1.1.1
```

COMMENT:
Hey @ankitaash
Did you fit the model with the training dataset?? Maybe that's why it is showing the error.

Labels: Bug, Needs Triage
Saving and loading calibratedclassifierCV model (ensemble)

### Describe the bug

Unable to load the saved calibratedclassifierCV model to a pickle file (.pkl) trained with cv=n
as that is a list of models

### Steps/Code to Reproduce

calibratedclassifier.dump('model.pkl')
model= pickle.load('model.pkl')

### Expected Results

Expected results - model object loaded

### Actual Results

Attribute error: _CalibratedClassifier has no attribute 'estimator'.

### Versions

```shell
1.1.1
```

COMMENT:
I'm closing this issue because I'm 99% sure that the pickling and unpickling is not done with the same scikit-learn version and thus some internal of the `CalibratedClassifierCV` changed in between. You need to use the exact same scikit-learn version for pickling and unpickling.

You can refer to this documentation page: https://scikit-learn.org/dev/model_persistence.html

@ankitaash if this is not the case and you have a minimal reproducer, feel free to share it and I'll reopen the issue.

Labels: Bug, Needs Triage
Add zero_division for single class prediction in MCC

### Describe the bug

I have found a potential edge case issue with _sklearn.metrics.matthews_corrcoef_. The example provided in the documentation works as expected:

```python
from sklearn.metrics import matthews_corrcoef
matthews_corrcoef([1, 1, 1, -1], [1, -1, 1, 1])  # returns -1/3 OK
```

However, edge cases appear when either _y_ or _y_true_ have only one single label.

### Steps/Code to Reproduce

```python
from sklearn.metrics import matthews_corrcoef
matthews_corrcoef([1, 1, 1, 1], [1, 1, 1, 1])  # returns 0 instead of 1
matthews_corrcoef([0, 0, 0, 0], [0, 0, 0, 0])  # returns 0 instead of 1
```

### Expected Results

Outputs should be 1, not 0.

### Actual Results

Outputs are 0.

### Versions

```shell
System:
    python: 3.11.9 (main, Apr  2 2024, 08:25:04) [Clang 16.0.6 ]
executable: /tmp/sklearn-test/.venv/bin/python
   machine: macOS-14.2.1-arm64-arm-64bit

Python dependencies:
      sklearn: 1.4.2
          pip: 24.0
   setuptools: 69.5.1
        numpy: 1.26.4
        scipy: 1.13.0
       Cython: None
       pandas: None
   matplotlib: None
       joblib: 1.4.2
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
    num_threads: 8
         prefix: libomp
       filepath: /tmp/sklearn-test/.venv/lib/python3.11/site-packages/torch/lib/libomp.dylib
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 8
         prefix: libopenblas
       filepath: /tmp/sklearn-test/.venv/lib/python3.11/site-packages/numpy/.dylibs/libopenblas64_.0.dylib
        version: 0.3.23.dev
threading_layer: pthreads
   architecture: armv8

       user_api: blas
   internal_api: openblas
    num_threads: 8
         prefix: libopenblas
       filepath: /tmp/sklearn-test/.venv/lib/python3.11/site-packages/scipy/.dylibs/libopenblas.0.dylib
        version: 0.3.26.dev
threading_layer: pthreads
   architecture: neoversen1

       user_api: openmp
   internal_api: openmp
    num_threads: 8
         prefix: libomp
       filepath: /tmp/sklearn-test/.venv/lib/python3.11/site-packages/sklearn/.dylibs/libomp.dylib
        version: None
```

COMMENT:
Basically the solution is to provide a `zero_division` parameter to have a better handling here because this is a corner case: https://github.com/scikit-learn/scikit-learn/pull/28509

Labels: Enhancement
Add zero_division for single class prediction in MCC

### Describe the bug

I have found a potential edge case issue with _sklearn.metrics.matthews_corrcoef_. The example provided in the documentation works as expected:

```python
from sklearn.metrics import matthews_corrcoef
matthews_corrcoef([1, 1, 1, -1], [1, -1, 1, 1])  # returns -1/3 OK
```

However, edge cases appear when either _y_ or _y_true_ have only one single label.

### Steps/Code to Reproduce

```python
from sklearn.metrics import matthews_corrcoef
matthews_corrcoef([1, 1, 1, 1], [1, 1, 1, 1])  # returns 0 instead of 1
matthews_corrcoef([0, 0, 0, 0], [0, 0, 0, 0])  # returns 0 instead of 1
```

### Expected Results

Outputs should be 1, not 0.

### Actual Results

Outputs are 0.

### Versions

```shell
System:
    python: 3.11.9 (main, Apr  2 2024, 08:25:04) [Clang 16.0.6 ]
executable: /tmp/sklearn-test/.venv/bin/python
   machine: macOS-14.2.1-arm64-arm-64bit

Python dependencies:
      sklearn: 1.4.2
          pip: 24.0
   setuptools: 69.5.1
        numpy: 1.26.4
        scipy: 1.13.0
       Cython: None
       pandas: None
   matplotlib: None
       joblib: 1.4.2
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
    num_threads: 8
         prefix: libomp
       filepath: /tmp/sklearn-test/.venv/lib/python3.11/site-packages/torch/lib/libomp.dylib
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 8
         prefix: libopenblas
       filepath: /tmp/sklearn-test/.venv/lib/python3.11/site-packages/numpy/.dylibs/libopenblas64_.0.dylib
        version: 0.3.23.dev
threading_layer: pthreads
   architecture: armv8

       user_api: blas
   internal_api: openblas
    num_threads: 8
         prefix: libopenblas
       filepath: /tmp/sklearn-test/.venv/lib/python3.11/site-packages/scipy/.dylibs/libopenblas.0.dylib
        version: 0.3.26.dev
threading_layer: pthreads
   architecture: neoversen1

       user_api: openmp
   internal_api: openmp
    num_threads: 8
         prefix: libomp
       filepath: /tmp/sklearn-test/.venv/lib/python3.11/site-packages/sklearn/.dylibs/libomp.dylib
        version: None
```

COMMENT:
/take

Labels: Enhancement