Talip7/scikit-learn-issues-multilabel huggingface dataset:
Enhancement: Add Summary Output for Linear Regression Models

### Describe the workflow you want to enable

While scikit-learn excels in predictive modeling, users often need detailed statistical summaries to interpret their regression results.
I propose we develop options for users wanting comprehensive statistical reports for models such as LinearRegression(), without impacting model performance.  

### Describe your proposed solution

**Modular Design:**
Introduce optional modules or mixins for secondary features.
Users can enable them explicitly when needed.
**Feature Flags:**
Allow users to toggle specific functionalities.
**Lazy Evaluation:**
Compute secondary features only when requested.

### Describe alternatives you've considered, if relevant

While statsmodels provides comprehensive summaries (including p-values!), having an integrated solution within scikit-learn would be valuable. The synergy between the two libraries benefits users seeking both prediction and statistical inference.
Using the existing metrics is inconvenient -- I often find myself copying the same code across projects for printing out all the evaluations. Statisticians would appreciate the full summary output. 

### Additional context

_No response_

COMMENT:
Could you provide the exact feature that you would like to be reported. I think this is more important than the design questions that will be a subsequent question.

Basically, if this is related to reporting of metrics and inspection method, I think that we want to explore something around model cards that should provide such reporting.

If it is related specifically to p-values of the linear models, then I recall that this discussion happened in the past and a decision was made to not include them.

That's why, I think it would be great to know the exact information that you would expect in the report.

Labels: New Feature, Needs Decision - Include Feature
Enhancement: Add Summary Output for Linear Regression Models

### Describe the workflow you want to enable

While scikit-learn excels in predictive modeling, users often need detailed statistical summaries to interpret their regression results.
I propose we develop options for users wanting comprehensive statistical reports for models such as LinearRegression(), without impacting model performance.  

### Describe your proposed solution

**Modular Design:**
Introduce optional modules or mixins for secondary features.
Users can enable them explicitly when needed.
**Feature Flags:**
Allow users to toggle specific functionalities.
**Lazy Evaluation:**
Compute secondary features only when requested.

### Describe alternatives you've considered, if relevant

While statsmodels provides comprehensive summaries (including p-values!), having an integrated solution within scikit-learn would be valuable. The synergy between the two libraries benefits users seeking both prediction and statistical inference.
Using the existing metrics is inconvenient -- I often find myself copying the same code across projects for printing out all the evaluations. Statisticians would appreciate the full summary output. 

### Additional context

_No response_

COMMENT:
x-refs for p-values #16802

Labels: New Feature, Needs Decision - Include Feature
Enhancement: Add Summary Output for Linear Regression Models

### Describe the workflow you want to enable

While scikit-learn excels in predictive modeling, users often need detailed statistical summaries to interpret their regression results.
I propose we develop options for users wanting comprehensive statistical reports for models such as LinearRegression(), without impacting model performance.  

### Describe your proposed solution

**Modular Design:**
Introduce optional modules or mixins for secondary features.
Users can enable them explicitly when needed.
**Feature Flags:**
Allow users to toggle specific functionalities.
**Lazy Evaluation:**
Compute secondary features only when requested.

### Describe alternatives you've considered, if relevant

While statsmodels provides comprehensive summaries (including p-values!), having an integrated solution within scikit-learn would be valuable. The synergy between the two libraries benefits users seeking both prediction and statistical inference.
Using the existing metrics is inconvenient -- I often find myself copying the same code across projects for printing out all the evaluations. Statisticians would appreciate the full summary output. 

### Additional context

_No response_

COMMENT:
this issue was referring to checking diagnostic plots, influence, and more detailed model reports. it is better to create a separate model for analysis to keep sklearn as efficient as possible

Labels: New Feature, Needs Decision - Include Feature
Add "scoring" argument to estimator's ``score`` method

### Describe the workflow you want to enable

I want to enable non-accuracy metrics to ``estimator.score``, and ultimately deprecate the default values of ``accuracy`` and ``r2``. I would call it ``scoring`` though it's a bit redundant but consistent.
That would allow us to get rid of the default scoring methods, which are objectively bad and misleading, and it would require the minimum amount of code changes for anyone. 

### Describe your proposed solution

Replace

```python
est.score(X, y)
```
with
```python
est.score(X, y, scoring="accuracy")
```
or rather
```python
est.score(X, y, scoring="recall_macro")
```
(or ``r2`` for regression).

### Describe alternatives you've considered, if relevant

- Keep current status, which is bad (both ``accuracy`` and ``r2`` are bad)
- Remove ``scoring`` method.

I can't think of any other alternatives tbh.

### Additional context

I think in theory this requires a slep, as it's changing shared API, right?

COMMENT:
Well there is already a separate module for importing metrics..
This score function uses default metric to predict based on ur estimator.


Can't we just make a bridge between the scoring parameter u want and the metrics sklearn module, such that it refers to the metric from sklearn module.

Besides this, there are lot of non accuracy metrics....which one should we focus upon

Labels: New Feature, API, Needs Decision, RFC, module:metrics
Add "scoring" argument to estimator's ``score`` method

### Describe the workflow you want to enable

I want to enable non-accuracy metrics to ``estimator.score``, and ultimately deprecate the default values of ``accuracy`` and ``r2``. I would call it ``scoring`` though it's a bit redundant but consistent.
That would allow us to get rid of the default scoring methods, which are objectively bad and misleading, and it would require the minimum amount of code changes for anyone. 

### Describe your proposed solution

Replace

```python
est.score(X, y)
```
with
```python
est.score(X, y, scoring="accuracy")
```
or rather
```python
est.score(X, y, scoring="recall_macro")
```
(or ``r2`` for regression).

### Describe alternatives you've considered, if relevant

- Keep current status, which is bad (both ``accuracy`` and ``r2`` are bad)
- Remove ``scoring`` method.

I can't think of any other alternatives tbh.

### Additional context

I think in theory this requires a slep, as it's changing shared API, right?

COMMENT:
I agree that current `score` behavior is at best useless, at worst bad.

But IMO, the question is a bit broader: does scikit-learn want to have more syntactic sugar or not?

If yes, then adding the possibility of choosing the metric to the `score` method should be the way to go. If not, removing the `score` method, which will be a breaking change, will ease the codebase and maintenance.

Labels: New Feature, API, Needs Decision, RFC, module:metrics
Add "scoring" argument to estimator's ``score`` method

### Describe the workflow you want to enable

I want to enable non-accuracy metrics to ``estimator.score``, and ultimately deprecate the default values of ``accuracy`` and ``r2``. I would call it ``scoring`` though it's a bit redundant but consistent.
That would allow us to get rid of the default scoring methods, which are objectively bad and misleading, and it would require the minimum amount of code changes for anyone. 

### Describe your proposed solution

Replace

```python
est.score(X, y)
```
with
```python
est.score(X, y, scoring="accuracy")
```
or rather
```python
est.score(X, y, scoring="recall_macro")
```
(or ``r2`` for regression).

### Describe alternatives you've considered, if relevant

- Keep current status, which is bad (both ``accuracy`` and ``r2`` are bad)
- Remove ``scoring`` method.

I can't think of any other alternatives tbh.

### Additional context

I think in theory this requires a slep, as it's changing shared API, right?

COMMENT:
> Can't we just make a bridge between the scoring parameter u want and the metrics sklearn module, such that it refers to the metric from sklearn module.

The [scorers are the bridge](https://scikit-learn.org/stable/modules/model_evaluation.html#common-cases-predefined-values) to `sklearn.metric`. The scorers add additional metadata on top of `sklearn.metrics` that tells the estimator what it needs to compute for a specific metric. For example:

https://github.com/scikit-learn/scikit-learn/blob/5e5cc3477025794c5b2ee6056a223d91adbfe925/sklearn/metrics/_scorer.py#L810-L814

> But IMO, the question is a bit broader: does scikit-learn want to have more syntactic sugar or not?

I'm okay with adding a `scorer` argument `est.score` because it makes it easier to switch between metrics. I think there is a API gap between using `est.score` to a scorer or a metric in `sklearn.metric`. For example, to use the ROC scorer:

```python
# User needs to know about `get_scorer`
from sklearn.metrics import get_scorer

roc_auc_scorer = get_scorer("roc_auc")
roc_auc_scorer(est, X_test, y_test)
```

Or using the metrics directly:

```python
from sklearn.metrics import roc_auc_score

# User needs to know that `roc_auc_score` requires `y_score`:
y_decision = est.decision_function(X_test)
roc_auc_score(y_test, y_decision)

# Of if estimator only has `predict_proba`:
roc_auc_score(y, clf.predict_proba(X)[:, 1])
```

With this proposal, we can write `est.score(X_test, y_test, scoring="roc_auc")`.

Labels: New Feature, API, Needs Decision, RFC, module:metrics
Add "scoring" argument to estimator's ``score`` method

### Describe the workflow you want to enable

I want to enable non-accuracy metrics to ``estimator.score``, and ultimately deprecate the default values of ``accuracy`` and ``r2``. I would call it ``scoring`` though it's a bit redundant but consistent.
That would allow us to get rid of the default scoring methods, which are objectively bad and misleading, and it would require the minimum amount of code changes for anyone. 

### Describe your proposed solution

Replace

```python
est.score(X, y)
```
with
```python
est.score(X, y, scoring="accuracy")
```
or rather
```python
est.score(X, y, scoring="recall_macro")
```
(or ``r2`` for regression).

### Describe alternatives you've considered, if relevant

- Keep current status, which is bad (both ``accuracy`` and ``r2`` are bad)
- Remove ``scoring`` method.

I can't think of any other alternatives tbh.

### Additional context

I think in theory this requires a slep, as it's changing shared API, right?

COMMENT:
I would be in favor of deprecating default scorers.

A few related points:

- Do we let users do something line `estimator.set_scorer("roc_auc")` and use it in `*SearchCV`? I rather not. Prefer to always require users to specify scorer when calling `score`.
- We can start by deprecating `scoring=None` in all search methods/classes.
- Can users pass anything that `check_scoring` accepts to `estimator.score`? I guess yes? While deprecating `_Passthrough` I'd say.

This probably does need a SLEP.

Labels: New Feature, API, Needs Decision, RFC, module:metrics
Add "scoring" argument to estimator's ``score`` method

### Describe the workflow you want to enable

I want to enable non-accuracy metrics to ``estimator.score``, and ultimately deprecate the default values of ``accuracy`` and ``r2``. I would call it ``scoring`` though it's a bit redundant but consistent.
That would allow us to get rid of the default scoring methods, which are objectively bad and misleading, and it would require the minimum amount of code changes for anyone. 

### Describe your proposed solution

Replace

```python
est.score(X, y)
```
with
```python
est.score(X, y, scoring="accuracy")
```
or rather
```python
est.score(X, y, scoring="recall_macro")
```
(or ``r2`` for regression).

### Describe alternatives you've considered, if relevant

- Keep current status, which is bad (both ``accuracy`` and ``r2`` are bad)
- Remove ``scoring`` method.

I can't think of any other alternatives tbh.

### Additional context

I think in theory this requires a slep, as it's changing shared API, right?

COMMENT:
I think it needs a SLEP because it is going to change deeply the API.

Regarding the default scorers, considering the difficulties to select the right metric for each use case, then not having a default might be better.

> Do we let users do something line estimator.set_scorer("roc_auc") and use it in *SearchCV

To me this is an extension of the API. We don't need to settle on this at first. Having the `scoring` parameter everywhere and behaving consistently would already be good.

> Can users pass anything that check_scoring accepts to estimator.score

I would say yes but it means that we need to properly define what is the scorer API. For instance, there already this difference between metric scorer and curve scorer (that does not exist yet but it probably should).

Labels: New Feature, API, Needs Decision, RFC, module:metrics
Add "scoring" argument to estimator's ``score`` method

### Describe the workflow you want to enable

I want to enable non-accuracy metrics to ``estimator.score``, and ultimately deprecate the default values of ``accuracy`` and ``r2``. I would call it ``scoring`` though it's a bit redundant but consistent.
That would allow us to get rid of the default scoring methods, which are objectively bad and misleading, and it would require the minimum amount of code changes for anyone. 

### Describe your proposed solution

Replace

```python
est.score(X, y)
```
with
```python
est.score(X, y, scoring="accuracy")
```
or rather
```python
est.score(X, y, scoring="recall_macro")
```
(or ``r2`` for regression).

### Describe alternatives you've considered, if relevant

- Keep current status, which is bad (both ``accuracy`` and ``r2`` are bad)
- Remove ``scoring`` method.

I can't think of any other alternatives tbh.

### Additional context

I think in theory this requires a slep, as it's changing shared API, right?

COMMENT:
Why call the argument `scoring`? For me something like `scorer` sounds better, maybe just because it isn't a verb :-/

> ```python
> est.score(X, y, scorer="accuracy")
> ```

> ```python
> est.score(X, y, scorer="recall_macro")
> ```

I am -1 on `set_scorer`.

I think providing a default metric as part of an estimator is useful for users. A bit like setting something like the number of trees in a random forest. While ten (I think the current default) is rarely the right value, I think it is useful that step 1 of using `RandomForestClassifier` isn't "think about a (random) value for each of its constructor arguments so that it starts working".

Same with the metric you use to evaluate your setup. You almost always want to think about it and make a deliberate choice, but the fact that there is a default to get you started is a good thing imho.

Labels: New Feature, API, Needs Decision, RFC, module:metrics
Add "scoring" argument to estimator's ``score`` method

### Describe the workflow you want to enable

I want to enable non-accuracy metrics to ``estimator.score``, and ultimately deprecate the default values of ``accuracy`` and ``r2``. I would call it ``scoring`` though it's a bit redundant but consistent.
That would allow us to get rid of the default scoring methods, which are objectively bad and misleading, and it would require the minimum amount of code changes for anyone. 

### Describe your proposed solution

Replace

```python
est.score(X, y)
```
with
```python
est.score(X, y, scoring="accuracy")
```
or rather
```python
est.score(X, y, scoring="recall_macro")
```
(or ``r2`` for regression).

### Describe alternatives you've considered, if relevant

- Keep current status, which is bad (both ``accuracy`` and ``r2`` are bad)
- Remove ``scoring`` method.

I can't think of any other alternatives tbh.

### Additional context

I think in theory this requires a slep, as it's changing shared API, right?

COMMENT:
I agree on not adding ``set_scorer``. Also agree with the rest of what @adrinjalali  suggested.

> I would say yes but it means that we need to properly define what is the scorer API. For instance, there already this difference between metric scorer and curve scorer (that does not exist yet but it probably should).

Can you elaborate on that? I think I'm not familiar with the proposal.

@betatim I suggested ``scoring`` because that's the argument in ``cross_validate`` and ``GridSearchCV`` and they are probably that just because we had ``scorer`` before with a different API and we had to do a deprecation cycle?

> While ten (I think the current default) is rarely the right value, I

It's 100, we fixed that quite a while ago :)

I think the default parameters and the default metric have indeed similar properties, though I'd say the consequences of the choice of metric are even worse.
The problem on both accounts is that people don't do the second step of actually thinking about it.

There's some hyper-parameters that have a huge impact, like kernel in SVC, but I think not tuning hypers is rarely as bad as using accuracy in an imbalanced classification problem (which is all classification problems).
Also, knowing reasonable defaults for all the model is not really feasible; but if you don't know how to pick at least a reasonable metric for your task, you probably shouldn't build a model yet.

I proposed two changes, adding they keyword arg, and removing the default. Just adding the keyword arg would at least smooth over the path from using accuracy to using something more relevant.

Labels: New Feature, API, Needs Decision, RFC, module:metrics
Add "scoring" argument to estimator's ``score`` method

### Describe the workflow you want to enable

I want to enable non-accuracy metrics to ``estimator.score``, and ultimately deprecate the default values of ``accuracy`` and ``r2``. I would call it ``scoring`` though it's a bit redundant but consistent.
That would allow us to get rid of the default scoring methods, which are objectively bad and misleading, and it would require the minimum amount of code changes for anyone. 

### Describe your proposed solution

Replace

```python
est.score(X, y)
```
with
```python
est.score(X, y, scoring="accuracy")
```
or rather
```python
est.score(X, y, scoring="recall_macro")
```
(or ``r2`` for regression).

### Describe alternatives you've considered, if relevant

- Keep current status, which is bad (both ``accuracy`` and ``r2`` are bad)
- Remove ``scoring`` method.

I can't think of any other alternatives tbh.

### Additional context

I think in theory this requires a slep, as it's changing shared API, right?

COMMENT:
What would be the default value of scorer?..

scorer = None ???

If you compare it to GridSearchCV etc.
They usually come with 'accuracy' as default scoring technique.

Moreover, scorer value would vary a lot in order to run multiple benchmark models.

My question is would you conclude to put None as default value?

Labels: New Feature, API, Needs Decision, RFC, module:metrics
Add "scoring" argument to estimator's ``score`` method

### Describe the workflow you want to enable

I want to enable non-accuracy metrics to ``estimator.score``, and ultimately deprecate the default values of ``accuracy`` and ``r2``. I would call it ``scoring`` though it's a bit redundant but consistent.
That would allow us to get rid of the default scoring methods, which are objectively bad and misleading, and it would require the minimum amount of code changes for anyone. 

### Describe your proposed solution

Replace

```python
est.score(X, y)
```
with
```python
est.score(X, y, scoring="accuracy")
```
or rather
```python
est.score(X, y, scoring="recall_macro")
```
(or ``r2`` for regression).

### Describe alternatives you've considered, if relevant

- Keep current status, which is bad (both ``accuracy`` and ``r2`` are bad)
- Remove ``scoring`` method.

I can't think of any other alternatives tbh.

### Additional context

I think in theory this requires a slep, as it's changing shared API, right?

COMMENT:
I think we're hurting people by not forcing people to think about their performance metrics. Those metrics have real life consequences and our defaults make no sense, and have nothing to do with the real life usecase at hand for the user.

We should have a much better documentation page on how to choose the metric for that purpose, and then we should force people to choose themselves.

Labels: New Feature, API, Needs Decision, RFC, module:metrics
Add "scoring" argument to estimator's ``score`` method

### Describe the workflow you want to enable

I want to enable non-accuracy metrics to ``estimator.score``, and ultimately deprecate the default values of ``accuracy`` and ``r2``. I would call it ``scoring`` though it's a bit redundant but consistent.
That would allow us to get rid of the default scoring methods, which are objectively bad and misleading, and it would require the minimum amount of code changes for anyone. 

### Describe your proposed solution

Replace

```python
est.score(X, y)
```
with
```python
est.score(X, y, scoring="accuracy")
```
or rather
```python
est.score(X, y, scoring="recall_macro")
```
(or ``r2`` for regression).

### Describe alternatives you've considered, if relevant

- Keep current status, which is bad (both ``accuracy`` and ``r2`` are bad)
- Remove ``scoring`` method.

I can't think of any other alternatives tbh.

### Additional context

I think in theory this requires a slep, as it's changing shared API, right?

COMMENT:
So....
1) add the scoring argument
2) add the documentation
3) deprecate the default?

Or 
1) add the documentation
2) add the scoring argument and deprecate the default?

I'm happy to write the SLEP after the NeurIPS deadline (and my vacation).

Labels: New Feature, API, Needs Decision, RFC, module:metrics
Add "scoring" argument to estimator's ``score`` method

### Describe the workflow you want to enable

I want to enable non-accuracy metrics to ``estimator.score``, and ultimately deprecate the default values of ``accuracy`` and ``r2``. I would call it ``scoring`` though it's a bit redundant but consistent.
That would allow us to get rid of the default scoring methods, which are objectively bad and misleading, and it would require the minimum amount of code changes for anyone. 

### Describe your proposed solution

Replace

```python
est.score(X, y)
```
with
```python
est.score(X, y, scoring="accuracy")
```
or rather
```python
est.score(X, y, scoring="recall_macro")
```
(or ``r2`` for regression).

### Describe alternatives you've considered, if relevant

- Keep current status, which is bad (both ``accuracy`` and ``r2`` are bad)
- Remove ``scoring`` method.

I can't think of any other alternatives tbh.

### Additional context

I think in theory this requires a slep, as it's changing shared API, right?

COMMENT:
I'd go more with the first version; it feels smoother. Happy to review the SLEP.

Labels: New Feature, API, Needs Decision, RFC, module:metrics