Talip7/scikit-learn-issues-multilabel huggingface dataset:
RFC Expose a plublic method to compute the objective function

I think that it would be valuable that all estimators that optimize some objective function expose a public method to compute it.
My main motivation is for the callbacks, for early stopping or monitoring, but I'm sure it would be useful in other contexts.

To really be practical, the signature should be the same across all estimators. What I have in mind is something like:

```py
def objective_function(self, X, y=None, *, sample_weight=None, normalize=False):
    y_pred = self.predict(X, y)
    # or Xt = self.transform(X) because some transformers do optimize an objective function.

    data_fit = <computation of the data fit term>
    penalization = <computation of the penalization term>

    if normalize:                     # allow to return a per sample
        data_fit /= X.shape[0]        # objective function
        penalization /= X.shape[0]    # for convenience
        # X.shape[0] probably needs to be replaced by sample_weight.sum()
 
    return data_fit + penalization, data_fit, penalization
```

If we want to compute the objective function of the training set during fitting, we could allow to provide the current variables that are required to compute it at a given iteration, encapsulated in a single argument (a dict):
```py
def objective_function(self, X, y=None, *, sample_weight=None, fit_state=None, normalize=False):
    if fit_state is None:
        y_pred = self.predict(X)
    else:
        y_pred = <compute y_pred using the information in fit_state>

    ...
```
where the content of fit_state would be estimator specific, and detailed in the docstring of `objective_function` for each estimator.

COMMENT:
I think the `normalize` option should be true by default to make the train and test set values comparable on the same scale even if the test set is 10x smaller than the training set.

We might not want not expose an estimator-specific `fit_state` argument as part of the public API of scikit-learn estimators. To achieve the same results, estimator could optionally implement a private `_objective_function()` with the freedom to pass estimator specific kwargs with meaningful names.

When this private method is present, the callbacks would attempt to use the private method first by looking up kwargs names from the signature from the fit state (or assume that they are present as attributes on self, updated at each iteration of `fit`). Or alternatively we keep the estimator `fit_state` argument, but we only expose it in the private version of method.

About the return value(s) of this method:

I think it should be more than a scalar or a fixed number of scalars, but instead a dataclass (or namedtuple):

- it should maybe communicate its name ("penalized_squared_error", "explained_variance", "inertia" ...),
- it should communicate if this is something to minimize or maximize,
- it should provide access to the aggregate scalar value,
- but also the individual terms (data fits, and each regularizer).

If we return a dataclass, we can have an HTML repr for notebooks.

Labels: Hard, RFC, Meta-issue
RFC Expose a plublic method to compute the objective function

I think that it would be valuable that all estimators that optimize some objective function expose a public method to compute it.
My main motivation is for the callbacks, for early stopping or monitoring, but I'm sure it would be useful in other contexts.

To really be practical, the signature should be the same across all estimators. What I have in mind is something like:

```py
def objective_function(self, X, y=None, *, sample_weight=None, normalize=False):
    y_pred = self.predict(X, y)
    # or Xt = self.transform(X) because some transformers do optimize an objective function.

    data_fit = <computation of the data fit term>
    penalization = <computation of the penalization term>

    if normalize:                     # allow to return a per sample
        data_fit /= X.shape[0]        # objective function
        penalization /= X.shape[0]    # for convenience
        # X.shape[0] probably needs to be replaced by sample_weight.sum()
 
    return data_fit + penalization, data_fit, penalization
```

If we want to compute the objective function of the training set during fitting, we could allow to provide the current variables that are required to compute it at a given iteration, encapsulated in a single argument (a dict):
```py
def objective_function(self, X, y=None, *, sample_weight=None, fit_state=None, normalize=False):
    if fit_state is None:
        y_pred = self.predict(X)
    else:
        y_pred = <compute y_pred using the information in fit_state>

    ...
```
where the content of fit_state would be estimator specific, and detailed in the docstring of `objective_function` for each estimator.

COMMENT:
I opened https://github.com/scikit-learn/scikit-learn/pull/28505 which implements it for NMF to help moving forward with this discussion.

> I think it should be more than a scalar or a fixed number of scalars, but instead a dataclass 

I agree and made it a dataclass

> When this private method is present, the callbacks would attempt to use the private method first by looking up kwargs names from the signature from the fit state

I think it's better to have it in the public method because callbacks are public and users can write custom callbacks so they should not have to rely on a private method.

> We might not want not expose an estimator-specific fit_state argument as part of the public API of scikit-learn estimators. To achieve the same results, estimator could optionally implement a private _objective_function() with the freedom to pass estimator specific kwargs with meaningful names.

Thinking more about it, I think that it's better to have 1 keyword param per fitted parameter of the model instead of a single param being a dict. The callback documentation will mention that if a callback needs to compute the objective function on the train set, the `fit_state` dict that the callback receives should be used: `model.objective_fuction(X, y, **fit_state)`.

Labels: Hard, RFC, Meta-issue
RFC Expose a plublic method to compute the objective function

I think that it would be valuable that all estimators that optimize some objective function expose a public method to compute it.
My main motivation is for the callbacks, for early stopping or monitoring, but I'm sure it would be useful in other contexts.

To really be practical, the signature should be the same across all estimators. What I have in mind is something like:

```py
def objective_function(self, X, y=None, *, sample_weight=None, normalize=False):
    y_pred = self.predict(X, y)
    # or Xt = self.transform(X) because some transformers do optimize an objective function.

    data_fit = <computation of the data fit term>
    penalization = <computation of the penalization term>

    if normalize:                     # allow to return a per sample
        data_fit /= X.shape[0]        # objective function
        penalization /= X.shape[0]    # for convenience
        # X.shape[0] probably needs to be replaced by sample_weight.sum()
 
    return data_fit + penalization, data_fit, penalization
```

If we want to compute the objective function of the training set during fitting, we could allow to provide the current variables that are required to compute it at a given iteration, encapsulated in a single argument (a dict):
```py
def objective_function(self, X, y=None, *, sample_weight=None, fit_state=None, normalize=False):
    if fit_state is None:
        y_pred = self.predict(X)
    else:
        y_pred = <compute y_pred using the information in fit_state>

    ...
```
where the content of fit_state would be estimator specific, and detailed in the docstring of `objective_function` for each estimator.

COMMENT:
>  The callback documentation will mention that if a callback needs to compute the objective function on the train set, the fit_state dict that the callback receives should be used: model.objective_fuction(X, y, **fit_state).

I suspect that there might be cases that we have in `fit_state` things that are not useful to pass as a kwarg to `objective_function`. Should we filter those out based on the signature of the objective function? Or make all objective function signature include an extra `**ignored_kwargs` (that sounds ugly)...

Labels: Hard, RFC, Meta-issue
RFC Expose a plublic method to compute the objective function

I think that it would be valuable that all estimators that optimize some objective function expose a public method to compute it.
My main motivation is for the callbacks, for early stopping or monitoring, but I'm sure it would be useful in other contexts.

To really be practical, the signature should be the same across all estimators. What I have in mind is something like:

```py
def objective_function(self, X, y=None, *, sample_weight=None, normalize=False):
    y_pred = self.predict(X, y)
    # or Xt = self.transform(X) because some transformers do optimize an objective function.

    data_fit = <computation of the data fit term>
    penalization = <computation of the penalization term>

    if normalize:                     # allow to return a per sample
        data_fit /= X.shape[0]        # objective function
        penalization /= X.shape[0]    # for convenience
        # X.shape[0] probably needs to be replaced by sample_weight.sum()
 
    return data_fit + penalization, data_fit, penalization
```

If we want to compute the objective function of the training set during fitting, we could allow to provide the current variables that are required to compute it at a given iteration, encapsulated in a single argument (a dict):
```py
def objective_function(self, X, y=None, *, sample_weight=None, fit_state=None, normalize=False):
    if fit_state is None:
        y_pred = self.predict(X)
    else:
        y_pred = <compute y_pred using the information in fit_state>

    ...
```
where the content of fit_state would be estimator specific, and detailed in the docstring of `objective_function` for each estimator.

COMMENT:
That's a good point. I guess I was planning to add a `**kwargs` param but I understand that it's not a very good pattern.

For the context, my motivation for being able to provide fit state parameters comes from when a callback would need to compute the objective function on the training set. It doesn't apply for any other metric and it doesn't apply for the validation set. And it's just to speed-up the computation, avoiding to call `predict` or `transform`.

So I'm wondering if we really need to have this/these param(s) at all. Maybe we can just accept that computing the objective function on the train set is as fast as computing it on the validation set, despite knowing that it could be done faster. What do you think ?

Labels: Hard, RFC, Meta-issue
RFC Expose a plublic method to compute the objective function

I think that it would be valuable that all estimators that optimize some objective function expose a public method to compute it.
My main motivation is for the callbacks, for early stopping or monitoring, but I'm sure it would be useful in other contexts.

To really be practical, the signature should be the same across all estimators. What I have in mind is something like:

```py
def objective_function(self, X, y=None, *, sample_weight=None, normalize=False):
    y_pred = self.predict(X, y)
    # or Xt = self.transform(X) because some transformers do optimize an objective function.

    data_fit = <computation of the data fit term>
    penalization = <computation of the penalization term>

    if normalize:                     # allow to return a per sample
        data_fit /= X.shape[0]        # objective function
        penalization /= X.shape[0]    # for convenience
        # X.shape[0] probably needs to be replaced by sample_weight.sum()
 
    return data_fit + penalization, data_fit, penalization
```

If we want to compute the objective function of the training set during fitting, we could allow to provide the current variables that are required to compute it at a given iteration, encapsulated in a single argument (a dict):
```py
def objective_function(self, X, y=None, *, sample_weight=None, fit_state=None, normalize=False):
    if fit_state is None:
        y_pred = self.predict(X)
    else:
        y_pred = <compute y_pred using the information in fit_state>

    ...
```
where the content of fit_state would be estimator specific, and detailed in the docstring of `objective_function` for each estimator.

COMMENT:
> Maybe we can just accept that computing the objective function on the train set is as fast as computing it on the validation set, despite knowing that it could be done faster. What do you think ?

Indeed, let's start simple and consider adding API complexity only when it is really justified.

Labels: Hard, RFC, Meta-issue
AttributeError: 'BinomialDeviance' object has no attribute 'get_init_raw_predictions'

### Describe the bug

I save GradientBoostingClassifier model by pickle at sklearn==0.20 ,loaded model at sklearn=0.22 and use it happend after error.

```pytb
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\ZJLAB\anaconda3\envs\pch\lib\site-packages\sklearn\ensemble\_gb.py", line 2214, in predict_proba
    raw_predictions = self.decision_function(X)
  File "C:\Users\ZJLAB\anaconda3\envs\pch\lib\site-packages\sklearn\ensemble\_gb.py", line 2121, in decision_function
    raw_predictions = self._raw_predict(X)
  File "C:\Users\ZJLAB\anaconda3\envs\pch\lib\site-packages\sklearn\ensemble\_gb.py", line 1655, in _raw_predict
    raw_predictions = self._raw_predict_init(X)
  File "C:\Users\ZJLAB\anaconda3\envs\pch\lib\site-packages\sklearn\ensemble\_gb.py", line 1649, in _raw_predict_init
    raw_predictions = self.loss_.get_init_raw_predictions(
AttributeError: 'BinomialDeviance' object has no attribute 'get_init_raw_predictions'
```

### Steps/Code to Reproduce

```python
with open(modelfile,"rb") as f:
    models = pickle.load(f)
    prob = models["model"].predict_proba(data)[:,1]
```

### Expected Results

e

### Actual Results

```python
>>> models["model"].predict_proba(data)[:,1]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\ZJLAB\anaconda3\envs\pch\lib\site-packages\sklearn\ensemble\_gb.py", line 2214, in predict_proba
    raw_predictions = self.decision_function(X)
  File "C:\Users\ZJLAB\anaconda3\envs\pch\lib\site-packages\sklearn\ensemble\_gb.py", line 2121, in decision_function
    raw_predictions = self._raw_predict(X)
  File "C:\Users\ZJLAB\anaconda3\envs\pch\lib\site-packages\sklearn\ensemble\_gb.py", line 1655, in _raw_predict
    raw_predictions = self._raw_predict_init(X)
  File "C:\Users\ZJLAB\anaconda3\envs\pch\lib\site-packages\sklearn\ensemble\_gb.py", line 1649, in _raw_predict_init
    raw_predictions = self.loss_.get_init_raw_predictions(
AttributeError: 'BinomialDeviance' object has no attribute 'get_init_raw_predictions'
```

### Versions

```shell
Python dependencies:
       pip: 23.3.1
setuptools: 68.2.2
   sklearn: 0.22
     numpy: 1.19.2
     scipy: 1.6.2
    Cython: None
    pandas: 1.2.4
matplotlib: None
    joblib: 1.2.0
```

COMMENT:
As mentioned in the documentation, we do not support unpickling with different versions:

https://scikit-learn.org/dev/model_persistence.html#security-maintainability-limitations

Therefore, you need to unpickle with 0.20 or retrain the model with the desired scikit-learn version.

Labels: Bug, Needs Triage
DOC Need Gini coefficient implementation

### Describe the workflow you want to enable

Hello there! 
For many reasons for classical binary classifications in different areas of business the Gini Score is used.
The Gini score is calculated as follows: Gini=2×ROC AUC−1
The recent implementation of sklearn.metrics.roc_auc_score https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html requires the formula above to be implemented.

The current functionality doesn't allow this. 

This simple fuction will allow many data scientists to clean up their code. 

Thanks in advance!

### Describe your proposed solution

My suggestion is to add this simple function gini_score in sklearn.metrics described as gini_score = 2*roc_auc_score - 1.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
You will find this simple formula implemented in some of our examples, too. Because it is so simple, I‘m against adding it to the already large metrics API.

We could, however, improve the doc and mention Gini in
- https://scikit-learn.org/stable/modules/model_evaluation.html#metrics-and-scoring-quantifying-the-quality-of-predictions and
- Docstring of `roc_auc_score`

Labels: New Feature
DOC Need Gini coefficient implementation

### Describe the workflow you want to enable

Hello there! 
For many reasons for classical binary classifications in different areas of business the Gini Score is used.
The Gini score is calculated as follows: Gini=2×ROC AUC−1
The recent implementation of sklearn.metrics.roc_auc_score https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html requires the formula above to be implemented.

The current functionality doesn't allow this. 

This simple fuction will allow many data scientists to clean up their code. 

Thanks in advance!

### Describe your proposed solution

My suggestion is to add this simple function gini_score in sklearn.metrics described as gini_score = 2*roc_auc_score - 1.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I strongly agree, that this fact can be written in docstring. How’s about adding Gini as attribute of roc_auc_score?

Labels: New Feature
DOC Need Gini coefficient implementation

### Describe the workflow you want to enable

Hello there! 
For many reasons for classical binary classifications in different areas of business the Gini Score is used.
The Gini score is calculated as follows: Gini=2×ROC AUC−1
The recent implementation of sklearn.metrics.roc_auc_score https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html requires the formula above to be implemented.

The current functionality doesn't allow this. 

This simple fuction will allow many data scientists to clean up their code. 

Thanks in advance!

### Describe your proposed solution

My suggestion is to add this simple function gini_score in sklearn.metrics described as gini_score = 2*roc_auc_score - 1.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I would not add as attribute since attribute means that this is the input parameters.

However, we can add a `Notes` section where we can mentioned the Gini coefficient and the equivalence with the ROC AUC.

Labels: New Feature
DOC Need Gini coefficient implementation

### Describe the workflow you want to enable

Hello there! 
For many reasons for classical binary classifications in different areas of business the Gini Score is used.
The Gini score is calculated as follows: Gini=2×ROC AUC−1
The recent implementation of sklearn.metrics.roc_auc_score https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html requires the formula above to be implemented.

The current functionality doesn't allow this. 

This simple fuction will allow many data scientists to clean up their code. 

Thanks in advance!

### Describe your proposed solution

My suggestion is to add this simple function gini_score in sklearn.metrics described as gini_score = 2*roc_auc_score - 1.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
@dzisandy do you want to make a pull-request?

Labels: New Feature
DOC Need Gini coefficient implementation

### Describe the workflow you want to enable

Hello there! 
For many reasons for classical binary classifications in different areas of business the Gini Score is used.
The Gini score is calculated as follows: Gini=2×ROC AUC−1
The recent implementation of sklearn.metrics.roc_auc_score https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html requires the formula above to be implemented.

The current functionality doesn't allow this. 

This simple fuction will allow many data scientists to clean up their code. 

Thanks in advance!

### Describe your proposed solution

My suggestion is to add this simple function gini_score in sklearn.metrics described as gini_score = 2*roc_auc_score - 1.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
@glemaitre it will be great. So we want to modify only the Notes section in doc string?

Labels: New Feature
DOC Need Gini coefficient implementation

### Describe the workflow you want to enable

Hello there! 
For many reasons for classical binary classifications in different areas of business the Gini Score is used.
The Gini score is calculated as follows: Gini=2×ROC AUC−1
The recent implementation of sklearn.metrics.roc_auc_score https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html requires the formula above to be implemented.

The current functionality doesn't allow this. 

This simple fuction will allow many data scientists to clean up their code. 

Thanks in advance!

### Describe your proposed solution

My suggestion is to add this simple function gini_score in sklearn.metrics described as gini_score = 2*roc_auc_score - 1.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
> So we want to modify only the Notes section in doc string?

Indeed. If no the `Notes` section does not exist we need to create one. We need to follow the numpydoc format: https://numpydoc.readthedocs.io/en/latest/format.html (the ordering is particularly important otherwise the CIs will fail).

Labels: New Feature
DOC Need Gini coefficient implementation

### Describe the workflow you want to enable

Hello there! 
For many reasons for classical binary classifications in different areas of business the Gini Score is used.
The Gini score is calculated as follows: Gini=2×ROC AUC−1
The recent implementation of sklearn.metrics.roc_auc_score https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html requires the formula above to be implemented.

The current functionality doesn't allow this. 

This simple fuction will allow many data scientists to clean up their code. 

Thanks in advance!

### Describe your proposed solution

My suggestion is to add this simple function gini_score in sklearn.metrics described as gini_score = 2*roc_auc_score - 1.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
@glemaitre created finally request.

Labels: New Feature
LedoitWolf.fit() crashes Jupyter Notebook

### Describe the bug

Hi, when i used LedoitWolf.fit(), my Jupyter Notebook crashes immediately. I am using the toy example given in the source code of 
LedoitWolf class


### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.covariance import LedoitWolf
real_cov = np.array([[.4, .2],[.2, .8]])
np.random.seed(0)
X = np.random.multivariate_normal(mean=[0, 0],cov=real_cov,size=50)
cov = LedoitWolf().fit(X)
```

### Expected Results

It is exptected to fit the estimator.

### Actual Results

However kernel crashes. sklearn.show_versions() also produce a WinError.

### Versions

```shell
Sklearn version is 1.3.2
```

COMMENT:
Can you provide the full traceback of the error of the code and the `show_versions`.

Labels: Bug, Needs Info