Talip7/scikit-learn-issues-multilabel huggingface dataset:
⚠️ CI failed on Linux_Nightly.pylatest_pip_scipy_dev ⚠️

**CI is still failing on [Linux_Nightly.pylatest_pip_scipy_dev](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=42983&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf)**
- test_missing_value_handling[est0-maxabs_scale-True-False-omit_kwargs0]
- test_missing_value_handling[est3-scale-True-False-omit_kwargs3]
- test_missing_value_handling[est6-quantile_transform-True-False-omit_kwargs6]
- test_missing_value_handling[est8-robust_scale-True-False-omit_kwargs8]

COMMENT:
The problem happens at fit time, the `scale_` attribute of `RobustScaler` has different values for the DIAgonal input data:

```
dense [1.35  0.525 3.55  1.65 ]
csr [1.35  0.525 3.55  1.65 ]
csc [1.35  0.525 3.55  1.65 ]
bsr [1.35  0.525 3.55  1.65 ]
coo [1.35  0.525 3.55  1.65 ]
dia [1.4 0.7 3.6 1.6]
```

Labels: Needs Triage
⚠️ CI failed on Linux_Nightly.pylatest_pip_scipy_dev ⚠️

**CI is still failing on [Linux_Nightly.pylatest_pip_scipy_dev](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=42983&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf)**
- test_missing_value_handling[est0-maxabs_scale-True-False-omit_kwargs0]
- test_missing_value_handling[est3-scale-True-False-omit_kwargs3]
- test_missing_value_handling[est6-quantile_transform-True-False-omit_kwargs6]
- test_missing_value_handling[est8-robust_scale-True-False-omit_kwargs8]

COMMENT:
Ok, I found a minimal reproducer:

```python
import numpy as np
import scipy.sparse as sp


X = np.ones(shape=(13, 4))
X[0, 0] = np.nan
print(sp.csc_matrix(X).indptr)
print(sp.csc_matrix(sp.dia_matrix(X)).indptr)
```

out:
```
[ 0 13 26 39 52]
[ 0 12 25 38 51]
```

Labels: Needs Triage
⚠️ CI failed on Linux_Nightly.pylatest_pip_scipy_dev ⚠️

**CI is still failing on [Linux_Nightly.pylatest_pip_scipy_dev](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=42983&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf)**
- test_missing_value_handling[est0-maxabs_scale-True-False-omit_kwargs0]
- test_missing_value_handling[est3-scale-True-False-omit_kwargs3]
- test_missing_value_handling[est6-quantile_transform-True-False-omit_kwargs6]
- test_missing_value_handling[est8-robust_scale-True-False-omit_kwargs8]

COMMENT:
I updated https://github.com/scipy/scipy/issues/16365 with a more minimal reproducer.

Labels: Needs Triage
⚠️ CI failed on Linux_Nightly.pylatest_pip_scipy_dev ⚠️

**CI is still failing on [Linux_Nightly.pylatest_pip_scipy_dev](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=42983&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf)**
- test_missing_value_handling[est0-maxabs_scale-True-False-omit_kwargs0]
- test_missing_value_handling[est3-scale-True-False-omit_kwargs3]
- test_missing_value_handling[est6-quantile_transform-True-False-omit_kwargs6]
- test_missing_value_handling[est8-robust_scale-True-False-omit_kwargs8]

COMMENT:
FYI: The bug is actually in NumPy: https://github.com/numpy/numpy/issues/21685

Labels: Needs Triage
⚠️ CI failed on Linux_Nightly.pylatest_pip_scipy_dev ⚠️

**CI is still failing on [Linux_Nightly.pylatest_pip_scipy_dev](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=42983&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf)**
- test_missing_value_handling[est0-maxabs_scale-True-False-omit_kwargs0]
- test_missing_value_handling[est3-scale-True-False-omit_kwargs3]
- test_missing_value_handling[est6-quantile_transform-True-False-omit_kwargs6]
- test_missing_value_handling[est8-robust_scale-True-False-omit_kwargs8]

COMMENT:
> FYI: The bug is actually in NumPy: https://github.com/numpy/numpy/issues/21685

Thanks. I should have checked I could reproduce with the latest stable numpy before reporting to scipy.

Labels: Needs Triage
⚠️ CI failed on Linux_Nightly.pylatest_pip_scipy_dev ⚠️

**CI is still failing on [Linux_Nightly.pylatest_pip_scipy_dev](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=42983&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf)**
- test_missing_value_handling[est0-maxabs_scale-True-False-omit_kwargs0]
- test_missing_value_handling[est3-scale-True-False-omit_kwargs3]
- test_missing_value_handling[est6-quantile_transform-True-False-omit_kwargs6]
- test_missing_value_handling[est8-robust_scale-True-False-omit_kwargs8]

COMMENT:
## CI is no longer failing! ✅

[Successful run](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=43021&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf)

Labels: Needs Triage
⚠️ CI failed on Linux_Nightly.pylatest_pip_scipy_dev ⚠️

**CI is still failing on [Linux_Nightly.pylatest_pip_scipy_dev](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=42983&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf)**
- test_missing_value_handling[est0-maxabs_scale-True-False-omit_kwargs0]
- test_missing_value_handling[est3-scale-True-False-omit_kwargs3]
- test_missing_value_handling[est6-quantile_transform-True-False-omit_kwargs6]
- test_missing_value_handling[est8-robust_scale-True-False-omit_kwargs8]

COMMENT:
Closing this one, I am guessing this did not get closed automatically because auto-close is off by default since https://github.com/scikit-learn/scikit-learn/pull/23155

Labels: Needs Triage
⚠️ CI failed on Linux_Nightly.pylatest_pip_scipy_dev ⚠️

**CI is still failing on [Linux_Nightly.pylatest_pip_scipy_dev](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=42983&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf)**
- test_missing_value_handling[est0-maxabs_scale-True-False-omit_kwargs0]
- test_missing_value_handling[est3-scale-True-False-omit_kwargs3]
- test_missing_value_handling[est6-quantile_transform-True-False-omit_kwargs6]
- test_missing_value_handling[est8-robust_scale-True-False-omit_kwargs8]

COMMENT:
@thomasjpfan this bot is really cool :)

Labels: Needs Triage
Confusion regarding BinomialDeviance

### Describe the issue linked to the documentation

The [BinomialDeviance](https://github.com/scikit-learn/scikit-learn/blob/80598905e517759b4696c74ecc35c6e2eb508cff/sklearn/ensemble/_gb_losses.py#L633) states that it computes the `Compute the deviance (= 2 * negative log-likelihood)` and then it proceeds by computing the following functions
```
# logaddexp(0, v) == log(1.0 + exp(v))
raw_predictions = raw_predictions.ravel()
if sample_weight is None:
    return -2 * np.mean( (y * raw_predictions) - np.logaddexp(0, raw_predictions) )
else:
    return ( -2 / sample_weight.sum() * np.sum( sample_weight * ( (y * raw_predictions) - np.logaddexp(0, raw_predictions) ) )
```
Let's take the first case in the if statement, `-2 * np.mean( (y * raw_predictions) - np.logaddexp(0, raw_predictions) )`, which seems to be computing `y * raw_predictions - sigmoid(raw_predictions)`. 

Shouldn't that instead be $y * \log(p) + (1-y) \log(1-p)$ where $p$ is a vector of probabilities obtained from `raw_inputs` when $y = 2$ classes?

Another thing that I don't fully follow is the `nagative_gradient` computation in the respective function which computes the following gradient `y - expit(raw_predictions.ravel())`. 

From the comment it states that it computes the gradient and then multiplies it by 1/2,  `Compute half of the negative gradient`.

Is that correct? Shouldn't the gradient of `y * raw_predictions - sigmoid(raw_predictions)` with respect to the `raw_predictions` be $y - \frac{e^{raw}}{1+e^{raw}}$ where `raw` refers to the `raw_predictions`?

The expression  $y - \frac{e^{raw}}{1+e^{raw}}$  can be simplified into  $y - \sigma(raw)(1-\sigma(raw))$ where $\sigma$ is the logistic function. 




### Suggest a potential alternative/fix

_No response_

COMMENT:
Overall, the `logaddexp` expression is not `sigmoid`. Starting with:

$$
y * \log(p) + (1-y)\log(1-p)
$$

and plugging in $p = \frac{1}{1+e^{-r}}$, where $r$ is the raw predictions. One gets:

$$
y * r - \log(1+e^r)
$$

which is the implementation. The gradient of the above expression with respect to $r$ is:

$$
y - \frac{1}{1+e^{-r}}
$$

which also matches the implementation.

Labels: Documentation, Needs Triage
RFC classifiers trained by minimizing the Brier loss

At the moment, our probablistic classifiers (e.g. logistic regression and gradient boosted trees) optimize the log loss, typically after taking a sigmoid or softmax inverse link function (typically part of the Cython loss).

However the log-loss is not the only proper-scoring rule to fit estimators of the expected conditional class probabilities, in particular the Brier loss is also a proper scoring rule and has the practical advantage of being upper bounded which should limit the impact of mislabeled examples in the training set.

Would it make sense to publicly such estimators in scikit-learn?

Note that for linear models, fitting the Brier loss is not equivalent to our `RidgeClassifier` because the latter does not take the softmax of its raw predictions before computing the square loss, hence it does **not** offer a well defined estimation of the conditional class probabilities.

Minimizing the Brier loss of a softmax linear model is no longer a convex optimization problem but I doubt that this would prevent a newton solver with a robust line search to converge in practice.

COMMENT:
Opinions @lorentzenchr @GaelVaroquaux @agramfort?

Labels: RFC
RFC classifiers trained by minimizing the Brier loss

At the moment, our probablistic classifiers (e.g. logistic regression and gradient boosted trees) optimize the log loss, typically after taking a sigmoid or softmax inverse link function (typically part of the Cython loss).

However the log-loss is not the only proper-scoring rule to fit estimators of the expected conditional class probabilities, in particular the Brier loss is also a proper scoring rule and has the practical advantage of being upper bounded which should limit the impact of mislabeled examples in the training set.

Would it make sense to publicly such estimators in scikit-learn?

Note that for linear models, fitting the Brier loss is not equivalent to our `RidgeClassifier` because the latter does not take the softmax of its raw predictions before computing the square loss, hence it does **not** offer a well defined estimation of the conditional class probabilities.

Minimizing the Brier loss of a softmax linear model is no longer a convex optimization problem but I doubt that this would prevent a newton solver with a robust line search to converge in practice.

COMMENT:
~~That's RidgeClassifier, isn't it?~~ Ooops, didn't read well.

I'd say: is there a good literature documenting practice (and theory) showing benefit of such classifiers. If not, I wouldn't prioritize this: we have a lot of things on our plate already.

Labels: RFC
RFC classifiers trained by minimizing the Brier loss

At the moment, our probablistic classifiers (e.g. logistic regression and gradient boosted trees) optimize the log loss, typically after taking a sigmoid or softmax inverse link function (typically part of the Cython loss).

However the log-loss is not the only proper-scoring rule to fit estimators of the expected conditional class probabilities, in particular the Brier loss is also a proper scoring rule and has the practical advantage of being upper bounded which should limit the impact of mislabeled examples in the training set.

Would it make sense to publicly such estimators in scikit-learn?

Note that for linear models, fitting the Brier loss is not equivalent to our `RidgeClassifier` because the latter does not take the softmax of its raw predictions before computing the square loss, hence it does **not** offer a well defined estimation of the conditional class probabilities.

Minimizing the Brier loss of a softmax linear model is no longer a convex optimization problem but I doubt that this would prevent a newton solver with a robust line search to converge in practice.

COMMENT:
I've never seen this done before. Do you have any ref who did this?

Labels: RFC
RFC classifiers trained by minimizing the Brier loss

At the moment, our probablistic classifiers (e.g. logistic regression and gradient boosted trees) optimize the log loss, typically after taking a sigmoid or softmax inverse link function (typically part of the Cython loss).

However the log-loss is not the only proper-scoring rule to fit estimators of the expected conditional class probabilities, in particular the Brier loss is also a proper scoring rule and has the practical advantage of being upper bounded which should limit the impact of mislabeled examples in the training set.

Would it make sense to publicly such estimators in scikit-learn?

Note that for linear models, fitting the Brier loss is not equivalent to our `RidgeClassifier` because the latter does not take the softmax of its raw predictions before computing the square loss, hence it does **not** offer a well defined estimation of the conditional class probabilities.

Minimizing the Brier loss of a softmax linear model is no longer a convex optimization problem but I doubt that this would prevent a newton solver with a robust line search to converge in practice.

COMMENT:
> I've never seen this done before. Do you have any ref who did this?

No ref, but it seems a logical interpolation point in the design space of ML estimators.

Anecdotally, it's used in the classification head of the YOLO Object Dection neural net architecture if I remember correctly.

Indeed, look at the last term of:

![image](https://user-images.githubusercontent.com/89061/171626035-468b432c-4d3c-4b41-b645-171770f4758b.png)

on the third page of:

https://arxiv.org/pdf/1506.02640.pdf

The `\hat{p}_i(c)` are softmax normalized outputs.

They do not mention the Brier loss in the paper but I suspect that they do not use the log loss on the classification part of the loss to keep it on the same scale as the regression terms of the loss throughout the training process to make optimization easier.

Actually, on page 2, it's written:

> We use sum-squared error because it is easy to optimize, however it does not perfectly align with our goal of maximizing average precision. It weights localization error equally with classification error which may not be ideal.

Labels: RFC
RFC classifiers trained by minimizing the Brier loss

At the moment, our probablistic classifiers (e.g. logistic regression and gradient boosted trees) optimize the log loss, typically after taking a sigmoid or softmax inverse link function (typically part of the Cython loss).

However the log-loss is not the only proper-scoring rule to fit estimators of the expected conditional class probabilities, in particular the Brier loss is also a proper scoring rule and has the practical advantage of being upper bounded which should limit the impact of mislabeled examples in the training set.

Would it make sense to publicly such estimators in scikit-learn?

Note that for linear models, fitting the Brier loss is not equivalent to our `RidgeClassifier` because the latter does not take the softmax of its raw predictions before computing the square loss, hence it does **not** offer a well defined estimation of the conditional class probabilities.

Minimizing the Brier loss of a softmax linear model is no longer a convex optimization problem but I doubt that this would prevent a newton solver with a robust line search to converge in practice.

COMMENT:
> I'd say: is there a good literature documenting practice (and theory) showing benefit of such classifiers. If not, I wouldn't prioritize this: we have a lot of things on our plate already.

I had no particular plans to work on this. It's that I had be thinking about this for a while but could not see a reason why this would not be common practice in our field, so I decided to dump the idea publicly:
- to collect feedback from others;
- so that we can link to it if the need appears again in another discussion.

Labels: RFC
RFC classifiers trained by minimizing the Brier loss

At the moment, our probablistic classifiers (e.g. logistic regression and gradient boosted trees) optimize the log loss, typically after taking a sigmoid or softmax inverse link function (typically part of the Cython loss).

However the log-loss is not the only proper-scoring rule to fit estimators of the expected conditional class probabilities, in particular the Brier loss is also a proper scoring rule and has the practical advantage of being upper bounded which should limit the impact of mislabeled examples in the training set.

Would it make sense to publicly such estimators in scikit-learn?

Note that for linear models, fitting the Brier loss is not equivalent to our `RidgeClassifier` because the latter does not take the softmax of its raw predictions before computing the square loss, hence it does **not** offer a well defined estimation of the conditional class probabilities.

Minimizing the Brier loss of a softmax linear model is no longer a convex optimization problem but I doubt that this would prevent a newton solver with a robust line search to converge in practice.

COMMENT:
In principle, this is a valid modelling strategy. One might suspect two advantages, as mentioned earlier:
- Computational more tractable (cheaper) than log loss.
- It is always finite, also for false certainty of predictions (predicting 1 when the outcome can be 0).

There are, however, practical disadvantages:
- One needs to constrain a model's output to the interval [0, 1]. This is often done by a link function such as expit/softmax. (For linear models, one could also use constraint optimization with the condition `0 <= X @ coef <= 1`.)
- The canonical link function for the Brier score/squared error is the identity. This does not limit the model's output.
- All non-canonical link functions will lead to larger model bias/bad calibration (in finite but possibly very large sample size).
- If the actual data is closer to a Binomial (on which I would bet for binary targets), estimation via log loss is more efficient (for finite sample size).

Labels: RFC
RFC classifiers trained by minimizing the Brier loss

At the moment, our probablistic classifiers (e.g. logistic regression and gradient boosted trees) optimize the log loss, typically after taking a sigmoid or softmax inverse link function (typically part of the Cython loss).

However the log-loss is not the only proper-scoring rule to fit estimators of the expected conditional class probabilities, in particular the Brier loss is also a proper scoring rule and has the practical advantage of being upper bounded which should limit the impact of mislabeled examples in the training set.

Would it make sense to publicly such estimators in scikit-learn?

Note that for linear models, fitting the Brier loss is not equivalent to our `RidgeClassifier` because the latter does not take the softmax of its raw predictions before computing the square loss, hence it does **not** offer a well defined estimation of the conditional class probabilities.

Minimizing the Brier loss of a softmax linear model is no longer a convex optimization problem but I doubt that this would prevent a newton solver with a robust line search to converge in practice.

COMMENT:
Thanks for the feedback. Do you have references for the last 3 points?

Labels: RFC
RFC classifiers trained by minimizing the Brier loss

At the moment, our probablistic classifiers (e.g. logistic regression and gradient boosted trees) optimize the log loss, typically after taking a sigmoid or softmax inverse link function (typically part of the Cython loss).

However the log-loss is not the only proper-scoring rule to fit estimators of the expected conditional class probabilities, in particular the Brier loss is also a proper scoring rule and has the practical advantage of being upper bounded which should limit the impact of mislabeled examples in the training set.

Would it make sense to publicly such estimators in scikit-learn?

Note that for linear models, fitting the Brier loss is not equivalent to our `RidgeClassifier` because the latter does not take the softmax of its raw predictions before computing the square loss, hence it does **not** offer a well defined estimation of the conditional class probabilities.

Minimizing the Brier loss of a softmax linear model is no longer a convex optimization problem but I doubt that this would prevent a newton solver with a robust line search to converge in practice.

COMMENT:
Pretty much any GLM literature (score equation for general links and canonical links is the key):
- McCullagh & Nelder, Generalized Linear Models (maybe still the best book about GLMs!)
  Chapters 2.2.3, 4.3.1
- https://dx.doi.org/10.2139/ssrn.3822407 Chapter 5.1.5 for the balance property of canonical links which implies good calibration.

For probabilistic classification in particular (no GLM context)
- Buja, Stuerzle & Shen, Loss Functions for Binary Class Probability Estimation
and Classification: Structure and Applications
   http://www-stat.wharton.upenn.edu/~buja/PAPERS/paper-proper-scoring.pdf

The last point about efficiency (for fitting) is a property of  Maximum Likelihood Estimation Theory (achieving the Cramer-Rao lower bound asymptotically).

Labels: RFC
RFC classifiers trained by minimizing the Brier loss

At the moment, our probablistic classifiers (e.g. logistic regression and gradient boosted trees) optimize the log loss, typically after taking a sigmoid or softmax inverse link function (typically part of the Cython loss).

However the log-loss is not the only proper-scoring rule to fit estimators of the expected conditional class probabilities, in particular the Brier loss is also a proper scoring rule and has the practical advantage of being upper bounded which should limit the impact of mislabeled examples in the training set.

Would it make sense to publicly such estimators in scikit-learn?

Note that for linear models, fitting the Brier loss is not equivalent to our `RidgeClassifier` because the latter does not take the softmax of its raw predictions before computing the square loss, hence it does **not** offer a well defined estimation of the conditional class probabilities.

Minimizing the Brier loss of a softmax linear model is no longer a convex optimization problem but I doubt that this would prevent a newton solver with a robust line search to converge in practice.

COMMENT:
As said earlier, training a model with Brier score and - ideally - identity as link function would work with constraint optimization, such that `max(abs(predict_proba)) <= 1`.
- For linear models, one could enforce this constraint with suitable optimizers.
- For trees and random forests, one predicts the empirical mean on each leaf. So this constraint is automatically fulfilled and already available in scikit-learn with MSE based trees on 0-1 encoded response y.
- For gradient boosting, it might be difficult to enforce it.
- For neural nets, ...

So overall, if someones needs a bachelor thesis or a pet project, this might be fun :wink:

Labels: RFC