Talip7/scikit-learn-issues-multilabel huggingface dataset:
RFC classifiers trained by minimizing the Brier loss

At the moment, our probablistic classifiers (e.g. logistic regression and gradient boosted trees) optimize the log loss, typically after taking a sigmoid or softmax inverse link function (typically part of the Cython loss).

However the log-loss is not the only proper-scoring rule to fit estimators of the expected conditional class probabilities, in particular the Brier loss is also a proper scoring rule and has the practical advantage of being upper bounded which should limit the impact of mislabeled examples in the training set.

Would it make sense to publicly such estimators in scikit-learn?

Note that for linear models, fitting the Brier loss is not equivalent to our `RidgeClassifier` because the latter does not take the softmax of its raw predictions before computing the square loss, hence it does **not** offer a well defined estimation of the conditional class probabilities.

Minimizing the Brier loss of a softmax linear model is no longer a convex optimization problem but I doubt that this would prevent a newton solver with a robust line search to converge in practice.

COMMENT:
https://stats.stackexchange.com/a/326381 also gives a nice explanation. In particular, the problem is not (guaranteed to be) convex anymore. I therefore close.

Labels: RFC
RFC classifiers trained by minimizing the Brier loss

At the moment, our probablistic classifiers (e.g. logistic regression and gradient boosted trees) optimize the log loss, typically after taking a sigmoid or softmax inverse link function (typically part of the Cython loss).

However the log-loss is not the only proper-scoring rule to fit estimators of the expected conditional class probabilities, in particular the Brier loss is also a proper scoring rule and has the practical advantage of being upper bounded which should limit the impact of mislabeled examples in the training set.

Would it make sense to publicly such estimators in scikit-learn?

Note that for linear models, fitting the Brier loss is not equivalent to our `RidgeClassifier` because the latter does not take the softmax of its raw predictions before computing the square loss, hence it does **not** offer a well defined estimation of the conditional class probabilities.

Minimizing the Brier loss of a softmax linear model is no longer a convex optimization problem but I doubt that this would prevent a newton solver with a robust line search to converge in practice.

COMMENT:
> As said earlier, training a model with Brier score and - ideally - identity as link function would work with constraint optimization, such that max(abs(predict_proba)) <= 1.

Note that this constraint alone would not be enough to give valid probabilistic interpretations of the outputs of such a predictive model: one would also need enforce:

- `(predict_proba.all(X) >= 0).all()`
- `assert_allclose(predict_proba(X).sum(axis=1), np.ones(X.shape[0]))`

Basically, constraining the predictions to lie on the simplex.

It's probably much easier to build that constraints directly into the model by using the softmax inverse link function (or a similar alternative).

Labels: RFC
RFC classifiers trained by minimizing the Brier loss

At the moment, our probablistic classifiers (e.g. logistic regression and gradient boosted trees) optimize the log loss, typically after taking a sigmoid or softmax inverse link function (typically part of the Cython loss).

However the log-loss is not the only proper-scoring rule to fit estimators of the expected conditional class probabilities, in particular the Brier loss is also a proper scoring rule and has the practical advantage of being upper bounded which should limit the impact of mislabeled examples in the training set.

Would it make sense to publicly such estimators in scikit-learn?

Note that for linear models, fitting the Brier loss is not equivalent to our `RidgeClassifier` because the latter does not take the softmax of its raw predictions before computing the square loss, hence it does **not** offer a well defined estimation of the conditional class probabilities.

Minimizing the Brier loss of a softmax linear model is no longer a convex optimization problem but I doubt that this would prevent a newton solver with a robust line search to converge in practice.

COMMENT:
Related thought: directly optimizing the Brier score instead of the negative log likelihood to fit a classification model might be interesting to make the fit more sample efficient and robust in the presence of "label noise" (e.g. when 10% of the labels are randomly flipped independently of X). Indeed, the per-sample Brier score is bounded above by 2.0 while the per-sample negative log likelihood is not bounded (unless the training procedure uses hacks such as [label smoothing](https://towardsdatascience.com/label-smoothing-as-another-regularization-trick-7b34c50dc0b9) which is quite popular to prevent neural networks to yield miscalibrated / overconfident predictions).

Indeed in the presence of a mislabeled sample (class B) in a region of the feature space densely populated with examples of the class A, I suspect that a nll-trained classifier will give a lot more importance to the mislabeled example than a Brier-score-trained classifier. As a result, the test performance (either measured with nll or BS) should be lower for the nll-trained classifier than for the Brier-score trained model.

The difference between the two procedures should vanish when the training set size increases since both loss functions are proper scoring rules, but in practice we always work in the finite-sample regime.

Labels: RFC
RFC classifiers trained by minimizing the Brier loss

At the moment, our probablistic classifiers (e.g. logistic regression and gradient boosted trees) optimize the log loss, typically after taking a sigmoid or softmax inverse link function (typically part of the Cython loss).

However the log-loss is not the only proper-scoring rule to fit estimators of the expected conditional class probabilities, in particular the Brier loss is also a proper scoring rule and has the practical advantage of being upper bounded which should limit the impact of mislabeled examples in the training set.

Would it make sense to publicly such estimators in scikit-learn?

Note that for linear models, fitting the Brier loss is not equivalent to our `RidgeClassifier` because the latter does not take the softmax of its raw predictions before computing the square loss, hence it does **not** offer a well defined estimation of the conditional class probabilities.

Minimizing the Brier loss of a softmax linear model is no longer a convex optimization problem but I doubt that this would prevent a newton solver with a robust line search to converge in practice.

COMMENT:
For really small sample sizes, I guess one needs a stochastic model (usually a strong assumption) about the data generating process to make any statement. The next best thing is to have an unbiased estimator which a GLM with canonical link such as logistic regression provides. The next best thing is maximum likelihood theory as it provides a consistent estimate that asymptotically approaches the Cramer-Rao lower bound (of the variance of the estimate).

Note that those statistical arguments are not in conflict with train-test evaluation strategies and cross-validation. The latter are just empirical methods to confirm or reject the statistical statements on real data and a specific model, e.g. detect overfitting.

>directly optimizing the Brier score instead of the negative log likelihood to fit a classification model might be interesting to make the fit more sample efficient and robust in the presence of "label noise"

_When in doubt, use a simulation._

Labels: RFC
RFC classifiers trained by minimizing the Brier loss

At the moment, our probablistic classifiers (e.g. logistic regression and gradient boosted trees) optimize the log loss, typically after taking a sigmoid or softmax inverse link function (typically part of the Cython loss).

However the log-loss is not the only proper-scoring rule to fit estimators of the expected conditional class probabilities, in particular the Brier loss is also a proper scoring rule and has the practical advantage of being upper bounded which should limit the impact of mislabeled examples in the training set.

Would it make sense to publicly such estimators in scikit-learn?

Note that for linear models, fitting the Brier loss is not equivalent to our `RidgeClassifier` because the latter does not take the softmax of its raw predictions before computing the square loss, hence it does **not** offer a well defined estimation of the conditional class probabilities.

Minimizing the Brier loss of a softmax linear model is no longer a convex optimization problem but I doubt that this would prevent a newton solver with a robust line search to converge in practice.

COMMENT:
> When in doubt, use a simulation.

That's my plan, both for linear models and gradient boosted models. One day, I will do it and report back here :)

Labels: RFC
Reference RBFSampler in SVM docs and docstring

### Describe the issue linked to the documentation

I'm sitting in a presentation where the speaker (an academic prof in applied sciences) mentions that RBF kernels with SVMs in scikit-learn was worked best for his problem, but were too slow, so he switched back to linear.

Seems like he should have investigated RBFSampler + linear models.

### Suggest a potential alternative/fix

Reference RBFSampler in various places (narrative docs, docstrings) where SVM (SVC/SVR) are described.

COMMENT:
Hi, I think we can add links of [Kernel Approximation](https://scikit-learn.org/dev/modules/kernel_approximation.html#rbf-kernel-approx) in the [Support Vector Machines](https://scikit-learn.org/dev/modules/svm.html#kernel-functions) documentation to let reader aware of the module. What do you think?

Labels: Documentation, help wanted
Reference RBFSampler in SVM docs and docstring

### Describe the issue linked to the documentation

I'm sitting in a presentation where the speaker (an academic prof in applied sciences) mentions that RBF kernels with SVMs in scikit-learn was worked best for his problem, but were too slow, so he switched back to linear.

Seems like he should have investigated RBFSampler + linear models.

### Suggest a potential alternative/fix

Reference RBFSampler in various places (narrative docs, docstrings) where SVM (SVC/SVR) are described.

COMMENT:
> Hi, I think we can add links of Kernel Approximation in the Support Vector Machines documentation to let reader aware of the module. What do you think?

I think that it would be useful indeed. Probably as a short text (one sentence?) in a "seealso" box)

Labels: Documentation, help wanted
Reference RBFSampler in SVM docs and docstring

### Describe the issue linked to the documentation

I'm sitting in a presentation where the speaker (an academic prof in applied sciences) mentions that RBF kernels with SVMs in scikit-learn was worked best for his problem, but were too slow, so he switched back to linear.

Seems like he should have investigated RBFSampler + linear models.

### Suggest a potential alternative/fix

Reference RBFSampler in various places (narrative docs, docstrings) where SVM (SVC/SVR) are described.

COMMENT:
I have opened a PR to add the link. Feel free to add any comments or suggestions!

Labels: Documentation, help wanted
Reference RBFSampler in SVM docs and docstring

### Describe the issue linked to the documentation

I'm sitting in a presentation where the speaker (an academic prof in applied sciences) mentions that RBF kernels with SVMs in scikit-learn was worked best for his problem, but were too slow, so he switched back to linear.

Seems like he should have investigated RBFSampler + linear models.

### Suggest a potential alternative/fix

Reference RBFSampler in various places (narrative docs, docstrings) where SVM (SVC/SVR) are described.

COMMENT:
It seems that we should check if the same note should be added in the docstrings of the SVC / SVR object.

Labels: Documentation, help wanted
Reference RBFSampler in SVM docs and docstring

### Describe the issue linked to the documentation

I'm sitting in a presentation where the speaker (an academic prof in applied sciences) mentions that RBF kernels with SVMs in scikit-learn was worked best for his problem, but were too slow, so he switched back to linear.

Seems like he should have investigated RBFSampler + linear models.

### Suggest a potential alternative/fix

Reference RBFSampler in various places (narrative docs, docstrings) where SVM (SVC/SVR) are described.

COMMENT:
I can open another PR to add the same note in the docstrings if needed.

Labels: Documentation, help wanted
Reference RBFSampler in SVM docs and docstring

### Describe the issue linked to the documentation

I'm sitting in a presentation where the speaker (an academic prof in applied sciences) mentions that RBF kernels with SVMs in scikit-learn was worked best for his problem, but were too slow, so he switched back to linear.

Seems like he should have investigated RBFSampler + linear models.

### Suggest a potential alternative/fix

Reference RBFSampler in various places (narrative docs, docstrings) where SVM (SVC/SVR) are described.

COMMENT:
@haochunchang please feel free to do so.

Labels: Documentation, help wanted
Reference RBFSampler in SVM docs and docstring

### Describe the issue linked to the documentation

I'm sitting in a presentation where the speaker (an academic prof in applied sciences) mentions that RBF kernels with SVMs in scikit-learn was worked best for his problem, but were too slow, so he switched back to linear.

Seems like he should have investigated RBFSampler + linear models.

### Suggest a potential alternative/fix

Reference RBFSampler in various places (narrative docs, docstrings) where SVM (SVC/SVR) are described.

COMMENT:
After checking the docstrings of the SVC / SVR object, I found that they already have links to the Kernel Approximation modules through:

> For large datasets consider using [LinearSVC](https://scikit-learn.org/dev/modules/generated/sklearn.svm.LinearSVC.html#sklearn.svm.LinearSVC) or [SGDClassifier](https://scikit-learn.org/dev/modules/generated/sklearn.linear_model.SGDClassifier.html#sklearn.linear_model.SGDClassifier) instead, possibly after a [Nystroem](https://scikit-learn.org/dev/modules/generated/sklearn.kernel_approximation.Nystroem.html#sklearn.kernel_approximation.Nystroem) transformer.
and

> For large datasets consider using [LinearSVR](https://scikit-learn.org/dev/modules/generated/sklearn.svm.LinearSVR.html#sklearn.svm.LinearSVR) or [SGDRegressor](https://scikit-learn.org/dev/modules/generated/sklearn.linear_model.SGDRegressor.html#sklearn.linear_model.SGDRegressor) instead, possibly after a [Nystroem](https://scikit-learn.org/dev/modules/generated/sklearn.kernel_approximation.Nystroem.html#sklearn.kernel_approximation.Nystroem) transformer.

Therefore, IMHO we don't need to add the same note to the docstrings.
Or, to add more context, we can add the note in the same line such as:

> For large datasets consider using [LinearSVR](https://scikit-learn.org/dev/modules/generated/sklearn.svm.LinearSVR.html#sklearn.svm.LinearSVR) or [SGDRegressor](https://scikit-learn.org/dev/modules/generated/sklearn.linear_model.SGDRegressor.html#sklearn.linear_model.SGDRegressor) instead, possibly after a [Nystroem](https://scikit-learn.org/dev/modules/generated/sklearn.kernel_approximation.Nystroem.html#sklearn.kernel_approximation.Nystroem) transformer or other [kernel approximation](https://scikit-learn.org/dev/modules/classes.html#module-sklearn.kernel_approximation).

Labels: Documentation, help wanted
Bayesian Ridge Regression

### Describe the bug

Using the Bayesian Ridge Regression with MultiOutputRegression(n_jobs=-1) consumes all the memory on the computer and the process dies. (Tested with 128G Ram + 128G Swap)

### Steps/Code to Reproduce

model = MultiOutputRegressor(sklearn.linear_model.BayesianRidge(), n_jobs=-1)
model = model.fit(train_lr, train_hr)
pred = model.predict(test_lr)

### Expected Results

No memory leak.

### Actual Results

Memory leak occurs

### Versions

```shell
tested on
scikit-learn==1.0.2
scikit-learn==0.24.2
```

COMMENT:
Can you provide information regarding the size of the matrix `X` and `y`?
How many processors do you have since you are using `n_jobs=-1`?

Looking at the code of the `BayesianRidge`, it involves an `SVD`. Therefore, with a huge matrix `X` and many outputs in `y`, you will have several decompositions in each process and it might blow up your memory. But we need more info to know.

Labels: module:linear_model, module:multioutput, Needs Investigation
Bayesian Ridge Regression

### Describe the bug

Using the Bayesian Ridge Regression with MultiOutputRegression(n_jobs=-1) consumes all the memory on the computer and the process dies. (Tested with 128G Ram + 128G Swap)

### Steps/Code to Reproduce

model = MultiOutputRegressor(sklearn.linear_model.BayesianRidge(), n_jobs=-1)
model = model.fit(train_lr, train_hr)
pred = model.predict(test_lr)

### Expected Results

No memory leak.

### Actual Results

Memory leak occurs

### Versions

```shell
tested on
scikit-learn==1.0.2
scikit-learn==0.24.2
```

COMMENT:
X.shape => (189, 12720)
Y.shape => (189, 35778)
20 Cores, 128G Ram, 256G Swap

Labels: module:linear_model, module:multioutput, Needs Investigation
Fix resize segfault

The following code segment should be adjusted:
```python
self.capacity = node_ndarray.shape[0]
if self._resize_c(self.capacity) != 0:
```
It should instead be just `if self._resize_c(node_ndarray.shape[0]) != 0:`. Allow me to explain why:

In the `resize_c` function, the initial check is `if capacity == self.capacity and self.nodes != NULL:` . If we are editing an existing tree (which is probably bad practice, but it has several use cases so it might as well be supported for people that know what they're doing right?), the nodes are not NULL. This means that, because we pass `self.capacity` as an argument, it will evaluate to true and thus return 0 without resizing.

If the replacement I suggest is implemented, the resize will be executed and - as an effect of the resize function - the capacity will still be updated, but at the right moment.

Currently trying to load the state of one (large) tree into the state of another (small) tree will result in a segfault on certain operations.

https://github.com/scikit-learn/scikit-learn/blob/8ea2997dfcd72126cdd6793a2c5293406407b1dc/sklearn/tree/_tree.pyx#L699

COMMENT:
If it does not introduce any regression elsewhere, why not implement this change.
Could you provide an example where the code will segfault?

Labels: module:tree, Needs Investigation
RFECV regression output of cv_results_ values always of length 11 independent of n_features_

I am trying to use RFECV on a regression task, with a dataset that has a high feature to samples ratio and contains a lot of noise. As expected, the calculated value of `n_features` is quite high.

However contrary to the documentation, the ndarrays contained in `cv_results` are *not* of shape `(n_features_,)`, but consistently seem to be of length 11 no matter how much I play around with the various script parameters (such as number of folds, example dataset properties, etc.)

I re-read the documentation several times to make sure that I am using this properly. Example codes available on various sources usually present this method in a classification context, so one guess is that this function is somehow not well suited for regression tasks?!

Below is a working minimal code example that shows versions numbers of installed packages as well as the observed behaviour:
```
import sys
import numpy as np
import sklearn
from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import KFold
from sklearn.feature_selection import RFECV
from sklearn.ensemble import RandomForestRegressor
from sklearn.datasets import make_regression

print('Python version:', sys.version)
print('Sklearn version:', sklearn.__version__)
print('Numpy version:', np.__version__)

seed=42

# approximate my real life dataset + preprocessing steps
X, y = make_regression(
    n_samples=60, n_features=1260, n_informative=250, n_targets=1, effective_rank=7, random_state=seed)
scaler_x = MinMaxScaler(feature_range=(-1, 1))
scaler_y = MinMaxScaler(feature_range=(0, 1))
X_sc = scaler_x.fit_transform(X)
y_sc = scaler_y.fit_transform(y.reshape(-1, 1))

# issue was tested on various numbers of cv folds
rf = RandomForestRegressor(n_estimators=100, n_jobs=-1, oob_score=True, bootstrap=True, random_state=seed)
kf = KFold(n_splits=5, shuffle=True, random_state=seed)

rfecv = RFECV(
    estimator=rf,
    step=0.1,
    cv=kf,
    scoring='neg_root_mean_squared_error'
)
rfecv.fit(X_sc, np.ravel(y_sc))

print('Optimal number of features :', rfecv.n_features_)
print('Length of cv_results_[mean_test_score]:', len(rfecv.cv_results_['mean_test_score']))
```

Output:

Python version: 3.10.4 | packaged by conda-forge | (main, Mar 30 2022, 08:38:02) [MSC v.1916 64 bit (AMD64)]
Sklearn version: 1.1.1
Numpy version: 1.22.4
Optimal number of features : 252
Length of cv_results_[mean_test_score]: 11

COMMENT:
The documentation for `cv_results_` is incorrect. It should be `n_subsets_of_features`. 

Note that `n_subsets_of_features` can be computed:

```python
import math

n_features = X.shape[0]

# since a float was used
step_int = rfecv.step * n_features
math.ceil((n_features - rfecv.min_features_to_select) / step_int) + 1
# 11
```

Do you want to open a PR to correct the documentation of `RFECV.cv_results_`?

Labels: Documentation, module:feature_selection