Talip7/scikit-learn-issues-multilabel huggingface dataset:
Ridge and RidgeCV has different result even if alpha is the same

### Describe the bug

(From discussion in https://github.com/scikit-learn/scikit-learn/discussions/26733)

In one of my dataset, I try to specify only one value in alphas in RidgeCV, and expects RidgeCV should give me the same result as using Ridge with same alpha value. But it would not (see the code and output below). And the RidgeCV result seems abnormal.

I found that the problem is related to some large values we used in sample_weight. I can reproduce it with make_regression.

I guess the large value in sample_weight may caused some overflow in RidgeCV intermediate calculation?
Given that RidgeCV and Ridge are using different solvers under the hood, I think it is not surprise that they may behave differently in this case? I just scaled down the weight and the problem is gone.



### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.linear_model import Ridge, RidgeCV
from sklearn.datasets import make_regression

X, y = make_regression(n_features=2)
wgt = np.random.randint(2e5, 2e11, size=100)

print('=== RidgeCV ===')
clf = RidgeCV(alphas=[0.001],fit_intercept=True).fit(
    X=X,
    y=y,
    sample_weight=wgt,
)
print(f'{clf.coef_ = }')
print(f'{clf.intercept_ = }')
print(f'{clf.alpha_ = }')

print('=== Ridge ===')
clf = Ridge(alpha=0.001,fit_intercept=True).fit(
    X=X,
    y=y,
    sample_weight=wgt
)
print(f'{clf.coef_ = }')
print(f'{clf.intercept_ = }')
```

### Expected Results

The output of RidgeCV and Ridge is the same.

### Actual Results

```
=== RidgeCV ===
'clf.coef_ = array([-95.63781897,  31.72140137])'
'clf.intercept_ = 2.6375015138569937'
'clf.alpha_ = 0.001'
=== Ridge ===
'clf.coef_ = array([41.8887401, 33.131992 ])'
'clf.intercept_ = -4.440892098500626e-16'
```

### Versions

```shell
I am using scikit-learn 1.2.1 on python 3.8 on Linux.
```

COMMENT:
If you set `cv` parameter in `RidgeCV`, you will get the same results. Smaller range of sample weights also gives identical values. Furthermore, if you set `gcv_mode` to `"eigen"` you will also get the same result. Thus bug is somewhere in `_RidgeGCV` in  `"svd"` mode (both sparse and dense options, wether we fit intercept or not does not make any difference).

Labels: Bug, module:linear_model, Needs Investigation, Numerical Stability
Ridge and RidgeCV has different result even if alpha is the same

### Describe the bug

(From discussion in https://github.com/scikit-learn/scikit-learn/discussions/26733)

In one of my dataset, I try to specify only one value in alphas in RidgeCV, and expects RidgeCV should give me the same result as using Ridge with same alpha value. But it would not (see the code and output below). And the RidgeCV result seems abnormal.

I found that the problem is related to some large values we used in sample_weight. I can reproduce it with make_regression.

I guess the large value in sample_weight may caused some overflow in RidgeCV intermediate calculation?
Given that RidgeCV and Ridge are using different solvers under the hood, I think it is not surprise that they may behave differently in this case? I just scaled down the weight and the problem is gone.



### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.linear_model import Ridge, RidgeCV
from sklearn.datasets import make_regression

X, y = make_regression(n_features=2)
wgt = np.random.randint(2e5, 2e11, size=100)

print('=== RidgeCV ===')
clf = RidgeCV(alphas=[0.001],fit_intercept=True).fit(
    X=X,
    y=y,
    sample_weight=wgt,
)
print(f'{clf.coef_ = }')
print(f'{clf.intercept_ = }')
print(f'{clf.alpha_ = }')

print('=== Ridge ===')
clf = Ridge(alpha=0.001,fit_intercept=True).fit(
    X=X,
    y=y,
    sample_weight=wgt
)
print(f'{clf.coef_ = }')
print(f'{clf.intercept_ = }')
```

### Expected Results

The output of RidgeCV and Ridge is the same.

### Actual Results

```
=== RidgeCV ===
'clf.coef_ = array([-95.63781897,  31.72140137])'
'clf.intercept_ = 2.6375015138569937'
'clf.alpha_ = 0.001'
=== Ridge ===
'clf.coef_ = array([41.8887401, 33.131992 ])'
'clf.intercept_ = -4.440892098500626e-16'
```

### Versions

```shell
I am using scikit-learn 1.2.1 on python 3.8 on Linux.
```

COMMENT:
This is also affecting `RidgeClassifier` and `RidgeClassifierCV`, since after `_check_gcv_mode` call, `"svd"` solver mode will be returned.

Labels: Bug, module:linear_model, Needs Investigation, Numerical Stability
Ridge and RidgeCV has different result even if alpha is the same

### Describe the bug

(From discussion in https://github.com/scikit-learn/scikit-learn/discussions/26733)

In one of my dataset, I try to specify only one value in alphas in RidgeCV, and expects RidgeCV should give me the same result as using Ridge with same alpha value. But it would not (see the code and output below). And the RidgeCV result seems abnormal.

I found that the problem is related to some large values we used in sample_weight. I can reproduce it with make_regression.

I guess the large value in sample_weight may caused some overflow in RidgeCV intermediate calculation?
Given that RidgeCV and Ridge are using different solvers under the hood, I think it is not surprise that they may behave differently in this case? I just scaled down the weight and the problem is gone.



### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.linear_model import Ridge, RidgeCV
from sklearn.datasets import make_regression

X, y = make_regression(n_features=2)
wgt = np.random.randint(2e5, 2e11, size=100)

print('=== RidgeCV ===')
clf = RidgeCV(alphas=[0.001],fit_intercept=True).fit(
    X=X,
    y=y,
    sample_weight=wgt,
)
print(f'{clf.coef_ = }')
print(f'{clf.intercept_ = }')
print(f'{clf.alpha_ = }')

print('=== Ridge ===')
clf = Ridge(alpha=0.001,fit_intercept=True).fit(
    X=X,
    y=y,
    sample_weight=wgt
)
print(f'{clf.coef_ = }')
print(f'{clf.intercept_ = }')
```

### Expected Results

The output of RidgeCV and Ridge is the same.

### Actual Results

```
=== RidgeCV ===
'clf.coef_ = array([-95.63781897,  31.72140137])'
'clf.intercept_ = 2.6375015138569937'
'clf.alpha_ = 0.001'
=== Ridge ===
'clf.coef_ = array([41.8887401, 33.131992 ])'
'clf.intercept_ = -4.440892098500626e-16'
```

### Versions

```shell
I am using scikit-learn 1.2.1 on python 3.8 on Linux.
```

COMMENT:
I think this is not specific to weights or even exclusive to the svd, but about numerical instability for very large values?

Try:

```
import numpy as np

from sklearn.linear_model._ridge import _RidgeGCV

est = _RidgeGCV(fit_intercept=False)

n = 1000
y = np.random.randn(n)
X = np.random.randn(n, 2)
X[:, 0] += y
X[:, 0] -= X[:, 0].mean()

sqrt_sw = np.ones(n, dtype=X.dtype)  # no weights
alpha = 1.

algos = {
    'eigen': (est._eigen_decompose_gram, est._solve_eigen_gram),
    'svd': (est._svd_decompose_design_matrix, est._solve_svd_design_matrix),
}

for scale in (1e0, 1e1, 1e7, 1e12):
    X_ = X * scale
    for algo, (decompose, solve) in algos.items():
        X_mean, *decomposition = decompose(X_, y, sqrt_sw)
        _, dual_coefs = solve(alpha, y, sqrt_sw, X_mean, *decomposition)
        print(f'Algorithm: {algo}, Scaling: {scale}, Coefs: {(dual_coefs.T @ X_) * scale}')
```
        
SVD breaks down faster but eventually eigen also breaks down.

The workaround would be rescaling the inputs *and* (if applicable) the weights.

Labels: Bug, module:linear_model, Needs Investigation, Numerical Stability
Ridge and RidgeCV has different result even if alpha is the same

### Describe the bug

(From discussion in https://github.com/scikit-learn/scikit-learn/discussions/26733)

In one of my dataset, I try to specify only one value in alphas in RidgeCV, and expects RidgeCV should give me the same result as using Ridge with same alpha value. But it would not (see the code and output below). And the RidgeCV result seems abnormal.

I found that the problem is related to some large values we used in sample_weight. I can reproduce it with make_regression.

I guess the large value in sample_weight may caused some overflow in RidgeCV intermediate calculation?
Given that RidgeCV and Ridge are using different solvers under the hood, I think it is not surprise that they may behave differently in this case? I just scaled down the weight and the problem is gone.



### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.linear_model import Ridge, RidgeCV
from sklearn.datasets import make_regression

X, y = make_regression(n_features=2)
wgt = np.random.randint(2e5, 2e11, size=100)

print('=== RidgeCV ===')
clf = RidgeCV(alphas=[0.001],fit_intercept=True).fit(
    X=X,
    y=y,
    sample_weight=wgt,
)
print(f'{clf.coef_ = }')
print(f'{clf.intercept_ = }')
print(f'{clf.alpha_ = }')

print('=== Ridge ===')
clf = Ridge(alpha=0.001,fit_intercept=True).fit(
    X=X,
    y=y,
    sample_weight=wgt
)
print(f'{clf.coef_ = }')
print(f'{clf.intercept_ = }')
```

### Expected Results

The output of RidgeCV and Ridge is the same.

### Actual Results

```
=== RidgeCV ===
'clf.coef_ = array([-95.63781897,  31.72140137])'
'clf.intercept_ = 2.6375015138569937'
'clf.alpha_ = 0.001'
=== Ridge ===
'clf.coef_ = array([41.8887401, 33.131992 ])'
'clf.intercept_ = -4.440892098500626e-16'
```

### Versions

```shell
I am using scikit-learn 1.2.1 on python 3.8 on Linux.
```

COMMENT:
I don't think it's reasonable to force scaling on input,  but it's possible to give a warning if input range/dispersion is too big. Scaling sample weights is needed but, there should be a consensus on how to do it correctly.

Labels: Bug, module:linear_model, Needs Investigation, Numerical Stability
Ridge and RidgeCV has different result even if alpha is the same

### Describe the bug

(From discussion in https://github.com/scikit-learn/scikit-learn/discussions/26733)

In one of my dataset, I try to specify only one value in alphas in RidgeCV, and expects RidgeCV should give me the same result as using Ridge with same alpha value. But it would not (see the code and output below). And the RidgeCV result seems abnormal.

I found that the problem is related to some large values we used in sample_weight. I can reproduce it with make_regression.

I guess the large value in sample_weight may caused some overflow in RidgeCV intermediate calculation?
Given that RidgeCV and Ridge are using different solvers under the hood, I think it is not surprise that they may behave differently in this case? I just scaled down the weight and the problem is gone.



### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.linear_model import Ridge, RidgeCV
from sklearn.datasets import make_regression

X, y = make_regression(n_features=2)
wgt = np.random.randint(2e5, 2e11, size=100)

print('=== RidgeCV ===')
clf = RidgeCV(alphas=[0.001],fit_intercept=True).fit(
    X=X,
    y=y,
    sample_weight=wgt,
)
print(f'{clf.coef_ = }')
print(f'{clf.intercept_ = }')
print(f'{clf.alpha_ = }')

print('=== Ridge ===')
clf = Ridge(alpha=0.001,fit_intercept=True).fit(
    X=X,
    y=y,
    sample_weight=wgt
)
print(f'{clf.coef_ = }')
print(f'{clf.intercept_ = }')
```

### Expected Results

The output of RidgeCV and Ridge is the same.

### Actual Results

```
=== RidgeCV ===
'clf.coef_ = array([-95.63781897,  31.72140137])'
'clf.intercept_ = 2.6375015138569937'
'clf.alpha_ = 0.001'
=== Ridge ===
'clf.coef_ = array([41.8887401, 33.131992 ])'
'clf.intercept_ = -4.440892098500626e-16'
```

### Versions

```shell
I am using scikit-learn 1.2.1 on python 3.8 on Linux.
```

COMMENT:
I think it might also be possible to rescale internally, transparently.

Although a warning internally would be the cheap and available option. `StandardScaler` for example also has such a warning.

Labels: Bug, module:linear_model, Needs Investigation, Numerical Stability
Ridge and RidgeCV has different result even if alpha is the same

### Describe the bug

(From discussion in https://github.com/scikit-learn/scikit-learn/discussions/26733)

In one of my dataset, I try to specify only one value in alphas in RidgeCV, and expects RidgeCV should give me the same result as using Ridge with same alpha value. But it would not (see the code and output below). And the RidgeCV result seems abnormal.

I found that the problem is related to some large values we used in sample_weight. I can reproduce it with make_regression.

I guess the large value in sample_weight may caused some overflow in RidgeCV intermediate calculation?
Given that RidgeCV and Ridge are using different solvers under the hood, I think it is not surprise that they may behave differently in this case? I just scaled down the weight and the problem is gone.



### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.linear_model import Ridge, RidgeCV
from sklearn.datasets import make_regression

X, y = make_regression(n_features=2)
wgt = np.random.randint(2e5, 2e11, size=100)

print('=== RidgeCV ===')
clf = RidgeCV(alphas=[0.001],fit_intercept=True).fit(
    X=X,
    y=y,
    sample_weight=wgt,
)
print(f'{clf.coef_ = }')
print(f'{clf.intercept_ = }')
print(f'{clf.alpha_ = }')

print('=== Ridge ===')
clf = Ridge(alpha=0.001,fit_intercept=True).fit(
    X=X,
    y=y,
    sample_weight=wgt
)
print(f'{clf.coef_ = }')
print(f'{clf.intercept_ = }')
```

### Expected Results

The output of RidgeCV and Ridge is the same.

### Actual Results

```
=== RidgeCV ===
'clf.coef_ = array([-95.63781897,  31.72140137])'
'clf.intercept_ = 2.6375015138569937'
'clf.alpha_ = 0.001'
=== Ridge ===
'clf.coef_ = array([41.8887401, 33.131992 ])'
'clf.intercept_ = -4.440892098500626e-16'
```

### Versions

```shell
I am using scikit-learn 1.2.1 on python 3.8 on Linux.
```

COMMENT:
According to the above simulations, things start breaking down around a scale of 100_000 for a feature.

Labels: Bug, module:linear_model, Needs Investigation, Numerical Stability
Ridge and RidgeCV has different result even if alpha is the same

### Describe the bug

(From discussion in https://github.com/scikit-learn/scikit-learn/discussions/26733)

In one of my dataset, I try to specify only one value in alphas in RidgeCV, and expects RidgeCV should give me the same result as using Ridge with same alpha value. But it would not (see the code and output below). And the RidgeCV result seems abnormal.

I found that the problem is related to some large values we used in sample_weight. I can reproduce it with make_regression.

I guess the large value in sample_weight may caused some overflow in RidgeCV intermediate calculation?
Given that RidgeCV and Ridge are using different solvers under the hood, I think it is not surprise that they may behave differently in this case? I just scaled down the weight and the problem is gone.



### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.linear_model import Ridge, RidgeCV
from sklearn.datasets import make_regression

X, y = make_regression(n_features=2)
wgt = np.random.randint(2e5, 2e11, size=100)

print('=== RidgeCV ===')
clf = RidgeCV(alphas=[0.001],fit_intercept=True).fit(
    X=X,
    y=y,
    sample_weight=wgt,
)
print(f'{clf.coef_ = }')
print(f'{clf.intercept_ = }')
print(f'{clf.alpha_ = }')

print('=== Ridge ===')
clf = Ridge(alpha=0.001,fit_intercept=True).fit(
    X=X,
    y=y,
    sample_weight=wgt
)
print(f'{clf.coef_ = }')
print(f'{clf.intercept_ = }')
```

### Expected Results

The output of RidgeCV and Ridge is the same.

### Actual Results

```
=== RidgeCV ===
'clf.coef_ = array([-95.63781897,  31.72140137])'
'clf.intercept_ = 2.6375015138569937'
'clf.alpha_ = 0.001'
=== Ridge ===
'clf.coef_ = array([41.8887401, 33.131992 ])'
'clf.intercept_ = -4.440892098500626e-16'
```

### Versions

```shell
I am using scikit-learn 1.2.1 on python 3.8 on Linux.
```

COMMENT:
As stated, this bug seems to be caused by the large sample weights. A possible solution could be to rescale the sample weights internally and therefore also the alpha values.

But before trying this, I would be interested to know where exactly the error happens.

Labels: Bug, module:linear_model, Needs Investigation, Numerical Stability
Where is CharNGramAnalyzer?

### Describe the issue linked to the documentation

In the example exercises for working with text data, the text says to use CharNGramAnalyzer. Unless I am missing something, like that this is part of some other class, I cannot find this anywhere in the docs, nor in the code.
Documentation:
https://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html#exercise-1-language-identification

### Suggest a potential alternative/fix

Based on what the answer is, whether this class does not exist anymore or refers to something else, I think the text exercise can be adapted. I can do a PR for this, once I know the answer. I also tried googling for an answer, so it might also be that I m missing something obvious that will explain it.

COMMENT:
The `CharNGramAnalyzer` has been removed in 2012. See [this issue](https://github.com/scikit-learn/scikit-learn/issues/990) for more information. 

The alternative to be used is the [sklearn.feature_extraction.text](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.feature_extraction.text) submodule, where users can feed n-grams directly into the [text.CountVectorizer](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html#sklearn.feature_extraction.text.CountVectorizer).

I'd propose to mention the `text.CountVectorizer `in the exercise description.

Labels: Documentation, Needs Triage
Where is CharNGramAnalyzer?

### Describe the issue linked to the documentation

In the example exercises for working with text data, the text says to use CharNGramAnalyzer. Unless I am missing something, like that this is part of some other class, I cannot find this anywhere in the docs, nor in the code.
Documentation:
https://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html#exercise-1-language-identification

### Suggest a potential alternative/fix

Based on what the answer is, whether this class does not exist anymore or refers to something else, I think the text exercise can be adapted. I can do a PR for this, once I know the answer. I also tried googling for an answer, so it might also be that I m missing something obvious that will explain it.

COMMENT:
Thanks for the report and the analysis of the problem. Would one of you be interested in opening a PR to fix the tutorial accordingly?

Labels: Documentation, Needs Triage
Where is CharNGramAnalyzer?

### Describe the issue linked to the documentation

In the example exercises for working with text data, the text says to use CharNGramAnalyzer. Unless I am missing something, like that this is part of some other class, I cannot find this anywhere in the docs, nor in the code.
Documentation:
https://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html#exercise-1-language-identification

### Suggest a potential alternative/fix

Based on what the answer is, whether this class does not exist anymore or refers to something else, I think the text exercise can be adapted. I can do a PR for this, once I know the answer. I also tried googling for an answer, so it might also be that I m missing something obvious that will explain it.

COMMENT:
I think @step21 wanted to make a PR for this. Otherwise I can step in.

Labels: Documentation, Needs Triage
Where is CharNGramAnalyzer?

### Describe the issue linked to the documentation

In the example exercises for working with text data, the text says to use CharNGramAnalyzer. Unless I am missing something, like that this is part of some other class, I cannot find this anywhere in the docs, nor in the code.
Documentation:
https://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html#exercise-1-language-identification

### Suggest a potential alternative/fix

Based on what the answer is, whether this class does not exist anymore or refers to something else, I think the text exercise can be adapted. I can do a PR for this, once I know the answer. I also tried googling for an answer, so it might also be that I m missing something obvious that will explain it.

COMMENT:
I can do it later today.

Labels: Documentation, Needs Triage
Where is CharNGramAnalyzer?

### Describe the issue linked to the documentation

In the example exercises for working with text data, the text says to use CharNGramAnalyzer. Unless I am missing something, like that this is part of some other class, I cannot find this anywhere in the docs, nor in the code.
Documentation:
https://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html#exercise-1-language-identification

### Suggest a potential alternative/fix

Based on what the answer is, whether this class does not exist anymore or refers to something else, I think the text exercise can be adapted. I can do a PR for this, once I know the answer. I also tried googling for an answer, so it might also be that I m missing something obvious that will explain it.

COMMENT:
Hey, I made the the PR. Feel free to suggest better wording, I saw that the solution uses Tfidf vectorizer directly, but I assume people can figure out that this includes CountVectorizer. Or should I rephrase it to use Tfidf vectoizer?

Labels: Documentation, Needs Triage
TargetEncoder has no way to select the columns being processed

### Describe the workflow you want to enable

`TargetEncoder` currently appears to try and look at the entire X dataframe. That's usually not what I want.

What I want to do is: just target encode one feature, or perhaps a small number of features in a dataset with many features.

Additionally, I want to be able to choose whether to keep the original columns that were target encoded, or drop them.

### Describe your proposed solution

`TargetEncoder` should have an argument indicating which columns (features) it should process. The default there could simply be all columns. The argument should allow the user to specify the features by column index / array index, or by column name.

It should have another option, allowing the user to choose whether they want to keep the original (untransformed) features, or drop them.

There should be a third option there, indicating an optional suffix to be added to the names of the transformed columns. If the original columns are dropped, the suffix could be the null string (no suffix). If the original columns are kept, the default suffix could be some reasonable string such as `_target_encoded`, and the user should be able to override it.

### Describe alternatives you've considered, if relevant

I can probably conjure the right combination of column selection in `ColumnTransformer` plus a bunch of other transformers, but it's unnecessarily complex.

### Additional context

_No response_

COMMENT:
I think there were a discussion about this, and your alternative was the option that was ultimately chosen as the way to go.

Labels: New Feature, Needs Triage