Talip7/scikit-learn-issues-multilabel huggingface dataset:
SplineTransformer(extrapolate="periodic") outputs nan values for constant features

While reviewing #24145 I discovered the following bug:

```python
In [1]: import numpy as np

In [2]: from sklearn.preprocessing import SplineTransformer

In [3]: SplineTransformer(extrapolation="periodic").fit_transform(np.ones(shape=(5, 1)))
Out[3]: 
array([[nan, nan, nan, nan],
       [nan, nan, nan, nan],
       [nan, nan, nan, nan],
       [nan, nan, nan, nan],
       [nan, nan, nan, nan]])
```

Batman!

This is caused by:

- https://github.com/scikit-learn/scikit-learn/blob/ea046f024694b9a558c882b8c2610c52dad95e29/sklearn/preprocessing/_polynomial.py#L962-L971

`spl.t[n] - spl.t[spl.k]` is typically zero for constant features.

While fixing it for constant features is probably easy (just use `x = X[:, i]`), I wonder if other related numerical stability problems can be triggered for nearly constant features.

But maybe we can solve this problem in two stages: first the nan problem caused by modulus by exact zero and then investigate behavior on nearly constant data.

COMMENT:
We probably need to detect near constant features similarly to what we do in `StandardScaler`:

https://github.com/scikit-learn/scikit-learn/blob/41b0bd82ffa90c00778c6b69f400d8ddf5775eb2/sklearn/preprocessing/_data.py#L74-L87

Labels: Bug
SplineTransformer(extrapolate="periodic") outputs nan values for constant features

While reviewing #24145 I discovered the following bug:

```python
In [1]: import numpy as np

In [2]: from sklearn.preprocessing import SplineTransformer

In [3]: SplineTransformer(extrapolation="periodic").fit_transform(np.ones(shape=(5, 1)))
Out[3]: 
array([[nan, nan, nan, nan],
       [nan, nan, nan, nan],
       [nan, nan, nan, nan],
       [nan, nan, nan, nan],
       [nan, nan, nan, nan]])
```

Batman!

This is caused by:

- https://github.com/scikit-learn/scikit-learn/blob/ea046f024694b9a558c882b8c2610c52dad95e29/sklearn/preprocessing/_polynomial.py#L962-L971

`spl.t[n] - spl.t[spl.k]` is typically zero for constant features.

While fixing it for constant features is probably easy (just use `x = X[:, i]`), I wonder if other related numerical stability problems can be triggered for nearly constant features.

But maybe we can solve this problem in two stages: first the nan problem caused by modulus by exact zero and then investigate behavior on nearly constant data.

COMMENT:
Hi @ogrisel ,

I am new to open source and would love to work on this issue with your help and guidance.

Labels: Bug
SplineTransformer(extrapolate="periodic") outputs nan values for constant features

While reviewing #24145 I discovered the following bug:

```python
In [1]: import numpy as np

In [2]: from sklearn.preprocessing import SplineTransformer

In [3]: SplineTransformer(extrapolation="periodic").fit_transform(np.ones(shape=(5, 1)))
Out[3]: 
array([[nan, nan, nan, nan],
       [nan, nan, nan, nan],
       [nan, nan, nan, nan],
       [nan, nan, nan, nan],
       [nan, nan, nan, nan]])
```

Batman!

This is caused by:

- https://github.com/scikit-learn/scikit-learn/blob/ea046f024694b9a558c882b8c2610c52dad95e29/sklearn/preprocessing/_polynomial.py#L962-L971

`spl.t[n] - spl.t[spl.k]` is typically zero for constant features.

While fixing it for constant features is probably easy (just use `x = X[:, i]`), I wonder if other related numerical stability problems can be triggered for nearly constant features.

But maybe we can solve this problem in two stages: first the nan problem caused by modulus by exact zero and then investigate behavior on nearly constant data.

COMMENT:
Hi @sqali 

Are you still interested in contributing to this issue? If so then you would need to see how _is_constant_feature is being used inside StandardScaler and apply it similarly here to detect for the constant features. Also you might want to first check the contributing guide for how to setup the project on your machine https://scikit-learn.org/stable/developers/contributing.html#contributing-code.

Labels: Bug
SplineTransformer(extrapolate="periodic") outputs nan values for constant features

While reviewing #24145 I discovered the following bug:

```python
In [1]: import numpy as np

In [2]: from sklearn.preprocessing import SplineTransformer

In [3]: SplineTransformer(extrapolation="periodic").fit_transform(np.ones(shape=(5, 1)))
Out[3]: 
array([[nan, nan, nan, nan],
       [nan, nan, nan, nan],
       [nan, nan, nan, nan],
       [nan, nan, nan, nan],
       [nan, nan, nan, nan]])
```

Batman!

This is caused by:

- https://github.com/scikit-learn/scikit-learn/blob/ea046f024694b9a558c882b8c2610c52dad95e29/sklearn/preprocessing/_polynomial.py#L962-L971

`spl.t[n] - spl.t[spl.k]` is typically zero for constant features.

While fixing it for constant features is probably easy (just use `x = X[:, i]`), I wonder if other related numerical stability problems can be triggered for nearly constant features.

But maybe we can solve this problem in two stages: first the nan problem caused by modulus by exact zero and then investigate behavior on nearly constant data.

COMMENT:
Hi @OmarManzoor ,

Yes, Thanks for guiding me. I will do the required tests and update.

Labels: Bug
SplineTransformer(extrapolate="periodic") outputs nan values for constant features

While reviewing #24145 I discovered the following bug:

```python
In [1]: import numpy as np

In [2]: from sklearn.preprocessing import SplineTransformer

In [3]: SplineTransformer(extrapolation="periodic").fit_transform(np.ones(shape=(5, 1)))
Out[3]: 
array([[nan, nan, nan, nan],
       [nan, nan, nan, nan],
       [nan, nan, nan, nan],
       [nan, nan, nan, nan],
       [nan, nan, nan, nan]])
```

Batman!

This is caused by:

- https://github.com/scikit-learn/scikit-learn/blob/ea046f024694b9a558c882b8c2610c52dad95e29/sklearn/preprocessing/_polynomial.py#L962-L971

`spl.t[n] - spl.t[spl.k]` is typically zero for constant features.

While fixing it for constant features is probably easy (just use `x = X[:, i]`), I wonder if other related numerical stability problems can be triggered for nearly constant features.

But maybe we can solve this problem in two stages: first the nan problem caused by modulus by exact zero and then investigate behavior on nearly constant data.

COMMENT:
Hey, I am working on this issue.

Labels: Bug
⚠️ CI failed on linux_arm64_wheel ⚠️

**CI failed on [linux_arm64_wheel](https://cirrus-ci.com/build/6501559970824192)** (May 15, 2023)

COMMENT:
## CI is no longer failing! ✅

[Successful run](https://cirrus-ci.com/build/4585352225095680) on May 16, 2023

Labels: Needs Triage
SequentialFeatureSelector in backward auto mode will always remove one feature

https://github.com/scikit-learn/scikit-learn/blob/364c77e047ca08a95862becf40a04fe9d4cd2c98/sklearn/feature_selection/_sequential.py#L273

The initial value of `old_score` is incorrect if `direction == 'backward'`. With the current initial value of `-np.inf, ((new_score - old_score) < self.tol)` would always be `False` no matter what `new_score` is returned by `self._get_best_new_feature_score()`, so that at least one feature would be removed in the first iteration.

`old_score` needs to be set to an initial value that is the `cross_val_score` with all the features in use if `direction == 'backward'`.

COMMENT:
We might also be able to do this by comparing the absolute value of the change in scores to the absolute value of the tolerance. The only catch is that we then lose the information about the direction of the delta.

I'm thinking about this a bit more, it would be nice to have a short bit of code that reproduces the problem. That would help with understanding the problem as well as be a basis for a non-regression test. Do you happen to have a snippet @sktin?

Labels: Bug, module:feature_selection
SequentialFeatureSelector in backward auto mode will always remove one feature

https://github.com/scikit-learn/scikit-learn/blob/364c77e047ca08a95862becf40a04fe9d4cd2c98/sklearn/feature_selection/_sequential.py#L273

The initial value of `old_score` is incorrect if `direction == 'backward'`. With the current initial value of `-np.inf, ((new_score - old_score) < self.tol)` would always be `False` no matter what `new_score` is returned by `self._get_best_new_feature_score()`, so that at least one feature would be removed in the first iteration.

`old_score` needs to be set to an initial value that is the `cross_val_score` with all the features in use if `direction == 'backward'`.

COMMENT:
You can use the following code to reproduce the issue.

```python
import numpy as np
import pandas as pd
from sklearn.model_selection import cross_val_score
from sklearn.feature_selection import SequentialFeatureSelector
from sklearn.model_selection import KFold
from sklearn.neighbors import KNeighborsRegressor

rng = np.random.default_rng(42)
X_train = pd.DataFrame(rng.normal(size=(5000,1)),columns=['x1'])
X_train['x2'] = X_train['x1'] + rng.normal(0,0.1,size=5000)
y_train = (X_train['x1']+X_train['x2'])**2

print(np.mean(cross_val_score(KNeighborsRegressor(),X_train,y_train,scoring='neg_mean_squared_error',
                               cv=KFold(5),n_jobs=-1)))
print(np.mean(cross_val_score(KNeighborsRegressor(),X_train[['x1']],y_train,scoring='neg_mean_squared_error',
                               cv=KFold(5),n_jobs=-1)))
print(np.mean(cross_val_score(KNeighborsRegressor(),X_train[['x2']],y_train,scoring='neg_mean_squared_error',
                               cv=KFold(5),n_jobs=-1)))

sfs = SequentialFeatureSelector(KNeighborsRegressor(), n_features_to_select='auto', tol=0, direction='backward', 
                                scoring='neg_mean_squared_error', cv=KFold(5),n_jobs=-1)
sfs.fit(X_train,y_train)
print(sfs.get_support())
```
Expected output:
```
-0.06691251179385828
-0.277988765632253
-0.2583318568604302
[False  True]
```
It shows that despite the fact that the CV score of using both features is smaller, `SequentialFeatureSelector` in backward mode would remove the first feature regardless.

Labels: Bug, module:feature_selection
Adding dark mode for pipeline diagram

### Describe the workflow you want to enable

The current diagram truly hurts eyes when everything else in dark mode. It would be a very nice feature to add.
![Screenshot 2023-05-14 100107](https://github.com/scikit-learn/scikit-learn/assets/95470962/bf9d7275-be3a-4864-82ce-362a7566d22e)


### Describe your proposed solution


Currently I do it this way, but I bet there could have been a better way.

`
def display_dark(pipeline, text_color="#3bd5eb"):
    replacements = {
        "color: black": f"color: {text_color};",
        "background-color: white": "background-color: transparent",
        'class="sk-toggleable__label sk-toggleable__label-arrow"': 'class="sk-toggleable__label sk-toggleable__label-arrow" style="background-color: #222831;"',
        "#696969;": "#e6e6e6;",
        "#d4ebff;": "#1a1a1a;",
        "background-color: #f0f8ff;": "background-color: #393e46",
        "background-color: #d4ebff;": "background-color: #1a1a1a;",
        "color: #e6e6e6;": "color: #b3b3b3;"
    }

    html = estimator_html_repr(pipeline)

    for old, new in replacements.items():
        html = html.replace(old, new)
    display(HTML(html))
`
![Screenshot 2023-05-14 100143](https://github.com/scikit-learn/scikit-learn/assets/95470962/7d12d2ad-3db7-4947-9fd5-41c588c8cbfe)


### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Nice feature

Labels: New Feature
Adding dark mode for pipeline diagram

### Describe the workflow you want to enable

The current diagram truly hurts eyes when everything else in dark mode. It would be a very nice feature to add.
![Screenshot 2023-05-14 100107](https://github.com/scikit-learn/scikit-learn/assets/95470962/bf9d7275-be3a-4864-82ce-362a7566d22e)


### Describe your proposed solution


Currently I do it this way, but I bet there could have been a better way.

`
def display_dark(pipeline, text_color="#3bd5eb"):
    replacements = {
        "color: black": f"color: {text_color};",
        "background-color: white": "background-color: transparent",
        'class="sk-toggleable__label sk-toggleable__label-arrow"': 'class="sk-toggleable__label sk-toggleable__label-arrow" style="background-color: #222831;"',
        "#696969;": "#e6e6e6;",
        "#d4ebff;": "#1a1a1a;",
        "background-color: #f0f8ff;": "background-color: #393e46",
        "background-color: #d4ebff;": "background-color: #1a1a1a;",
        "color: #e6e6e6;": "color: #b3b3b3;"
    }

    html = estimator_html_repr(pipeline)

    for old, new in replacements.items():
        html = html.replace(old, new)
    display(HTML(html))
`
![Screenshot 2023-05-14 100143](https://github.com/scikit-learn/scikit-learn/assets/95470962/7d12d2ad-3db7-4947-9fd5-41c588c8cbfe)


### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Happy to see a PR for this feature.

Labels: New Feature
Adding dark mode for pipeline diagram

### Describe the workflow you want to enable

The current diagram truly hurts eyes when everything else in dark mode. It would be a very nice feature to add.
![Screenshot 2023-05-14 100107](https://github.com/scikit-learn/scikit-learn/assets/95470962/bf9d7275-be3a-4864-82ce-362a7566d22e)


### Describe your proposed solution


Currently I do it this way, but I bet there could have been a better way.

`
def display_dark(pipeline, text_color="#3bd5eb"):
    replacements = {
        "color: black": f"color: {text_color};",
        "background-color: white": "background-color: transparent",
        'class="sk-toggleable__label sk-toggleable__label-arrow"': 'class="sk-toggleable__label sk-toggleable__label-arrow" style="background-color: #222831;"',
        "#696969;": "#e6e6e6;",
        "#d4ebff;": "#1a1a1a;",
        "background-color: #f0f8ff;": "background-color: #393e46",
        "background-color: #d4ebff;": "background-color: #1a1a1a;",
        "color: #e6e6e6;": "color: #b3b3b3;"
    }

    html = estimator_html_repr(pipeline)

    for old, new in replacements.items():
        html = html.replace(old, new)
    display(HTML(html))
`
![Screenshot 2023-05-14 100143](https://github.com/scikit-learn/scikit-learn/assets/95470962/7d12d2ad-3db7-4947-9fd5-41c588c8cbfe)


### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
/take

Labels: New Feature
Adding dark mode for pipeline diagram

### Describe the workflow you want to enable

The current diagram truly hurts eyes when everything else in dark mode. It would be a very nice feature to add.
![Screenshot 2023-05-14 100107](https://github.com/scikit-learn/scikit-learn/assets/95470962/bf9d7275-be3a-4864-82ce-362a7566d22e)


### Describe your proposed solution


Currently I do it this way, but I bet there could have been a better way.

`
def display_dark(pipeline, text_color="#3bd5eb"):
    replacements = {
        "color: black": f"color: {text_color};",
        "background-color: white": "background-color: transparent",
        'class="sk-toggleable__label sk-toggleable__label-arrow"': 'class="sk-toggleable__label sk-toggleable__label-arrow" style="background-color: #222831;"',
        "#696969;": "#e6e6e6;",
        "#d4ebff;": "#1a1a1a;",
        "background-color: #f0f8ff;": "background-color: #393e46",
        "background-color: #d4ebff;": "background-color: #1a1a1a;",
        "color: #e6e6e6;": "color: #b3b3b3;"
    }

    html = estimator_html_repr(pipeline)

    for old, new in replacements.items():
        html = html.replace(old, new)
    display(HTML(html))
`
![Screenshot 2023-05-14 100143](https://github.com/scikit-learn/scikit-learn/assets/95470962/7d12d2ad-3db7-4947-9fd5-41c588c8cbfe)


### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
@adrinjalali @solo-driven hi please help review too if this suits your needs https://github.com/scikit-learn/scikit-learn/pull/26862 thanks!

Labels: New Feature
Adding dark mode for pipeline diagram

### Describe the workflow you want to enable

The current diagram truly hurts eyes when everything else in dark mode. It would be a very nice feature to add.
![Screenshot 2023-05-14 100107](https://github.com/scikit-learn/scikit-learn/assets/95470962/bf9d7275-be3a-4864-82ce-362a7566d22e)


### Describe your proposed solution


Currently I do it this way, but I bet there could have been a better way.

`
def display_dark(pipeline, text_color="#3bd5eb"):
    replacements = {
        "color: black": f"color: {text_color};",
        "background-color: white": "background-color: transparent",
        'class="sk-toggleable__label sk-toggleable__label-arrow"': 'class="sk-toggleable__label sk-toggleable__label-arrow" style="background-color: #222831;"',
        "#696969;": "#e6e6e6;",
        "#d4ebff;": "#1a1a1a;",
        "background-color: #f0f8ff;": "background-color: #393e46",
        "background-color: #d4ebff;": "background-color: #1a1a1a;",
        "color: #e6e6e6;": "color: #b3b3b3;"
    }

    html = estimator_html_repr(pipeline)

    for old, new in replacements.items():
        html = html.replace(old, new)
    display(HTML(html))
`
![Screenshot 2023-05-14 100143](https://github.com/scikit-learn/scikit-learn/assets/95470962/7d12d2ad-3db7-4947-9fd5-41c588c8cbfe)


### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I'm closing this issue because we already have an improved theme adaptation for some IDE.

Labels: New Feature
⚠️ CI failed on linux_arm64_wheel ⚠️

**CI failed on [linux_arm64_wheel](https://cirrus-ci.com/build/6515525694521344)** (May 14, 2023)

COMMENT:
## CI is no longer failing! ✅

[Successful run](https://cirrus-ci.com/build/6501559970824192) on May 15, 2023

Labels: Needs Triage
Selecting Lasso via cross-validation data leakage

### Describe the issue linked to the documentation

```py
model = make_pipeline(StandardScaler(), LassoCV(cv=20)).fit(X, y)
```

Optimizing Alpha for lasso involves data leakage because standard scaling is applied on X and y in one go, instant for each fold.

It is used in the example Selecting Lasso via cross-validation
https://scikit-learn.org/stable/auto_examples/linear_model/plot_lasso_model_selection.html




### Suggest a potential alternative/fix

_No response_

COMMENT:
Thanks for bringing this up, I've been wondering about our `*CV` classes too. Unless certain conditions are met, we always have leakage in those pipelines, and that's true for all of them (except `*SearchCV` taking the whole pipeline).

We should probably make that very clear, and make sure none of our examples have this issue. And I think if the user has enough CPU cycles at their hand, it's almost always safer to use a `*SearchCV`.

I wonder what @agramfort thinks here.

Labels: Documentation, module:linear_model
Selecting Lasso via cross-validation data leakage

### Describe the issue linked to the documentation

```py
model = make_pipeline(StandardScaler(), LassoCV(cv=20)).fit(X, y)
```

Optimizing Alpha for lasso involves data leakage because standard scaling is applied on X and y in one go, instant for each fold.

It is used in the example Selecting Lasso via cross-validation
https://scikit-learn.org/stable/auto_examples/linear_model/plot_lasso_model_selection.html




### Suggest a potential alternative/fix

_No response_

COMMENT:
do you have empirical evidence that the alpha obtained is one option or the other is very different and that the test performance degrades with the approach `model = make_pipeline(StandardScaler(), LassoCV(cv=20)).fit(X, y)` ?

Labels: Documentation, module:linear_model
Selecting Lasso via cross-validation data leakage

### Describe the issue linked to the documentation

```py
model = make_pipeline(StandardScaler(), LassoCV(cv=20)).fit(X, y)
```

Optimizing Alpha for lasso involves data leakage because standard scaling is applied on X and y in one go, instant for each fold.

It is used in the example Selecting Lasso via cross-validation
https://scikit-learn.org/stable/auto_examples/linear_model/plot_lasso_model_selection.html




### Suggest a potential alternative/fix

_No response_

COMMENT:
@agramfort as soon as there are outliers, they're not the same anymore, here's an example:

```py
from sklearn.datasets import make_regression
from sklearn.linear_model import Lasso, LassoCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import GridSearchCV

X, y = make_regression(n_samples=200, n_features=500, n_informative=5, random_state=0)
X[:2, ] = X[:2, ] +20

est = make_pipeline(StandardScaler(), LassoCV(cv=5)).fit(X, y)

est_gs = GridSearchCV(
    make_pipeline(StandardScaler(), Lasso(random_state=0)),
    param_grid={"lasso__alpha": est[-1].alphas_},
    cv=5,
).fit(X, y)
print(est[-1].alpha_)
print(est_gs.best_estimator_[-1].alpha)
```

Gives these two alphas:

```
44.79962005897805
51.508676553970744
```

Labels: Documentation, module:linear_model
Selecting Lasso via cross-validation data leakage

### Describe the issue linked to the documentation

```py
model = make_pipeline(StandardScaler(), LassoCV(cv=20)).fit(X, y)
```

Optimizing Alpha for lasso involves data leakage because standard scaling is applied on X and y in one go, instant for each fold.

It is used in the example Selecting Lasso via cross-validation
https://scikit-learn.org/stable/auto_examples/linear_model/plot_lasso_model_selection.html




### Suggest a potential alternative/fix

_No response_

COMMENT:
two remarks here:
- Using a GridSearchCV for a Lasso is a terrible waste of time as you start from scratch for each alpha. I suspect a 100x difference in running time unless you accept to have your screened filled with "ConvergenceWarning: Objective did not converge."
- if you have outliers maybe use a RobustScaler? Using a RobustScaler with the snippet above I get as alphas:
66.89463732039701
66.89463732039701
So it seems to do the job.

my 2c

Labels: Documentation, module:linear_model
Selecting Lasso via cross-validation data leakage

### Describe the issue linked to the documentation

```py
model = make_pipeline(StandardScaler(), LassoCV(cv=20)).fit(X, y)
```

Optimizing Alpha for lasso involves data leakage because standard scaling is applied on X and y in one go, instant for each fold.

It is used in the example Selecting Lasso via cross-validation
https://scikit-learn.org/stable/auto_examples/linear_model/plot_lasso_model_selection.html




### Suggest a potential alternative/fix

_No response_

COMMENT:
I think my point is that people need to really know what they're doing if they use something like `LassoCV`. We should have very clear warnings and describe why using is might be okay, and when it can be used.

Labels: Documentation, module:linear_model