Talip7/scikit-learn-issues-multilabel huggingface dataset:
Create estimators for inference only

### Describe the workflow you want to enable

Allow a trained estimator to be converted into a form suitable only for predict/transform type operations and not fitting. In many cases, the estimator could be made more compact or performant as part of this transformation. 

For instance, feature selection steps in a pipeline may rely on complex models during training, but at inference they simply drop unused features. When deploying the model, conversion of the feature selection step to a simpler form could save memory and model load time. 

### Describe your proposed solution

Add a new method to BaseEstimator `prep_for_inference(self)` which returns a model which retains all predict/transform methods but does not necessarily support fitting. By default it would return `self`. Estimators and transformers could override this as necessary. Pipeline would convert each step. 

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I think the original intention of the OP is mostly satisfied now and we can close. The issue of making the estimators more "compact" or optimised for the purpose of inference, is a much larger one, which we can get back to when it becomes our focus.

Labels: New Feature
Include a lower bound attribute of BaseMixture

### Describe the workflow you want to enable

Currently, there exists a `lower_bound_` attribute in the `fit_predict` method of `BaseMixture`.  However, the entire sequence of lower bounds is not accessible, which makes a convergence analysis more difficult to a user. 

### Describe your proposed solution

In addition to the `lower_bound_` attribute, create a new attribute called `lower_bounds_`, which is a list of floats where each float is a lower bound set in https://github.com/scikit-learn/scikit-learn/blob/92c9b1866fab77412d8fe93cb3716c03d80ad8ed/sklearn/mixture/_base.py#L248.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Creating the list and appending to it would increase memory costs (but not by much). Is this a possible concern?

COMMENT:
Sounds like a nice addition! I'll work on modifying the code to add this feature with some samples.

Labels: New Feature
Include a lower bound attribute of BaseMixture

### Describe the workflow you want to enable

Currently, there exists a `lower_bound_` attribute in the `fit_predict` method of `BaseMixture`.  However, the entire sequence of lower bounds is not accessible, which makes a convergence analysis more difficult to a user. 

### Describe your proposed solution

In addition to the `lower_bound_` attribute, create a new attribute called `lower_bounds_`, which is a list of floats where each float is a lower bound set in https://github.com/scikit-learn/scikit-learn/blob/92c9b1866fab77412d8fe93cb3716c03d80ad8ed/sklearn/mixture/_base.py#L248.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Creating the list and appending to it would increase memory costs (but not by much). Is this a possible concern?

COMMENT:
@a-wozniakowski I have added the lower bounds list feature [here](https://github.com/myenugula/scikit-learn/tree/GaussianMixture-lower_bounds). Please give it a shot and let me know. I'll make a PR if it's all good.

Here's a code sample for using it on Iris dataset:
```
from sklearn import datasets
from sklearn.mixture import GaussianMixture
import matplotlib.pyplot as plt
import pandas as pd

iris = datasets.load_iris()
X = iris.data

d = pd.DataFrame(X)

gmm = GaussianMixture(n_components=3, n_init=10)
gmm.fit(d)

lower_bounds = gmm.lower_bounds_
n_iter = gmm.n_iter_

plt.figure(figsize=(10, 6))
plt.plot(range(1, n_iter + 1), lower_bounds, marker='o', linestyle='-')
plt.title('GMM Lower Bounds Across Iterations')
plt.xlabel('Iteration')
plt.ylabel('Lower Bound')
plt.grid(True)
plt.show()
``` 

<img width="980" alt="image" src="https://github.com/scikit-learn/scikit-learn/assets/127900888/4fb05fc7-9779-4499-b432-1afc64828845">

Labels: New Feature
Include a lower bound attribute of BaseMixture

### Describe the workflow you want to enable

Currently, there exists a `lower_bound_` attribute in the `fit_predict` method of `BaseMixture`.  However, the entire sequence of lower bounds is not accessible, which makes a convergence analysis more difficult to a user. 

### Describe your proposed solution

In addition to the `lower_bound_` attribute, create a new attribute called `lower_bounds_`, which is a list of floats where each float is a lower bound set in https://github.com/scikit-learn/scikit-learn/blob/92c9b1866fab77412d8fe93cb3716c03d80ad8ed/sklearn/mixture/_base.py#L248.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Creating the list and appending to it would increase memory costs (but not by much). Is this a possible concern?

COMMENT:
Yep we could have this feature.

Labels: New Feature
Improve "polars" integration (error, warning & linting examples)

### Describe the workflow you want to enable

using polars data (DataFrame, Series) is already supported in many places which is awesome, thank you!!

But in many places there are still
- errors / crashes -> required conversion to numpy/pandas
- warnings -> requires conversion to numpy/pandas
- linting/type problems -> requires updates to typing signalture?

# Examples

## Code

```python
import pandas as pd
import polars as pl
from sklearn.datasets import make_classification
from sklearn.inspection import permutation_importance
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, cross_validate

X, y = make_classification()

# X = pd.DataFrame(X)
# y = pd.Series(y)

# X = pl.DataFrame(X)
# y = pl.Series(y)

clf = LogisticRegression()

clf.fit(
    X=X,
    y=y,
)

clf.score(
    X=X,  # Lint/Type Problem
    y=y,
)

cross_val_score(
    estimator=clf,
    X=X,  # Lint/Type Problem
    y=y,  # ERROR with polars
)

cross_validate(
    estimator=clf,
    X=X,  # Lint/Type Problem
    y=y,  # ERROR with polars
)

permutation_importance(
    estimator=clf,
    X=X,  # WARNING with polars + Lint/Type Problem
    y=y,
)

clf.predict(X)
```

## Errors / crashes using polars

Both `cross_val_score` and `cross_validate` crash using polars Series with message:
- `TypeError: cannot use `__getitem__` on Series of dtype Int32 with argument (array([18, 21, 22, 23, 24, 25,...`

### Temporary solution

use
- `to_numpy()` -> works with numpy array
- `to_pandas()` -> works with pandas Series

## Warnings

`permutation_importance` creates "UserWarnings" using polars DataFrame with message:
- `UserWarning: X does not have valid feature names, but LogisticRegression was fitted with feature names`

### Temporary Solution

use
- `to_pandas()` -> works with pandas DataFrame

## Linting / Type problems

numpy arrays and pandas DataFrame have "full support" while polars looks a little sad 😆 

![image](https://github.com/scikit-learn/scikit-learn/assets/25177421/b0aa54c2-52e0-4d4c-8c7c-448a9fbb978e)











### Describe your proposed solution

would love to see further support for polars DataFrame / Series types.

Not sure why some things error, because I thought polars provides all the required apis.

Any way to assist / help?

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Wild guess would be that `_safe_indexing` does not support the polars maybe. I need to check deeper.

Labels: Bug
Improve "polars" integration (error, warning & linting examples)

### Describe the workflow you want to enable

using polars data (DataFrame, Series) is already supported in many places which is awesome, thank you!!

But in many places there are still
- errors / crashes -> required conversion to numpy/pandas
- warnings -> requires conversion to numpy/pandas
- linting/type problems -> requires updates to typing signalture?

# Examples

## Code

```python
import pandas as pd
import polars as pl
from sklearn.datasets import make_classification
from sklearn.inspection import permutation_importance
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, cross_validate

X, y = make_classification()

# X = pd.DataFrame(X)
# y = pd.Series(y)

# X = pl.DataFrame(X)
# y = pl.Series(y)

clf = LogisticRegression()

clf.fit(
    X=X,
    y=y,
)

clf.score(
    X=X,  # Lint/Type Problem
    y=y,
)

cross_val_score(
    estimator=clf,
    X=X,  # Lint/Type Problem
    y=y,  # ERROR with polars
)

cross_validate(
    estimator=clf,
    X=X,  # Lint/Type Problem
    y=y,  # ERROR with polars
)

permutation_importance(
    estimator=clf,
    X=X,  # WARNING with polars + Lint/Type Problem
    y=y,
)

clf.predict(X)
```

## Errors / crashes using polars

Both `cross_val_score` and `cross_validate` crash using polars Series with message:
- `TypeError: cannot use `__getitem__` on Series of dtype Int32 with argument (array([18, 21, 22, 23, 24, 25,...`

### Temporary solution

use
- `to_numpy()` -> works with numpy array
- `to_pandas()` -> works with pandas Series

## Warnings

`permutation_importance` creates "UserWarnings" using polars DataFrame with message:
- `UserWarning: X does not have valid feature names, but LogisticRegression was fitted with feature names`

### Temporary Solution

use
- `to_pandas()` -> works with pandas DataFrame

## Linting / Type problems

numpy arrays and pandas DataFrame have "full support" while polars looks a little sad 😆 

![image](https://github.com/scikit-learn/scikit-learn/assets/25177421/b0aa54c2-52e0-4d4c-8c7c-448a9fbb978e)











### Describe your proposed solution

would love to see further support for polars DataFrame / Series types.

Not sure why some things error, because I thought polars provides all the required apis.

Any way to assist / help?

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
About linting / typing issues you mentioned, the type annotations are not maintained by the scikit-learn project so they come from somewhere else, maybe https://github.com/microsoft/python-type-stubs/tree/main/stubs/sklearn?

Labels: Bug
Improve "polars" integration (error, warning & linting examples)

### Describe the workflow you want to enable

using polars data (DataFrame, Series) is already supported in many places which is awesome, thank you!!

But in many places there are still
- errors / crashes -> required conversion to numpy/pandas
- warnings -> requires conversion to numpy/pandas
- linting/type problems -> requires updates to typing signalture?

# Examples

## Code

```python
import pandas as pd
import polars as pl
from sklearn.datasets import make_classification
from sklearn.inspection import permutation_importance
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, cross_validate

X, y = make_classification()

# X = pd.DataFrame(X)
# y = pd.Series(y)

# X = pl.DataFrame(X)
# y = pl.Series(y)

clf = LogisticRegression()

clf.fit(
    X=X,
    y=y,
)

clf.score(
    X=X,  # Lint/Type Problem
    y=y,
)

cross_val_score(
    estimator=clf,
    X=X,  # Lint/Type Problem
    y=y,  # ERROR with polars
)

cross_validate(
    estimator=clf,
    X=X,  # Lint/Type Problem
    y=y,  # ERROR with polars
)

permutation_importance(
    estimator=clf,
    X=X,  # WARNING with polars + Lint/Type Problem
    y=y,
)

clf.predict(X)
```

## Errors / crashes using polars

Both `cross_val_score` and `cross_validate` crash using polars Series with message:
- `TypeError: cannot use `__getitem__` on Series of dtype Int32 with argument (array([18, 21, 22, 23, 24, 25,...`

### Temporary solution

use
- `to_numpy()` -> works with numpy array
- `to_pandas()` -> works with pandas Series

## Warnings

`permutation_importance` creates "UserWarnings" using polars DataFrame with message:
- `UserWarning: X does not have valid feature names, but LogisticRegression was fitted with feature names`

### Temporary Solution

use
- `to_pandas()` -> works with pandas DataFrame

## Linting / Type problems

numpy arrays and pandas DataFrame have "full support" while polars looks a little sad 😆 

![image](https://github.com/scikit-learn/scikit-learn/assets/25177421/b0aa54c2-52e0-4d4c-8c7c-448a9fbb978e)











### Describe your proposed solution

would love to see further support for polars DataFrame / Series types.

Not sure why some things error, because I thought polars provides all the required apis.

Any way to assist / help?

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
It seems that `_safe_indexing` uses polars indexing only for polars dataframes but not for polars series, which causes `cross_validate` and `cross_val_score` (that uses `cross_validate`) to fail.

The warning of `permutation_importance` is caused by converting polars DataFrame to numpy array (while we detect pandas DataFrame and avoid such a conversion).

Labels: Bug
Allow `NaN` in feature selectors when estimator does Imputation

### Describe the workflow you want to enable

I would like to perform feature selection (e.g.: `SequentialFeatureSelector`) on data containing "NaN/null" values where my estimator does the imputation in a pipeline.

example
```python
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline

X, y = make_classification()
X[0][0] = float("nan")  # set 1 value to "nan"

pipline = make_pipeline(
    SimpleImputer(strategy="mean"),
    LogisticRegression(),
)

pipline.fit(X, y)  # works!
cross_val_score(pipline, X, y) # works!

selector = SequentialFeatureSelector(
    estimator=pipline,
    n_features_to_select=2,
)
selector.fit(X, y)
# >>> ValueError: Input X contains NaN.
```

### Describe your proposed solution

As far as I understand this could work as is!?

In the docs it says that:
> This Sequential Feature Selector adds (forward selection) or removes (backward selection) features to form a feature subset in a greedy fashion. At each stage, this estimator chooses the best feature to add or remove based on the cross-validation score of an estimator.

As shown the estimator can deal with "nan" values and can compute the cv score.
Is there is reason this currently is not working/supported or can this be considered a "bug" that it crashes although it would work?

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Indeed this is a duplicate of https://github.com/scikit-learn/scikit-learn/issues/27903. We need pipeline to properly delegate that `nan` are allowed.

I'm closing the issue for this reason but you can check https://github.com/scikit-learn/scikit-learn/issues/27903 for resolution of this issue.

Labels: New Feature, Needs Triage
Suggesting updates on the doc of `sklearn.linear_model.PassiveAggressiveRegressor`

### Describe the issue linked to the documentation

Hi,

We discover an inconsistency issue between documentation and code in the class [`sklearn.linear_model.PassiveAggressiveRegressor`](https://scikit-learn.org/dev/modules/generated/sklearn.linear_model.PassiveAggressiveRegressor.html#sklearn-linear-model-passiveaggressiveregressor). As mentioned in the description of parameter `validation_fraction`.

> **validation_fraction: float, default=0.1**
The proportion of training data to set aside as validation set for early stopping. Must be between 0 and 1. **_Only used if early_stopping is True._**

However, I did not see any constraint in the source code or even in the parent class.

Could you please check it?

### Suggest a potential alternative/fix

Maybe you can update the doc to make it clear.

COMMENT:
Seems that this is a false positive.

`validation_fraction` is only used in `BaseSGD._make_validation_split()`.

```python
    def _make_validation_split(self, y, sample_mask):
        n_samples = y.shape[0]
        validation_mask = np.zeros(n_samples, dtype=np.bool_)
        if not self.early_stopping: # <--------------------------------
            # use the full set for training, with an empty validation set
            return validation_mask
        if is_classifier(self):
            splitter_type = StratifiedShuffleSplit
        else:
            splitter_type = ShuffleSplit
        cv = splitter_type(
            test_size=self.validation_fraction, random_state=self.random_state # <--------------------------------
        )
        etc. etc.
```
And if `early_stopping` is false, the rest of the method is not run and `validation_fraction` is never used.

So, our doc for `PassiveAggressiveRegressor` is correct.

Labels: Documentation
Suggesting updates on the doc of `sklearn.linear_model.PassiveAggressiveRegressor`

### Describe the issue linked to the documentation

Hi,

We discover an inconsistency issue between documentation and code in the class [`sklearn.linear_model.PassiveAggressiveRegressor`](https://scikit-learn.org/dev/modules/generated/sklearn.linear_model.PassiveAggressiveRegressor.html#sklearn-linear-model-passiveaggressiveregressor). As mentioned in the description of parameter `validation_fraction`.

> **validation_fraction: float, default=0.1**
The proportion of training data to set aside as validation set for early stopping. Must be between 0 and 1. **_Only used if early_stopping is True._**

However, I did not see any constraint in the source code or even in the parent class.

Could you please check it?

### Suggest a potential alternative/fix

Maybe you can update the doc to make it clear.

COMMENT:
It is in the parent class (`BaseSGD`) of the parent class (`BaseSGDRegressor`). I agree that the documentation is correct, so closing this issue.

Labels: Documentation
Suggesting updates on the doc of `sklearn.compose.TransformedTargetRegressor`

### Describe the issue linked to the documentation

Hi,

We discover an inconsistency issue between documentation and code in the class [`sklearn.compose.TransformedTargetRegressor`](https://scikit-learn.org/dev/modules/generated/sklearn.compose.TransformedTargetRegressor.html#sklearn-compose-transformedtargetregressor). As mentioned in the description of parameter `func`.

> **func: function, default=None**
Function to apply to y before passing to fit. Cannot be set at the same time as transformer. The function needs to return a 2-dimensional array. If func is None, the function used will be the identity function.

The most relevant piece of source code looks like this:
```
if self.func is not None and self.inverse_func is None:
    raise ValueError(
        "When 'func' is provided, 'inverse_func' must also be provided"
    )
```

The error in the code mentioned that when `func` is provided, `inverse_func` must also be provided. The constraint is not mentioned in the document. 

Could you please check it?

### Suggest a potential alternative/fix

Maybe you can add the constraint into the document to avoid unnecessary misuse and extra debug efforts.

COMMENT:
This is correct, it should be mentioned in the documentation.
I will make a PR to add that.

Labels: Documentation, Needs Triage
Suggesting updates on the doc of `sklearn.decomposition.KernelPCA`

### Describe the issue linked to the documentation

Hi,

We discover an inconsistency issue between documentation and code in the class [`sklearn.decomposition.KernelPCA`](https://scikit-learn.org/dev/modules/generated/sklearn.decomposition.KernelPCA.html#sklearn.decomposition.KernelPCA). As mentioned in the description of parameter `gamma`.

> **gamma: float, default=None**
**_Kernel coefficient for rbf, poly and sigmoid kernels. Ignored by other kernels_**. If gamma is None, then it is set to 1/n_features.

The most relevant piece of source code looks like this:
```
"kernel": [
    StrOptions({"linear", "poly", "rbf", "sigmoid", "cosine", "precomputed"}),
    callable,
]

def _get_kernel(self, X, Y=None):
    if callable(self.kernel):
        params = self.kernel_params or {}
    else:
        params = {"gamma": self.gamma_, "degree": self.degree, "coef0": self.coef0}
    return pairwise_kernels(
        X, Y, metric=self.kernel, filter_params=True, n_jobs=self.n_jobs, **params
    )
```

It seems that `gamma` will be ignored not only when `kernel` is `rbf`, `poly`, `sigmoid` but all kernels.

The same situation also happened in `degree` and `coef0`. As mentioned in the description of parameter `degree` and `coef0`.

> **degree: float, default=3**
**_Degree for poly kernels. Ignored by other kernels._**. 

> **coef0: float, default=1**
**_Independent term in poly and sigmoid kernels. Ignored by other kernels._**. 


Could you please check it?

### Suggest a potential alternative/fix

Maybe you can reconstruct the if-else statement to cover the situation or update the doc to make it clear.

COMMENT:
This is another false positive.

I went through `sklearn/metrics/pairwise.py` to verify that the kernel functions actually so use these params exactly as the documentation states, and they really do.

Labels: Documentation, Needs Triage
Suggesting updates on the doc of `sklearn.decomposition.KernelPCA`

### Describe the issue linked to the documentation

Hi,

We discover an inconsistency issue between documentation and code in the class [`sklearn.decomposition.KernelPCA`](https://scikit-learn.org/dev/modules/generated/sklearn.decomposition.KernelPCA.html#sklearn.decomposition.KernelPCA). As mentioned in the description of parameter `gamma`.

> **gamma: float, default=None**
**_Kernel coefficient for rbf, poly and sigmoid kernels. Ignored by other kernels_**. If gamma is None, then it is set to 1/n_features.

The most relevant piece of source code looks like this:
```
"kernel": [
    StrOptions({"linear", "poly", "rbf", "sigmoid", "cosine", "precomputed"}),
    callable,
]

def _get_kernel(self, X, Y=None):
    if callable(self.kernel):
        params = self.kernel_params or {}
    else:
        params = {"gamma": self.gamma_, "degree": self.degree, "coef0": self.coef0}
    return pairwise_kernels(
        X, Y, metric=self.kernel, filter_params=True, n_jobs=self.n_jobs, **params
    )
```

It seems that `gamma` will be ignored not only when `kernel` is `rbf`, `poly`, `sigmoid` but all kernels.

The same situation also happened in `degree` and `coef0`. As mentioned in the description of parameter `degree` and `coef0`.

> **degree: float, default=3**
**_Degree for poly kernels. Ignored by other kernels._**. 

> **coef0: float, default=1**
**_Independent term in poly and sigmoid kernels. Ignored by other kernels._**. 


Could you please check it?

### Suggest a potential alternative/fix

Maybe you can reconstruct the if-else statement to cover the situation or update the doc to make it clear.

COMMENT:
Closing this as it is a false positive.

We can re-open if needed.

Labels: Documentation, Needs Triage