Talip7/scikit-learn-issues-multilabel huggingface dataset:
Add Warnings when trying to use estimator.fit()  functions with Pandas objects with different indices.

### Describe the workflow you want to enable

Because sklearn uses numpy.array objects but accepts `pandas.DataFrame` / `pandas.Series` objects as input to its many functions, passing two pandas objects with different indices to a `estimator.fit()` function allows to unadvertedly work on shuffled X and y arrays. 

Let's construct an `X` `pandas.DataFrame` and an `y` `pandas.Series`.

```
import pandas as pd

x1 = [3, 3, 4, 5, 3, 5]
x2 = [0.1, 0.2, -2, -3, -3, -2]

y = ['a', 'a', 'b', 'b', 'b', 'b']

X_index = [1, 2, 3, 4, 5, 6]
y_index = [6, 5, 4, 3, 2, 1]

X = pd.DataFrame({'A': x1, 'B': x2}, index=X_index)
y = pd.Series(y, index=y_index)
```
Different pre-processing protocols (such as the use of `pandas.DataFrame.merge()`) can unadvertedly change the index of one of the two `pandas` objects (usually `X`, which is normally subject to more modifications than `y`).

Passing two incompatible indices like the ones above to an `estimator.fit()` function will yield no kind of warning. The model will be fit on two data objects which can potentially be in different orders, without expliciting the fact that the two indices are not equal. 

```
from sklearn.linear_model import LogisticRegression

model = LogisticRegression()
model.fit(X, y)
>>> LogisticRegression()

X.index == y.index
>>> array([False, False, False, False, False, False])
```

Checking if the two `pandas` objects' indices are equal and warning if they are not would allow users to understand that their `X` and `y` inputs may not be compatible. This can potentially save _a lot_ of time spent modeling on randomly shuffled data without warnings of any kind. Git knows I would've saved a lot of time. 

### Describe your proposed solution

For all fitting functions, implementing a simple check that  `X.index == y.index` is true for all cases before turning the `pandas` objects to `numpy`, and printing a warning informing that the indices are not compatible if it is not the case, urging the user to check that the order of their data is correct.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I think if pandas DFs are accepted as input it is not great for users if silently the "wrong thing" happens. However, when I try the example from the top comment it seems like the index is ignored. I looked at the value of `X` and `y` after input validation in `fit`. It looks like they are both in the order that was used to construct the DataFrame and Series. As if the index didn't exist/is ignored during conversion to ndarrays.

So maybe the thing to add is a check that neither input uses an index and emit a warning/error.

There is already some code specifically for detecting pandas DFs and Series in `check_array` so we could hook in there for a warning.

Labels: New Feature, Needs Decision
Add Warnings when trying to use estimator.fit()  functions with Pandas objects with different indices.

### Describe the workflow you want to enable

Because sklearn uses numpy.array objects but accepts `pandas.DataFrame` / `pandas.Series` objects as input to its many functions, passing two pandas objects with different indices to a `estimator.fit()` function allows to unadvertedly work on shuffled X and y arrays. 

Let's construct an `X` `pandas.DataFrame` and an `y` `pandas.Series`.

```
import pandas as pd

x1 = [3, 3, 4, 5, 3, 5]
x2 = [0.1, 0.2, -2, -3, -3, -2]

y = ['a', 'a', 'b', 'b', 'b', 'b']

X_index = [1, 2, 3, 4, 5, 6]
y_index = [6, 5, 4, 3, 2, 1]

X = pd.DataFrame({'A': x1, 'B': x2}, index=X_index)
y = pd.Series(y, index=y_index)
```
Different pre-processing protocols (such as the use of `pandas.DataFrame.merge()`) can unadvertedly change the index of one of the two `pandas` objects (usually `X`, which is normally subject to more modifications than `y`).

Passing two incompatible indices like the ones above to an `estimator.fit()` function will yield no kind of warning. The model will be fit on two data objects which can potentially be in different orders, without expliciting the fact that the two indices are not equal. 

```
from sklearn.linear_model import LogisticRegression

model = LogisticRegression()
model.fit(X, y)
>>> LogisticRegression()

X.index == y.index
>>> array([False, False, False, False, False, False])
```

Checking if the two `pandas` objects' indices are equal and warning if they are not would allow users to understand that their `X` and `y` inputs may not be compatible. This can potentially save _a lot_ of time spent modeling on randomly shuffled data without warnings of any kind. Git knows I would've saved a lot of time. 

### Describe your proposed solution

For all fitting functions, implementing a simple check that  `X.index == y.index` is true for all cases before turning the `pandas` objects to `numpy`, and printing a warning informing that the indices are not compatible if it is not the case, urging the user to check that the order of their data is correct.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Assuming we have 100 features, this check would typically accounts for less than 1% of the `_assert_all_finite` cost, so we might consider that this additional check is not introducing too much overhead comparatively.

On the otherhand, in the [dataframe API spec](https://data-apis.org/dataframe-protocol/latest/API.html), there are no indices. So when scikit-learn will accept generic dataframes via this API instead of being pandas-specific we will face the same problem.

Labels: New Feature, Needs Decision
[Refactor Request Tree] Make sorting and splitting utility functions cimportable by including them in the pxd files

Hi,

I was wondering if it is possible to define the relevant splitter utility functions in the `.pxd` file so that it is cimportable from 3rd party applications?

## Motivation
3rd party applications will typically leverage the Cython code within scikit-learn even though it is not a publicly supported API. Refactoring the tree code has been discussed, but has been moving slowly due to lack of time from maintainers. In the meantime, one might want to use copy/modify some of the existing tree code to implement a new tree model. 

In the splitter, they will typically still use common functions such as "sort", "extract_nnz_binary_search", "extract_nnz_index_to_samples", "sparse_swap" (I may be missing others). 

## Solution
Just include their headers in the `sklearn.tree._splitter.pxd` file. This is similar to how common utility functions are also included in `sklearn.tree._utils.pxd`. 

This would be a reasonably small PR, with no side effects.

COMMENT:
I am -1 because these functions were not designed to be public.

From the triaging meeting, I think it's best to wait till have a clearer picture of what third party needs before exposing and supporting a Cython API.

Labels: API, Needs Decision, module:tree
[Refactor Request Tree] Make sorting and splitting utility functions cimportable by including them in the pxd files

Hi,

I was wondering if it is possible to define the relevant splitter utility functions in the `.pxd` file so that it is cimportable from 3rd party applications?

## Motivation
3rd party applications will typically leverage the Cython code within scikit-learn even though it is not a publicly supported API. Refactoring the tree code has been discussed, but has been moving slowly due to lack of time from maintainers. In the meantime, one might want to use copy/modify some of the existing tree code to implement a new tree model. 

In the splitter, they will typically still use common functions such as "sort", "extract_nnz_binary_search", "extract_nnz_index_to_samples", "sparse_swap" (I may be missing others). 

## Solution
Just include their headers in the `sklearn.tree._splitter.pxd` file. This is similar to how common utility functions are also included in `sklearn.tree._utils.pxd`. 

This would be a reasonably small PR, with no side effects.

COMMENT:
Sounds good. Makes sense!

Labels: API, Needs Decision, module:tree
InteractionTransformer

### Describe the workflow you want to enable

The latest 1.2 release is full of great features, e.g., full column name support. This brings me to one of my most desired features regarding building strong and realistic linear models: Interactions!

It is currently very hard to add interaction terms between two feature groups, especially if they involve 1 to m transforms like OHE or SplineTransformers.

### Describe your proposed solution

An idea of @lorentzenchr that I try to summarize: Create the transforms like a ColumnTransformer,  but also adding interaction terms between columns generated by each transformer. The resulting columns could be glued to an other ColumnTransformer using `FeatureUnion`.

## Sketch of the API

```
InteractionTransformer(
    transformers,
    interaction_only=False
    include_bias=True,
    verbose_feature_names_out=True
)

# transformers: list of tuples
#     List of (name, transformer, columns) tuples specifying the transformer objects to be applied to subsets of the data.
#     Each transformer would specify a feature (group) that would interact with other transformers.
# interaction_only: If False, also return main effects
```

## Example 1: Interactions between two 1-m transforms

Here, we would let each dummy variable of "f1" interact with each spline basis of "f2":

```
InteractionTransformer(
    transformers=[
        ("f1_ohe", OneHotEncoder(drop="first"), ["f1"]),
        ("f2_spline", SplineTransformer(), ["f2"]),
    ]
)
```

## Example 2: Interaction between OHE and other features

Each column generated by OHE "f1" would interact with numeric feature "f2" and also with numeric feature "f3".

```
InteractionTransformer(
    transformers=[
        ("f1_ohe", OneHotEncoder(drop="first"), ["f1"]),
        ("f2_others", "passthrough", ["f2", "f3"]),
    ]
)
```

## Example 3: Interaction between two OHE, further features linear

```
interactor = InteractionTransformer(
    transformers=[
        ("f1_ohe", OneHotEncoder(drop="first"), ["f1"]),
        ("f2_ohe", OneHotEncoder(drop="first"), ["f2"]),
    ]
)

preprocessor = FeatureUnion(
  interactor,
  ColumnTransformer("other", "passthrough", ["f3", "f4"])
)

```


### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Let's see a pull request then :)

It sounds like a pragmatic answer to a need that many people expressed over the years.

Labels: New Feature, module:preprocessing
InteractionTransformer

### Describe the workflow you want to enable

The latest 1.2 release is full of great features, e.g., full column name support. This brings me to one of my most desired features regarding building strong and realistic linear models: Interactions!

It is currently very hard to add interaction terms between two feature groups, especially if they involve 1 to m transforms like OHE or SplineTransformers.

### Describe your proposed solution

An idea of @lorentzenchr that I try to summarize: Create the transforms like a ColumnTransformer,  but also adding interaction terms between columns generated by each transformer. The resulting columns could be glued to an other ColumnTransformer using `FeatureUnion`.

## Sketch of the API

```
InteractionTransformer(
    transformers,
    interaction_only=False
    include_bias=True,
    verbose_feature_names_out=True
)

# transformers: list of tuples
#     List of (name, transformer, columns) tuples specifying the transformer objects to be applied to subsets of the data.
#     Each transformer would specify a feature (group) that would interact with other transformers.
# interaction_only: If False, also return main effects
```

## Example 1: Interactions between two 1-m transforms

Here, we would let each dummy variable of "f1" interact with each spline basis of "f2":

```
InteractionTransformer(
    transformers=[
        ("f1_ohe", OneHotEncoder(drop="first"), ["f1"]),
        ("f2_spline", SplineTransformer(), ["f2"]),
    ]
)
```

## Example 2: Interaction between OHE and other features

Each column generated by OHE "f1" would interact with numeric feature "f2" and also with numeric feature "f3".

```
InteractionTransformer(
    transformers=[
        ("f1_ohe", OneHotEncoder(drop="first"), ["f1"]),
        ("f2_others", "passthrough", ["f2", "f3"]),
    ]
)
```

## Example 3: Interaction between two OHE, further features linear

```
interactor = InteractionTransformer(
    transformers=[
        ("f1_ohe", OneHotEncoder(drop="first"), ["f1"]),
        ("f2_ohe", OneHotEncoder(drop="first"), ["f2"]),
    ]
)

preprocessor = FeatureUnion(
  interactor,
  ColumnTransformer("other", "passthrough", ["f3", "f4"])
)

```


### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
How would this design interact with hyper-parameter tuning? We should try to craft a design that will lead to a nice UX assuming we move forward with #21784.

Labels: New Feature, module:preprocessing
correct and reasonable new example to replace the old one

https://github.com/scikit-learn/scikit-learn/blob/98cf537f5c538fdbc9d27b851cf03ce7611b8a48/sklearn/feature_extraction/image.py#L496

The problem of the old example is that it did not consider the "n_samples" dimension of the function `from sklearn.feature_extraction.image.PatchExtractor` and therefore caused a strange, unreasonable and confusing result that had no channel dimension.

The new example corrects the issue and gives a clear, reasonable demonstration of how to use `PatchExtractor` sensibly. It is shown as following:

```
>>> from sklearn.datasets import load_sample_images
>>> from sklearn.feature_extraction import image
# Use the array data from the second image in this dataset:
>>> X = load_sample_images().images[1]
>>> X = X[np.newaxis,:] # make X has a shape (n_samples, image_height, image_width, n_channels). Very important!!
>>> print('Image shape: {}'.format(X.shape))
Image shape: (1, 427, 640, 3)
>>> pe = image.PatchExtractor(patch_size=(10, 10))
>>> pe_fit = pe.fit(X) # Do nothing and return the estimator unchanged.
>>> pe_trans = pe.transform(X)
>>> print('Patches shape: {}'.format(pe_trans.shape))
Patches shape: (263758, 10, 10, 3)
```
Note, this line `X = X[np.newaxis,:]` adds a new axis as the "n_samples" dimension, which makes X has a correct shape (n_samples, image_height, image_width, n_channels). It is the key difference from the old example. This change is very important, because, otherwise, "image_height" will be treated as "n_samples", and other dimensions will also be messed up similarly. That is why the result in the old example look strange and confusing, with only 3 dimensions but without the channel dimension.

It would be good to add the following reconstruction scripts to verify that the original image can be reconstructed from the extracted patches using `reconstruct_from_patches_2d` function. This makes the example more complete, similar to the example of using the `extract_patches_2d` function. Again, it is important to keep in mind the "n_samples" dimension.
```
>>> X_reconstructed = image.reconstruct_from_patches_2d(pe_trans, X.shape[1:])
>>> print(X_reconstructed.shape)
>>> np.testing.assert_array_equal(X[0], X_reconstructed)
(427, 640, 3)
```

COMMENT:
I agree with the change to add in `n_samples`, I'll recommend something like this:

```python
from sklearn.datasets import load_sample_images
from sklearn.feature_extraction import image
# Use the array data from the second image in this dataset:
X = load_sample_images()["images"][1]
X = X[None, ...]
print(f"Image shape: {X.shape}")

pe = image.PatchExtractor(patch_size=(10, 10))
pe_trans = pe.transform(X)
print(f"Patches shape: {pe_trans.shape}")

X_reconstructed = image.reconstruct_from_patches_2d(pe_trans, X.shape[1:])
print(f"Reconstructed shape: {X_reconstructed.shape}")
```

There is no need to call `fit` before transform. Our docstring tests will assert that the reconstructed shape is correct.

@Song-Pingfan Are you interested in opening a pull request to update the docstring?

Labels: Documentation, module:feature_extraction
Feature request: for GridSearchCV and RandomizedSearchCV, add a kwarg for a preprocessing step after CV splits have been made

### Describe the workflow you want to enable

See solution

### Describe your proposed solution

Add a new kwarg to cross-validators such as GridSearchCV and RandomizedSearchCV that allows you to use preprocessing steps (such as scalers) on the train & test sets created during each cross validation fold. For example, if I wanted to use StandardScaler()'s `.fit_transform()` method on the training set and the `.transform()` method on the test set each time a new cross validation split is made, I'd imagine a new kwarg (I'll call it "post_split" here) looking like:

```
# Instantiate a scaler object
scaler = StandardScaler()

# Use the scaler every CV fold
grid = GridSearchCV(post_split={"training_set": scaler.fit_transform(),
                                "test_set": scaler.transform()},
                    **kwargs)

# Fit the data
grid.fit(X, y)
```

Here, the "training_set" key signifies the training set created by that cross-validation split, and the "test_set" key would signify the test set created by that cross-validation split. The StandardScaler's `.fit_transform()` method only gets called on the cross validation training data, and the `.transform()` method only gets called on the cross validation test data. Once that is done, that cross-validation 

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

**The problem:**
When the current implementation of GridSearchCV and RandomizedSearchCV perform their train-test splits on scaled data, they have to do it on data that has been scaled together with the `.fit_transform()` method of a scaler. This creates a data leak between the training and test sets created for cross-validation folds because these sets originate from a single python object passed into the cross-validator's `.fit()` function.

COMMENT:
Using properly `sklearn.pipeline.Pipeline` will not leak data as documented here: https://scikit-learn.org/dev/common_pitfalls.html#data-leakage

More generally, we are against adding `kwargs`. If this is linked to some metadata, SLEP006 would be the solution.

Labels: New Feature, Needs Triage
OrdinalEncoder with option to mention the start index

### Describe the workflow you want to enable

Hi,
Can OrdinalEncoder be provided with an argument to decide what index to start encoding on


### Describe your proposed solution


For example
``` 
import numpy as np
from sklearn.preprocessing import OrdinalEncoder
oe = OrdinalEncoder(encode_start=0)
oe.fit_transform(np.array(['a','b','c','d','e']))
# Gives us [0,1,2,3,4]

oe = OrdinalEncoder(encode_start=1)
oe.fit_transform(np.array(['a','b','c','d','e']))
# Gives us [1,2,3,4,5]

```



### Describe alternatives you've considered, if relevant

_No response_

### Additional context

The Rationale Behind this issue is that users may be able to set unknown values to 0 if encoding is 1-indexed

COMMENT:
I am -1 for this feature. I don't think that the benefit of the feature is worth the maintenance and the addition of the new keyword.

Note that this behaviour can easily be achieved with the following pipeline:

```python
encoder_with_shift = make_pipeline(
    OrdinalEncoder(unknown_value=-1),
    FunctionTransformer(
        func=lambda x: x + 1,
        inverse_func=lambda x: x - 1,
    )
)
```

Labels: New Feature, Needs Triage
OrdinalEncoder with option to mention the start index

### Describe the workflow you want to enable

Hi,
Can OrdinalEncoder be provided with an argument to decide what index to start encoding on


### Describe your proposed solution


For example
``` 
import numpy as np
from sklearn.preprocessing import OrdinalEncoder
oe = OrdinalEncoder(encode_start=0)
oe.fit_transform(np.array(['a','b','c','d','e']))
# Gives us [0,1,2,3,4]

oe = OrdinalEncoder(encode_start=1)
oe.fit_transform(np.array(['a','b','c','d','e']))
# Gives us [1,2,3,4,5]

```



### Describe alternatives you've considered, if relevant

_No response_

### Additional context

The Rationale Behind this issue is that users may be able to set unknown values to 0 if encoding is 1-indexed

COMMENT:
I agree with @glemaitre and -1 on this feature request. With that in mind, I am closing this issue. Thank you for the feature request!

Labels: New Feature, Needs Triage
OrdinalEncoder with option to mention the start index

### Describe the workflow you want to enable

Hi,
Can OrdinalEncoder be provided with an argument to decide what index to start encoding on


### Describe your proposed solution


For example
``` 
import numpy as np
from sklearn.preprocessing import OrdinalEncoder
oe = OrdinalEncoder(encode_start=0)
oe.fit_transform(np.array(['a','b','c','d','e']))
# Gives us [0,1,2,3,4]

oe = OrdinalEncoder(encode_start=1)
oe.fit_transform(np.array(['a','b','c','d','e']))
# Gives us [1,2,3,4,5]

```



### Describe alternatives you've considered, if relevant

_No response_

### Additional context

The Rationale Behind this issue is that users may be able to set unknown values to 0 if encoding is 1-indexed

COMMENT:
> I am -1 for this feature. I don't think that the benefit of the feature is worth the maintenance and the addition of the new keyword.
> 
> Note that this behaviour can easily be achieved with the following pipeline:
> 
> ```python
> encoder_with_shift = make_pipeline(
>     OrdinalEncoder(unknown_value=-1),
>     FunctionTransformer(
>         func=lambda x: x + 1,
>         inverse_func=lambda x: x - 1,
>     )
> )
> ```

If use `FunctionTransformer` to assign the start index, we will have problem saving the pipeline, see the following code:

```python
pickle.dump(encoder_with_shift , open(save_path, 'wb'))
```
It will raise an error, look like the following:
`_pickle.PicklingError: Can't pickle <function <lambda> at 0x7f32c82b1900>: it's not found as xxx.<lambda> `

How to fix the issue?

Labels: New Feature, Needs Triage