Talip7/scikit-learn-issues-multilabel huggingface dataset:
Instantiate tunedthresholdCV on google collab failed

### Describe the bug

I updated scikit-learn version into google Collab and instantiate TunedThresholdClassifierCV by  importing  name 'TunedThresholdClassifierCV' from 'sklearn.model_selection' but I got this error 

```python

ImportError: cannot import name 'TunedThresholdClassifierCV' from 'sklearn.model_selection' (/usr/local/lib/python3.10/dist-packages/sklearn/model_selection/__init__.py)
```
### Steps/Code to Reproduce

```python

from sklearn.model_selection import TunedThresholdClassifierCV

```

### Expected Results

merely instantiation the new version of scikit learn for TunedThresholdClassifierCV object to not fail.

### Actual Results

```python
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
[<ipython-input-342-0c5de440dc79>](https://localhost:8080/#) in <cell line: 51>()
     49 from sklearn.cluster import KMeans
     50 from sklearn.preprocessing import StandardScaler
---> 51 from sklearn.model_selection import TunedThresholdClassifierCV
     52 
     53 #UMAP

ImportError: cannot import name 'TunedThresholdClassifierCV' from 'sklearn.model_selection' (/usr/local/lib/python3.10/dist-packages/sklearn/model_selection/__init__.py)
```

### Versions

```shell
1.5.1
```

COMMENT:
I finally solved it by going to this solution by restarting the [kernel](https://stackoverflow.com/questions/59339251/how-to-update-scikit-learn-in-google-colab). but then i got new error on instantiating imblearn  with this error 
```python 
ImportError: cannot import name '_MissingValues' from 'sklearn.utils._param_validation' (/usr/local/lib/python3.10/dist-packages/sklearn/utils/_param_validation.py)
```

it seems there was no update for imblearn for scikit-learn version 1.5.1

Labels: Bug, Needs Triage
Grid search with predetermined parameter order

### Describe the workflow you want to enable

I often have a pipeline with expensive preprocessing, and I tune hyperparameters both for preprocessing and downstream classifier. A toy example with PCA and logistic regression:
```
pipeline = Pipeline([
    ("pca", PCA()),
    ("logreg", LogisticRegression()),
])
cv = GridSearchCV(
    pipeline,
    param_grid={"pca__n_components": [100, 200, 300], "logreg__C": [1, 10, 100]},
    n_jobs=-1
)
```
In this setting, preprocessing is recomputed as many times as there are grid elements. This is very wasteful, as for given preprocessing parameters, the params of the downstream classifier can be optimized. The order would then be:
```
1. pca__n_components=100, check logreg__C = 1, 10, 100
2. pca__n_components=200, check logreg__C = 1, 10, 100
3. pca__n_components=300, check logreg__C = 1, 10, 100
```
For logistic regression, this behavior is actually possible by using `LogisticRegressionCV`. However, e.g. for Random Forest, it is not (to the best of my knowledge). In fact, this can be always done for any pipeline with estimator at the end, since there can only be one classifier (or estimator in general).

This would probably only be relevant to grid search and halving search. For randomized search, a cache for preprocessing output would achieve similar results.

### Describe your proposed solution

Additional parameter or setting to grid search, optimizing parameter order.

Note that this is not equivalent to pipeline caching: https://scikit-learn.org/stable/auto_examples/compose/plot_compare_reduction.html#caching-transformers-within-a-pipeline. Caching only saves the fitting time, whereas here I propose caching the actual transformed output and reusing it for tuning classifier hyperparameters. https://github.com/scikit-learn/scikit-learn/issues/10068 mentions something about caching transform output, but I haven't found confirmation of this behavior anywhere.

### Describe alternatives you've considered, if relevant

This can easily implemented manually in a simple version, but it's less trivial to handle `n_jobs` fully properly. Assuming that we have 100 different parameter sets to check for preprocessing, and 5 sets for classifier, then using n_jobs=6 should always use 6 workers in parallel. But checking only the "current" preprocessing result, we could only use 5 workers. Handling this and starting next batch (i.e. checking estimator hyperparameters for next set of preprocessing parameters) is less trivial, and it would also require keeping 2 (or possibly more) preprocessing results in memory.

### Additional context

This is highly relevant to graph kernels, where preprocessing (computing the kernel) takes ~90% of time. In general, kernel methods would greatly benefit from this.

COMMENT:
@adrinjalali just to make sure - is there any way currently to cache the transformer output? I have stateless transformers with an empty `.fit()`, and all heavy lifting in the `.transform()`, so I need to cache outputs really.

Labels: New Feature, Needs Triage
Grid search with predetermined parameter order

### Describe the workflow you want to enable

I often have a pipeline with expensive preprocessing, and I tune hyperparameters both for preprocessing and downstream classifier. A toy example with PCA and logistic regression:
```
pipeline = Pipeline([
    ("pca", PCA()),
    ("logreg", LogisticRegression()),
])
cv = GridSearchCV(
    pipeline,
    param_grid={"pca__n_components": [100, 200, 300], "logreg__C": [1, 10, 100]},
    n_jobs=-1
)
```
In this setting, preprocessing is recomputed as many times as there are grid elements. This is very wasteful, as for given preprocessing parameters, the params of the downstream classifier can be optimized. The order would then be:
```
1. pca__n_components=100, check logreg__C = 1, 10, 100
2. pca__n_components=200, check logreg__C = 1, 10, 100
3. pca__n_components=300, check logreg__C = 1, 10, 100
```
For logistic regression, this behavior is actually possible by using `LogisticRegressionCV`. However, e.g. for Random Forest, it is not (to the best of my knowledge). In fact, this can be always done for any pipeline with estimator at the end, since there can only be one classifier (or estimator in general).

This would probably only be relevant to grid search and halving search. For randomized search, a cache for preprocessing output would achieve similar results.

### Describe your proposed solution

Additional parameter or setting to grid search, optimizing parameter order.

Note that this is not equivalent to pipeline caching: https://scikit-learn.org/stable/auto_examples/compose/plot_compare_reduction.html#caching-transformers-within-a-pipeline. Caching only saves the fitting time, whereas here I propose caching the actual transformed output and reusing it for tuning classifier hyperparameters. https://github.com/scikit-learn/scikit-learn/issues/10068 mentions something about caching transform output, but I haven't found confirmation of this behavior anywhere.

### Describe alternatives you've considered, if relevant

This can easily implemented manually in a simple version, but it's less trivial to handle `n_jobs` fully properly. Assuming that we have 100 different parameter sets to check for preprocessing, and 5 sets for classifier, then using n_jobs=6 should always use 6 workers in parallel. But checking only the "current" preprocessing result, we could only use 5 workers. Handling this and starting next batch (i.e. checking estimator hyperparameters for next set of preprocessing parameters) is less trivial, and it would also require keeping 2 (or possibly more) preprocessing results in memory.

### Additional context

This is highly relevant to graph kernels, where preprocessing (computing the kernel) takes ~90% of time. In general, kernel methods would greatly benefit from this.

COMMENT:
If calculating the input hash is not computationally intensive, would an `lru_cache` work for you?

Labels: New Feature, Needs Triage
Grid search with predetermined parameter order

### Describe the workflow you want to enable

I often have a pipeline with expensive preprocessing, and I tune hyperparameters both for preprocessing and downstream classifier. A toy example with PCA and logistic regression:
```
pipeline = Pipeline([
    ("pca", PCA()),
    ("logreg", LogisticRegression()),
])
cv = GridSearchCV(
    pipeline,
    param_grid={"pca__n_components": [100, 200, 300], "logreg__C": [1, 10, 100]},
    n_jobs=-1
)
```
In this setting, preprocessing is recomputed as many times as there are grid elements. This is very wasteful, as for given preprocessing parameters, the params of the downstream classifier can be optimized. The order would then be:
```
1. pca__n_components=100, check logreg__C = 1, 10, 100
2. pca__n_components=200, check logreg__C = 1, 10, 100
3. pca__n_components=300, check logreg__C = 1, 10, 100
```
For logistic regression, this behavior is actually possible by using `LogisticRegressionCV`. However, e.g. for Random Forest, it is not (to the best of my knowledge). In fact, this can be always done for any pipeline with estimator at the end, since there can only be one classifier (or estimator in general).

This would probably only be relevant to grid search and halving search. For randomized search, a cache for preprocessing output would achieve similar results.

### Describe your proposed solution

Additional parameter or setting to grid search, optimizing parameter order.

Note that this is not equivalent to pipeline caching: https://scikit-learn.org/stable/auto_examples/compose/plot_compare_reduction.html#caching-transformers-within-a-pipeline. Caching only saves the fitting time, whereas here I propose caching the actual transformed output and reusing it for tuning classifier hyperparameters. https://github.com/scikit-learn/scikit-learn/issues/10068 mentions something about caching transform output, but I haven't found confirmation of this behavior anywhere.

### Describe alternatives you've considered, if relevant

This can easily implemented manually in a simple version, but it's less trivial to handle `n_jobs` fully properly. Assuming that we have 100 different parameter sets to check for preprocessing, and 5 sets for classifier, then using n_jobs=6 should always use 6 workers in parallel. But checking only the "current" preprocessing result, we could only use 5 workers. Handling this and starting next batch (i.e. checking estimator hyperparameters for next set of preprocessing parameters) is less trivial, and it would also require keeping 2 (or possibly more) preprocessing results in memory.

### Additional context

This is highly relevant to graph kernels, where preprocessing (computing the kernel) takes ~90% of time. In general, kernel methods would greatly benefit from this.

COMMENT:
You could have a meta-estimator that wraps around transformers, and implements `transform` with `lru_cache`, and wrap them before putting in your pipeline.

Labels: New Feature, Needs Triage
Boundary value problem of `SequentialFeatureSelector` and a suggestion on the document

### Describe the issue linked to the documentation

Hi devs of scikit-learn,

I found a potential slight boundary value problem in [`sklearn.feature_selection.SequentialFeatureSelector`](https://github.com/scikit-learn/scikit-learn/blob/70fdc843a/sklearn/feature_selection/_sequential.py#L232). The code here is looks like:

```python
if self.tol is not None and self.tol < 0 and self.direction == "forward":
    raise ValueError("tol must be positive when doing forward selection")
```

The boundary value judgment of `tol` is incomplete because 0 is not considered, although `tol` usually takes a very small value, such as 1e-4. 

By the way, the related document of [`sklearn.feature_selection.SequentialFeatureSelector`](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SequentialFeatureSelector.html#sequentialfeatureselector) can also be improved. In `tol` part, the document only claims that **`tol` can be negative when removing features using `direction="backward"`**. Why not add the information in the error message to the document? This will help users better understand the relationship between `tol` and `direction`, especially for newcomers.


### Suggest a potential alternative/fix

Maybe you can add **"tol must be positive when doing forward selection"** in the documentation and change `self.tol < 0` to `self.tol <= 0` in if branch.

COMMENT:
Changing from `self.tol < 0` to `self.tol <= 0` is a change of behaviour. In practice, it would allow to had feature that does not improve the score and it might not be the right thing to do since the feature is not useful. Since we are still able to pass a very small value `1e-12`, I don't think we should change the behaviour here.

What I would propose is to improve both the documentation and the error message to mention that `tol` is required to be *strictly* positive when doing forward selection.

@ParsifalXu Do you want to contribute a PR to improve the documentation and error message.

Labels: Documentation, spam
Boundary value problem of `SequentialFeatureSelector` and a suggestion on the document

### Describe the issue linked to the documentation

Hi devs of scikit-learn,

I found a potential slight boundary value problem in [`sklearn.feature_selection.SequentialFeatureSelector`](https://github.com/scikit-learn/scikit-learn/blob/70fdc843a/sklearn/feature_selection/_sequential.py#L232). The code here is looks like:

```python
if self.tol is not None and self.tol < 0 and self.direction == "forward":
    raise ValueError("tol must be positive when doing forward selection")
```

The boundary value judgment of `tol` is incomplete because 0 is not considered, although `tol` usually takes a very small value, such as 1e-4. 

By the way, the related document of [`sklearn.feature_selection.SequentialFeatureSelector`](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SequentialFeatureSelector.html#sequentialfeatureselector) can also be improved. In `tol` part, the document only claims that **`tol` can be negative when removing features using `direction="backward"`**. Why not add the information in the error message to the document? This will help users better understand the relationship between `tol` and `direction`, especially for newcomers.


### Suggest a potential alternative/fix

Maybe you can add **"tol must be positive when doing forward selection"** in the documentation and change `self.tol < 0` to `self.tol <= 0` in if branch.

COMMENT:
@glemaitre I'm glad to make a PR.

Labels: Documentation, spam
Boundary value problem of `SequentialFeatureSelector` and a suggestion on the document

### Describe the issue linked to the documentation

Hi devs of scikit-learn,

I found a potential slight boundary value problem in [`sklearn.feature_selection.SequentialFeatureSelector`](https://github.com/scikit-learn/scikit-learn/blob/70fdc843a/sklearn/feature_selection/_sequential.py#L232). The code here is looks like:

```python
if self.tol is not None and self.tol < 0 and self.direction == "forward":
    raise ValueError("tol must be positive when doing forward selection")
```

The boundary value judgment of `tol` is incomplete because 0 is not considered, although `tol` usually takes a very small value, such as 1e-4. 

By the way, the related document of [`sklearn.feature_selection.SequentialFeatureSelector`](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SequentialFeatureSelector.html#sequentialfeatureselector) can also be improved. In `tol` part, the document only claims that **`tol` can be negative when removing features using `direction="backward"`**. Why not add the information in the error message to the document? This will help users better understand the relationship between `tol` and `direction`, especially for newcomers.


### Suggest a potential alternative/fix

Maybe you can add **"tol must be positive when doing forward selection"** in the documentation and change `self.tol < 0` to `self.tol <= 0` in if branch.

COMMENT:
Closing, xref: https://github.com/scikit-learn/scikit-learn/issues/29509#issuecomment-2236806105

Labels: Documentation, spam
DOC: Add missing solver in the doc of `LogisticRegressionCV`

### Describe the issue linked to the documentation

Hi,

I found a potential code-doc inconsistency issue in [`sklearn.linear_model.LogisticRegression`](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html#logisticregression) and [`sklearn.linear_model.LogisticRegressionCV`](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegressionCV.html#sklearn.linear_model.LogisticRegressionCV).

In the description of `multi_class`, the document claims that **Multinomial is unavailable when solver='liblinear'**. However, in the code:

```python
if multi_class == "multinomial" and solver in ("liblinear", "newton-cholesky"):
    raise ValueError("Solver %s does not support a multinomial backend." % solver)
```
 
Apparently, `multinomial` is unavailable not only when `solver=liblinear` but also when `solver='newton-cholesky'`.

### Suggest a potential alternative/fix

Maybe you can add "newton-cholesky" in the corresponding document.

COMMENT:
Indeed, we mentioned in other places the solver that are supported but this is true that we are not enough explicit in the `multi_class` description. Even though, we are going to remove this parameter soonish, I think we can quickly solve this issue.

@ParsifalXu Do you want to make a PR to improve the documentation in this regard.

Labels: Documentation, spam
DOC: Add missing solver in the doc of `LogisticRegressionCV`

### Describe the issue linked to the documentation

Hi,

I found a potential code-doc inconsistency issue in [`sklearn.linear_model.LogisticRegression`](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html#logisticregression) and [`sklearn.linear_model.LogisticRegressionCV`](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegressionCV.html#sklearn.linear_model.LogisticRegressionCV).

In the description of `multi_class`, the document claims that **Multinomial is unavailable when solver='liblinear'**. However, in the code:

```python
if multi_class == "multinomial" and solver in ("liblinear", "newton-cholesky"):
    raise ValueError("Solver %s does not support a multinomial backend." % solver)
```
 
Apparently, `multinomial` is unavailable not only when `solver=liblinear` but also when `solver='newton-cholesky'`.

### Suggest a potential alternative/fix

Maybe you can add "newton-cholesky" in the corresponding document.

COMMENT:
> Indeed, we mentioned in other places the solver that are supported but this is true that we are not enough explicit in the `multi_class` description. Even though, we are going to remove this parameter soonish, I think we can quickly solve this issue.
> 
> @ParsifalXu Do you want to make a PR to improve the documentation in this regard.

@glemaitre Thanks for quick confirm. Sure, I'm glad to make a PR.

Labels: Documentation, spam
DOC: Add missing solver in the doc of `LogisticRegressionCV`

### Describe the issue linked to the documentation

Hi,

I found a potential code-doc inconsistency issue in [`sklearn.linear_model.LogisticRegression`](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html#logisticregression) and [`sklearn.linear_model.LogisticRegressionCV`](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegressionCV.html#sklearn.linear_model.LogisticRegressionCV).

In the description of `multi_class`, the document claims that **Multinomial is unavailable when solver='liblinear'**. However, in the code:

```python
if multi_class == "multinomial" and solver in ("liblinear", "newton-cholesky"):
    raise ValueError("Solver %s does not support a multinomial backend." % solver)
```
 
Apparently, `multinomial` is unavailable not only when `solver=liblinear` but also when `solver='newton-cholesky'`.

### Suggest a potential alternative/fix

Maybe you can add "newton-cholesky" in the corresponding document.

COMMENT:
Please don‘t open a PR and better consider reviewing #28840 which adds multinomial multiclass to newton-cholesky.

Labels: Documentation, spam
DOC: Add missing solver in the doc of `LogisticRegressionCV`

### Describe the issue linked to the documentation

Hi,

I found a potential code-doc inconsistency issue in [`sklearn.linear_model.LogisticRegression`](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html#logisticregression) and [`sklearn.linear_model.LogisticRegressionCV`](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegressionCV.html#sklearn.linear_model.LogisticRegressionCV).

In the description of `multi_class`, the document claims that **Multinomial is unavailable when solver='liblinear'**. However, in the code:

```python
if multi_class == "multinomial" and solver in ("liblinear", "newton-cholesky"):
    raise ValueError("Solver %s does not support a multinomial backend." % solver)
```
 
Apparently, `multinomial` is unavailable not only when `solver=liblinear` but also when `solver='newton-cholesky'`.

### Suggest a potential alternative/fix

Maybe you can add "newton-cholesky" in the corresponding document.

COMMENT:
Closing as #28840 should also resolve this, and that this seems AI generated. More context here: https://github.com/scikit-learn/scikit-learn/issues/29509

Labels: Documentation, spam
MAINT, RFC Simplify the Cython code in `sklearn/tree/` by splitting the "Splitter" and "Partitioner" code

### Summary of the problem
There are quite a number of GH issues with the label `tree` (https://github.com/scikit-learn/scikit-learn/issues?page=2&q=is%3Aopen+is%3Aissue+label%3Amodule%3Atree).

However, the code is a bit hard to approach as there's so many moving parts. For example, see: https://github.com/scikit-learn/scikit-learn/issues/18448#issuecomment-702598316, where users are intimidated by the Cython codebase. A large part of the complexity comes in the splitter, which contains the most algorithmic logic. I think with the work done by @thomasjpfan recently, we were able to pull apart the idea of a "partitioner" and "splitter" in the trees. 

This problem also arises because https://github.com/scikit-learn/scikit-learn/pull/29437/files#diff-e2cca285e1e883ab1d427120dfa974c1ba83eb6e2f5d5f416bbd99717ca5f5fc makes the code-diff very large inside `_splitter.pyx` file making it also harder to review and understand what changes affect the "splitter" and what changes affect the "partitioner".

### Proposed change
I further propose to split these into separate files, so it is easier to maintain, read and determine what is affecting what. In addition, we can decrease the code complexity by creating an abstract base class for the `DensePartitioner` and `SparsePartitioner`.

See the following [PR](https://github.com/scikit-learn/scikit-learn/pull/29458) for an example of what would change. The `_splitter.pyx` file would decrease by over 800 LOC, while it gets moved to `_partitioner.pyx`.

COMMENT:
Would ppl be open to this simplification?

I'm a bit biased as even from my personal experience, it was a headache reading through the entire `_splitter.*` files to understand what was going on. Separating the files semantically would make the code much more approachable.

Assuming there is no performance degradation.

Labels: module:tree, cython