Talip7/scikit-learn-issues-multilabel huggingface dataset:
Ensure RandomizedSearchCV (and other optimizers) skips duplicated hyperparameter combinations

### Describe the workflow you want to enable

RandomizedSearchCV and similar hyperparameter tuners need to handle duplicate hyperparameter combinations. This issue is particularly noticeable when a user has a small number of hyperparameters, especially when they are integers or categorical values.

### Describe your proposed solution

A set of frozendict objects representing previously tried combinations should be maintained. When a new combination is generated (or retrieved), it should be skipped if it exists in this set, as running cross-validation on the same hyperparameters would be redundant.

### Describe alternatives you've considered, if relevant

There are no such alternatives.

### Additional context

I was surprised to learn that RandomizedSearchCV in sklearn 1.5.1 allows duplicate hyperparameters combinations.

`model.cv_results_['params']`

> [{'binningprocess__max_n_prebins': 7},
>   {'binningprocess__max_n_prebins': 14},
>   {'binningprocess__max_n_prebins': 22},
>   {'binningprocess__max_n_prebins': 14},
>   {'binningprocess__max_n_prebins': 22},
>   {'binningprocess__max_n_prebins': 11},
>   {'binningprocess__max_n_prebins': 13},
>   {'binningprocess__max_n_prebins': 22},
>   {'binningprocess__max_n_prebins': 20},
>   {'binningprocess__max_n_prebins': 20}]

COMMENT:
While `RandomizedSearchCV` was originally intended for parameter spaces that are much larger than what is typically fed to a `GridSearchCV` case, I think that would you propose (rejecting explored candidates) would improve the user experience and would not cost much to maintain.

However we need to be careful not to introduce an infinity loop in the process. If more than 10 sampled candidates are rejected in a row, I would rather raise a `ConvergenceWarning` with an informative error message and stop the search priori to reaching the `n_iter` target search budget.

EDIT The warning message could point the user to either use a richer / finer search space, or decrease the value of `n_iter` or switch to `GridSearchCV` for an exhaustive search of small parameter spaces.

Labels: New Feature
Ensure RandomizedSearchCV (and other optimizers) skips duplicated hyperparameter combinations

### Describe the workflow you want to enable

RandomizedSearchCV and similar hyperparameter tuners need to handle duplicate hyperparameter combinations. This issue is particularly noticeable when a user has a small number of hyperparameters, especially when they are integers or categorical values.

### Describe your proposed solution

A set of frozendict objects representing previously tried combinations should be maintained. When a new combination is generated (or retrieved), it should be skipped if it exists in this set, as running cross-validation on the same hyperparameters would be redundant.

### Describe alternatives you've considered, if relevant

There are no such alternatives.

### Additional context

I was surprised to learn that RandomizedSearchCV in sklearn 1.5.1 allows duplicate hyperparameters combinations.

`model.cv_results_['params']`

> [{'binningprocess__max_n_prebins': 7},
>   {'binningprocess__max_n_prebins': 14},
>   {'binningprocess__max_n_prebins': 22},
>   {'binningprocess__max_n_prebins': 14},
>   {'binningprocess__max_n_prebins': 22},
>   {'binningprocess__max_n_prebins': 11},
>   {'binningprocess__max_n_prebins': 13},
>   {'binningprocess__max_n_prebins': 22},
>   {'binningprocess__max_n_prebins': 20},
>   {'binningprocess__max_n_prebins': 20}]

COMMENT:
> However we need to be careful not to introduce an infinity loop in the process. If more than 10 sampled candidates are rejected in a row, I would rather raise a `ConvergenceWarning` with an informative error message and stop the search priori to reaching the `n_iter` target search budget.

Thanks! I was thinking the same. Maybe even make that 10 a configurable parameter, like **max_repeating_combinations: int=10**.

max_repeating_combinations=-1 could have a special meaning = "do not use explored candidates control"
max_repeating_combinations=0 could mean "do not check wasted candidates". 

Also the (potentially infinite) set of explored candidates could be replaced by some limited container like collections.deque or ExpiringDict to control RAM usage when the number of combinations to check is huge (in this case, a few extra init parameters would be needed, how many max entries and for what time to keep in the container).

Or maybe we can use already existing data structure that ends up being returned in the 'params' field, I haven't checked the source codes.

Labels: New Feature
Discrepancy between .fit_transform() and .transform() methods in the LLE module

### Describe the bug

A user would expect the same result from  
- `.fit(X)` and then `.transform(X)`
- `.fit_transformX()`

But this is not the case in the current code for `LocallyLinearEmbedding`. 

### Steps/Code to Reproduce

```python
from sklearn.manifold import LocallyLinearEmbedding
from sklearn.datasets import make_s_curve
import numpy as np

X,_ = make_s_curve(100)
methods = ["standard", "hessian", "ltsa", "modified"]
for method in methods:
    lle = LocallyLinearEmbedding(method=method, n_neighbors=12)
    fit_transform = lle.fit_transform(X)  
    fit_then_transform = lle.transform(X)
    equal = np.any(fit_transform == fit_then_transform)
    close_count = np.isclose(fit_transform ,fit_then_transform).sum()
    print(f"For {method} it is {equal} that f_t and f_then_t are equal.")
    print(f"Only {close_count} are close.\n" )
```

### Expected Results

```text
For {method} it is True that `fit_transform(X) == transform(x)`.
```

### Actual Results

```text
For standard, it is False that `fit_transform(X) == transform(x)`.
Only 2 are close.

For hessian, it is False that `fit_transform(X) == transform(x)`.
Only 1 are close.

For ltsa, it is False that `fit_transform(X) == transform(x)`.
Only 1 are close.

For modified, it is False that `fit_transform(X) == transform(x)`.
Only 0 are close.
```

### Versions

```shell
System:
    python: 3.10.14 | packaged by conda-forge | (main, Mar 20 2024, 12:51:49) [Clang 16.0.6 ]
executable: /Users/wonderman/miniforge3/envs/sklearn_dev_env/bin/python
   machine: macOS-14.6.1-arm64-arm-64bit

Python dependencies:
      sklearn: 1.6.dev0
          pip: 24.2
   setuptools: 72.2.0
        numpy: 2.1.0
        scipy: 1.14.1
       Cython: 3.0.11
       pandas: None
   matplotlib: None
       joblib: 1.4.2
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 8
         prefix: libopenblas
       filepath: /Users/wonderman/miniforge3/envs/sklearn_dev_env/lib/libopenblas.0.dylib
        version: 0.3.27
threading_layer: openmp
   architecture: VORTEX

       user_api: openmp
   internal_api: openmp
    num_threads: 8
         prefix: libomp
       filepath: /Users/wonderman/miniforge3/envs/sklearn_dev_env/lib/libomp.dylib
        version: None
```

COMMENT:
I believe the way code is written was intentional to get some performance benefit from reusing the precomputed `embedding_` attribute for the training data points. But I agree this is a problem if the numeric discrepancy is too large.

Instead of using `np.isclose` with the implicit/default tolerance level, could you please compute the euclidean distance of the difference divided by the maximum norm of the two ways to compute the transformation?

Labels: Bug
Include T-Processes Subclass of Gaussian-Processes

This is a feature request to implement T-process. Moving the discussion into the issue tracker to get more visibility.

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/28942

<div type='discussions-op-text'>

<sup>Originally posted by **conradstevens** May  3, 2024</sup>
I am implementing T-Processes (TP)s In addition to Gaussian Processes (GP)s.

Due to the great similarities in structure and functionality I have made a new class _TProcessRegressor_ a subclass of _GaussianProcessRegressor_. However, this will result in duplicate code. 

If it where up to me, there would be a class __stochasticProcessRegressor_ that GPs and TPs would both inherit from. This would minimize duplicate code and make integrating other stochastic processes (eg Gamma Processes) more straight forward. However, this architecture change would complicate an eventual pull request. 

I am interested to hear peoples thoughts on implementing TPs and the pros and cons of the two architectures. 

T-Process Background:
https://arxiv.org/abs/1402.4306</div>

COMMENT:
@conradlee Looking at the original paper, it seems that it would pass the inclusion criterion for year since publication and the number of citations.

Could you provide compelling examples in which it is interesting to have this technique and where Gaussian Process will not be enough.

@snath-xoc since you know much better Gaussian Process than me, are you aware of this technique?

Labels: New Feature
Include T-Processes Subclass of Gaussian-Processes

This is a feature request to implement T-process. Moving the discussion into the issue tracker to get more visibility.

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/28942

<div type='discussions-op-text'>

<sup>Originally posted by **conradstevens** May  3, 2024</sup>
I am implementing T-Processes (TP)s In addition to Gaussian Processes (GP)s.

Due to the great similarities in structure and functionality I have made a new class _TProcessRegressor_ a subclass of _GaussianProcessRegressor_. However, this will result in duplicate code. 

If it where up to me, there would be a class __stochasticProcessRegressor_ that GPs and TPs would both inherit from. This would minimize duplicate code and make integrating other stochastic processes (eg Gamma Processes) more straight forward. However, this architecture change would complicate an eventual pull request. 

I am interested to hear peoples thoughts on implementing TPs and the pros and cons of the two architectures. 

T-Process Background:
https://arxiv.org/abs/1402.4306</div>

COMMENT:
@glemaitre Funny enough I was looking for methods like this to represent phenomena that are not so thinly tailed (Student-t distribution allows for fatter tails). I know of the R version ([STReg](https://cran.r-project.org/web/packages/StReg/StReg.pdf) ) will give the paper a proper look... on the implementation side I agree that it would be better to think of having e.g., class _stochasticProcessRegressor

Labels: New Feature
Include T-Processes Subclass of Gaussian-Processes

This is a feature request to implement T-process. Moving the discussion into the issue tracker to get more visibility.

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/28942

<div type='discussions-op-text'>

<sup>Originally posted by **conradstevens** May  3, 2024</sup>
I am implementing T-Processes (TP)s In addition to Gaussian Processes (GP)s.

Due to the great similarities in structure and functionality I have made a new class _TProcessRegressor_ a subclass of _GaussianProcessRegressor_. However, this will result in duplicate code. 

If it where up to me, there would be a class __stochasticProcessRegressor_ that GPs and TPs would both inherit from. This would minimize duplicate code and make integrating other stochastic processes (eg Gamma Processes) more straight forward. However, this architecture change would complicate an eventual pull request. 

I am interested to hear peoples thoughts on implementing TPs and the pros and cons of the two architectures. 

T-Process Background:
https://arxiv.org/abs/1402.4306</div>

COMMENT:
@glemaitre - I haven't contributed to scikit-learn in a while.  I think you've got the wrong Conrad.  Maybe you meant to tag @conradstevens ?

Labels: New Feature
Include T-Processes Subclass of Gaussian-Processes

This is a feature request to implement T-process. Moving the discussion into the issue tracker to get more visibility.

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/28942

<div type='discussions-op-text'>

<sup>Originally posted by **conradstevens** May  3, 2024</sup>
I am implementing T-Processes (TP)s In addition to Gaussian Processes (GP)s.

Due to the great similarities in structure and functionality I have made a new class _TProcessRegressor_ a subclass of _GaussianProcessRegressor_. However, this will result in duplicate code. 

If it where up to me, there would be a class __stochasticProcessRegressor_ that GPs and TPs would both inherit from. This would minimize duplicate code and make integrating other stochastic processes (eg Gamma Processes) more straight forward. However, this architecture change would complicate an eventual pull request. 

I am interested to hear peoples thoughts on implementing TPs and the pros and cons of the two architectures. 

T-Process Background:
https://arxiv.org/abs/1402.4306</div>

COMMENT:
Glad to hear there is some interest. For some context, I am just finishing up my Masters on the topic, hopefully I'll be able to share some relevant implementations soon. 

As for the model architecture, in retrospect there didn't end up being that much duplicate code... Again, happy to discuss. 

Conrad

Labels: New Feature
Include T-Processes Subclass of Gaussian-Processes

This is a feature request to implement T-process. Moving the discussion into the issue tracker to get more visibility.

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/28942

<div type='discussions-op-text'>

<sup>Originally posted by **conradstevens** May  3, 2024</sup>
I am implementing T-Processes (TP)s In addition to Gaussian Processes (GP)s.

Due to the great similarities in structure and functionality I have made a new class _TProcessRegressor_ a subclass of _GaussianProcessRegressor_. However, this will result in duplicate code. 

If it where up to me, there would be a class __stochasticProcessRegressor_ that GPs and TPs would both inherit from. This would minimize duplicate code and make integrating other stochastic processes (eg Gamma Processes) more straight forward. However, this architecture change would complicate an eventual pull request. 

I am interested to hear peoples thoughts on implementing TPs and the pros and cons of the two architectures. 

T-Process Background:
https://arxiv.org/abs/1402.4306</div>

COMMENT:
> I think you've got the wrong Conrad. Maybe you meant to tag @conradstevens ?

Indeed :)

Labels: New Feature
Include T-Processes Subclass of Gaussian-Processes

This is a feature request to implement T-process. Moving the discussion into the issue tracker to get more visibility.

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/28942

<div type='discussions-op-text'>

<sup>Originally posted by **conradstevens** May  3, 2024</sup>
I am implementing T-Processes (TP)s In addition to Gaussian Processes (GP)s.

Due to the great similarities in structure and functionality I have made a new class _TProcessRegressor_ a subclass of _GaussianProcessRegressor_. However, this will result in duplicate code. 

If it where up to me, there would be a class __stochasticProcessRegressor_ that GPs and TPs would both inherit from. This would minimize duplicate code and make integrating other stochastic processes (eg Gamma Processes) more straight forward. However, this architecture change would complicate an eventual pull request. 

I am interested to hear peoples thoughts on implementing TPs and the pros and cons of the two architectures. 

T-Process Background:
https://arxiv.org/abs/1402.4306</div>

COMMENT:
> Glad to hear there is some interest. For some context, I am just finishing up my Masters on the topic, hopefully I'll be able to share some relevant implementations soon.
> 
> As for the model architecture, in retrospect there didn't end up being that much duplicate code... Again, happy to discuss.
> 

Great, looking forward to seeing the implementations. I've see the associated PR and will go through glad to hear that the code implementation was simpler than initially thought.

Labels: New Feature
Big problem with scikit-learn on Python311 when installing (FreeBSD)

### Describe the bug

[long log.txt](https://github.com/user-attachments/files/16875744/long.log.txt)
https://github.com/man-group/dtale/issues/877#issuecomment-2329784822

### Steps/Code to Reproduce

After `pip install -U scikit-learn==1.1.3`
https://github.com/man-group/dtale/issues/877

### Expected Results

Just install and use with other  

### Actual Results

 problem

### Versions

```shell
scikit-learn==1.1.3
```

COMMENT:
This does not seem like a scikit-learn issue, closing this one.

Quickly looking at your log, it looks like it is trying to compile scipy from source and failing.

Labels: Bug, Needs Triage
Big problem with scikit-learn on Python311 when installing (FreeBSD)

### Describe the bug

[long log.txt](https://github.com/user-attachments/files/16875744/long.log.txt)
https://github.com/man-group/dtale/issues/877#issuecomment-2329784822

### Steps/Code to Reproduce

After `pip install -U scikit-learn==1.1.3`
https://github.com/man-group/dtale/issues/877

### Expected Results

Just install and use with other  

### Actual Results

 problem

### Versions

```shell
scikit-learn==1.1.3
```

COMMENT:
Project don't care about correct compilation?
Where are compiled versions?

Labels: Bug, Needs Triage
Running RFECV.fit inside joblib.Parallel causes ValueError or AttributeError

```py
from sklearn.datasets import make_classification
from sklearn.feature_selection import RFECV
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold

from joblib import Parallel, delayed



X, y = make_classification(
    n_samples=500,
    n_features=15,
    n_informative=3,
    n_redundant=2,
    n_repeated=0,
    n_classes=8,
    n_clusters_per_class=1,
    class_sep=0.8,
    random_state=0,
)

min_features_to_select = 1  # Minimum number of features to consider
clf = LogisticRegression()
cv = StratifiedKFold(5)

def fit():
    rfecv = RFECV(
        estimator=clf,
        step=1,
        cv=cv,
        scoring="accuracy",
        min_features_to_select=min_features_to_select,
        n_jobs=2,
    )

    rfecv.fit(X, y)


Parallel(n_jobs=2)(delayed(fit)() for _ in range(5))
```

You can get two types of errors:
```
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (5,) + inhomogeneous part.
```
or
```
AttributeError: 'LogisticRegression' object has no attribute 'coef_'
```

I don't quite understand what is happening yet but it seems like there is a side-effect somewhere I would have thought that the inner parallelism would do copy but apparently not. Using `clone` in https://github.com/scikit-learn/scikit-learn/blob/e04142cbe0f4f854272f877eb9692053b0a6bcf8/sklearn/feature_selection/_rfe.py#L886-L889

seems to fix it:
```diff
diff --git a/sklearn/feature_selection/_rfe.py b/sklearn/feature_selection/_rfe.py
index 8ccbffce9b..99aa8e2b4f 100644
--- a/sklearn/feature_selection/_rfe.py
+++ b/sklearn/feature_selection/_rfe.py
@@ -886,7 +886,7 @@ class RFECV(RFE):
             func = delayed(_rfe_single_fit)
 
         scores_features = parallel(
-            func(rfe, self.estimator, X, y, train, test, scorer, routed_params)
+            func(clone(rfe), self.estimator, X, y, train, test, scorer, routed_params)
             for train, test in cv.split(X, y, **routed_params.splitter.split)
         )
         scores, step_n_features = zip(*scores_features)
```

This was seen in https://github.com/scikit-learn/scikit-learn/pull/29614 (and also in private testing of CPython 3.13 free-threaded with default joblib backend set to threading but I thought it was threading related).

COMMENT:
Please check my pull request i have resolved this bug

Labels: Bug
CI CUDA CI not running in lock-file update automated PR

Discussed in https://github.com/scikit-learn/scikit-learn/pull/29576#issuecomment-2255371442, for now we need to remember to unset "CUDA CI" label and set it again manually on automated array API lock-file PRs like https://github.com/scikit-learn/scikit-learn/pull/29766.

From https://github.com/scikit-learn/scikit-learn/issues/29576#issuecomment-2328452652, maybe this is due to triggering a workflow from a workflow limitations, see [doc](https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/triggering-a-workflow#triggering-a-workflow-from-a-workflow). Apparently you need a PAT rather than the default GITHUB_TOKEN when you are in this case. The last example in the doc is very similar to our use case I think:

> Conversely, the following workflow uses GITHUB_TOKEN to add a label to an issue. It will not trigger any workflows that run when a label is added.

I don't remember if we have a PAT for some of the CI that we could try to see whether that fixes the issue ...

COMMENT:
Indeed this sounds like a valid explanation.

> I don't remember if we have a PAT for some of the CI that we could try to see whether that fixes the issue ...

`.github/workflows/update_tracking_issue.yml` uses `${{ secrets.BOT_GITHUB_TOKEN }}` which might be a PAT although I am not sure.

It's also used in `.github/workflows/update-lock-files.yml` by the `peter-evans/create-pull-request` action.

EDIT: this might be a PAT based on https://github.com/peter-evans/create-pull-request?tab=readme-ov-file#token.

Labels: Build / CI
CI CUDA CI not running in lock-file update automated PR

Discussed in https://github.com/scikit-learn/scikit-learn/pull/29576#issuecomment-2255371442, for now we need to remember to unset "CUDA CI" label and set it again manually on automated array API lock-file PRs like https://github.com/scikit-learn/scikit-learn/pull/29766.

From https://github.com/scikit-learn/scikit-learn/issues/29576#issuecomment-2328452652, maybe this is due to triggering a workflow from a workflow limitations, see [doc](https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/triggering-a-workflow#triggering-a-workflow-from-a-workflow). Apparently you need a PAT rather than the default GITHUB_TOKEN when you are in this case. The last example in the doc is very similar to our use case I think:

> Conversely, the following workflow uses GITHUB_TOKEN to add a label to an issue. It will not trigger any workflows that run when a label is added.

I don't remember if we have a PAT for some of the CI that we could try to see whether that fixes the issue ...

COMMENT:
Based on https://github.com/scikit-learn/scikit-learn/blob/390ce98e6b1ff1c509bfff11dc274cb5175ae344/maint_tools/update_tracking_issue.py#L8-L11 it seems that this is a PAT from @scikit-learn-bot that only has `repo:public_repo`.

It might not be enough to change a label on its own PR but it's worth a try. Otherwise we could elevate the permissions of the bot account and/or create a new PAT with elevated permissions to set the label.

Maybe @thomasjpfan knows better.

Labels: Build / CI