Talip7/scikit-learn-issues-multilabel huggingface dataset:
RFC BayesSearchCV in scikit-learn

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/26141

<div type='discussions-op-text'>

<sup>Originally posted by **earlev4** April 11, 2023</sup>
**Hi! First off, thanks so much to the excellent work done by all the scikit-learn contributors! The project is truly a gift and your work is greatly appreciated!**

I still consider myself a novice when it comes to scikit-learn. In my usage, I typically will attempt to use GridSearchCV when searching for the best parameters. However, depending on the search space, GridSearchCV might not always be the best option and can be computationally expensive and time-consuming in some scenarios. RandomizedSearchCV can be an alternative in these situations, but does not always seem to provide the best parameters compared to scikit-optimize BayesSearchCV. In my humble opinion, scikit-optimize BayesSearchCV seems to be a nice compromise between GridSearchCV and RandomizedSearchCV, providing good parameters in a reasonable time.

Unfortunately, scikit-optimize BayesSearchCV seems to be no longer supported. The last commit was in 2021. As of NumPy 1.24, NumPy now results in an error, unless a workaround of `np.int = int` is used. This is just one example. There are numerous issues that have not been touched since 2021. It would be a shame to lose a project such as scikit-optimize BayesSearchCV and _humbly ask the contributors of scikit-learn if a similar version could be implemented in scikit-learn_.

**Thanks so much for your consideration! Looking forward to the discussion.**</div>

----------------------------------------

I thought `scikit-optimize` was a maintained project, but it seems it isn't, and I agree it'd be nice for the community to have a maintained `BayesSearchCV` available. 

I'm not sure if we should include it here, or `scikit-learn-extra`, but would be nice to have it.

COMMENT:
As a one of the scikit-optimize creators: if someone who has experience maintaining open-source projects wants to take over maintenance of scikit-optimize please get in touch!

---

Given the additional complexity of using a tool like `BayesSearchCV` (you trade tuning your estimator for tuning the regression model used by `BayesSearchCV`) and the cheapness of additional compute: how does `RandomSearchCV`/`HalvingRandomSearchCV` with more CPU and RAM compare to bayes search?

The last time we (the scikit-optimize authors) paid attention to the literature/benchmarks related to this topic our conclusion was that renting more CPU+RAM was way less complicated.

Labels: module:model_selection, Needs Decision - Include Feature
RFC BayesSearchCV in scikit-learn

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/26141

<div type='discussions-op-text'>

<sup>Originally posted by **earlev4** April 11, 2023</sup>
**Hi! First off, thanks so much to the excellent work done by all the scikit-learn contributors! The project is truly a gift and your work is greatly appreciated!**

I still consider myself a novice when it comes to scikit-learn. In my usage, I typically will attempt to use GridSearchCV when searching for the best parameters. However, depending on the search space, GridSearchCV might not always be the best option and can be computationally expensive and time-consuming in some scenarios. RandomizedSearchCV can be an alternative in these situations, but does not always seem to provide the best parameters compared to scikit-optimize BayesSearchCV. In my humble opinion, scikit-optimize BayesSearchCV seems to be a nice compromise between GridSearchCV and RandomizedSearchCV, providing good parameters in a reasonable time.

Unfortunately, scikit-optimize BayesSearchCV seems to be no longer supported. The last commit was in 2021. As of NumPy 1.24, NumPy now results in an error, unless a workaround of `np.int = int` is used. This is just one example. There are numerous issues that have not been touched since 2021. It would be a shame to lose a project such as scikit-optimize BayesSearchCV and _humbly ask the contributors of scikit-learn if a similar version could be implemented in scikit-learn_.

**Thanks so much for your consideration! Looking forward to the discussion.**</div>

----------------------------------------

I thought `scikit-optimize` was a maintained project, but it seems it isn't, and I agree it'd be nice for the community to have a maintained `BayesSearchCV` available. 

I'm not sure if we should include it here, or `scikit-learn-extra`, but would be nice to have it.

COMMENT:
@betatim Thank you so much for your effort in `scikit-optimize`! If sklearn maintainers have decided to include `BayesSearchCV` mentioned above, we @Charlie-XIAO @ROMEEZHOU would like to help move that feature to sklearn. We are also willing to help find and solve the issues that have arisen since 2021. Are there specific smaller tasks that you recommend starting with?

Labels: module:model_selection, Needs Decision - Include Feature
RFC BayesSearchCV in scikit-learn

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/26141

<div type='discussions-op-text'>

<sup>Originally posted by **earlev4** April 11, 2023</sup>
**Hi! First off, thanks so much to the excellent work done by all the scikit-learn contributors! The project is truly a gift and your work is greatly appreciated!**

I still consider myself a novice when it comes to scikit-learn. In my usage, I typically will attempt to use GridSearchCV when searching for the best parameters. However, depending on the search space, GridSearchCV might not always be the best option and can be computationally expensive and time-consuming in some scenarios. RandomizedSearchCV can be an alternative in these situations, but does not always seem to provide the best parameters compared to scikit-optimize BayesSearchCV. In my humble opinion, scikit-optimize BayesSearchCV seems to be a nice compromise between GridSearchCV and RandomizedSearchCV, providing good parameters in a reasonable time.

Unfortunately, scikit-optimize BayesSearchCV seems to be no longer supported. The last commit was in 2021. As of NumPy 1.24, NumPy now results in an error, unless a workaround of `np.int = int` is used. This is just one example. There are numerous issues that have not been touched since 2021. It would be a shame to lose a project such as scikit-optimize BayesSearchCV and _humbly ask the contributors of scikit-learn if a similar version could be implemented in scikit-learn_.

**Thanks so much for your consideration! Looking forward to the discussion.**</div>

----------------------------------------

I thought `scikit-optimize` was a maintained project, but it seems it isn't, and I agree it'd be nice for the community to have a maintained `BayesSearchCV` available. 

I'm not sure if we should include it here, or `scikit-learn-extra`, but would be nice to have it.

COMMENT:
+1 for community maintained implementation either in `scikit-optimize`, `scikit-learn-extra` or a dedicated, single estimator package under the `scikit-learn-contrib` organization.

However as @betatim said, I think this is too much complexity to be maintained as part of the main `scikit-learn` repo.

Labels: module:model_selection, Needs Decision - Include Feature
RFC BayesSearchCV in scikit-learn

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/26141

<div type='discussions-op-text'>

<sup>Originally posted by **earlev4** April 11, 2023</sup>
**Hi! First off, thanks so much to the excellent work done by all the scikit-learn contributors! The project is truly a gift and your work is greatly appreciated!**

I still consider myself a novice when it comes to scikit-learn. In my usage, I typically will attempt to use GridSearchCV when searching for the best parameters. However, depending on the search space, GridSearchCV might not always be the best option and can be computationally expensive and time-consuming in some scenarios. RandomizedSearchCV can be an alternative in these situations, but does not always seem to provide the best parameters compared to scikit-optimize BayesSearchCV. In my humble opinion, scikit-optimize BayesSearchCV seems to be a nice compromise between GridSearchCV and RandomizedSearchCV, providing good parameters in a reasonable time.

Unfortunately, scikit-optimize BayesSearchCV seems to be no longer supported. The last commit was in 2021. As of NumPy 1.24, NumPy now results in an error, unless a workaround of `np.int = int` is used. This is just one example. There are numerous issues that have not been touched since 2021. It would be a shame to lose a project such as scikit-optimize BayesSearchCV and _humbly ask the contributors of scikit-learn if a similar version could be implemented in scikit-learn_.

**Thanks so much for your consideration! Looking forward to the discussion.**</div>

----------------------------------------

I thought `scikit-optimize` was a maintained project, but it seems it isn't, and I agree it'd be nice for the community to have a maintained `BayesSearchCV` available. 

I'm not sure if we should include it here, or `scikit-learn-extra`, but would be nice to have it.

COMMENT:
> The last time we (the scikit-optimize authors) paid attention to the literature/benchmarks related to this topic our conclusion was that renting more CPU+RAM was way less complicated.

That basically means putting more money behind it, and I don't think we can expect everybody to be able to do that.

I've heard over and over that people want BayesSearchCV in scikit-learn, so from that perspective, I'd be in favor of having it in the core library. Maybe a draft PR wouldn't hurt to see how complex inclusion of it in scikit-learn would be?

Labels: module:model_selection, Needs Decision - Include Feature
RFC BayesSearchCV in scikit-learn

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/26141

<div type='discussions-op-text'>

<sup>Originally posted by **earlev4** April 11, 2023</sup>
**Hi! First off, thanks so much to the excellent work done by all the scikit-learn contributors! The project is truly a gift and your work is greatly appreciated!**

I still consider myself a novice when it comes to scikit-learn. In my usage, I typically will attempt to use GridSearchCV when searching for the best parameters. However, depending on the search space, GridSearchCV might not always be the best option and can be computationally expensive and time-consuming in some scenarios. RandomizedSearchCV can be an alternative in these situations, but does not always seem to provide the best parameters compared to scikit-optimize BayesSearchCV. In my humble opinion, scikit-optimize BayesSearchCV seems to be a nice compromise between GridSearchCV and RandomizedSearchCV, providing good parameters in a reasonable time.

Unfortunately, scikit-optimize BayesSearchCV seems to be no longer supported. The last commit was in 2021. As of NumPy 1.24, NumPy now results in an error, unless a workaround of `np.int = int` is used. This is just one example. There are numerous issues that have not been touched since 2021. It would be a shame to lose a project such as scikit-optimize BayesSearchCV and _humbly ask the contributors of scikit-learn if a similar version could be implemented in scikit-learn_.

**Thanks so much for your consideration! Looking forward to the discussion.**</div>

----------------------------------------

I thought `scikit-optimize` was a maintained project, but it seems it isn't, and I agree it'd be nice for the community to have a maintained `BayesSearchCV` available. 

I'm not sure if we should include it here, or `scikit-learn-extra`, but would be nice to have it.

COMMENT:
Together with a PR (or maybe even before) it would be interesting to see a comparison to random search or sequential halving to evaluate how big an improvement you get (in terms of time spent to find a "nearly optimal point").

That way, no matter what the outcome, we can either add it to scikit-learn or write a blog post + docs to point out why it is not needed.

Labels: module:model_selection, Needs Decision - Include Feature
RFC BayesSearchCV in scikit-learn

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/26141

<div type='discussions-op-text'>

<sup>Originally posted by **earlev4** April 11, 2023</sup>
**Hi! First off, thanks so much to the excellent work done by all the scikit-learn contributors! The project is truly a gift and your work is greatly appreciated!**

I still consider myself a novice when it comes to scikit-learn. In my usage, I typically will attempt to use GridSearchCV when searching for the best parameters. However, depending on the search space, GridSearchCV might not always be the best option and can be computationally expensive and time-consuming in some scenarios. RandomizedSearchCV can be an alternative in these situations, but does not always seem to provide the best parameters compared to scikit-optimize BayesSearchCV. In my humble opinion, scikit-optimize BayesSearchCV seems to be a nice compromise between GridSearchCV and RandomizedSearchCV, providing good parameters in a reasonable time.

Unfortunately, scikit-optimize BayesSearchCV seems to be no longer supported. The last commit was in 2021. As of NumPy 1.24, NumPy now results in an error, unless a workaround of `np.int = int` is used. This is just one example. There are numerous issues that have not been touched since 2021. It would be a shame to lose a project such as scikit-optimize BayesSearchCV and _humbly ask the contributors of scikit-learn if a similar version could be implemented in scikit-learn_.

**Thanks so much for your consideration! Looking forward to the discussion.**</div>

----------------------------------------

I thought `scikit-optimize` was a maintained project, but it seems it isn't, and I agree it'd be nice for the community to have a maintained `BayesSearchCV` available. 

I'm not sure if we should include it here, or `scikit-learn-extra`, but would be nice to have it.

COMMENT:
Hi, have maintainers decided to work on this or this is open to public? Me and @jiawei-zhang-a have been interested in this for a while, if possible we would like to help with drafting a PR.

Labels: module:model_selection, Needs Decision - Include Feature
RFC BayesSearchCV in scikit-learn

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/26141

<div type='discussions-op-text'>

<sup>Originally posted by **earlev4** April 11, 2023</sup>
**Hi! First off, thanks so much to the excellent work done by all the scikit-learn contributors! The project is truly a gift and your work is greatly appreciated!**

I still consider myself a novice when it comes to scikit-learn. In my usage, I typically will attempt to use GridSearchCV when searching for the best parameters. However, depending on the search space, GridSearchCV might not always be the best option and can be computationally expensive and time-consuming in some scenarios. RandomizedSearchCV can be an alternative in these situations, but does not always seem to provide the best parameters compared to scikit-optimize BayesSearchCV. In my humble opinion, scikit-optimize BayesSearchCV seems to be a nice compromise between GridSearchCV and RandomizedSearchCV, providing good parameters in a reasonable time.

Unfortunately, scikit-optimize BayesSearchCV seems to be no longer supported. The last commit was in 2021. As of NumPy 1.24, NumPy now results in an error, unless a workaround of `np.int = int` is used. This is just one example. There are numerous issues that have not been touched since 2021. It would be a shame to lose a project such as scikit-optimize BayesSearchCV and _humbly ask the contributors of scikit-learn if a similar version could be implemented in scikit-learn_.

**Thanks so much for your consideration! Looking forward to the discussion.**</div>

----------------------------------------

I thought `scikit-optimize` was a maintained project, but it seems it isn't, and I agree it'd be nice for the community to have a maintained `BayesSearchCV` available. 

I'm not sure if we should include it here, or `scikit-learn-extra`, but would be nice to have it.

COMMENT:
@Charlie-XIAO we would really appreciate it if you could start with what @betatim suggests and a draft PR.

Labels: module:model_selection, Needs Decision - Include Feature
RFC BayesSearchCV in scikit-learn

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/26141

<div type='discussions-op-text'>

<sup>Originally posted by **earlev4** April 11, 2023</sup>
**Hi! First off, thanks so much to the excellent work done by all the scikit-learn contributors! The project is truly a gift and your work is greatly appreciated!**

I still consider myself a novice when it comes to scikit-learn. In my usage, I typically will attempt to use GridSearchCV when searching for the best parameters. However, depending on the search space, GridSearchCV might not always be the best option and can be computationally expensive and time-consuming in some scenarios. RandomizedSearchCV can be an alternative in these situations, but does not always seem to provide the best parameters compared to scikit-optimize BayesSearchCV. In my humble opinion, scikit-optimize BayesSearchCV seems to be a nice compromise between GridSearchCV and RandomizedSearchCV, providing good parameters in a reasonable time.

Unfortunately, scikit-optimize BayesSearchCV seems to be no longer supported. The last commit was in 2021. As of NumPy 1.24, NumPy now results in an error, unless a workaround of `np.int = int` is used. This is just one example. There are numerous issues that have not been touched since 2021. It would be a shame to lose a project such as scikit-optimize BayesSearchCV and _humbly ask the contributors of scikit-learn if a similar version could be implemented in scikit-learn_.

**Thanks so much for your consideration! Looking forward to the discussion.**</div>

----------------------------------------

I thought `scikit-optimize` was a maintained project, but it seems it isn't, and I agree it'd be nice for the community to have a maintained `BayesSearchCV` available. 

I'm not sure if we should include it here, or `scikit-learn-extra`, but would be nice to have it.

COMMENT:
Thanks for the information @betatim I'll definitely take a look.

Labels: module:model_selection, Needs Decision - Include Feature
RFC BayesSearchCV in scikit-learn

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/26141

<div type='discussions-op-text'>

<sup>Originally posted by **earlev4** April 11, 2023</sup>
**Hi! First off, thanks so much to the excellent work done by all the scikit-learn contributors! The project is truly a gift and your work is greatly appreciated!**

I still consider myself a novice when it comes to scikit-learn. In my usage, I typically will attempt to use GridSearchCV when searching for the best parameters. However, depending on the search space, GridSearchCV might not always be the best option and can be computationally expensive and time-consuming in some scenarios. RandomizedSearchCV can be an alternative in these situations, but does not always seem to provide the best parameters compared to scikit-optimize BayesSearchCV. In my humble opinion, scikit-optimize BayesSearchCV seems to be a nice compromise between GridSearchCV and RandomizedSearchCV, providing good parameters in a reasonable time.

Unfortunately, scikit-optimize BayesSearchCV seems to be no longer supported. The last commit was in 2021. As of NumPy 1.24, NumPy now results in an error, unless a workaround of `np.int = int` is used. This is just one example. There are numerous issues that have not been touched since 2021. It would be a shame to lose a project such as scikit-optimize BayesSearchCV and _humbly ask the contributors of scikit-learn if a similar version could be implemented in scikit-learn_.

**Thanks so much for your consideration! Looking forward to the discussion.**</div>

----------------------------------------

I thought `scikit-optimize` was a maintained project, but it seems it isn't, and I agree it'd be nice for the community to have a maintained `BayesSearchCV` available. 

I'm not sure if we should include it here, or `scikit-learn-extra`, but would be nice to have it.

COMMENT:
The original [Hyperband paper](https://jmlr.org/papers/volume18/16-558/16-558.pdf) that uses successive halving has benchmarks against Bayesian methods. To experiment with, Optuna already has a [Search CV](https://optuna.readthedocs.io/en/stable/reference/generated/optuna.integration.OptunaSearchCV.html) to use their strategies. For reference, [here](https://arxiv.org/pdf/2201.06433.pdf) is a recent paper comparing Optuna and HyperOpt. 

Given the third party options out there, I am not too excited about adding a Bayes Search CV. I rather push people to use successive halving and further improve it. For example, we can take advantage of warm starting: https://github.com/scikit-learn/scikit-learn/issues/15125

@amueller may have more insights on this topic

Labels: module:model_selection, Needs Decision - Include Feature
RFC BayesSearchCV in scikit-learn

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/26141

<div type='discussions-op-text'>

<sup>Originally posted by **earlev4** April 11, 2023</sup>
**Hi! First off, thanks so much to the excellent work done by all the scikit-learn contributors! The project is truly a gift and your work is greatly appreciated!**

I still consider myself a novice when it comes to scikit-learn. In my usage, I typically will attempt to use GridSearchCV when searching for the best parameters. However, depending on the search space, GridSearchCV might not always be the best option and can be computationally expensive and time-consuming in some scenarios. RandomizedSearchCV can be an alternative in these situations, but does not always seem to provide the best parameters compared to scikit-optimize BayesSearchCV. In my humble opinion, scikit-optimize BayesSearchCV seems to be a nice compromise between GridSearchCV and RandomizedSearchCV, providing good parameters in a reasonable time.

Unfortunately, scikit-optimize BayesSearchCV seems to be no longer supported. The last commit was in 2021. As of NumPy 1.24, NumPy now results in an error, unless a workaround of `np.int = int` is used. This is just one example. There are numerous issues that have not been touched since 2021. It would be a shame to lose a project such as scikit-optimize BayesSearchCV and _humbly ask the contributors of scikit-learn if a similar version could be implemented in scikit-learn_.

**Thanks so much for your consideration! Looking forward to the discussion.**</div>

----------------------------------------

I thought `scikit-optimize` was a maintained project, but it seems it isn't, and I agree it'd be nice for the community to have a maintained `BayesSearchCV` available. 

I'm not sure if we should include it here, or `scikit-learn-extra`, but would be nice to have it.

COMMENT:
> Regardless, I don't think I'm as familiar with this topic as maintainers (just providing some information and personal thoughts here). Maybe I should first wait for you to reach an agreement?

Yea, from what I've seen, people have moved away from bayesian approaches to approaches that takes training time into consideration.

> I'm not really sure what scikit-learn wants to have. An algorithm that has been quite popular,

From memory, I think successive halving is close enough to Hyperband and Hyperband is better than Bayesian approaches unless you have a lot of time to search the space. At the end of the day, I'll always recommend `HalvingSearchCV` even if `BayesianSearchCV` existed. I rather people get a good enough model quickly and move on to evaluation than to spend a long time searching for hyper parameters.

Labels: module:model_selection, Needs Decision - Include Feature