RFC BayesSearchCV in scikit-learn

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/26141

<div type='discussions-op-text'>

<sup>Originally posted by **earlev4** April 11, 2023</sup>
**Hi! First off, thanks so much to the excellent work done by all the scikit-learn contributors! The project is truly a gift and your work is greatly appreciated!**

I still consider myself a novice when it comes to scikit-learn. In my usage, I typically will attempt to use GridSearchCV when searching for the best parameters. However, depending on the search space, GridSearchCV might not always be the best option and can be computationally expensive and time-consuming in some scenarios. RandomizedSearchCV can be an alternative in these situations, but does not always seem to provide the best parameters compared to scikit-optimize BayesSearchCV. In my humble opinion, scikit-optimize BayesSearchCV seems to be a nice compromise between GridSearchCV and RandomizedSearchCV, providing good parameters in a reasonable time.

Unfortunately, scikit-optimize BayesSearchCV seems to be no longer supported. The last commit was in 2021. As of NumPy 1.24, NumPy now results in an error, unless a workaround of `np.int = int` is used. This is just one example. There are numerous issues that have not been touched since 2021. It would be a shame to lose a project such as scikit-optimize BayesSearchCV and _humbly ask the contributors of scikit-learn if a similar version could be implemented in scikit-learn_.

**Thanks so much for your consideration! Looking forward to the discussion.**</div>

----------------------------------------

I thought `scikit-optimize` was a maintained project, but it seems it isn't, and I agree it'd be nice for the community to have a maintained `BayesSearchCV` available. 

I'm not sure if we should include it here, or `scikit-learn-extra`, but would be nice to have it.

COMMENT:
As for benchmarking, optuna developers seem to use their own [kurobako](https://github.com/optuna/kurobako) and there is also [HBOBench](https://github.com/automl/HPOBench).

> I rather push people to use successive halving and further improve it.

This does seem reasonable. For instance Hyperband (based on successive halving) seems to perform better than Bayesian optimization with small~medium budget (because Bayesian is like random initially and only stands out with larger budgets). However, there is also [BOHB](https://arxiv.org/pdf/1807.01774.pdf) combining Bayesian optimization and Hyperband that again outperforms Hyperband on most tasks (though the [BOHB repo](https://github.com/automl/HpBandSter) is no longer maintained). Here is an [interesting post](https://www.automl.org/blog_bohb/) that might be worth reading.

I have also seen [DEHB](https://arxiv.org/pdf/2105.09821.pdf) based on differential evolution and Hyperband that claims to beat BOHB. The repo is [here](https://github.com/automl/DEHB).

I'm not really sure what scikit-learn wants to have. An algorithm that has been quite popular, a state-of-the-art algorithm, or something else?

> Given the third party options out there

I'm thinking that scikit-learn is still nowadays the first choice when dealing with machine learning problems especially for beginners (and beginners may not even know `optuna`).

Regardless, I don't think I'm as familiar with this topic as maintainers (just providing some information and personal thoughts here). Maybe I should first wait for you to reach an agreement?

Labels: module:model_selection, Needs Decision - Include Feature