RFC BayesSearchCV in scikit-learn

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/26141

<div type='discussions-op-text'>

<sup>Originally posted by **earlev4** April 11, 2023</sup>
**Hi! First off, thanks so much to the excellent work done by all the scikit-learn contributors! The project is truly a gift and your work is greatly appreciated!**

I still consider myself a novice when it comes to scikit-learn. In my usage, I typically will attempt to use GridSearchCV when searching for the best parameters. However, depending on the search space, GridSearchCV might not always be the best option and can be computationally expensive and time-consuming in some scenarios. RandomizedSearchCV can be an alternative in these situations, but does not always seem to provide the best parameters compared to scikit-optimize BayesSearchCV. In my humble opinion, scikit-optimize BayesSearchCV seems to be a nice compromise between GridSearchCV and RandomizedSearchCV, providing good parameters in a reasonable time.

Unfortunately, scikit-optimize BayesSearchCV seems to be no longer supported. The last commit was in 2021. As of NumPy 1.24, NumPy now results in an error, unless a workaround of `np.int = int` is used. This is just one example. There are numerous issues that have not been touched since 2021. It would be a shame to lose a project such as scikit-optimize BayesSearchCV and _humbly ask the contributors of scikit-learn if a similar version could be implemented in scikit-learn_.

**Thanks so much for your consideration! Looking forward to the discussion.**</div>

----------------------------------------

I thought `scikit-optimize` was a maintained project, but it seems it isn't, and I agree it'd be nice for the community to have a maintained `BayesSearchCV` available. 

I'm not sure if we should include it here, or `scikit-learn-extra`, but would be nice to have it.

COMMENT:
Depending on your prior about the outcome of a benchmark, it might be wise to start with that instead of a PR.

Something I don't know how to benchmark/measure is the "you trade tuning your estimator for tuning the regression model used by `BayesSearchCV`" part. As a user of `BayesSearchCV` I have to make choices about the regression model used. In particular if you use gaussian processes (GPs) as your model my impression is that you quickly end up in a world that is unfamiliar to the average person. It is a fascinating world, if you are into that kind of stuff, but it is by no means "free", like picking points at random is. A key problem to think about and try and solve is how to keep this cost under control or even reduce it so much that it is something the average user doesn't have to think about. I don't know how clever we can be about picking the right kernels for each hyper-parameter, etc. Part of the reason I spent time learning about and using trees as the regressor instead of GPs in scikit-optimize is that it looks like they perform well and are quite easy to reason about. There is a group at Freiburg university in Germany, the professor is called Hutter(?) or similar, who are very active in the autoML world and have used decision tree based models to win a lot of competitions.

Another line of though is to investigate what the popular optimisers/models are in tools like https://optuna.org/. Optuna seems like it is a popular choice. I don't think it contains any "secret sauce" that people couldn't get elsewhere, so understanding why people choose it and which part of it they use could give you an unfair advantage in terms of making a hyper-parameter optimiser that is simple to maintain, simple to use and used a lot.

But this moves the goal from "create a BayesSearchCV" to "create a simple and powerful hyper-parameter optimiser". Whether this is a good thing or not depends on people's priors (I guess).

A (very) old paper comparing methods https://proceedings.neurips.cc/paper_files/paper/2011/file/86e8f7ab32cfd12577bc2619bc635690-Paper.pdf

Labels: module:model_selection, Needs Decision - Include Feature