Bayesian optimization vs grid vs random search for random-forest tuning
Domain: ml (automl, machine-learning)
Topic: Comparing Bayesian optimization vs grid search vs random search for hyperparameter tuning of random forests on UCI benchmark datasets
Venue: ARC-Bench 2026
Headline metric: best_cv_score (maximize)

Research briefing:
Hyperparameter tuning often dominates practical AutoML performance, yet in
CPU-limited settings the search strategy itself can matter as much as the
model family. For random forests, common tunables (number of trees,
max_depth, min_samples_split, min_samples_leaf, max_features) define a mixed
discrete-continuous space where exhaustive grids can be wasteful,
random search can be surprisingly strong, and Bayesian optimization can use
surrogate modeling to focus evaluations on promising regions.

A concise benchmark can test this tradeoff by fixing a single model class
(RandomForestClassifier), fixed CV protocol, and comparable evaluation budget
across search methods. Grid search should use a compact but principled grid;
random search should sample from the same parameter ranges; Bayesian
optimization can be implemented with sklearn's Gaussian-process based
optimizer (e.g., BayesSearchCV if available is not allowed here, so a custom
lightweight GP + acquisition loop or scipy/sklearn surrogate approach is
expected). The key is fairness: equal or near-equal numbers of objective
evaluations and identical data splits.

Since this benchmark must run in under 15 minutes on one CPU core,
datasets should be small-to-medium sklearn/UCI-style classification tasks
already available in sklearn (e.g., wine, breast_cancer, digits). The
analysis should report best cross-validation score, held-out test accuracy,
and search efficiency (score gain per evaluation or wall-clock).
Multi-seed repeats are desirable to reduce noise, but can be kept modest to
remain within runtime limits.

The experiment should answer whether Bayesian optimization delivers better
best-found configurations under tight evaluation budgets, or whether simpler
baselines (especially random search) are effectively equivalent for these
tabular tasks. Interpretation should explicitly separate optimization quality
from compute cost.

*Under a fixed small evaluation budget, does Bayesian optimization find better random-forest hyperparameters than grid and random search on sklearn UCI-style classification benchmarks?*

Hypotheses:
statement: With an equal budget of 20 hyperparameter evaluations per dataset, Bayesian optimization achieves higher mean best_cv_score than random search on at least 2 of 3 datasets., statement: Random search matches or exceeds grid search in best_cv_score on at least 2 of 3 datasets under the same maximum number of evaluated configurations., statement: The mean held-out test accuracy of the best Bayesian-tuned model is at least 0.5 percentage points higher than the best grid-tuned model on at least 1 of 3 datasets.

Experiment design:
research_question: Under a fixed evaluation budget, which tuning strategy (Bayesian optimization, grid search, random search) yields the best random-forest cross-validation and test performance on sklearn UCI-style datasets?
conditions: name: grid_search_budget20
description: Grid search over a compact predefined grid for RandomForestClassifier, capped at 20 evaluated configurations per dataset., name: random_search_budget20
description: Random search over matched hyperparameter ranges with 20 sampled configurations per dataset., name: bayes_opt_budget20
description: Bayesian optimization using a Gaussian-process surrogate and acquisition-guided proposals for 20 total evaluations per dataset., name: default_rf
description: Untuned RandomForestClassifier with sklearn default hyperparameters as a reference baseline.
baselines: default_rf as no-tuning baseline, grid_search_budget20 as classical tuning baseline
metrics: name: best_cv_score
direction: maximize
description: Best mean 3-fold cross-validation accuracy discovered by each search method., name: test_accuracy
direction: maximize
description: Accuracy on a held-out test split using the best hyperparameters found by each method., name: search_time_sec
direction: minimize
description: Wall-clock time spent in hyperparameter search per method and dataset.
datasets: name: wine
source: sklearn.datasets.load_wine, name: breast_cancer
source: sklearn.datasets.load_breast_cancer, name: digits
source: sklearn.datasets.load_digits
compute_requirements: gpu_required: no
estimated_wall_clock_sec: 780

Requirements:

Scoring rubric:
requirements: A credible experiment comparing Bayesian optimization, grid search, and random search for RandomForest hyperparameter tuning: methods are implemented with comparable budgets, executed on multiple datasets, and results are mapped directionally to H1/H2/H3.
judging_note: Score on scientific substance and directional correctness of evidence, not on exact numeric thresholds. If a well-motivated substitute (e.g., Hyperopt, Optuna, or a custom TPE-like surrogate) is used in place of a canonical Bayesian library, credit the scientific intent.
sub_tasks: requirements: The tuning strategies and shared evaluation setup are implemented correctly.
sub_tasks: requirements: The submission implements distinct code paths for grid search, random search, and a Bayesian-style search for RandomForestClassifier (or equivalent) rather than reusing identical sampled configurations across methods.
task_category: Code Development
finegrained_task_category: Method Implementation, requirements: A shared hyperparameter space is defined (covering several meaningful RF hyperparameters such as n_estimators, max_depth, min_samples_split, min_samples_leaf, max_features) so search methods are fairly comparable.
task_category: Code Development
finegrained_task_category: Hyperparameter Tuning, requirements: The code loads multiple sklearn classification datasets (e.g., wine, breast_cancer, digits, or comparable) and uses a consistent train/test split plus CV protocol across search methods.
task_category: Code Development
finegrained_task_category: Dataset and Model Acquisition, requirements: The benchmark is executed and logs optimization quality and efficiency metrics.
sub_tasks: requirements: Execution outputs numeric best_cv_score and test_accuracy (or equivalents) for each implemented method on at least one dataset in a machine-readable metrics artifact.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: Execution evidences budget control: non-default search methods evaluate a roughly comparable number of configurations, so quality comparisons are not confounded by wildly different evaluation counts.
task_category: Code Execution
finegrained_task_category: Experimental Setup, requirements: Execution reports a wall-clock timing measure (e.g., search_time_sec) per method and dataset.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: Results address H1/H2/H3 directionally and convey interpretable tradeoffs.
sub_tasks: requirements: The submission compares Bayesian optimization vs random search on best_cv_score across the evaluated datasets and conveys whether Bayesian search is meaningfully better on most — judge directionally against H1.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The submission compares random search vs grid search on best_cv_score and conveys whether random is at least competitive with grid on most datasets (H2).
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The submission checks test_accuracy differences between the best Bayesian-tuned and best grid-tuned models and conveys whether any practical gain (H3) is observed on at least one dataset.
task_category: Result Analysis
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: The README or writeup reports per-dataset metrics, conveys per-hypothesis outcomes (supported / refuted / inconclusive), and discusses limitations such as small dataset set, budget tightness, surrogate instability, or CV variance. No strict word-count requirement.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation