Bagging vs boosting vs stacking for noisy non-linear regression
Domain: ml (machine-learning, ensemble-methods)
Topic: Evaluating ensemble methods (bagging, boosting, stacking) for regression on synthetic non-linear datasets with varying noise levels
Venue: ARC-Bench 2026
Headline metric: rmse (minimize)

Research briefing:
Ensemble learning is often presented as a broadly reliable way to improve
predictive performance, but different ensemble families can react very
differently to observation noise in regression. Bagging primarily reduces
variance by averaging many weakly correlated models. Boosting is sequential
and can aggressively fit residuals, which can help on structured non-linear
signals but may overfit high-noise targets if regularization is not tuned.
Stacking combines heterogeneous base learners through a meta-model and may
benefit from diversity, but it also introduces extra fitting stages that can
be sensitive to data size and leakage handling.

A focused CPU-scale benchmark can probe these tradeoffs using synthetic and
lightweight sklearn regression datasets where non-linearity and noise are
controllable. By varying noise level explicitly in generated data, the study
can test whether relative ranking among bagging, boosting, and stacking is
stable or changes materially as signal-to-noise degrades. This design also
encourages careful evaluation with repeated random seeds and test-set metrics
that capture both average error and fit quality.

A credible implementation should compare at least one representative for each
family (e.g., RandomForestRegressor for bagging, GradientBoostingRegressor
for boosting, StackingRegressor with diverse base estimators for stacking),
include a single-model baseline, and report RMSE-centered conclusions. Since
this benchmark is intended for fast autonomous execution, datasets and models
must remain small enough to run in minutes on one CPU core while still
producing clear numeric contrasts across noise regimes.

The key outcome is not reproducing a known paper result but determining when
each ensemble strategy is most robust for noisy non-linear regression in a
constrained practical setting.

*How does predictive robustness (RMSE) of bagging, boosting, and stacking change as regression noise increases on small non-linear benchmarks?*

Hypotheses:
statement: On the high-noise synthetic dataset (noise >= 25), bagging (RandomForestRegressor) achieves lower mean RMSE than boosting (GradientBoostingRegressor), averaged over at least 5 seeds., statement: On at least 2 of 3 evaluated datasets, at least one ensemble method (bagging, boosting, or stacking) improves RMSE by >= 10% relative to the single DecisionTreeRegressor baseline., statement: StackingRegressor attains the best (lowest) mean RMSE on at least 1 of 3 datasets while not being worst on more than 1 dataset, averaged over at least 5 seeds.

Experiment design:
research_question: How does RMSE performance of bagging, boosting, and stacking vary with increasing noise in non-linear regression tasks under CPU-only constraints?
conditions: name: decision_tree_baseline
description: Single DecisionTreeRegressor (max_depth tuned over a small grid) as non-ensemble baseline., name: bagging_random_forest
description: RandomForestRegressor representing bagging-style averaging across trees., name: boosting_gbrt
description: GradientBoostingRegressor representing sequential boosting on residuals., name: stacking_heterogeneous
description: StackingRegressor with base learners {kNN regressor, ridge regression, shallow random forest} and a linear meta-regressor.
baselines: decision_tree_baseline is the single-model baseline, bagging_random_forest serves as a strong classical ensemble reference
metrics: name: rmse
direction: minimize
description: Root Mean Squared Error on held-out test split, averaged over seeds., name: mae
direction: minimize
description: Mean Absolute Error on held-out test split, averaged over seeds., name: r2
direction: maximize
description: Coefficient of determination on held-out test split, averaged over seeds.
datasets: name: friedman1_low_noise
source: sklearn.datasets.make_friedman1 (n_samples=1200, n_features=10, noise=1.0), name: friedman1_high_noise
source: sklearn.datasets.make_friedman1 (n_samples=1200, n_features=10, noise=30.0), name: diabetes
source: sklearn.datasets.load_diabetes
compute_requirements: gpu_required: no
estimated_wall_clock_sec: 420

Requirements:

Scoring rubric:
requirements: A credible experiment studying bagging vs boosting vs stacking for noisy non-linear regression: the main ensemble families are implemented, execution covers multiple regression datasets with reasonable seed coverage, and the analysis addresses H1/H2/H3 using RMSE-centered evidence.
judging_note: Score on scientific substance and directional correctness of evidence, not on exact threshold satisfaction. Condition-name variants (e.g., 'RandomForest' vs 'bagging_random_forest') should not be penalized when the underlying method is clearly the intended one.
sub_tasks: requirements: The regression ensemble conditions and datasets are implemented in a way that supports a fair comparison.
sub_tasks: requirements: The submission implements the main ensemble families relevant to the hypotheses — a bagging-style model (e.g., RandomForestRegressor), a boosting-style model (e.g., GradientBoostingRegressor), and a stacking-style model (e.g., StackingRegressor) — alongside a non-ensemble baseline such as a single DecisionTreeRegressor. Equivalent well-motivated choices are acceptable.
task_category: Code Development
finegrained_task_category: Method Implementation, requirements: The submission uses multiple regression datasets that allow a noise-robustness comparison — for example, friedman1 at varying noise levels and/or diabetes from sklearn — with a reasonable train/test split.
task_category: Code Development
finegrained_task_category: Dataset and Model Acquisition, requirements: The pipeline uses a consistent preprocessing and evaluation setup across conditions, with controlled random-state handling and no obvious test-set leakage into training or stacking meta-features.
task_category: Code Development
finegrained_task_category: Experimental Setup, requirements: Execution logs the core metrics needed to evaluate the hypotheses.
sub_tasks: requirements: Execution produces a machine-readable metrics artifact containing numeric RMSE for the implemented (dataset, condition) cells, plus at least one complementary error metric (MAE or R²). Complete coverage is preferred; partial coverage should be scored proportional to how much of the hypothesis-relevant grid is populated.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: Reported dataset-condition results are aggregated over multiple random seeds with some form of dispersion reporting (std, stderr, CI, or min/max). More seeds are better, but an honest small-seed run with reported variance is preferable to a single deterministic run.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: Results analysis addresses all three hypotheses with quantitative evidence.
sub_tasks: requirements: The submission compares bagging (e.g., Random Forest) vs boosting (e.g., GBRT) on RMSE in a high-noise regression regime and conveys whether bagging is clearly better, clearly worse, or comparable — judge directionally against H1.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The submission conveys whether ensembles deliver a meaningful RMSE improvement over the single-tree baseline on most of the evaluated datasets. Exact percentage thresholds are not required; a clear comparison grounded in the reported numbers suffices.
task_category: Result Analysis
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: The submission ranks ensemble and baseline conditions by RMSE across the evaluated datasets and discusses whether stacking is competitive (best or near-best on some datasets, not systematically worst) — judge directionally against H3.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The README or writeup describes the methods and setup, presents the key RMSE/MAE/R² findings, conveys per-hypothesis outcomes (supported / refuted / inconclusive), and notes limitations. No strict word-count requirement.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation