Bandit algorithm robustness under stationary and drifting reward regimes
Domain: ml (reinforcement-learning, bandits)
Topic: Comparing multi-armed bandit algorithms (epsilon-greedy, UCB1, Thompson sampling, Exp3) on synthetic Bernoulli and Gaussian arms under stationary and drift regimes
Venue: ARC-Bench 2026
Headline metric: cumulative_regret (minimize)

Research briefing:
Multi-armed bandit algorithms encode different assumptions about reward
generation and non-stationarity. Epsilon-greedy is simple and adaptable with
persistent exploration; UCB1 is optimism-driven and often strong in
stationary stochastic settings; Thompson sampling can be highly sample
efficient when model assumptions are matched; Exp3 is designed for
adversarial settings and may trade off stochastic efficiency for robustness.
In small CPU-constrained studies, these differences are often discussed
qualitatively but not tested with matched synthetic environments.

A meaningful benchmark should evaluate both Bernoulli and Gaussian reward
arms, because posterior/model assumptions and noise scale can materially
change outcomes. It should also include stationary and drifting regimes:
methods tuned for fixed means can degrade when the best arm changes over
time. Regret, not just final reward, is the right primary metric because it
captures online learning efficiency throughout the horizon.

A credible experiment here implements the four algorithms with consistent
action/reward interfaces, runs repeated simulations over multiple seeds, and
reports cumulative regret trajectories and endpoint statistics. To keep the
study CPU-friendly, horizons and arm counts should be modest (e.g., 5-10 arms,
500-2000 steps), with vectorized numpy updates where possible.

The key analytical value is comparative: identify where UCB1/Thompson excel
in stationary stochastic settings, and whether epsilon-greedy or Exp3 is more
resilient under drift. The objective is not reproducing a single paper, but
testing algorithm-environment fit under controlled synthetic shifts.

*How do epsilon-greedy, UCB1, Thompson sampling, and Exp3 trade off cumulative regret across Bernoulli vs Gaussian arms and stationary vs drifting reward regimes?*

Hypotheses:
statement: In stationary Bernoulli environments, either UCB1 or Thompson sampling achieves at least 10% lower mean cumulative regret than epsilon-greedy at horizon T on at least 2 of 3 datasets (averaged over >=20 seeds)., statement: In drifting environments, epsilon-greedy or Exp3 achieves lower mean cumulative regret than UCB1 on at least 2 of 3 datasets at horizon T (averaged over >=20 seeds)., statement: Across all evaluated datasets, no single algorithm is best (lowest mean cumulative regret) on every dataset, indicating environment-dependent performance.

Experiment design:
research_question: How do epsilon-greedy, UCB1, Thompson sampling, and Exp3 compare in cumulative regret across stationary versus drifting synthetic bandit tasks with Bernoulli and Gaussian rewards?
conditions: name: epsilon_greedy_eps0.1
description: Epsilon-greedy with constant epsilon=0.1 and sample-mean value estimates., name: ucb1
description: UCB1 with exploration bonus sqrt(2 log t / n_a)., name: thompson_sampling
description: Thompson sampling using Beta-Bernoulli updates for Bernoulli arms and Gaussian posterior sampling (known variance assumption) for Gaussian arms., name: exp3_gamma0.07
description: Exp3 with gamma=0.07 and importance-weighted reward estimates.
baselines: epsilon_greedy_eps0.1 as simple stochastic baseline, ucb1 as canonical optimism baseline
metrics: name: cumulative_regret
direction: minimize
description: Mean cumulative regret at final horizon T relative to oracle best arm per round, averaged over seeds., name: instantaneous_regret_auc
direction: minimize
description: Area under per-round regret curve over time (lower is better)., name: best_arm_selection_rate
direction: maximize
description: Fraction of rounds selecting the current optimal arm (for drift: time-varying optimum).
datasets: name: bernoulli_stationary_k10
source: synthetic: 10 Bernoulli arms with fixed means sampled in [0.05, 0.95] and sorted gap >=0.05, name: bernoulli_drift_k10
source: synthetic: 10 Bernoulli arms with piecewise-constant means; best arm switches at predefined changepoints, name: gaussian_drift_k8
source: synthetic: 8 Gaussian arms (sigma=1) with linearly drifting means and periodic rank reversals
compute_requirements: gpu_required: no
estimated_wall_clock_sec: 420

Requirements:

Scoring rubric:
requirements: A credible experiment comparing bandit algorithms (epsilon-greedy, UCB1, Thompson sampling, Exp3, or equivalents) under stationary and drifting synthetic regimes: algorithms are implemented with a common interface, experiments cover multiple environments with repeated seeds, and results address H1/H2/H3 directionally using cumulative regret.
judging_note: Score on scientific substance and directional correctness of evidence, not on exact numeric thresholds. Well-motivated algorithm variants (e.g., UCB-V, linear Thompson) should be credited when they test the same scientific question.
sub_tasks: requirements: The bandit algorithms and synthetic environments are implemented correctly.
sub_tasks: requirements: The submission implements multiple distinct algorithm code paths — typically including epsilon-greedy, UCB1, Thompson sampling, and/or Exp3 — with per-round action selection and update logic that are not identical wrappers.
task_category: Code Development
finegrained_task_category: Method Implementation, requirements: The submission defines multiple synthetic bandit environments including at least one stationary and one drifting regime, with reproducible seed control and oracle best-arm rewards per round for regret computation.
task_category: Code Development
finegrained_task_category: Experimental Setup, requirements: If Thompson sampling and/or Exp3 are included, implementation uses sampling-based posterior decisions (Thompson) and probability-weighted action selection with importance-weighted updates (Exp3), rather than greedy mean selection.
task_category: Code Development
finegrained_task_category: Method Implementation, requirements: Execution produces regret metrics across algorithms and datasets.
sub_tasks: requirements: Execution runs multiple seeds per (algorithm, dataset) cell for multiple environments and logs final cumulative-regret values with mean and dispersion. Honest small-seed runs with variance reported are preferable to a single run.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: A machine-readable results artifact is produced containing dataset-wise metrics for each implemented algorithm, including cumulative regret.
task_category: Code Execution
finegrained_task_category: Logging, Analysis & Presentation, requirements: Findings address H1/H2/H3 directionally and summarize implications.
sub_tasks: requirements: The submission compares stationary-regime cumulative regret between epsilon-greedy and {UCB1, Thompson} and conveys whether the principled algorithms are meaningfully better — judge directionally against H1.
task_category: Result Analysis
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: The submission evaluates drifting-regime cumulative regret and conveys whether more exploratory algorithms (epsilon-greedy or Exp3) outperform UCB1 on most drifting datasets (H2).
task_category: Result Analysis
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: The submission conveys whether any single algorithm dominates across all environments or whether winners are mixed (H3), with supporting tables or summaries.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The README or writeup describes setup, reports key cumulative-regret results per environment, conveys per-hypothesis outcomes (supported / refuted / inconclusive), and notes limitations (horizon length, hyperparameter sensitivity, synthetic-only scope). No strict word-count requirement.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation