Classical forecaster robustness on synthetic seasonal time series
Domain: ml (time-series, forecasting)
Topic: Benchmarking classical time-series forecasters (AR, ARIMA, SARIMAX, ETS, Theta) on synthetic seasonal series with varying noise, trend, and seasonality amplitudes
Venue: ARC-Bench 2026
Headline metric: smape (minimize)

Research briefing:
Classical univariate forecasting methods remain widely used because they are
interpretable and lightweight, yet their comparative behavior can flip under
different data-generating regimes. AR models often perform well when dynamics
are mostly autoregressive and weakly seasonal; ARIMA can absorb trend and
differencing structure; SARIMAX can explicitly represent seasonal lag effects;
ETS captures error-trend-seasonal decomposition; and Theta is often a strong
low-variance baseline. In small CPU-constrained settings, practitioners need
practical guidance about which method is robust to changing signal-to-noise
and seasonality strength.

A synthetic benchmark is appropriate here because we can precisely control
trend slope, seasonal amplitude, and observation noise while keeping compute
modest. By generating multiple families of monthly-like series with known
periodicity (e.g., period 12), we can evaluate whether model rankings are
stable or regime-dependent. This avoids overfitting conclusions to one
real-world dataset and enables direct stress testing under high-noise versus
high-seasonality conditions.

A credible study should compare at least four of {AR, ARIMA, SARIMAX, ETS,
Theta} on 2–3 synthetic dataset families, using rolling-origin or holdout
forecasts with a fixed horizon. It should report scale-robust error metrics
(especially sMAPE), include at least one baseline (e.g., seasonal naive), and
aggregate over multiple random seeds. The analysis should explicitly test
whether one method is consistently best overall or whether the best choice
depends on data regime.

The key outcome is actionable selection logic: if seasonal structure is strong
and noise is low, do seasonal/state-space methods dominate; and if noise is
high, do simpler methods become competitive? These are concrete decisions an
autonomous forecasting pipeline can apply when only limited data and CPU are
available.

*How do AR, ARIMA, SARIMAX, ETS, and Theta compare in sMAPE robustness across synthetic seasonal series with controlled trend, seasonality amplitude, and noise levels?*

Hypotheses:
statement: SARIMAX or ETS achieves the lowest mean sMAPE on at least 2 of 3 synthetic dataset families when seasonality amplitude is high (amplitude >= 8) and noise is low (sigma <= 1.0), averaged over >=5 seeds., statement: Under high-noise settings (sigma >= 3.0), the sMAPE gap between the best advanced method (ARIMA/SARIMAX/ETS/Theta) and a seasonal_naive baseline is <= 5 percentage points on at least 2 of 3 dataset families., statement: No single method among {AR, ARIMA, SARIMAX, ETS, Theta} ranks first in mean sMAPE on all evaluated dataset families, indicating regime-dependent winners.

Experiment design:
research_question: Which classical forecasting method is most robust in sMAPE across controlled synthetic regimes of trend, seasonality amplitude, and noise?
conditions: name: ar_lag12
description: Autoregressive model with lag order selected from {3,6,12} by AIC on training split., name: arima_auto_small
description: Non-seasonal ARIMA with (p,d,q) searched over small grid p,q in [0,2], d in [0,1], selected by AIC., name: sarimax_seasonal12
description: Seasonal ARIMA/SARIMAX with seasonal period 12 and small grid over (p,d,q)x(P,D,Q), selected by AIC., name: ets_additive
description: Exponential smoothing with additive trend/seasonality (period 12), damped trend optional by AIC., name: theta
description: ThetaModel forecast with default theta decomposition for univariate series.
baselines: seasonal_naive (forecast y_t = y_{t-12}) as a simple seasonal baseline, naive_last (random walk) as a non-seasonal baseline
metrics: name: smape
direction: minimize
description: Symmetric Mean Absolute Percentage Error on forecast horizon, averaged over seeds and series instances., name: mae
direction: minimize
description: Mean Absolute Error on forecast horizon., name: fit_time_sec
direction: minimize
description: Per-method wall-clock fit+forecast runtime in seconds.
datasets: name: syn_low_noise_high_season
source: Synthetic monthly series: length 180, period 12, trend slope in [0.0,0.2], season amplitude in [8,12], gaussian noise sigma in [0.5,1.0]., name: syn_medium_noise_medium_season
source: Synthetic monthly series: length 180, period 12, trend slope in [0.1,0.4], season amplitude in [4,8], gaussian noise sigma in [1.5,2.5]., name: syn_high_noise_low_season
source: Synthetic monthly series: length 180, period 12, trend slope in [0.0,0.3], season amplitude in [1,4], gaussian noise sigma in [3.0,4.0].
compute_requirements: gpu_required: no
estimated_wall_clock_sec: 720

Requirements:

Scoring rubric:
requirements: A credible experiment benchmarking classical forecasters (AR, ARIMA, SARIMAX, ETS, Theta, or equivalents) on synthetic seasonal series: model conditions are implemented, execution covers multiple synthetic regimes with repeated seeds, and results address H1/H2/H3 directionally using sMAPE-centered analysis.
judging_note: Score on scientific substance and directional correctness of evidence, not on exact numeric thresholds. Alternative forecaster implementations or substitutes that test the same scientific question should be credited.
sub_tasks: requirements: The forecasting conditions and synthetic data generation pipeline are implemented correctly.
sub_tasks: requirements: The submission implements multiple distinct forecasting methods — typically AR/ARIMA, a seasonal model (SARIMAX or ETS), and Theta or equivalents — as separate code paths, and includes at least one explicit naive/seasonal-naive baseline.
task_category: Code Development
finegrained_task_category: Method Implementation, requirements: The submission generates multiple synthetic seasonal dataset families with controllable seasonality, trend, and noise, each with a train/test split suitable for forecasting.
task_category: Code Development
finegrained_task_category: Dataset and Model Acquisition, requirements: Forecasting setup uses a fixed holdout horizon (or rolling-origin equivalent) and computes comparable forecasts for every (method, dataset, seed) cell.
task_category: Code Development
finegrained_task_category: Experimental Setup, requirements: Benchmark execution logs required metrics with repeated trials.
sub_tasks: requirements: Execution produces a structured metrics artifact containing numeric sMAPE and MAE (or equivalents) for each implemented method on at least one dataset family.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: Each reported method-dataset metric is aggregated over multiple random seeds with dispersion statistics. Honest small-seed runs with variance reported are preferable to a single run.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: The run logs per-method wall-clock timing and completes within a CPU-only budget.
task_category: Code Execution
finegrained_task_category: Logging, Analysis & Presentation, requirements: Results analysis addresses H1/H2/H3 directionally with quantitative comparisons.
sub_tasks: requirements: The submission isolates high-seasonality low-noise results and conveys whether SARIMAX or ETS tends to attain the best mean sMAPE — judge directionally against H1.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The submission compares the best advanced method against seasonal-naive under high-noise settings and conveys whether the sMAPE gap collapses to a small margin (H2).
task_category: Result Analysis
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: The submission ranks methods by mean sMAPE per dataset family and conveys whether no single method dominates across all families (H3).
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The README or writeup reports key metric tables, conveys per-hypothesis outcomes (supported / refuted / inconclusive), and discusses limitations (synthetic realism, grid-search scope, seed count, horizon sensitivity). No strict word-count requirement.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation