Reservoir computing vs MLP vs GP for short-horizon Lorenz-63 forecasting
Domain: ml (dynamical-systems, machine-learning, reservoir-computing)
Topic: Testing whether a random-feature echo state network (reservoir computing) predicts Lorenz-63 short-horizon trajectories more accurately than a matched-parameter MLP and a Gaussian Process at small training-data sizes (N=200-2000 points)
Venue: ARC-Bench 2026
Headline metric: valid_prediction_time (maximize)

Research briefing:
Reservoir computing (a.k.a. echo state networks, ESN) is a lightweight
approach for learning dynamical systems: a fixed random recurrent network
projects the input into a high-dimensional reservoir state, and only a
linear readout layer is trained. For chaotic systems, ESNs have repeatedly
been reported to match or outperform trainable recurrent nets on
short-horizon trajectory prediction, despite training only a ridge
regression on the readout.

A credible CPU-scale study of this claim must compare (i) a reservoir
network with a random sparse internal matrix and fixed spectral radius,
(ii) a parameter-matched fully-connected MLP that reads a fixed window of
past states, and (iii) a Gaussian Process regressor with an RBF kernel on
the same windowed input. The target signal is Lorenz-63 integrated at
dt=0.02 with the classical (σ=10, ρ=28, β=8/3) parameters. Training-set
sizes N ∈ {200, 500, 1000, 2000} are small enough that the standard
reservoir-computing folklore — "ESNs win when data is scarce" — can
actually be tested rather than assumed.

The key dynamics-aware metric is the *valid prediction time* (VPT):
the first time step at which the normalized prediction error exceeds a
threshold (commonly 0.4 of the attractor standard deviation). Unlike
per-step RMSE, VPT tracks how long a model's forecast remains useful on
the chaotic attractor. Reporting both RMSE and VPT reveals whether any
observed "superiority" is numerical or dynamical.

The research question is: *does a small reservoir (N_res ≤ 500 units) beat
a matched-parameter MLP and a Gaussian Process on valid prediction time
for Lorenz-63, and at which training-set size does the gap appear?*

Hypotheses:
statement: An echo-state network with spectral radius ≈ 0.9 and ≤ 500 reservoir units achieves a longer mean Valid Prediction Time (VPT ≥ 1.2×) than a parameter-matched MLP on Lorenz-63 at training-set sizes N ≤ 500., statement: The Gaussian Process regressor is competitive with (VPT within ±20% of) the ESN at N=200 but is decisively beaten (ESN VPT at least 2× the GP's) at N=2000, reflecting the GP's O(N^3) training bottleneck and the ESN's ability to exploit more data without re-inverting a Gram matrix., statement: Per-step RMSE at one-step-ahead prediction is NOT a reliable proxy for VPT: at least one condition exists where model A has lower one-step RMSE than model B but a shorter VPT, showing that per-step error understates the divergence on chaotic attractors.

Experiment design:
research_question: On Lorenz-63 short-horizon forecasting, does a small echo-state network achieve a longer valid prediction time than a parameter-matched MLP and a Gaussian Process regressor, and how does the ranking depend on training-set size?
conditions: name: esn_N200
description: Echo-state network with 300 reservoir units, spectral radius 0.9, input scaling 1.0, leak rate 0.3, ridge regression readout (alpha=1e-5). Trained on N=200 consecutive Lorenz-63 samples., name: esn_N500
description: Same ESN configuration, N=500 training samples., name: esn_N1000
description: Same ESN configuration, N=1000 training samples., name: esn_N2000
description: Same ESN configuration, N=2000 training samples., name: mlp_N{200,500,1000,2000}
description: Fully-connected MLP with 2 hidden layers whose total parameter count is within ±10% of the ESN readout parameter count. Reads a 5-step window of past (x,y,z) states as input. Adam optimizer, early stopping on a 10% validation split., name: gp_N{200,500,1000,2000}
description: Gaussian Process regressor with an RBF kernel and WhiteNoise term, hyperparameters optimised by marginal likelihood maximisation. Reads the same 5-step window as the MLP. At N=2000 the GP may time out or run out of memory; record the failure.
baselines: Persistence baseline: predicted next state = previous state. Establishes a floor for VPT., Linear autoregressive baseline: one-step-ahead linear regression on the 5-step window. Establishes whether the non-linear methods' gains are real.
metrics: name: valid_prediction_time
direction: maximize
description: First forecast step (in units of integration dt) at which the normalized L2 error exceeds 0.4 × attractor std. Averaged over 10 random initial conditions on an independent test trajectory., name: one_step_rmse
direction: minimize
description: Root-mean-square error of one-step-ahead prediction on the test trajectory., name: train_wall_clock_sec
direction: minimize
description: Wall-clock time to fit the model, including hyperparameter optimisation where applicable., name: param_count
direction: report
description: Number of trainable parameters. Must be reported to demonstrate that the MLP is parameter-matched to the ESN.
datasets: name: lorenz63_train
source: synthetic — integrate dxdt=σ(y-x), dy/dt=x(ρ-z)-y, dz/dt=xy-βz at dt=0.02 with σ=10, ρ=28, β=8/3. Use scipy.integrate.solve_ivp or a hand-rolled RK4., name: lorenz63_test
source: independent Lorenz-63 trajectory (different IC, 10× longer warmup) with 10 evaluation windows for VPT averaging.
compute_requirements: gpu_required: no
estimated_wall_clock_sec: 720

Requirements:

Scoring rubric:
requirements: A credible CPU-scale study of reservoir computing (echo-state network), a matched-parameter MLP, and a Gaussian Process on Lorenz-63 short-horizon forecasting: Lorenz-63 is integrated, the three model families are implemented, multiple training-set sizes are swept, valid-prediction-time is reported, and results address H1/H2/H3 directionally.
judging_note: Score on scientific substance and directional correctness of evidence, not on exact numeric thresholds. Reasonable substitutes (e.g., alternative chaotic system, library-based ESN when clearly documented, Matern-kernel GP) that test the same scientific question should be credited. A GP running out of memory at large N is a valid reported outcome.
sub_tasks: requirements: The three model families and the dynamical system are implemented.
sub_tasks: requirements: The submission integrates the Lorenz-63 system with classical parameters at a small time step using scipy.integrate or a hand-rolled Runge-Kutta integrator, and uses separate train/test trajectories.
task_category: Code Development
finegrained_task_category: Dataset and Model Acquisition, requirements: An echo-state network is implemented with a sparse random reservoir scaled to a target spectral radius below ~1.1, driven by the input sequence with a leak-rate update and a linear readout fit by ridge regression. A from-scratch implementation is preferred; a clearly documented library-based ESN that exposes the same mechanics is acceptable.
task_category: Code Development
finegrained_task_category: Method Implementation, requirements: An MLP is implemented that reads a fixed window of past states and predicts the next state, with its total parameter count reported and roughly matched to the ESN readout parameter count.
task_category: Code Development
finegrained_task_category: Method Implementation, requirements: A Gaussian Process regressor with an RBF or Matern kernel is fit on the same windowed input, with marginal-likelihood hyperparameter optimisation enabled.
task_category: Code Development
finegrained_task_category: Method Implementation, requirements: Execution produces per-method, per-N quantitative results.
sub_tasks: requirements: The experiment sweeps multiple training-set sizes for each of the three methods and reports numeric results per cell.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: A valid-prediction-time (VPT) metric is computed per method per N — the first step at which the normalised prediction error exceeds a threshold — averaged over multiple independent initial conditions or seeds.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: A one-step-ahead RMSE (or equivalent pointwise error) is computed per method per N on the test trajectory and reported in the metrics file.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: The writeup addresses H1/H2/H3 directionally and contextualises numeric findings.
sub_tasks: requirements: The submission reports VPT for the ESN vs MLP at small training sizes and conveys whether the ESN meaningfully outperforms the MLP in the small-N regime — judge directionally against H1.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The submission compares ESN and GP VPT at both small-N and larger-N and conveys whether the GP is competitive at small N and falls behind (or becomes infeasible) at large N (H2). Reporting a GP memory/time failure at large N is a valid outcome.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The submission examines whether one-step RMSE ranking tracks VPT ranking across method-N cells and conveys whether at least one rank inversion is observed (H3).
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The README or writeup describes the Lorenz-63 setup, the three methods, the N-sweep, the VPT and RMSE numbers, conveys per-hypothesis outcomes (supported / refuted / inconclusive), and notes limitations (single dynamical system, no Lyapunov-time normalisation, GP memory cliff). No strict word-count requirement.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation