Title: Designing Time Series Experiments in A/B Testing with Transformer Reinforcement Learning
Primary area: causal reasoning
Keywords: Experiment Designs, A/B Testing, Reinforcement Learning

Abstract: A/B testing has become a gold standard for modern technological companies to
conduct policy evaluation. Yet, its application to time series experiments, where
treatments are sequentially assigned over time, remains challenging. Existing designs suffer from two limitations: (i) they do not fully leverage the entire history for treatment allocation; (ii) they rely on strong assumptions to approximate the objective function (e.g., the mean squared error of the estimated treatment effect) for optimizing the design. We first establish an impossibility theorem showing
that failure to condition on the full history leads to suboptimal designs, due to the dynamic dependencies in time series experiments. To address both limitations simultaneously, we next propose a transformer reinforcement learning (RL) approach which leverages transformers to condition treatment allocation on the entire history and employs RL to directly optimize the MSE without relying on
restrictive assumptions. Empirical evaluations on synthetic data, a publicly available dispatch simulator, and a real-world ridesharing dataset demonstrate that our proposal consistently outperforms existing designs.

TL;DR: Transformer reinforcement learning is all you need for designing time series experiments.