NeurIPS 2025

Forecasting in Ofﬂine Reinforcement Learning for
Non-stationary Environments
Suzan Ece Ada1,2
Georg Martius2
Emre Ugur1
Erhan Oztop3,4
1Bogazici University, Türkiye
2University of Tübingen, Germany
3Ozyegin University, Türkiye
4Osaka University, Japan
ece.ada@bogazici.edu.tr
Abstract
Ofﬂine Reinforcement Learning (RL) provides a promising avenue for training poli-
cies from pre-collected datasets when gathering additional interaction data is infea-
sible. However, existing ofﬂine RL methods often assume stationarity or only con-
sider synthetic perturbations at test time, assumptions that often fail in real-world
scenarios characterized by abrupt, time-varying offsets. These offsets can lead to
partial observability, causing agents to misperceive their true state and degrade per-
formance. To overcome this challenge, we introduce Forecasting in Non-stationary
Ofﬂine RL (FORL), a framework that uniﬁes (i) conditional diffusion-based candi-
date state generation, trained without presupposing any speciﬁc pattern of future
non-stationarity, and (ii) zero-shot time-series foundation models. FORL targets
environments prone to unexpected, potentially non-Markovian offsets, requiring
robust agent performance from the onset of each episode. Empirical evaluations on
ofﬂine RL benchmarks, augmented with real-world time-series data to simulate
realistic non-stationarity, demonstrate that FORL consistently improves perfor-
mance compared to competitive baselines. By integrating zero-shot forecasting
with the agent’s experience, we aim to bridge the gap between ofﬂine RL and the
complexities of real-world, non-stationary environments.
1
Introduction
Episodes
Offset time series
Figure 1: Setting. The agent does not know
its location in the environment because its
perception is offset every episode j by an un-
known offset bj (only vertical offsets are illus-
trated). FORL leverages historical offset data
and ofﬂine RL data (from a stationary phase)
to forecast and correct for new ...