Forecasting in Offline Reinforcement Learning for Non-stationary Environments
Authors: Suzan Ece Ada
Georg Martius
Emre Ugur
Erhan Oztop
Venue: NeurIPS 2025 spotlight ()
Keywords: Reinforcement learning
Robust Reinforcement learning
Offline Reinforcement Learning
Diffusion Models
Primary area: reinforcement_learning
TL;DR: We introduce Forecasting in Non-stationary Offline RL (FORL), a novel framework designed to be robust to passive non-stationarities, leveraging diffusion probabilistic models and time-series forecasting foundation models.

Abstract: Offline Reinforcement Learning (RL) provides a promising avenue for training policies from pre-collected datasets when gathering additional interaction data is infeasible. However, existing offline RL methods often assume stationarity or only consider synthetic perturbations at test time, assumptions that often fail in real-world scenarios characterized by abrupt, time-varying offsets. These offsets can lead to partial observability, causing agents to misperceive their true state and degrade performance. To overcome this challenge, we introduce Forecasting in Non-stationary Offline RL (FORL), a framework that unifies (i) conditional diffusion-based candidate state generation, trained without presupposing any specific pattern of future non-stationarity, and (ii) zero-shot time-series foundation models. FORL targets environments prone to unexpected, potentially non-Markovian offsets, requiring robust agent performance from the onset of each episode. Empirical evaluations on offline RL benchmarks, augmented with real-world time-series data to simulate realistic non-stationarity, demonstrate that FORL consistently improves performance compared to competitive baselines. By integrating zero-shot forecasting with the agent's experience, we aim to bridge the gap between offline RL and the complexities of real-world, non-stationary environments.

Forecasting in Offline Reinforcement Learning for Non-stationary Environments
Suzan Ece Adal.2 Georg Martius Emre Ugur' Erhan Oztop 3.4
Bogazici University, Türkiye
University of Tübingen, Germany
Ozyegin University, Türkiye ece. ada@bogazici eew.iiiiieedd........
*Osaka University, Japan
Abstract
Offline Reinforcement Learning (RL) provides a promising avenue for training poli- cies from pre-collected datasets when gathering additional interaction data is infea- sible. However, existing offline RL methods often assume stationarity or only con- sider synthetic perturbations test time, assumptions that often fail in real-world scenarios characterized by abrupt, time-varying offsets. These offsets can lead to partial observability, causing agents to misperceive their true state and degrade per- formance. To overcome this challenge, we introduce Forecasting in Non- Non-stationary Offine RL (FORL), a framework that unifies (i) conditional diffusion-baser candi- date state generation, trained without presupposing any specific pattern of future non-stationarity, and (ii) zero-shot time-series foundation models. FORL targets environments prone to unexpected, potentially non non-Markovian offsets, requiring robust agent performance from the onset of each episode. Empirical evaluations on offline RL benchmarks, augmented with real-world time-series data to simulate realistic non-stationarity, demonstrate that FORL consistently improves perfor- mance compared to competitive baselines. By integrating pero-shot forecasting with the agent's 's experience, we aim bridge the gap between offline RL and the complexities of real-world, non -on-ssaioonary environments.
Introduction
Offline Reinforcement Learning (RL) leverages static datasets to avoid costly or risky online interactions [12] Yet, agents trained on fully observable states often fail when deployed with noisy or corrupted ob- servations. While robust offline RL. methods address
Offiret time series
test-time perturbations, such as sensor noise or adver- sarial attacks J critical gap persists in addressing non-stationarity within the observation function-a challenge that fundamentally alters the agent's 's per- ception of the environment over time.
Episodes
Figure Setting. The agent does not know its location in the environment because its perception is offset every episode j by an un- known offset & (only vertical offsets are illus- trated). FORL leverages historical offset data and offline RL data (from stationary phase) to forecast and correct for new offsets at test time. Ground-truth offsets (1.1) hidden throughout the evaluation episodes.
Prior online algorithms that consider the scope of non-stationarity as the observation function focus on learning agent morphologies [4] and generaliza- tion in Block MDPs (5). While this scope of non- stationarity holds significant potential for real-world applications (6) it remains underexplored. We focus on the episodic evolution of the observation function at test-time in offline RL. In our setup, each dimen-
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

sion of an agent's s state is influenced by unknown, time- deneedeeendent constant additive offset that remains fixed within single operational interval (an "episode"). This leads to a stream of evolving observation functions ? extending across multiple future episodes. where the offsets remain hidden throughout the prediction window. For instance, industrial robots might apply a daily calibration offset to each joint, while sensors can exhibit deviation until the next scheduled recalibra- tion. Similarly, in healthcare or finance, data may be partially aggregated or withheld to comply with regulations, effectively obscuring finer-grained variations and leaving single offset the dominant factor per episode. By only storing these representative offsets, we circumvent the challenges of continuous interaction buffers in bandwidth-consstriied
privacy-sensitie environments. Because the offset can differ across state dimensions (e.g.. different sensor or actuation channels), each state dimension can be affected by different unknown bias that stays constant for that episode but evolves differently across episodes. Approaches that assume predefined perturbations can struggle with these abrupt, episodic shifts because such offsets violate the typical assumption of smoothly varying obser- vation functiion. Frequent retraining, hyperparameter optimization, or extensive online adaptation to new observation function evolution patterns costly, risky (due trial-and-error in safety-critical settings), and may be infeasible if these patterns no longer reflect assumptions made during training. By separating offset data (episodic calibration values) from the massive replay buffers, zero-shot forecasting-base baadd approach can anticipate each new offset from the beginning of the episode without requiring policy updates or making assumptions on task evolution test time S Modeling these multidimensional additive offsets as stable, per-episode constants presents a robust and efficient way handle time-varying conditions in non-stationary environments where the evolution of tasks follows Mar-Maakovinn, time-series pattern 9 mitigating the risks of online exploration.
We consider an offline RL setting during train- ing where we have only access to a standard offline RL dataset collected from stationary environment (3 with fully observable states, Initial data may be collected under near-ideal conditions and then gradually affected by wear, tear, or other natural shifts-ever even as the underly- ing physical laws (dynamics) remain unchanged. At test time, however, we evaluate in non- stationary environment where both the observa- tion function and the observation space change due to time-dependent external factors. This setup can be interpreted as environments shift- ing along observation space dimensions while the initial state of the agent sampled from uniform distribution over the state space. A sim- plified version of this setup for an offset affect- ing only one dimension of the state is illustrated in Figure I Here, the agent "knows" is in maze but does not know where it is in the maze. Furthermore, it will remain uncertain of its location across all episodes at test-time, in every episode, a new offset leads to systematic misalignment between perceived and actual positions. Importantly, these offsets may conform to Gaussian or Markovian assumptions; instead, they may stem directly from complex, real-world time-series data M and remain constant throughout each episode. As result, standard noise-driven or parametric state-estimation techniques, which typically rely on smoothly varying or randomly perturbed functions, cannot reliably identify these persistent, episode-wide offsets that are not avail- able after the episode terminates. While zero-shot forecasting can adjust observation offsets, its performance depends on the forecaster's accuracy. Similarly, integrating zero-shot forecasting into model-based offline RL approach [3] can underperform when real-world offsets deviate from predefined assumptions about future observation functions. Our approach uses the insight that the belief of the true states can be refined from a sequence of actions and effects. For instance, in maze navigation, if an agent misjudges its location and hits a wall, analyzing its actions and delta observations leading to the collision can provide evidence for likely locations within the maze.
Policy (a(3)
$
Environment
Candidate Selection
{
Forecasting Foundation
{s
10) J
Trajectory Buffer
Diffusion Model
Tixa) [[Ahe- Gr-p) (a,, (x-)))
Figure 2: Overview of the proposed FORL frame- work at test-time. The observations are processed by both the trajectory buffer and the time-series forecasting foundation module [TC] Observa- tion changes and actions sampled from the policy (Ao, a), are stored in the trajectory buffer. The dif- fusion model generates candidate states {s!")), } conditioned on (t..ar). The candidate selection module then generates the estimated 5c.

---

We propose the Forecasting in Non-stationary Offline RL (FORL) framework (Figure for test- rest-time adaptation non non-stationary environments where the observation function is perturbed by an arbitrary time-series. Our framework has two main ingredients: forecasting offsets with zero-shot time-series forecasting model [T]] from past episode offsets (ground truth offsets after the episode terminates are not accessible at test-time) and within-episode update of the state estimation using a conditional diffusion model [ trained on offline stationary data.
Contributions. We unify the strengths of probabilistic forecasting and decision-making under uncertainty to enable continuous adaptation when the environment diverges from predictions. Conse- quently, our framework: (1) accommodates future offsets without assuming specific non non-stationarity patterns during training. eliminating the need for retraining and hyperparameter tuning when the agent encounters new. unseen non non-stationary patterns at test time, and (2) targets non-trivial non- stationarities af test time without requiring environment interaction or knowledge of POMDPs during training. (3) FORI. introduces a novel, modular framework combining conditional diffusion model (FORI--DM) for multimodal belief generation with a lightweight Dimension-wi
Closest Match (DCM) fusion strategy, validated by extensive experiments no-access to past offsets, policy- agnostic plug-and-play, offset magnitude-scaling, and inter/intra-episode drifts. (4) We propose a novel benchmark that integrates offsets from real-world time-series datasets with standard offline RL benchmarks and demonstrate that FORL consistently outperforms baseline methods.
Background: Diffusion Models Denoising diffusion models [12 13 aim to model the data dis- tribution with (o()))) := f Po (oo 1x from samples zo in the dataset. The joint distribution follows the Markov Chain (Zo:N) == Nxx: 0, II
sample for diffusion timestep n and pe (In-1 xx)) := n= Pe (In-1 [n)) where is the noisy
(In-1: My (I,, Eg (In, n)). During
training, we use the samples from the distribution q (z,, æn) = N (xn; vā, To. 1 ā,,) I) where
On II= og [T.. General information on diffusion models is given in Section
B.3
2 Method
In this section, we formulate our problem statement and describe our FORL diffusion model trained on the offline RL dataset to predict plausible states. Then, we introduce our online state estimation method, Dimension- sireesioo-wiseeeee wise Closest Match that uses plausible states predicted by the multimodal FORL diffusion model (DM) and the states predicted from past episodes prior to evaluation by a probabilistic unimodal zero-shot time-series foundation model.
2.1 Problem Statement
Training (Offline Stationary MDP) We begin with an episodic, stationary Markov Decision Pro- cess (MDP) Mirain = (S, A, R. po), where the initial state distribution Po is uniform distribution over the state space S. We only have access an offline RL dataset D {(s at +f42,r{) with transitions collected from this MDP. Crucially, our FORL. diffusion model and a diffusion policy [14] are trained offline using this dataset, such as the standard D4RL benchmark [15], without making any assumptions about how the environment might become non-stationary at test time.
Test Environment (Sequence of POMDPs) At test time, the agent faces an infinite sequence of POMDPs (M,) Each POMDP is described by 7-tule [16] M, = (S, A, Oj, T, R. Po. x,). where the state space S, action space A, transition function T, and reward function R remain iden- tical to the training MDP. x is the observation function, where we restrict ourselves to deterministic versions (x:
C I7 Non-stationary environments can be formulated in different ways. In Khetarpal et al. 6 general non-stationary RL formulation is put forward, which allows each compo- nent of the underlying MDP or POMDP to evolve over time, i.c. (S(t), A(t), T(t), R(t), x(t). O(t)). A set specifies which of these components vary, and a driver determines how they evolve. In particular, passive drivers of non-stationarity imply that exogenous factors alone govern the evolu- tion of the environment, independent of the agent's 's actions. In this work, we consider the scope of stationtion [6] (Section B.2) to be the observation function and the observation space, i.e. O). We consider the case where non-stationarity unfolds over episodes and where the observation function x, is different in each episode j. The change in the observation function is assumed to have an
O)
3

---

additive structure and is independent of the agent's is actions (passive non non- -on-stationarity 6 the function x, offsets states s, by fixed offset & E R=:
Concretely,
Oj {s+bb
S}, xy(s) = s + b¹.
Importantly, the sequence (b²) can evolve under arbitrary real-world time-series data, and the agent does not have access to the ground-truth information throughout the evaluation-simile to scenarios where observations are only available periodically and shifts occur between these intervals. Thus, the episodes have temporal order, relating to Non-Stationary Decision Processes (NSDP), defining a sequence of POMDPs 7
(Section B.2
Partial Observability and Historical Context into the future, each M, is a POMDP. The agent receives only the offset-shifted observations (ot). where + &2 without any noise. Moreover, the agent may have access to limited historical context of previous offsets future offsets (b², & ..... (.) Hence, the agent must forecast and/or adapt to unknown future offsets without prior non non-stationary training.
Since b is never directly observed for P Pepisodes
(......"'))))) at discrete intervals P. but no direct information about
Partial Identifiability Despite observing = + b3 the agent cannot generally disentangle st from b²_ For any single observation, there are infinite possible pairs of state and offset that yield og = s' d'+ Additionally, the initial state distribution p is uniform and does not provide information about b. Thus, we can only form a belief over 8, and refine that belief based on two sources of information: a) the sequence of actions and effects observed within an episode and b) the sequence of past identified offsets. To exploit source a, we utilize a predictive model of commonly expected outcomes based on a diffusion model, which will be explained next. To make use of source b, model (see Section 2.3 for details). Afterwards, both pieces of information are fused (Section 2.4
zero-shot forecasting
observation
la (9) 1x
state
2.2 FORL Diffusion Model
In our setting, we assume that the offsets added the states are unobservable at test time, while the transition dy- namics of the evaluation environment remain unchanged.
S
To eventually reduce the uncertainty about the underlying state, we perform filtering or belief updates using the se- quence of past interactions. To understand why the history of interactions indicative of a particular ground truth state, consider the following example in maze environ- ment illustrated in Fig. 5 When the agent moves north for three steps and then bumps into wall, the possible ground truth states can only be those three steps south of any wall. The agent cannot observe the hidden green trajectory of ground-trr -ruth states; it only has access to the sequence of observation changes (Ao) and action vectors, which narrows down the possible positions to four candidate regions exaastyy those identified by our model. Clearly, the distribution of possible states is highly multimodal, such that we propose using a diffusion model as a flexible predictive model of plausible states given the observed actions and outcomes. Diffusion models excel at capturing multimodal distributions [14), making them well-suited for our task. We train the diffusion model on offline data without offsets (b, = 0) which we detail below.
Figure 3: Candidate states generated by FORL Diffusion Model.
To distinguish between the trajectory timesteps in reinforcement learning (RL) and the timesteps in the diffusion process, we use the subscript E (O..... T) to refer to RL timesteps and n E {O.....Ny N} for diffusion timesteps. We first begin by defining subsequence of a trajectory
......,T)
(1) where delta observations So, = 0 - 0 = st - st-1 denote the state changes (effects), w the
((),w) = ((or- Or- or)
(Aor, (4=1)]
window size. Using conditional diffusion model, we aim to model the distribution (8r ((..))) For that, we define the reverse process (denoising) as
N
((5
II
(n-1) (n) pa 88
8 8
, ....w)). p(0/51) = -<( 0.
(2)

---

and pe is modeled as the distribution NV [Vss (na, able mean and variance. We could directly supervise the training of je using the forward (diffusion)
[g(s, (t,,w)) n)) Eg(s;") : we) n)) with learn-
process. Following Ho et al. Song et al. [8] we compute a noisy sample (x) based on the true sample 8 = $t (0)
vā(n)s s +
(3)
where E ~ NO I) is the noise, a(n) I a(i) and the weighting factors a(n)
(Bmm + Baar) 2m-1) where mmux = 10 and Siiin = 0.1 are parameters introduced for empiri-
cal reasons 19
We can equally learn to predict the true samples by learning a noise model 20 Hence, we train
noise model ea (s( (t.ww), n)) that learns predict the noise vector e. By using the conditional version of the simplified surrogate objective from ΑЛЛ. we minimize
(p) n,. ||e-col co (s(m), [tt,w)) m)) )1
where 8t is the state sampled from the dataset D for t Ur Ufw..... T-1}), s(n) computed according to Eq. (3). is the noise, and n Un((1....,NN)) ) is the uniform distribution used for sampling the diffusion timestep.
We use the true data sample 8, from the offline RI. dataset to obtain the noisy sample in Eq. ( Leveraging our model's capacity learn multimodal distributions, we generate a set of k samples
{s{") } as our predicted state candidates in parallel from the reverse diffusion chain. We use the
noise prediction model [1]] with the reverse diffusion chain st (n-1) ((()) formulated as
(m) a())
Eg(s," (n) -Tt,w)), n) + vI (n))
(5)
(n)
Va(m)
where (0, I) for N,.... 1, and c for candidates are used during an episode.
Below, detail how the state
2.3 Forecasting using Zero-Shot Foundation Model
Because we assume that the offsets b' originate from a time series, we propose using a probabilistic zero- shot forecasting foundation model
Algorithm Candidate Selection
Sample {(b)..... [b]{²} Zero-Shot FM
2: for each episode p = 1, 1,,... , do
(Zero-Shot FM) such as Lag-Llama
IC] to forecast future offsets from past ones. We assume that after P episodes, the true offsets are revealed, and we predict the offsets for the fol- lowing P episodes. Using the prob- abilistic Zero-Shot FM we generate
4: Reset environment ~ E
5:
8 < op -6b}
6: Initialize T(t,w)
while not done do
8: Sample (0) ~ T (a|3)
9:
Take action a in E, observe 0++1
6//.....6++. b/+P-1). where (1) denotes the number of samples generated for each episode (timestamp). Since Lag-
10:
(5/41) + 0:+1 {b]
11:
T(+11w) PUSH (T(..w)) (Aoct1, ((1))
Llama is a probabilistic model, it can generate multiple samples per times- tamp, conditioned on C number of past contexts "o......bbb b5-1). In practice, we forecast every dimension of b independently since the Shro-Shot
12:
T(++1w) = POP (T(++1,))) (Ao- at-w))
13:
if w then
14:
Sample { }b from FORL by Eq.
5 5 DCDM([[8 (9) Jas (8)41))
15:
16:
else
17:
+ o+11 {b)
FM (Lag-Lit -lama (T0) is a univariate probabilistic model.
18:
end if
19:
f f- 1++1
20: end while
2.4 FORL State Estimation
end for
The next step in our method is to fuse the information from the forecaster and the diffusion model into a state estimate used for control at test time.
5

---

At the beginning of an episode, no information can be obtained from the diffusion model, so for the first w steps we only rely on the forecaster's 's mean prediction, i.e. & = B where the mean is taken over the I samples.
As soon as wo steps are taken, our FORI. State Estimation improves on the inferred state de- tailed below. Figure 2 offers an overview of the entire system and Algorithm provides detailed pseudocode.
Dimension-wid wise wise Closest Match
Multimodal
DOX samplea
Preduct
PDF
Usieodal
To recap, the diffusion model generates samples (s())) } from the in-episode history т, Eq. q These samples represent multimodal distri- bution of plausible state regions. The Zero- Shot FM generates 1 samples of offsets {b}1 from which we compute forecasted states us-
0
2
Figure 4: Distribution of samples produced by DCM (histograms for 10k samples for illustration).
ing { 0r {6}.
FORL: Dimension-wiss Closest Match (DCM) We propose a lightweight approach to sample good estimate based on the samples from the multimodal (diffussion model {s; (0) }&b and unimodal
(Zero-Shot FM { distributions. Let D Dillusion = (xI....,Xy}) D Dimeseries {yi..... yr),
where x, , E P". Then DCM constructs z E R" by
2d (d),d where jd(d) arg min min Fz,d yy.a))
where d = 1...n. n. other words, for each dimension d, we choose the sample from D Dimeseries that has the closest sample in Daiffusion- The process is straightforward yet effective, and under ideal sampling conditions for a toy dataset in Fig. DCM approximately samples from the product distribution. DCM uses non-parametric search find the forecast sample with the highest score, which corresponds to the minimum dimension-win distance. DCM's prediction error is governed by the accuracy of the forecast samples in the unimodal D Pumeseries that achieves this best score. As we will demonstrate in the experiments, this approach empirically yields lower maximum errors and is more stable compared to other methods.
FORL Algorithm Algorithm D summarizes the entire inference process at test time. We begin the episode by relying on the forecasted states šg. As more transitions (Ao,, at-1) become available, the FORL. diffusion model proposes candidate states (s( & through satroppcction-rrasoonii over the past in-episode experience to adapt state estimation on the fly when they begin to diverge from predictions. We then invoke DCM blend the diffusion model's candidates with the foundation model's 's unimodal forecasts and obtain the final state estimate š, We use an off-the-shelf offline RL policy such as Diffusion-QQL (DQL) 14 select the agent's action de.
resl-data-A 1
Deel-data-D(Srree 1).
Summary By combining powerful zero-shot forecasting model with a conditional diffusion mechanism, FORL addresses partial observabil- ity in continuous state and action space when ground-truit offsets are unavailable. This proce- dure is performed in the absence of ground-truth offsets for past, current, and future episodes over the interval j j+P P at test time. DCM provides a computationally inexpensive yet effective way of using the multimodal diffusion candidates and unimodal time-series forecasts. This robust adaptation approach yields state estimate št, aligned with the agent's 's retrospective experience the stationary offline RL dataset, incorporating a prospective external offset forecast.
Offeet
M
urraet
WW
Episodea
Egiaadea
Standard
Tiacrosciva
Purucuat
Figure 5: Llama (T0] for the first univariate series (plotted) from the real-data-A. D datasets; experiments use the first two series from each dataset.
Zero-shot forecasting results of Lag-

---

Experiments
Table I: Normalized scores (mean = std.) for FORL framework and the baselines. Bold are the best values, and those not significantly different (p > 0.05, Welch's t-test).
We evaluate FORL across navigation and ma- nipulation tasks in D4RL [5 and OGBench
[1 offline RL environments, each augmented with five real-world non-stationarity domains sourced from 22 Fig. D presents the ground truth, forecast mean, and standard deviation from Lag-Llama [TO] for the first series of real-data-A A and real-data-1 Our experi- ments address the following questions: (1) Does FORL maintain state-of-the-art performance when confronted with unseen non-stationary offsets? (2) How
maze2d mazd2d-nadiun
DQL.
DQI.+LAG-3
(nars)
A
30.2 1..5
30.2 x5
25.1 .4
63.1
Bata-B
real-data-4
-2.3
56.7 1: 181
56.9
86.3 157
real-data-
36.9
38.5
103.4
real-data-E
3.5
2.00
114=28
51.2
Average
10.0
37.2
34.6
74.1
moze2d-larga
real-data
16.2
2.4
4,2
42,9 4:1
real-data-B
10
5.5
15.0=145
34.9
use FORL when we have no access to delayed past ground truth offsets?
real-data-D
real-data-C
0.9.17
3.0
16.6
8.6
26.8 11
13.4
45.6 4.1
=8.4
real-data-E
12.0
(3) How does DCM compare to other fusion approaches? (4) Can FORL han- dle intra-episode non-stationarity? (5) How gracefully does performance degrade offset magnitude is scaled from 0 (no offset) 1 (our evaluation setup)? (6) Can FORL serve as a plug-an aud--aa module for different offline RL algorithms without retraining? Extended results, forecasts for the remaining series, and implementation details are provided in the Ap- pendix. Results average seeds, unless noted.
Average
3.5
7.1
12.1
38.8
-iierse
real-data-A
22
41.0
=
65.3 =
real-data-B
24
1.
62.5
74.2
real-data-C
21.7
504
60.4399
78.8 =
real-data-D
5.8
26.7 1.1
29.2 19
15.8 1x
real-data-E
6.0.68
58.0
59.3-76
-1.3=
Average
16.1
44.9
51.4
75.1
antraza-rediuu--o
real-data-A
31.0
40.0 1.
39.7 4B
44.0 7.
real-data-B
23.3
48.3
43.3
55.8 77
real-data-G
10.0
real-data
11.7+554
46.7
41.7=66
64.2 86
real-data-F
18.7
26.0=
26.7
Average
18.9
42.1
40.1
48.7
entaaze large-diveree diveree
Baselines We compare approach with the following baselines: DQL [4. Flow Q-learning (FQL) A are diffusion-based and flow-based offline RL policies, respec- tively, that do not incorporate forecast infor- mation. DQL+LAG-3, FQL+LAG-S extend DQL and FQL by using the sample mean of the forecasted states { at each timestep (using the constant per-episode predicted 6)). DQL+LAGG& similarly extends DQL using the median. DMBP+LAG is variant of the Diffu-
A
11.0
11.3 1:49
9.0 4:
34.3 A3
B
5.8
9.2
8.3
46.7 16.7:100
22.1
17.9
33.8
real-data-D
2.5
14.2
14.2=
46.7 126
real-data-E
5,3+
3.3+24
3.3=
11.3
Average
12.0
10.5
34.6
kitchen-complete
real-data-
16.6 14
7.2
8.7
12.0=
real-data-B
12.914
32.7 1.5
20.0
33.1 5.
13.4
23.9
=0
=3
real-data-D
7,5
24.0
28.1=aa
27.1=101
real-data-E
18.5
2.8211
6.2
10.3
Average
miba-single-play
FQI.
FQL.+LAG--
FORL-P (ours)
sion Model- Based Predictor (DMBP) S (a ro- bust offline RL algorithm designed to mitigate state-observation perturbations at test time, de- tailed Appendix D that integrates forecasted states from Shro-Shot FM [10] into its state pre- diction module. By using the model Icarned from the offline data, DMBP+LAG aims to refine the forecasted states to make robust state estima- tions. The underlying policies throughout our experiments are identical policy checkpoints for both our method and the baselines.
real-data-A
0.0
0.0
23.7
real-data-8
0.0
15.0
60.0 :
real-data-G
real-data-D
0.0
0
70,0
real-data-E
Average
0.1
5.2
45.7
antmaze-large-
real-data-A
22,7
1.3
24.3
real-data-B
21.7 5A
29.2
40.0
real-data-C
5.
34.6
55.8
real-data-D real-data-8
10.0 :4
3,3 AD
15.3
Average
12.0
21.2
observation
DOLATAG-R
PORL
sttate
Illustrative Example Figs. and I6 illusttre an agent navigating the maze2d- arge environment where the true position is labeled as "state". The agent receives an observation indicating where be- lieves it is located due to unknown time-dependent factors. The candidate states predicted by the FORL diffusion model are shown circles. Importantly, the agent's s (Ao, a) -ttrrajectory can reveal possible states for the agent. FORL.S 's diffusion model (DM) compo-
SVPe 502
Figure 6: Visualization of states, predicted states as the agent navigates the environment.
7

---

nent predicts these candidate states by using observation changes (Ao) and corresponding actions (a). The possible candidate regions where the agent can be are limited, and our model successfully captures these locations. FORL.'s candidate selection module (DCM) uses the samples from the forecaster and the diffusion model to recover a close estimate for the state. In contrast, the baseline DQL+LAG-S relies on the forecaster [C for state predictions, which are significantly farther from the actual state. Consistent with the results in Fig. 7 FORL reduces prediction errors at test-time, thereby improving performance.
Prediction Error(})
3.1 Results
FORL outperforms both pure forecasting (DQL+LAG-S) and the two- stage strategy that first predicts offsets with a time-series model and then applies a noise-robust offline RL algorithm (DMBP+LAG). Its advantage is consistent across previously unseen non-stationary per- turbations from five domains, each introducing distinct univariate series into a separate state dimension at test time. We present the av- erage normalized scores over the prediction length P across multiple
DQL. DQL+LAG4 DQL
DamPeLAG
FOBL.
Figure 7: Prediction Error in recovering true agent state.
episodes run in the D4RL. [15] and OGBench [21] for each time-series in Table We conduct
pairwise Welch's t-tests across all settings. Figure 7 plots the lz norm between the ground-truth states 8 and those predicted by FORL and the baselines in the antmaze and maze2d environments. Consistent with the average scores, FORL achieves the lowest prediction error on average.
Score(t)
3.1.1 No Access to Past Offsets
25
We evaluate different variants of using DM and Lero-Shot FM when we do not have any access to past offsets in Fig. 8 FORL-DM
20
(DM): Diffusion Model utilizes the candidate states generated by the FORL'SS 's diffusion model component (Section 2.2 which can be a multimodal distribution (Fig. 5 Compared to DM, the full FORI. framework yields a 97.8% relative performance improvement. No- tably, DM performs on par with our extended baselines that incorpo- rate historical offsets and Prrecasting-DMBPPP LLAG DQL+LAG-8, and DQL+LAG-S, Moreover, without access to historical offset information before evaluation, DM achieves 151.4% improvement over DQL, demonstrating its efficacy as standalone module trained solely on a standard, stationary offline RL dataset without offset labels. H-LAG: We maintain a history of offsets generated by DM over the most recent C' episodes (excluding the evaluation interval P. since offsets are not revealed after episode termination at test-time). We then feed this history into the Zero-Shot FM to generate offset samples for the next P evaluation episodes. These samples are applied directly at test time. H-LAG+DCM: We initially follow the same procedure in H-LAG to obtain predictions from Shro-Shot FM. Then, we apply DCM to these predicted offsets and the candidate states generated by FORL'S diffusion model. We also compare against MED+DCM and MED+NOISE, simpler median-based heuristics detailed in Section G Empirically, H-LAG+DCM outperforms H-LAG, demonstrating that DCM with FORL's diffusion model can improve robustness. Overall, scores and prediction errors indicate that just using the samples from DM has better scores on average, while H-LAG+DCM is more stable in Fig. 15
15
MED+Notse
DQL
MED+DCM
H-LAG
H-LAG+OCM
FORL-DM -DM
Figure 8: DM Ablations
Score(t)
3.1.2 Dimension-wise Closest Match (DCM) Ablations
55
50
We compare FORL (DCM) against four alternative fusion strategies. (KDE:: For each dimension, we fit a kernel density estimator
45
(KDE) on Daiifusion = { }x and then we evaluate that probability density function for each point in Dumeseries Then, we take the product of these densities in each dimension to obtain the weight for each sample s/. We obtain a single representative sample by taking the weighted average of samples in D Dumeseries. To ensure stability, when the sum of the weights near zero, we use the mean of the Diimeseries as the states. We use Scott's rule [2 to compute the bandwidth. DM-FS-8, DM-FS-$ select the closest prediction from DM to the mean and median of the Shoo FM's predictions, respectively. FORL (MAX) constructs diagonal multivariate distribution from the dimension-win mean and standard deviation of the
40
DCM
KDE
DM-FS-E DM-FS-S
MAX
Figure 9: Candidate Selection
8

---

Table 2: Normalized scores (mean std.) for FORL and baselines maze2d-large Bolds denote the best scores and those not significantly different (Welch's 's t-test, p > 0.05). Suffixes -T and -R denote the use of TD3+BC 2 and RORL 25 respectively.
To3Bc Policy
RORL. Policy
mazge
To3Bc
TDSBC+LAG-s
DMEP+LAG-T
FORL (ours)-T
RORl
RORL+LAGGS
DMBP+LAG-R
FORL (ours)-R
real -
14.7
19
20.7 1:
12.2
13.0.30
4.31.0
MM
r
-0.9
11.7+
56.8
=.2
13.1
217
98.5
resl-data-C
0.8 19
21.6
19.5
56.9 146
3.1 us
60.6
139,0
-all-aat-D
2.4
149
14
29.5
-1.6_07
17.9.
17.5
33.1 1
real-data-8
-2
1.0
2.0
-0.9
3.3
2.2455
32.2
Average
3.0
8.9
12.5
34.4
2.8
21.6
18.4
71.9
forecasted states, then selects the sample predicted by our diffusion model with the highest likelihood under that distribution. Although all baselines fuse information using the same two sets generated by the diffusion model and Zero-Shot FM, DCM has higher performance. In Table X we compute the maximum, minimum, and mean prediction error values over the test episodes used in Fig. 6 FORL (DCM) yields significantly stable prediction errors (Maximum Error (:2.40) for both maximum error and mean error compared to FORL (MAX) (Maximum Error 4:9.33) demonstrating its robustness.
Score f=50 (1)
3.1.3 Intra-episode Non-stationarity
Our framework can natively handle intra-episode offsets, where the offset changes every [ 50 timesteps. this setting, the offsets become available after the episode terminates, but the agent is subject to a dime-dependent unknown offset within the episode. Zero-shot forecasting foundation module can generate samples before the episode begins. Our diffusion model (FORL-DM) itself does not rely on the forecasts of the foundation module and only tracks observation changes and actions which are invariant to the offsets. The DCM can adaptively fuse information from both models at each timestep without requiring any hyperparameters. Table TO and Fig. [0] show the average scores for DQL vs. FORL-DM and DQL+LAG-S vs. FORL. Among the algorithms that do not use any past ground truth offsets DQL and FORL-DM, only using the diffusion model of FORL significantly increases performance. When have access past offsets, FORL obtains superior performance. This shows that method covers both cases, when information is available and not available, even when offsets are not constant throughout the episode.
40
20
DQL+LAG-3
DQL
FORL-DN DD
FORL.
Figure 10: Intra- Intra-Episode
Performance
3.1.4 Offset-Scaling
Froe
DumP-LLac
DQL+Laae
NQL
100
We scale the offsets with a across all maze experiments. We conduct experiments in 5 environments (all antmaze and maze2d used in Table D across 5 time-series dataset setups with a E(0, (0..0 0.25, 0.5, 0.75, 1.0), where = 0 is the standard offline RL environment used during training and = 1.0 is our evaluation setup. The results show that FORL outperforms the baselines, confirming its robustness. Even small scaling of 0.25 results in sudden drop in performance, whereas FORL only experiences a gradual decrease in Figure 11 Detailed results for each environment and pairs provided in Appendix Figure K
50
0.25
0.5
0.75
Scilin parameter (x)
Figure 11: Impact of offset scaling (a)
on average normalized scores.
3.1.5 Policy-Agnostic
In the maze2d-large -arge experiments (in Table 2 maze2d in Appendix Table 3 we use Robust Offline RL (RORl) [25]. and TD3BC 2 offline RL algorithms instead of DQL [4 to analyze the effect of offline RL policy choice during evaluation. RORL+LAG-S and TD3BC+LAG-S extend RORL and TD3BC by using the sample mean of the forecasted states { at each timestep (using the constant per-episode predicted b²). Results indicate that using a robust offline RL algorithm during training significantly increases performance (71.9) compared to DQL (38.8) and TD3BC (34.4) at test time when used with FORL., no increase when used alone, and a marginal increase with Lag-Llama and DMBP+LAG. We observe similar performance gains when applying FORL to other

---

policies (Section E and Table
including Implicit Q-Learning (IQL) 26 and FQL 223 as detailed in Table
4 Related Work
Reinforcement Learning in Non-Stationary Environments Existing works in non-stationary reinforcement learning (RL) predominantly focus on adapting to changing transition dynamics and reward functions. Ackermann et al. ( propose an offline RL framework that incorporates structured non-stationarity in reward and transition functions by learning hidden task representations and predicting them at test. time. Although our work also investigates the intersection of non-stationary environments and offline RL, we assume stationarity during training. To learn adaptive policies online, meta-learning algorithms have been proposed as promising approach [28 300 Al-Shedivat et al.
20 explores a competitive multi-agent environment where transition dynamics change. While these approaches provide valuable insights, they often require samples from the current environment and struggle non-trivial non-stationarity, highlighting the need for more future-oriented methods p 3T Examples of such inture-oriented approaches include Proactively Synchronizing Tempo (ProST) 99 and Prognosticator [II, which address the evolution of transition and reward functions over time. ProST leverages forecaster, namely, Auto-Regressee model predictor to optimize for future policies in environments to overcome the time-synchronization issue in time-elapsing MDPs. This approach aligns with focus time-varying environments and similarly utilizes real-world finance (e.g.. stock price) time-series datasets to model con-stationarity. Both ProST and Prognosticator assume that states are fully observable during testing and that online interaction with the environment is possible during in the real world. Instead, our approach assumes that states are not fully observable and that direct interaction with the environment during training is not feasible, necessitating that the policy be learned exclusively from a pre-collected dataset.
Integrated Moving Average (ARIMA), and
that are not always feasible
3 RORL 25 examine scenarios where foise-free, stationary dataset is used for training. but corruption is introduced during testing. which assumes a corrupted training
Robust offline RL Testing-time robust offline RL methods DMBP
This is distinct from B dataset. Both RORL 25 and DMBP k] assume access only a clean, uncorrupted offline RL dataset, as FORL, and they are evaluated in perturbed environment. To the best of our knowledge, FORL is non-Markovian, time-evolving, non-stationary deployment
training- time robust offline RL
32 6X)
the first work to extend this setting environment. We focus on time-dependent exogenous factors from real-data that are aligned with the definition of a non-stationary environment
Diffusion models in offline RL Diffusion models 12 have seen widespread adoption in RL B4 (35) due to their remarkable expressiveness, particularly in representing multimodal distributions, scalability, and stable training properties. In the context of offline RL, diffusion models have been
used for representing policies [14
36 36-38 planners [39 40), data synthesis 41 1pp and removing
noise 3 Notably, Diffusion Q-learning [4]] leverages conditional diffusion model policies to learn
from offfine RL datasets, maintaining proximity to behavior policy while utilizing Q-value function guidance. In contrast, our method harnesses diffusion models to learn from a sequence of actions and effect tuples. leveraging the multimodal capabilities of diffusion models to identify diverse candidate locations of the hidden states.
5 Conclusion
We introduce Forecasting in Non-stationary Offline RL (FORL), novel framework designed to be robust to passive non-stationarities that arise at test time. This is crucial when an agent trained on an offline RL dataset deployed in a non-stationary environment or when the environment begins to exhibit partial observability due to unknown, time-varying factors. FORL leverages diffusion probabilistic models and zero-shot time series foundation models to correct unknown offsets in obser- vations, thereby enhancing the adaptability of learned policies. Our empirical results across diverse time-series datasets, OGBench (11) and D4RL (IS) benchmarks, demonstrate that FORL not only bridges the gap between forecasting and non-stationary offline RL. but also consistently outperforms the baselines. Our approach currently limited by the assumption of additive perturbations. For future work, we plan to extend our work to more general observation transformations.
10

---

Acknowledgments and Disclosure of Funding
Georg Martius a member of the Machine Learning Cluster of Excellence, EXC number 2064/1 Project number 390727645. Co-funded by the European Union (ERC, REAL-RL, 101045454). Views and opinions expressed are, however, those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union the granting authority can be held responsible for them. This work was supported by the German Federal Ministry of Education and Research (BMBF)) Tübingen Al Center, FKZ: was in part supported by the INVERSE project (101 136067) funded by the European Union and JSPS KAKENHI Grant Numbers JP23K24926, AP25H01236, The numerical calculations reported in this paper were partially performed at TUBITAK ULAKBIM, High Performance and Grid Computing Center (TRUBA resources). The authors would like to thank René Geist, Tomáš Daniš, Ji Shi, and Leonard Franz for their valuable comments on the manuscript.
This work
References
[1] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. CoRR, abs/2005.01643, 2020. URL
[2] Scott Fujimoto and Shixiang Gu. A minimalist approach to offline reinforcement learning. In
A. Beygelzimer, Y. Dauphin, P. Liang, and Wortman Vaughan, editors, Advances Neural Information Processing Systems, 2021.
[3] YANG Zhihe and Yunjian Xu. Dmbp: Diffusion model-based predictor for robust offline reinforcement learning against state observation perturbations. In The Twelfth International Conference on Learning Representations, 2024.
[4] Brandon Trabucco, Mariano Phielipp, and Glen Berseth. Anymorph: polices by inferring agent morphology. In International Conference on Machine Learning. pages 21677-21691. PMLR, 2022.
Learning transferable
[5] Amy Zhang, Clare Lyle, Shagun Sodhani, Angelos Filos, Marta Kwiatkowska, Joelle Pineau, Yarin Gal, and Doina Precup. Invariant causal prediction for block mdps. In International Conference Machine Learning, pages 11214 1224. PMLR, 2020.
[6] Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual rein- forcement learning: review and perspectives. Journal of Artificial Intelligence Research, 75: 1401-1476, 2022.
[7] Yash Chandak. Reinforcement Learning for Non-stationary problems. PhD thesis, PhD thesis, University of Massachusetts Amherst, 2022.
[8] Annie Xie, James Harrison, and Chelsea Finn. Deep reinforcement learning amidst continual structured non-stationarity. In International Conference on Machine Learning, pages 1 11393- 1 11403, PMLR, 2021.
[9] Hyunin Lee, Yuhao Ding, Jongmin Lee, Ming Jin, Javad Lavaei, and Somayeh Sojoudi. Tempo adaptation in non-stationary reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[10] Kashif Rasul, Arjun Ashok, Andrew Robert Williams, Hena Ghonia, Rishika Bhagwatkar, Arian Khorasani, Mohammad Javad Darvishi Bayazi, George Adamopoulos, Roland Riachi, Nadhir Hassen, Marin Biloš, Sahil Garg, Anderson Schneider, Nicolas Chapados, Alexandre Drouin, Valentina Zantedeschi, Yuriy Nevmyvaka, and Irina Rish. Lag-Ilama: Towards foundation 2023.
models for time series forecasting. arXiv preprint
[11] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances Neural Information Processing Systems, volume 33, pages 6840-6851. Curran Associates, Inc., 2020.

---

[12] Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. CoRR, URI.https:
2015.
03585
[13] Calvin Luo. Understanding diffusion models: A unified perspective. arXiv preprint WMXi::22081119700 2022. URL httthtttp//raatt wwwrriv...og/aae.. aabs/2208 11970
[14] Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023.
[15] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020. URL
07219
[16] Leslic Pack Kaclbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):9--34, 1998.
[17] Blai Bonet. Deterministic pomdps revisited. arXiv preprint arXiv:1205.266: 2012. URL
nttps://arxiv. wwwpariiv..rrgaaae
2659
[18] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13455 2020.
[19] Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans. arXiv preprint arXiv:2112.07804 2021.
[20] Calvin Luo. Understanding diffusion models: A unified perspective, 2022. URL https
//arxiv.org www.oorg///////eeeeeeeoo 11970
[21] Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. OGBench: Bench-
marking offline goal-conditioned RI. In The Thirteenth International Conference on Learning Representations, 2025. URL http:::/t///// /
[22] Alexander Alexandrov, Konstantinos Benidis. Michael Bohlke-Schneider, Valentin Flunkert, Jan Gasthaus, Tim Januschowski, Danielle Maddix, Syama Rangapuram, David Salinas, Jasper Schulz, Lorenzo Stella, Ali Caner Türkmen, and Yuyang Wang. GluonTS: Probabilistic and Neural Time Series Modeling in Python. Journal of Machine Learning Research, 21(116):
1-6, 2020.
[23] Scohong Park, Qiyang Li. and Sergey Levine. Flow q-learning In Forty-second Interna- tional Conference on Machine Learning, 2025. URL 1d=KV12SPLip1
[24] David W Scott. Multivariate density estimation: theory, practice, and visualization. John Wiley & Sons, 2015.
[25] Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. Rorl: Ro- bust offline reinforcement learning via conservative smoothing. Advances neural information processing systems, 35:23851-23866 2022.
[26] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. reinforcmen learning with implicit q-leaming. CoRR, abs/2110.06166 2021. URL https:t/arxxxi.coo org abs/2110.....e/.e./.. 06169
[27] Johannes Ackermann, Takayuki Osa, and Masashi Sugiyama. Offline reinforcement learning from datasets with structured non-stationarity. In Reinforcement Learning Conference, 2024.
[28] Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta-lcarning. In International conference on machine learning, pages 1920-1930. PMLR, 2019.
[29] Maruan Al-Shedivat, Trapit Bansal, Yura Burda, Ilya Sutskever, Igor Mordatch, and Pieter Abbeel. Continuous adaptation via meta-learning in nonstationary and competitive environments. In International Conference on Learning Representations, 2018.
12

---

[30] Suzan Ece Ada and Emre Ugur. Unsupervised meta-testing with conditional neural processes for hybrid meta-reinforeement learning. IEEE Robotics and Automation Letters, 9(1)):4477-444, 2024. doi: 10. 1.099..RA.222.3444444444
[31] Yash Chandak, Georgios Theocharous, Shiv Shankar, Martha White, Sridhar Mahadevan, and Philip Thomas. Optimizing for the future in non-stationary mdps. In International Conference on Machine Learning, pages 1414-1425. PMLR, 2020.
[32] Chenlu Ye, Rui Yang, Quanquan Gu, and Tong Zhang. Corruption-robust offline reinforcement learning with general function approximation. Advances in Neural Information Processing Systems, 36:36208-36666 -62221, 2023.
[33] Xuezhou Zhang, Yiding Chen, Xiaojin Zhu, and Wen Sun. Corruption-robust offline rein- forcement learning, In International Conference on Artificial Intelligence and Statistics, pages 5757-5773. PMLR, 2022.
[34] Zhengbang Zhu, Hanye Zhao, Haoran He, Yichao Zhong, Shenyu Zhang, Yong Yu, and Weinan Zhang. Diffusion models for reinforcement learning: A survey. arXiv preprint arXiv:2311.0222 2023.
[35] Jiayu Chen, Bhargav Ganguly, Yang Xu, Yongsheng Mei, Tian Lan, and Vaneet Aggarwal. Deep generative models for offline policy learning: Tutorial, survey, and perspectives on future directions. arXiv preprint urXiv:2402.13777 2024.
[36] Longxiang He, Linrui Zhang, Junbo Tan, and Xueqian Wang. Diffeps: Diffusion model based constrained policy search for offline reinforcement learning. arXiv preprint 30X0v:23/0.0533,, 2023.
[37] Philippe Glansnn-Estruce Ilya Kostrikov. Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304. 10573, 2023. /httts:///artee www./rgg//////eeeee...eeeeeeeeee
10573
[38] Suzan Ece Ada, Erhan Oztop, and Emre Ugur. Diffusion policies for out-of-distribtiio generalization in offline reinforcement learning. IEEE Robotics and Automation Letters, 94): 3116-3123, 2024. doi: RA111099/...222..33..
[39] Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning. 2022.
[40] Matthew Coleman, Olga Russakovsky, Christine Allen-Blanchette, and Ye Zhu. Discrete diffusion reward guidance methods for offline reinforcement learning. In ICML 2023 Workshop: Sampling and Optimization in Discrete Space, 2023.
[41] Haoran He, Chenjia Bai, Kang Xu, Zhuoran Yang, Weinan Zhang, Dong Wang, Bin Zhao, and Xuelong Li. Diffusion model is effective planner and data synthesizer for multi-task reinforcement learning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 64896-64917. Curran Associates, Inc., 2023.
[42] Zhixuan Liang. Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, and Ping Luo. Adaptdif- fuser: Diffusion models as adaptive evolvvg planners. arXiv preprint arXiv:2302.01877 2023. URL //axxiv. org/ abs 2302 01877.
[43] Scott Fujimoto, David Meger, and Doina Precup. Off- Off-policy deep reinforcement learning without exploration. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Leurning, volume 97 of Proceedings of Machine Learning Research, pages 2052-2066 PMLR, 09-15 Jun 2019. httpst:s:: http:s/////ccceee//g// html
mlr. ppress/v97/fujimotooo
[44] Noah Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavior modelling priors for offline reinforcement learning. In International Conference Learning Representations, 2020.
13

---

[45] Yecheng Jason Ma. Dinesh Jayaraman, and Osbert Bastani. Conservative offline distri- butional reinforcement learning. In A. Beygelzimer, Y. Dauphin, Liang, and J. Wort- man Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL httpss//pppaareeeeeee.errrrrrrrp
[46] Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences dialog. arXiv preprint urXiv:197.000450 2019. URL
arXiv preprint http://orga/bbbb//egooee .00566
[47] Yifan Wu, George Tucker, and Ofir Nachum. 2019. URL
offiine 11361 11361
learning.
[48] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weiighte regression Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00111 2019.
[49] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359) 2020.
[50] Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. Advances in neural information processing systems, 29, 2016.
[51] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- policy q-learning via bootstrapping error reduction. Advances in neural information processing systems, 32, 2019.
[52] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. COMBO: conservative offline model-based policy optimization. CoRR, abs/2102.08363, 2021. URL httphhtts///aa
httg//bb/21002... 2202. 08363
[53] Seohong Park, Kevin Frans, Sergey Levine, and Aviral Kumar. Is value learning really the main bottleneck in offline ri? arXiv preprint arXiv:2406.003229 09329, 2024. URL CRL.https:// https://arxx: org/ abs 0bs/2406 093299
[54] Bogdan Mazoure, Ilya Kostrikov, Ofir Nachum, and Jonathan J Tompson. Improving zero-shot generalization in offline reinforcement learning using generalized similarity functions. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 25088-25101. Curran Associates, Inc., 2022.
[55] Martin I Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
[56] Zhihe Yang and Yunjian Xu. DMBP: Diffusion model-based predictor for robust offine reinforcement learning against state observation perturbations (official implementation). https http//treeemmiin 2024.
//gthub. wewsthub.comzyyaaatee wttp://hhaaggg22///////2 hwtp:///222//////ee
[57] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
[58] Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez. Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Repre- sentations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
[59] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. International conference on machine learning, pages 1587-1596. PMLR, 2018
[60] Jan Peters and Stefan Schaal. Reinforcement learning by reward- weighted regression for operational space control. Proceedings of the 24th international conference on Machine learning, pages 745-750, 2007.
14

---

[61] Qing Wang. Jiechao Xiong. Lei Han, peng sun, Han Liu, and Tong Zhang. Ex- ponentially weighted imitation learning for batched historical data. In S. Bengio. H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL httphttts//proceeeiiie neurips. cc/ cc/paper.p.... fiwes/llepael ptt.e////ppprr//////ieeee pdf
[62] Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. CORL: besearch-orientee deep offline reinforcement learning library. In 3rd Offline RL Workshop: Offline RL as a "Launchpad", 2022. URL htttps://ppeareviev.. wtwp:evviiiineeeffuuuum.
1d=SyAS49BBcv
[63] Garrett Thomas. Implicit q-learning (iql) in pytorch. nttps:///ithub.ccmm wwt.:/o/m/ww/aa...... IQL-PyTorch] 2021.
[64] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint 0at::229900003, 2022.
[65] Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, and qiang liu. Instatlow: One step is enough for high-quality diffusion based text-to-image -moge generation. In The Twelfth International Conference on Learning Representations, 2024. /ttttts:///pprreeeeeot wtwp:eeviiii.neeeffreeeeeee formm?
id (d11kybbbDq.
[66] Kevin Frans, Danijar Hafner, Sergey Levine, and Pieter Abbeel. One step diffusion via shortcut models. arXiv preprint arXiv:2410. 12557, 2024.
[67] Zihan Ding, Chi Jin, Difan Liu, Haitian Zheng, Krishna Kumar Singh, Qiang Zhang, Yan Kang, Zhe Lin, and Yuchen Liu. Dollar: Few-step video generation via distillation and latent reward optimization. arXiv preprint arXiv:2412.15669, 2024.
[68] Jiachen Li, Weixi Feng, Wenhu Chen, and William Yang Wang. Reward guided latent consis- tency distillation. arXiv preprini arXiv:2403. /027, 2024.
[69] Elvezio M Ronchetti and Peter Huber. Robust statistics. John Wiley & Sons Hoboken, NJ, USA, 2009.
[70] Martin A. Fischler and Robert C. Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM, 24(6): 381-395, June 1981. ISSN 0001-0782.
[71] B. P. Welford. Note on method for calculating corrected sums of squares and products. Technometrics, 420,, 1962.
[72] Donald Ervin Knuth. The art of computer programming, volume 3. Pearson Education, 1997.
[73] Rakshitha Wathsadini Godahewa, Christoph Bergmeir, Geoffrey I. Webb, Rob Hyndman, and Pablo Moontero-Manso Monash time series forecasting archive. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[74] Dua Dheeru and Ef Karra Taniskidou. Uci machine learning repository. http://archiv/
ics. uci. edu/ ml 2017.
[75] David Salinas, Michael Bohlke-Schneider, Laurent Callot, Roberto Medico, and Jan Gasthaus. Hligh-dimensional multivariate forecasting with low-rank gaussian copula processes. Advances in neural information processing systems, 32, 2019.
[76] Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research development in information retrieval, pages 95-104, 2018
[77] A. Alexandrov, K. Benidis, M. Bohike-Schneider
V. Flunkert, J. Gasthaus, T. Januschowski, D. C. Maddix, S. Rangapuram, D. Salinas, J. Schulz, L. Stella, A. C. Türkken, and Y. Wang. GluonTS: Probabilistic Time Series Modeling Python. arXiv preprint arXiv: 19066952664 2019.
15

---

[78] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. Medical image computing and computer- assisted intervention-MICCAl 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part 18, pages 241. Springer, 2015.
[79] Zhendong Wang. Diffusion policies for offline rl official pytorch implementation. https www.fssioooooliiiiee..oooeoeoeee ooto://tieeoooogggaagg/g/iiiiooo
//github. cww. Seomm/Zeenonng-Waa wang//gggaaa/iiiiiaa
for -ffoorfffflnnee RL 2023.
[80] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua
Bengio and Yann LeCun, editors, ICLR (Poster), 2015. URL http://bblp...u uni-hblpp..uniiii wni-triieee de/
tb/confficclr/ic122015 httmm #Kiggaiii
[81] Ilya Kostrikov. Offline reinforceementt learning with implicit q-learning (official implementation).
http://hho.c.ooooooororoooooooog leaariing 2021.
[82] Seohong Park. Fql: Flow q-learning (official implementation). https: tiuth....... com/
ff 2025.
[83] Scott Fujimoto. A minimalist approach to offline reinforcement learning pytorch implementation.
https://github. wwwccooo/uuuu/ sfmp///ujjmmm///// TD3 BC 2018.
[84] Rui Yang. Rorl: Robust offline reinforcement learning via conservative smoothing code repository. https:/ thub. com www.oo//nnnnnnn////nnoo.. 2022.
[85] Emanuel Todorov, Tom Erez, and Yuval Tassa Mujoco: A physics engine for model-based control. Intelligent Robots and Systems (IROS), 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pages
IEEE, 2012.
[86] Artur Trindade.
12014. UCI Machine Learning Repository, 2015.
DOI: https://l..or//1..33333/CCCCC3C
[87] John Hunter. Matplotlib: A 2d graphics environment. Computing in science engineering, 9(33):90-95 2007.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract and introduction supported by experimental results and compared to the baselines.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We disclose all datasets and environments used (all publicly available), the model architecture hyperparameters, implementation details, and pseudocode of the algo- rithms.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
18

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We plan to provide open access to code in the future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (httpss//nips... cc/
mublic//gudes//CddSSuiiiis sorrooddSabmmisssoooooooroo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ 'nipsscc/public//uui /otee//guudddeeeedSuuuubmiibii ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer:
[Yes]
Justification: We specify all training and test details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We report statistical significance tests in this paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
19

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide computational hardware required to reproduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http://aarrr...cccco/ll///aaaaaa
Answer: [Yes]
Justification:
The research conducted fully conforms with the NeurIPS Ethics Guidelines
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: To the best of our knowledge there are no direct negative societal impacts of our work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not involve releasing any high risk models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the original owners and included the licenses.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paaperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not introduce new dataset.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing experiments nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsourcing experiments nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
22

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: We did not use LLMs for important, original, or non-standard component of the core methods in this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standarr components.
Please refer to our LLM policy battps://neuree cetparriisscccoofeeeeeeeee / for what should or should be described.
23


Reviews:
[{"review_id": "lx2xnvV0BC", "reviewer": "Reviewer_VWZ4", "summary": "This paper addresses a nonstationary offline RL setting in which the the offline dataset contains no nonstationarity, but during deployment there is an unknown, additive offset added to the state.\nThis offset evolves between episodes but is constant within each episode, which is also known as a Dynamic-Parameter MDP setting.\nThe authors propse to address the problem of identifying the offset by 1) a learned environment model (based on diffusion) and 2) a zero-shot time-series model, that uses the offsets from prior episodes.\nIn experiments on maze, antmaze, and in a manipulation task, the authors show that their method performs favorably compared to baselines, both in terms of achieved reward as well as prediction error.", "questions": "* Why can't the proposed method be extended to other transformations of the state? It seems that at least arbitrary monotonous transformations of the state should be within the scope of the proposed method?\n * Currently the knowledge of the offsets of past episodes is considered known. Is it possible to replace this with the output of the diffusion model for prior episodes? This would significantly increase the applicability of the proposed method. \n * Could you explain the decision to use \"Dimension-Wise Closest Match\" in your approach? There are many different ways to combine multiple different state estimators, so this choice currently feels a bit arbitrary.\n\nMinor comments:\n * Equation (5) and (6) are referred to in the main text but only appear in the appendix.\n * L108 \"FORL makes informed decisions at the start of each episode\" does not seem like a contribution\n * L240: Maybe use \"diffusion\" and \"timeseries\" instead of \"multimodal\" and \"unimodal\"\n * I would recommend more clearly separating your proposed method from background material, such as the explanation of diffusion models.", "limitations": "The assumption of the past offsets being revealed after the first few episodes seems strong. It would be helpful to evaluate the method in cases where it is violated.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n * The proposed problem setting of additive offsets during deployment with none being present during training is well motivated and challenging. Dynamics that differ during deployment are explored thoroughly in the robust RL literature and unknown transformations of the true state are throughly explored in the POMDP literature. However, the inclusion of unknown but predictably evolving transformations /offsets of the state is novel, to my knowledge.\n * The experimental validation shows a clear advantage of the proposed method\n\nNeutral:\n * the application of zero-shot prediction models is relatively straight-forward. On the other hand, it is also very reasonable and it is unclear what else could be done in this setting.\n\nWeakness:\n * the combination of the learned dynamics model and time-series model seems relatively ad-hoc. In essence, the proposed method obtains multiple estimates of the state from the dynamics model and chooses the closest pair among both sets, for each dimension. Thus the final estimate may have each dimension from a different sample drawn from the dynamics model, such that the combination could be entirely outside the support of the dynamics model. It is not clear why this method was used instead of any other way to select the state estimate. This is an important part of the algorithm and should be justified better.\n\n * The problem setting is limited to additive offsets, which is not a huge issue, but it is not clear to me why the method can not be extended to any other state transformations\n\n* Experiments: Hyperparameter tuning is limited. While some hyperparameters of DMBP were tuned, for other baselines it seems like the default hyperparameters were used without tuning, which makes the comparison somewhat unfair as the proposed problem is new. Further, it would be helpful to evaluate more realistic cases of additive offsets, outside of maze-like. The method is also evaluated on a manipulation task, but there the advantage compared to baselines are less clear.\n \n\n* The current method assumes knowledge of the true offset in past episodes, after P episodes have passed. This is a significant limitation and should be addressed in the paper, but is currently not listed in the limitations.", "was_revised": true, "final_justification": "The rebuttal has mostly resolved my concerns about the choice of the \"Dimensionwise Closest Match\" mechanism, by providing additional alternative baseline methods.\n\nIt has also fully resolved my concerns about the assumption of knowledge of the true offsets of past episodes, by showing that even without this assumption the method is useful.\n\nI do have some remaining concerns about the clarity of the writing, but with some cleanup based on the rebuttals it should not be a major issue. \n\nAs such, I am raising my score 4->5 and recommending acceptance.", "rebuttal": "We sincerely thank the reviewer for their inspiring and detailed comments. Your recognition of our **novel**, **well-motivated** and **challenging** problem setting and the experimental advantages of our method is truly encouraging. Your insightful questions and suggestions have highlighted key opportunities to strengthen our paper's contributions and structure.\n\n> Could you explain the decision to use \"Dimension-Wise Closest Match\" in your approach?.. currently feels a bit arbitrary. \n\nThank you for this insightful question, and attention to detail. Although our choice of DCM was a principled one tailored to zero-shot adaptation, we agree it deserves a more thorough justification and analysis.\n\nThe primary advantages of DCM are that it is simple, yet effective (Fig. 4), data-driven and non-parametric. In a test-time adaptation setting, where an agent may face entirely new non-stationary dynamics, methods that require tuning (e.g., the bandwidth of a Kernel Density Estimator) can be impractical. DCM provides a robust, data-driven way to fuse information from the two models without any tuning. It is designed for our setting where each state dimension can be affected by an independent non-stationarity to test the robustness of our framework, making a dimension-wise fusion a natural choice. We believe DCM can contribute to the research of fusion from different sources of information.\n\nTo justify this choice empirically, we have conducted a new, extensive ablation study comparing DCM against five alternative fusion strategies (across 5 random seeds), including a Bayesian approach using KDEs (Bayes (KDE)) and methods that explicitly **ensure the final estimate lies within the support of the diffusion model's samples** (MAX(Appendix F.4), DM-mean(x)_(timeseries), DM-med(x)_(timeseries)). Maximum-likelihood approach was discussed in the supplementary material F.4 & F.5 (L1144), we include the numerical results for your convenience.\n\n- **Bayes KDE**: For each dimension, we fit a kernel density estimator (KDE) on D_(diffusion)=$\\{\\boldsymbol{s}t^{(0)}\\}k$ and then we evaluate that probability density function for each point in D_(timeseries) Then, we multiply these densities to obtain the weight for each sample $\\hat{s}^b_t$. We obtain a single representative sample from these distributions for taking the weighted average of samples in D_(timeseries) \n- **DM-mean(x)_(timeseries), DM-med(x)_(timeseries)** where we take the closest prediction from DM to the mean/median of the FM's predictions. \n\n| Method | Score| \n| :--- | :--- | \n| DCM (ours) | **54.3** | \n| Bayes (KDE) | 51.5 | \n| DM-mean(x)_(timeseries) | 37.4 |\n| DM-med(x)_(timeseries) | 38.7 |\n| MAX | 37.67 |\n\nThe results show that our simple, hyperparameter-free DCM is highly competitive with, and often superior to, other methods. While the KDE-based `Bayes` method performs well, it requires bandwidth selection, and fallback mechanism for numerically unstable results. This highlights DCM's practical advantage.\n\nWe will add a new subsection to the appendix detailing these alternative fusion methods and presenting the full comparison table. We will summarize these findings in the main text to provide a clear and compelling justification for our choice of DCM\n\n> Why can't the proposed method be extended to other transformations of the state? \n\nOur diffusion model component uniquely uses the $\\Delta s, a$ pairs during training so  $\\Delta s, a$ =  $\\Delta o, a$ holds at test time.  Thus, other transformations are challenging.\nNevertheless, we empirically evaluate our framework for scaling and offset transformations. Following the reviewer's suggestion, we conducted experiments with scaling using another time-series from the existing real-data domains. Our results for scaling and offsets in maze2d-large environment across all real-data-A,B,C,D,E are provided below:\n\n|  | $\\mathbf{DQL}$ | $\\mathbf{DQL+MeanLag}$ | $\\mathbf{FORL}$ |\n|:---|:---:|:---:|:---:|\n| Score | 1.8 | 7.7 | **34.0** |\n\n> Hyperparameter tuning is limited.\n\nThe underlying policies (TD3+BC, RORL, Diffusion-QL), and the forecasts predicted by the Zero-Shot FM are **identical policy checkpoints** for both our method and the baselines. We directly deploy the same policies without any modification in the policy modules of DMBP and FORL framework in Fig.2 as both have plug-and-play policy module. Therefore, any tuning of the policy would benefit all methods. Our focus on a zero-shot, test-time adaptation setting follows the testing-time robust offline RL setup. We will clarify this in the experimental setup section.\n\n\n> Further, it would be helpful to evaluate more realistic cases of additive offsets, outside of maze-like.\n\nWe agree that testing on more realistic offsets is an important direction, which we intend to explore for future work.\n\n> Currently the knowledge of the offsets of past episodes is considered known. \n\nWe apologize for not making this clearer in the main text: **our core contribution is highly effective even when we have no access to true offsets in past episodes.** The full FORL framework, which uses the forecasting foundation module, is designed for scenarios where this information is available. However, our use of a conditional diffusion model (DM) for candidate state generation conditioned on action‑$\\Delta o$ trajectories trajectories offers a novel contribution distinct from typical diffusion model applications in reinforcement learning. DM can operate **without any access to historical offset information**, we wil rename DM as FORL-DM to be more precise. Specifically Fig.11 show that our Diffusion Module (DM) alone, without any historical offsets, outperforms all baselines that do not use historical ground-truth offsets.  \n\n> Is it possible to replace this with the output of the diffusion model for prior episodes? This would significantly increase the applicability of the proposed method.\n\nWe investigated this exact scenario with our H‑LAG+DCM, (L1140) (We correct the typo here where L1140 refers to H-LAG+DCM and 1136 refers to H-LAG where results for both algorithms are shown in Fig. 11). H‑LAG+DCM replaces true offsets with DM‑generated estimates and performs similarly to DM. This directly addresses the question of replacing known offsets with DM outputs. Our findings show comparable performance to the standalone DM. \n\nWe will use a different coloring for methods that do not use past historical ground truth offsets (DQL, DM, MED-DCM, MED+NOISE, H-LAG+DCM and H-Lag) in Fig. 11, and move the results for DM and H-LAG+DCM to main text. \n\n| Method| Score   |\n|--------------------|--------:|\n| $\\texttt{DM}$ (renamed to FORL-DM)| **27.4** |\n| $\\texttt{H-Lag-DCM}$ | 26.3|\n| $\\texttt{DQL}$| 10.9|\n| $\\texttt{Med-DCM}$ | 17.7|\n| $\\texttt{Med-Noise}$| 15.5|\n| $\\texttt{H-Lag}$| 17.3|\n\nIf past offsets are available, our modular design allows leveraging zero‑shot foundation models with DCM, so improvements in these models will directly benefit FORL. We will report results for all datasets and environments in a dedicated table for methods that do not use past historical ground truth offsets. Furthermore, we have extended  (MOD, Running-Mean,Running-Mean-p and Random Sample Consensus (RANSAC)) in maze2d‑medium environments with all real-data-A,B,C,D,E over five random seeds, which we will include in the supplementary. \n\n- MAD: computes the coordinate-wise median of the differences between the observation and each denoiser prediction, discards any sample whose absolute deviation from this median exceeds $\\epsilon$ x Median Absolute Deviation (MAD), takes the median of the remaining inliers to obtain a robust offset estimate, and subtracts that offset from the observation\n- Running-$\\mu$:computes a numerically stable, global running mean of offsets from DM across all episodes \n- Running-$\\mu$-p: Running-$\\mu$ per episode $p$\n- RANSAC: calculates the offsets from DM, sets an adaptive per-dimension threshold as $\\epsilon$ x MAD, runs RANSAC by repeatedly sampling one offset candidate and choosing the candidate whose inlier set (differences within that threshold) is largest, averages those inliers to estimate offset.\n\n| Method   | Score   |\n|--------------------|--------:|\n| FORL-DM ($\\texttt{DM}$ in main paper)  | **57.1** |\n| FORL-DM-MAD| 45.6  |\n| FORL-DM-Running-$\\mu$ |42.9 |\n| FORL-DM-Running-$\\mu$-p | 45.5  |\n| FORL-DM-RANSAC   | 47.6  |\n\n> L108 \"FORL makes informed decisions at the start of each episode\" does not seem like a contribution\n\nUsing a zero-shot forecasting foundation model, our agent has an information on the unknown non-stationarity of the new episode, unlike methods that start \"blind\" and must learn from scratch within the episode. We will instead list FORL-DM in our contributions.\n\n> L240: Maybe use \"diffusion\" and \"timeseries\" instead of \"multimodal\" and \"unimodal\"\n\nWe agree with the reviewer's suggestion, and will use D_(diffusion) and D_(timeseries) for clarity.\n\n> background material, such as the explanation of diffusion models.\n> Equation (5) and (6) appear in the appendix.\n\nTo the best of our knowledge, no prior work trains a diffusion model on sequences of $\\Delta s$ and actions to generate multimodal candidate states, a design choice that substantially enhances our model’s robustness. To clarify the training procedure, we explicitly detail the conditioning variables and the training procedure. Following reviewer's suggestion we will move the core components of the diffusion model to a dedicated background section, which will also include Equations (5) and (6) previously in the Appendix."}, {"review_id": "ztaB7fGajk", "reviewer": "Reviewer_VqtP", "summary": "This paper focuses on robust offline reinforcement learning, in the sense that at training time, the agent will receive clean, fully observable data, while at test time, the agent's data can be offset by certain (possibly non-Markovian) factors in the environment, rendering it partially observable. The proposed framework, FORL, plans using a diffusion model and diffusion policy learned over clean, offline trajectory data, and allows for forecasting without certain assumptions being made on either the nature of perturbations at test time, and handles nonstationarities without the need of explicitly modeling this problem as a POMDP.", "questions": "I think that there are some things to clean up in general about the paper -- I can understand how sampling is done with this model, but there is no Equation (6) for instance that is mentioned in Algorithm 1 on page 5. I am also not entirely sure what this DCM procedure is -- it seems like some nearest neighbor-style thing to generate good samples, but I don't understand what the unimodal sample dataset does here (because as far as I can understand, this unimodal dataset contains offsets from the time-series FM, not true samples). Can this be cleared up a bit? It's also not clear what the initial samples from the FM do in FORL in Algorithm 1 -- it seems like they just aren't used in the algorithm at all?", "limitations": "Not explicitly, but it may not be needed.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper is reasonably clear -- I don't have a background in this specific area of offline RL, but I could understand the overall method after a couple of passes through it. The paper's motivation is also quite straightforward -- if it is possible to do forecasting (e.g. with the time series foundation model) to estimate the offsets, while also having an expressive generative model (e.g. the diffusion model) model the transition dynamics, the hope is that you can generalize well out of distribution by doing online planning by generating perturbed trajectories from the combination of these two models. Experimental results look pretty good too across a diverse suite of MuJoCo tasks common for offline RL.\n\nI think from an empirical standpoint, the one weakness I could see is that D4RL is a somewhat saturated set of domains -- there are new, difficult tasks that are also fully-observable (such as OGBench) that would be great to see results for.", "was_revised": true, "final_justification": "Given the OGBench results provided by the authors, I am happy to raise my score. Sorry this is coming so late -- I was at a conference last week.", "rebuttal": "We sincerely thank the reviewer for their valuable suggestions and encouraging feedback on our work. We are glad they found the paper's motivation straightforward, the experimental results strong. The reviewer's thoughtful questions have highlighted several areas where we can significantly improve the clarity and empirical scope of our paper. \n\n> I think from an empirical standpoint, the one weakness I could see is that D4RL is a somewhat saturated set of domains -- there are new, difficult tasks that are also fully-observable (such as OGBench) that would be great to see results for.\n\nOGbench is an excellent recent benchmark that was primarily developed for goal-conditioned offline RL but also includes standard offline RL experiments. Thank you for suggesting it. The current paper's scope was an extensive study to establish the method's core properties across different non-stationarities.\nMost testing-time robust offline RL algorithms use D4RL (RORL,DMBP), so hyperparameter optimization at this stage may not be possible for all methods, as the navigation environments are brittle against hyperparameters. However, we believe our empirical results remain valid since all baselines and testing-time robust offline RL algorithms use these established hyperparameters on D4RL tasks.\n\n> I think that there are some things to clean up in general about the paper -- I can understand how sampling is done with this model, but there is no Equation (6) for instance that is mentioned in Algorithm 1 on page 5.\n\nThank you for your suggestion. We agree moving Eq.6 from the Supplementary Material (C.1) to the main text is important for the clarity of our paper. Eq.6 show the reverse diffusion chain and how it is formulated to generate the candidates states conditioned on $\\tau_{t,w}$. The final samples of this reverse diffusion chain $s^0$ form our multimodal set from the diffusion model $\\{s_{t+1}^{(0)}\\}_k$. \n\n> I am also not entirely sure what this DCM procedure is -- it seems like some nearest neighbor-style thing to generate good samples, but I don't understand what the unimodal sample dataset does here (because as far as I can understand, this unimodal dataset contains offsets from the time-series FM, not true samples). Can this be cleared up a bit?\n\n\nWe thank the reviewer for giving us the opportunity to clarify the datasets used in our `Dimension-Wise Closest Match` (DCM) procedure. The reviewer's understanding of DCM is **correct**. The unimodal distribution in here is the one predicted by the time-series FM. We are able to obtain samples from it by querying the model. We confirm and clarify below: \n- DCM is a nearest neighbor-style, parameter-free, data-driven approach method to generate a sample using information from a multimodal and a unimodal distribution.\n- Unimodal dataset does not contain true samples, as correctly pointed out by the reviewer. The unimodal dataset contains the offset removed states, where the offsets are predicted by the forecasting module. Time-series FM is a probabilistic forecasting, foundation model thus it proposes $l$ number of offsets for each episode. These offsets $\\{\\hat{b}\\}_l$ are then subtracted from the observation $o_t$. The resulting offset removed states $\\{\\hat{s}^b_t\\}_l$ forms the set of plausable states hypothesized by the forecasting module. The \"unimodal\" dataset contains forecasted offset removed states. We refer to the lines 235-L236 in our current paper below: \n\n\"The *Zero-Shot FM* generates $l$ samples of offsets $\\{\\hat{b}\\}_l$ from which we compute $\\{\\hat{s}^b_t\\}_l = o_t - \\{\\hat{b}\\}_l$.\"\n\nWe will include this part explicitly in Algorithm 1 to be clearer. Furthermore, we will rename the Datasets useed in the DCM to explicitly refer which dataset it refers to instead of using *D*_(uni) and *D*_(multi). \n\nTo summarize, as our diffusion model is learning a multimodal distribution, the candidate states generated by the diffusion model are multimodal (*D*_(diffusion), illustrated as filled orange circles in Fig.3), while the zero-shot time-series FM is unimodal (*D*_(timeseries)). We will denote *D*_(uni) by *D*_(timeseries) and *D*_(multi) by *D*_(diffusion). \n\n> It's also not clear what the initial samples from the FM do in FORL in Algorithm 1 -- it seems like they just aren't used in the algorithm at all?\n\nWe are very grateful to the reviewer for their detailed reading and for spotting a crucial typo in Algorithm 1. The variables should be $\\hat{b}$ (for bias/offset), not $\\hat{d}$. These forecasted offsets $\\{\\hat{b}\\}^p_l$ for the current episode $p$ are essential and used in lines Alg. 5,14, and 16. We correct this typo below, where the first line of the algorithm is changed to:\n\nSample $\\{\\{\\hat{b}\\}^1_l, \\dots, \\{\\hat{b}\\}^P_l\\}\\sim $ *Zero-Shot FM*\n\nTo obtain offset removed states from the forecaster we compute $\\tilde{s}\\leftarrow o_0-\\{\\hat{b}\\}^p_l$ where $p$ is the episode, and $l$ is the number of forecasts samples per episode. Because we use a probabilistic forecasting module, the FM gives us multiple samples for that specific episode. \n\nWe use initial samples from the FM in lines Alg. 5,14, and 16 where we update $\\tilde{s}\\leftarrow o_0-\\{\\hat{b}\\}^p_l$ where p is the episode. We will make it clearer in the algorithm by adding $\\tilde{s}\\leftarrow o_0-\\{\\hat{b}\\}^p_l$ after Line 4, Line 11 in the algorithm to explicitly show how the Samples from Zero-Shot FM are used to obtain the states from the forecasting module."}, {"review_id": "PmIsEUh6A9", "reviewer": "Reviewer_cynB", "summary": "This paper proposes FORL, a framework for offline RL under test-time observation non-stationarity, where each episode is affected by a latent, fixed offset. The method uses a time-series model to forecast future offsets based on past episodes, and a diffusion model to correct state estimates during testing. A fixed offline-trained policy then operates on the corrected states. Experiments on D4RL benchmarks with synthetic offsets demonstrate performance gains over baseline methods in perturbed settings.", "questions": "1. If the test-time environment is a POMDP, why is the training data assumed to be from a stationary, fully observable MDP? Isn’t that an unrealistic assumption given the real-world motivation?\n2. Why is the additive offset model (i.e., ot=st+bj) considered sufficient to represent the broad class of real-world observation non-stationarities?\n3. How exactly is the forecasting model trained or used, given that ground-truth offset values are not observable during deployment? Where do these values come from?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper focuses on a realistic and underexplored problem: test-time observation non-stationarity in offline RL. This is an important deployment issue that is often ignored by current benchmarks and methods.\n2. The combination of diffusion-based state inference and time-series forecasting is novel and modular.\n\nWeaknesses:\n1. The paper assumes that the training data comes from a fully observable, stationary MDP, while the test-time deployment setting is a POMDP with unknown episode-wise observation offsets. If the real world is truly non-stationary (as the paper claims), then the offline dataset itself should already reflect these dynamics—i.e., it should be collected under a POMDP setting. The assumption that we have clean, unshifted ground-truth states during training but encounter sudden, unobservable bias at test time feels not well justified.\n2. The forecasting component depends on a sequence of ground-truth offset values from past episodes but these offsets are fundamentally unobservable in practice. The model assumes they \"become available\" after each episode, which breaks the zero-shot narrative and introduces information leakage. In real deployment, these values are never accessible, making the forecasting model infeasible in practice.\n3. Although the paper claims to tackle realistic non-stationarity, the experiments are all conducted on standard D4RL environments—clean MDPs—augmented by synthetic additive offsets on state vectors. These do not reflect real-world POMDP complexity like occlusion, information loss, non-linear emissions, or sensor dropout. The claim that this method handles realistic partial observability is not strongly supported.\n4. The paper only compare against DQL and its variants, omitting widely-used and stronger baselines like IQL, or methods related to Decision Transformer.The modularity of the proposed method would allow easy integration with other offline RL methods, but this is not well tested.", "was_revised": true, "final_justification": "The score is updated based on the rebuttal.", "rebuttal": "We thank the reviewer for their insightful and constructive feedback on our paper. We are encouraged that the reviewer recognizes the **novelty and modularity** of our approach and the importance of our problem setting. Following the reviewer's suggestions, we extend our empirical analysis, clarify our positioning and demonstrate the significance of our setup. \n\n> \"The assumption that we have clean, unshifted ground-truth states during training but encounter sudden, unobservable bias at test time feels not well justified.\"\n\nWe will revise our related work to reflect how our problem setting fits into the existing research on robust offline RL. Our choice of a stationary training dataset and a non-stationary test environment is a deliberate one that positions our work as a significant extension of the important and well-established subfield of (i) **testing-time robust offline RL**[RORL,DMBP]. This is distinct from, (ii) **training-time robust offline RL** frameworks [1,2], which assume a corrupted training dataset. The \"clean train, perturbed test\" is standard practice to evaluate testing-time robustnes. Both RORL and DMBP used in our baselines assume only having access to a clean uncorrupted offline RL dataset as FORL, and they are then evaluated on a perturbed environment. RORL can be integrated with DMBP and FORL which we provide results for both cases in Table 2 & 4 (L1099). To the best of our knowledge, **FORL is the first work to extend testing-time robust offline RL to a non-Markovian, time-evolving, non-stationary deployment environment.** We focus on time-dependent exogenous factors from real-data which is aligned with the non-stationary environment definition.\n\n> \"If the test-time environment is a POMDP, why is the training data assumed to be from a stationary, fully observable MDP? Isn’t that an unrealistic assumption given the real-world motivation?\"\n\nOur problem formulation, which utilizes a stationary training dataset and a non-stationary test environment, is motivated by scenarios where data collection and deployment occur under different conditions. Offline datasets are often collected during periods of stationary conditions, since the goal is not to collect the dataset again and only rely on a static pre-collected dataset. Hence, data may be collected using calibrated hardware, in controlled laboratory settings, or from validated simulators. However long-term deployment can expose the agent to systematic shifts, which were not present in the training data. Furthermore we may not have access to all possible offsets evolution patterns that can occur during training, which is the focus of **testing-time robust offline RL**. Our setup enables the agent to learn the core task dynamics from the clean offline data while treating the non-stationarity as a separate problem to be solved zero-shot at test time, decoupling policy and dynamics learning from the challenge of adapting to unseen non-stationary patterns during deployment.\n\nThe key advantage of our testing-time adaptation approach is its **generalization to unseen non-stationarity**. An approach trained on a dataset that already contains specific non-stationary patterns would learn to model *those specific patterns*. If a new, previously unseen pattern of non-stationarity emerges at test-time, such a model would can fail, necessitating a costly cycle of **new data collection, retraining, and hyperparameter tuning**. \n\nFORL, by contrast, is designed to be robust to this scenario, which aligns with the general aim of offline RL where additional data collection is restricted. It learns the time-invariant dynamics of the task from the clean, stationary data. The adaptation to a different non-stationary pattern is then handled at test-time by its modular components. This makes our framework significantly more flexible and practical for real-world deployment, where the future evolution of the environment can be fundamentally unknown. We will clarify this crucial distinction in the related work section.\n\n[1]Ye, Chenlu, Rui Yang, Quanquan Gu, and Tong Zhang. \"Corruption-robust offline reinforcement learning with general function approximation.\" NeurIPS 2023\n\n[2]Zhang, X., Chen, Y., Zhu, X. and Sun, W., Corruption-robust offline reinforcement learning. AISTATS 2022\n\n> Why is the additive offset model (i.e., ot=st+bj) considered sufficient to represent the broad class of real-world observation non-stationarities? & \"episode-wise observation-offsets\"\n\nWe do not claim that the additive offset model is universally sufficient for all possible real-world non-stationarities. Instead, we propose this formulation as a critical and more generalizable first step beyond the restrictive assumptions in prior literature. Existing work models non-stationarity using predefined, often simplistic, functional forms. A common approach is to use a sinusoidal function where the episode index is the input, which implicitly assumes a long-term, repetitive pattern in the environmental changes. Our work makes no such assumption of repetition. Our framework, by contrast, conceptualizes the non-stationarity as a time-series component. Although we agree that $o_t=s_t+b_j$ is general, the offset term $b_j$ can be **large**, complex, and non-Markovian as illustrated in Fig. 3 Fig.6. We position our work as a foundational framework that is designed to scale with rapid advancements in sequential modeling. As the capabilities of foundation models improve, enabling longer-horizon predictions, the class of non-stationarities our method can effectively handle will expand accordingly.\n\n>  The forecasting component depends on a sequence of ground-truth offset values from past episodes \n\nWe thank the reviewer for raising this critical point, and we apologize if our description was unclear. We will revise the text to be more precise. \n\nThe assumption is **not that offsets become available after every episode, but after a long interval of P episodes** (e.g., P=60 in our experiments). This models scenarios like periodic maintenance checks on a robot or quarterly reconciliation of financial data, where a ground-truth signal becomes available, but only infrequently. In many deployments offsets are logged after scheduled calibration (e.g., daily robot zeroing, weekly clinical lab alignment). Hence they are **not revealed after every episode** and there is no leakage.\n\nOur framework can handle intra-episode offsets, where the offset changes every 50 timesteps, but the offsets are revealed after each episode. We share our results for this setting below across all antmaze, maze2d environments and real-data-A,B,C,D,E with 5 random seeds.\n\n| Method| Score|\n|--------------------|--------:|\n| **FORL(ours)**|**61.5** |\n| FORL-DM(ours)| 33.0|\n| DQL(Mean+Lag)|46.9|\n| DQL | 17.4 |\n\nOur framework's core contribution is **robust even when offsets are never revealed**. DM component, proposed as part of our FORL framework,generates candidate states from the sequence of action‑$\\Delta_o$, does not require access to historical offset information. As shown in our results in Figure 11 and Section F.3 (L1121), the DM (renamed to FORL-DM) module by itself outperforms DQL baseline. We will move the DM and H-LAG-DCM to the main text to make it more visible.\n\n| Method| Score|\n|--------------------|--------:|\n| $\\texttt{DM}$ (FORL)| **27.4** |\n| $\\texttt{DQL}$| 10.9|\n| $\\texttt{Med-DCM}$  | 17.7|\n| $\\texttt{Med-Noise}$| 15.5|\n| $\\texttt{H-Lag-DCM}$| 26.3|\n| $\\texttt{H-Lag}$| 17.3|\n\nThis demonstrates that our primary innovation—the within-episode state estimation based on action-effect history—is highly effective as a standalone module for the most challenging scenario where ground-truth offsets are never accessible. The full FORL framework (**54.25**), including the forecaster, is presented as an enhanced version for settings where infrequent, periodic ground-truthing is feasible. We thus provide a practical solution for both cases. Our \"zero-shot\" claim refers to the fact that the agent requires no training on non-stationary data and can adapt to unseen non-stationary patterns from the first timestep. We will clarify this distinction between the DM and full FORL settings in the paper.\n\n> How exactly is the forecasting model trained or used, given that ground-truth offset values are not observable during deployment? Where do these values come from?\n\nThe forecaster is a pre‑trained Lag‑Llama foundation model. We use the open-source pretrained model and predict the future P offsets zero-shot from the same model for all data, based on only the context length of ground truth offsets. We will make this more explicit in the main text.\n\n> The paper only compare against DQL and its variants, omitting widely-used and stronger baselines like IQL, or methods related to Decision Transformer.The modularity of the proposed method would allow easy integration with other offline RL methods, but this is not well tested.\n\nWe compared with DQL, RORL and TD3BC in our main paper. Results in DQL paper show that DQL achieves performance superior to IQL, and DT across D4RL benchmarks.\n\nDT: Antmaze and maze2d datasets require stitching, hence it is suggested in [3] to use Q-learning based methods instead. We did consider DT as a baseline, as it is a widely used and strong offline RL baseline, however our results ranging from (-4.02 to 0) confirm the near zero scores reported in [3], hence we omit DT at this time. Results below are across all real-data-A,B,C,D,E, and 5 seeds for environments maze2d-medium,large, antmaze-umaze and medium for IQL. \n\n|  | $\\mathbf{IQL}$ | $\\mathbf{IQL+MeanLag}$ | $\\mathbf{DMBP+Lag-I}$ | $\\mathbf{FORL-I}$ |\n|:---|:---:|:---:|:---:|:---:|\n| Score | 9.1 | 22.7 | 23.3 | **35.9** |\n\n[3]Bhargava, P., Chitnis, R., Geramifard, A., Sodhani, S. and Zhang, A., 2023. When should we prefer decision transformers for offline reinforcement learning?.ICLR,2024"}, {"review_id": "8FnFOZU5cJ", "reviewer": "Reviewer_x1rM", "summary": "This paper introduces Forecasting in Offline RL (FORL), a framework for enabling offline RL agents to handle a specific kind of non-stationarity at test time: unobserved observation offsets that vary between episodes but are constant within episodes. FORL uses a zero-shot time-series foundation model to sample candidate offsets for the current episode, assuming past ones are revealed after a delay. In parallel, it uses a conditional diffusion model to sample candidate Markov states given a within-episode sequence of actions and \"delta observations\" ($o_t-o_{t-1})$. The two sets of samples are combined via a \"dimension-wise closest match\" heuristic to give a final state estimate that is fed to the policy. The authors demonstrate FORL's effectiveness on standard offline RL benchmarks augmented with observation offsets derived from real-world time-series data, showing improved performance over existing methods.", "questions": "1. Clearly, the main limitation of this work is its technical assumptions: episode-constant offsets and revealing of ground truth offsets after a delay. Could you discuss the technical barriers that prevent the current FORL framework from handling continuously changing non-stationarity, and ideally bypassing the dependency on delayed ground truth offset information? What architectural changes would be necessary to tackle this more realistic and general problem?\n\n2. Given the framework's dependency on an external forecaster, have you performed any analysis on the sensitivity of FORL to forecasting errors? How does performance degrade as the forecast becomes less accurate? This would provide important context for the method's robustness.", "limitations": "Yes. Because the authors are very transparent about their restricted problem definition, I don't think a lack of discussion is an issue. However, as outlined above, I think this limitation *itself* is a major drawback of the paper and my primary reason for a lowered score.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper does a good job of formally defining a specific and challenging sub-problem within the broader space of non-stationary RL. The scope of the work is clear throughout, which allows for a focused solution and rigorous evaluation.\n- The core idea of combining a long-range, inter-episode forecaster with a short-range, intra-episode state inference model seems quite original. Using a conditional diffusion model for belief state generation based on action-delta trajectories is a technically innovative approach that is distinct from typical uses of diffusion models in RL.\n- Within the confines of the specific problem being studied (episode-constant offsets), FORL significantly outperforms relevant baselines. The use of real-world time series data to generate the offsets makes the evaluation more compelling than using simple synthetic noise.\n\nWeaknesses:\n- The paper's biggest weakness is its exclusive focus on observation offsets that are constant within an episode. This is a significant constraint that does not capture most real-world non-stationarities (e.g. sensor drift, mechanical wear, environmental changes), which typically evolve continually during a task. The entire technical approach relies on this assumption and would likely fail dramatically if it was broken. This makes the current method a solution to a highly stylised and arguably artificial variant of the non-stationarity problem.\n- Moreover, the zero-shot forecasting model makes a further assumption that ground truth episode offsets are revealed after a delay (of $P$ episodes). It is unclear why this assumption is reasonable, and it doesn't appear to be justified anywhere in the paper.\n- The method's performance is heavily dependent on the quality of the pre-trained time-series foundation model. The paper does not analyse the sensitivity of FORL to forecasting errors or to environmental dynamics that are out-of-distribution for the forecaster, which is a crucial consideration for any real-world deployment.\n- (Minor) The method's name \"Forecasting in Offline RL\" is not particularly informative as it doesn't include any reference to non-stationarity. The authors should consider a different name.", "was_revised": true, "final_justification": "The authors' detailed reply has largely addressed my concerns about relying on episode-constant offsets and the visibility of ground truth offsets after a delay. These appeared to be major limitations, but don't in fact seem to be fundamental. This new information is sufficient for me to raise my score to a 4. I suggest that the authors spend time rewriting the paper to highlight this, to prevent future readers from having similar concerns.", "rebuttal": "We thank the reviewer for their inspiring and constructive feedback on our paper, and recognizing the **clarity** of our problem formulation, the novelty of our conditional diffusion-based approach, and our **rigorous evaluation**.\n\n>The paper's biggest weakness is its exclusive focus on observation offsets that are constant within an episode. This is a significant constraint that does not capture most real-world non-stationarities (e.g. sensor drift, mechanical wear, environmental changes), which typically evolve continually during a task. The entire technical approach relies on this assumption and would likely fail dramatically if it was broken. This makes the current method a solution to a highly stylised and arguably artificial variant of the non-stationarity problem.\n\nWe sincerely thank the reviewer for raising this critical point, and highlighting its importance throughout their review. The reviewer’s insightful comment prompted us to conduct new experiments to empirically validate the generality of our framework. The challenge of intra-episode non-stationarity is indeed a crucial frontier for the field, and we are grateful for the opportunity to demonstrate our method's capabilities in this more general setting. We are pleased to report that it natively handles timestep-wise non-stationarities without requiring any modifications to the architecture or the framework. The results presented below confirm that our method remains highly effective with in-episode changes to observation offsets. We believe this demonstrates a significant step beyond prior work that relies on simpler parametric functions, as our method successfully adapts to challenging dynamics from real-world time-series data.\n\n**Performance on Timestep-Wise Additive Offsets**\n\nZero-shot forecasting foundation module can continuously generate samples, our diffusion model itself does not rely on the the forecasts of the foundation module and only tracks observation changes and actions which are invariant to the offsets, our DCM module already samples a good estimate at each timestep without requiring any hyperparameters. This shows that our algorithm can solve intra-episode offsets, we will add this to our contributions.\n\nBelow we present results for the case where the offsets change every $f$ timesteps inside the episode. We take $f=50$ for the experiments below. We observe that our framework can handle also the within episode change counterpart. It can also handle every timestep changing drifts, which we believe is a very important problem and fits not only into the sequential episodic scenario but also to the time-step evolving scenario, where the offsets become available after the episode terminates, but the agent is subject to a time-dependent non-markovian shifts within the episode. Episodic changes are usually addressed in the literature due to their tractability but we uncover that time-step wise changes is also possible which makes our framework foundational in terms of extendability. \n\n| Method             | Score   |\n|--------------------|--------:|\n| **FORL(ours)**    |**61.5** |\n| FORL-DM(ours)    | 33.0|\n| DQL(Mean+Lag)     |46.9    |\n| DQL            | 17.4 |\n\nAmong the algorithms that do not use any past ground truth offsets $FORL-DM$ and DQL,  only using the diffusion model of FORL $FORL-DM$ significantly increases performance. Integration of the forecasting FM significant increase in performance, when we have access to this information FORL as a whole obtains a superior performance, this shows that our method covers both cases when information is available and not available even when there is intra-episode changes in the offsets.\n\n> Moreover, the zero-shot forecasting model makes a further assumption that ground truth episode offsets are revealed after a delay (of  episodes). It is unclear why this assumption is reasonable, and it doesn't appear to be justified anywhere in the paper.\n\nWe thank the reviewer for this question, as it allows us to highlight a crucial aspect of our method. We would like to clarify that the assumption of receiving delayed ground truth offsets is **not required** for our method to outperform baselines.\n\nOur proposed diffusion module, even without access to any ground truth offsets, demonstrates superior performance in environments with continuously changing non-stationarities. This is empirically validated in our ablation studies presented in Supplementary Material F.3 (Figures 11). Given the importance of this finding, we will move these figures and the corresponding discussion to the main body of the paper to make this capability explicit. Below we share the numerical values of the plot in the main paper:\n\n| Method             | Score   |\n|--------------------|--------:|\n| $\\texttt{DM}$ (renamed to FORL-DM)| **27.4** |\n| $\\texttt{DQL}$                   | 10.9    |\n| $\\texttt{Med-DCM}$               | 17.7     |\n| $\\texttt{Med-Noise}$             | 15.5     |\n| $\\texttt{H-Lag-DCM}$             | 26.3     |\n| $\\texttt{H-Lag}$                 | 17.3     |\n\n\n> Clearly, the main limitation of this work is its technical assumptions: episode-constant offsets and revealing of ground truth offsets after a delay. Could you discuss the technical barriers that prevent the current FORL framework from handling continuously changing non-stationarity, and ideally bypassing the dependency on delayed ground truth offset information? What architectural changes would be necessary to tackle this more realistic and general problem?\n> I think this limitation itself is a major drawback of the paper and my primary reason for a lowered score.\n\n1.  **Handling Continuously Changing Non-Stationarity:** As demonstrated in our first response, the FORL framework is not fundamentally limited to episode-constant offsets. The primary barrier is the predictive capability of the external forecaster. To handle more complex, continuously changing dynamics over long horizons, one simply needs a more powerful time-series foundation model. The architecture of FORL itself does not need to change; it is designed to integrate any such forecaster.\n2.  **Bypassing Dependency on Ground Truth Offsets:** We wish to re-emphasize that **no architectural changes are necessary** to operate without delayed ground truth information. Our DM is inherently capable of functioning in a purely zero-shot manner, **without** leveraging the forecasts directly. As empirically demonstrated in our ablations (Figures 11 in the appendix), this configuration already outperforms existing baselines, and heuristics.\n\n>The method's performance is heavily dependent on the quality of the pre-trained time-series foundation model. The paper does not analyse the sensitivity of FORL to forecasting errors or to environmental dynamics that are out-of-distribution for the forecaster, which is a crucial consideration for any real-world deployment.\n>Given the framework's dependency on an external forecaster, have you performed any analysis on the sensitivity of FORL to forecasting errors? How does performance degrade as the forecast becomes less accurate? This would provide important context for the method's robustness.\n\nThe time-series dataset plots with mean, ground truth values are available in the supplementary, we report results in each dataset so we can see the degradation in FORL and baselines\nWhile a full sensitivity analysis is implicitly present in our results (forecasting errors for each dataset in Table 1, for their corresponding plot in Fig. 13). Our experiments are conducted across five different real-world time-series datasets, each presenting a unique forecasting challenge. Table 3 shows the comparison of prediction errors, where the $DQL-Lag-\\bar{s}$ uses just the forecasting module, which is detailed in Supplementary Material Section E. For instance for the real-data-E where the forecasts are highly inaccurate performance in all methods degrade, whereas FORL is comparatively more robust. We are currently working on alternative ways of analysis following reviewer's feedback.\n\n>  (Minor) The method's name \"Forecasting in Offline RL\" is not particularly informative as it doesn't include any reference to non-stationarity. The authors should consider a different name.\n\nWe agree and will rename our method to either \"Prospective Offline RL\" or \"Anticipatory Offline RL.\" We chose the original name to attract interest from the time-series forecasting community, as we believe time-series forecasting is highly relevant to the general problem of Non-stationary RL."}]

Author rebuttal:

Meta-review: This paper studies nonstationary offline RL in a setting where training data are stationary but deployment involves an unknown, additive offset to the state that varies across episodes while remaining constant within each episode. The authors propose a method combining a diffusion-based learned environment model with a zero-shot time-series model that forecasts future offsets from past ones. Experiments demonstrate clear advantages over strong baselines in terms of both prediction error and achieved reward.

Reviewers agree that this is an important and timely problem and that the proposed techniques are novel and empirically strong. The rebuttal strengthened the case for acceptance by providing additional baselines and experiments in OGBench. There was also constructive discussion on how restrictive some of the assumptions are. Reviewers were broadly convinced by the authors’ arguments.

A main concern shared across reviewers relates to the clarity and structure of the paper. The presentation could be streamlined, and assumptions and limitations—such as the treatment of offsets—should be explained more clearly upfront. I recommend the authors address this in the camera-ready version. Overall, I recommend acceptance.
Decision: Accept (spotlight)