Belief-Based Offline Reinforcement Learning for Delay-Robust Policy Optimization
Authors: Simon Sinong Zhan
Qingyuan Wu
Philip Wang
Frank Yang
Xiangyu Shi
Chao Huang
Qi Zhu
Venue: ICLR 2026 Poster (2026)
Keywords: Delayed Reinforcement Learning
Offline-to-Online Adaption
Primary area: reinforcement learning

Abstract: Offline–to–online deployment of reinforcement learning (RL) agents often stumbles over two fundamental gaps: (1) the sim-to-real gap, where real-world systems exhibit latency and other physical imperfections not captured in simulation; and (2) the interaction gap, where policies trained purely offline face out-of-distribution (OOD) issues during online execution, as collecting new interaction data is costly or risky. As a result, agents must generalize from static, delay-free datasets to dynamic, delay-prone environments.
In this work, we propose $\textbf{DT-CORL}$ ($\textbf{D}$elay-$\textbf{T}$ransformer belief policy $\textbf{C}$onstrained $\textbf{O}$ffline $\textbf{RL}$), a novel framework for learning delay-resilient policies solely from static, delay-free offline data. DT-CORL introduces a transformer-based belief model to infer latent states from delayed observations and jointly trains this belief with a constrained policy objective, ensuring that value estimation and belief representation remain aligned throughout learning. Crucially, our method does not require access to delayed transitions during training and outperforms naive history-augmented baselines, SOTA delayed RL methods, and existing belief-based approaches.
Empirically, we demonstrate that DT-CORL achieves strong delay-robust generalization across both locomotion and goal-conditioned tasks in the D4RL benchmark under varying delay regimes. Our results highlight that joint belief-policy optimization is essential for bridging the sim-to-real latency gap and achieving stable performance in delayed environments.

University of Southampton Published as a conference paper at ICLR 2026
BELIEF-BASED OFFLINE REINFORCEMENT LEARNING FOR DELAY-ROBUST POLICY OPTIMIZATION
Simon Sinong Zhan
Qingyuan 2u
Philip Wang'
Frank Yang'
Xiangyu Shi
Chao Huang? Qi Zhu
Northwestern University
ABSTRACT
Offline-to-oni to-oiine deployment of reinforcement learning (RL) agents often stumbles over two fundamental gaps: (1) the sim-to-real gap, where real- world systems exhibit latency and other physical imperfections not captured in simulation; and (2) the interaction gap, where policies trained purely offline face nut-of-distribution (OOD) issues during online execution, as collecting new interaction data is costly risky. As a result, agents must generalize from static, delay-free datasets into dynamic, delay-prone environments. In this work, we propose DT-CORL (Delay- Transformer belief policy Constrained Offline RL), novel framework for learning delay-robust policies solely from static, delay-free offline data. DT-CORL intro- duces a transformer-basee belief model to infer latent states from delayed obser- vations and jointly trains this belief with a constrained policy objective, ensuring that value estimation and belief representation remain aligned throughout learning. Crucially, our method does not require access to delayed transitions during training and outperforms naive iistory-auumeeter baselines, sfate-of-the-aat delayed RL methods, and existing belief- based approaches. Empirically, demonstrate that DT-CORL achieves strong delay-robus generalization across both locomotion and goal-conditioned tasks in the D4RL benchmark under varying delay regimes. Our results highlight that joint belief-policy optimization is essential for bridging the sim-to-real latency gap and achieving stable performance delayed environments.
INTRODUCTION
woald autonomy, ranging from embodied assistants to autonomous robots (Jiao et al. 2024 Wang et al., 2023a 66. Wei al. 2017 Zhan et al., (2024), routinely faces observation and actuation delays, where sensing lags and control commands arrive late due to computation communica- tion (Cao et al. 2020; Mahmood et al. 2018 Sun et al. 2022). Such delays violate the Markov assumption and
severe performance degeneration. Existing solutions often approximate the
problem with Delayed Differential Equations (DDEs) (Bellen & Zennaro 2013 Zhu et al. 2021 or augmented MDPs Altman & Nain 1992).
Collecting online interaction data under delayed dynamics is unsafe, costly, or impractical (Li et al. 2023 Yang al.) 2024 Zou al. 2015). In contrast, many systems already provide large delay- free datasets from simulators or legacy controllers (Fu et al.] [2200] Mu et al., [202]). which omit deploy latencies. This mismatch is especially pronounced various applications: simula- tions and idealized hardware logs (Makoviychuk et al. 2021 Krishnan et al.) (20TT) are delay-free, whereas real platforms, such as drones, warehouse manipulators, cloud-robot systems, or even high- frequency trading pipelines, inevitably suffer delays from computation and communication (Gupta) Chow (2009) 'Tang et al., 2025). Since collecting trajectories with realistic delays is infeasible, the key challenge is to leverage delay-free data while ensuring robustness to delayed dynamics at deployment. Consequently, we investigate the following question:
"Equal contribution, Contact Emails: Sinom/2han222888euurtooowoe http://wssssseeccccccceeetteeeee @phh@nonntwesttem..di

---

Published as a conference paper at ICLR 2026
How can we train robust policy uffli only pre-collected, delay- free trajectories- that it performs reliably when executed in a delayed real-world environment?
This question extends to two pressing needs: (i) leveraging static offline data without further interac- tion with the environment (offline RL), and (ii) robustly coping with latency at online deployment (delayed RL). Achieving both simultaneously promises safer development cycles, reduced sample complexity, and smoother sim-to-real transfer for latency-sensitive robotic and autonomous systems.
Delayed RL arises when perceptual or actuation latencies break the Markov property, forcing agents to reason over the gap between true system states and delayed observations. Existing solutions fall into two categories. State augmentation methods (Kim et al., [0033] Liotet et al. 2022 Wu et al. 2024b a extend the state with stacked histories, but incur severe sample inefficieccy as dimensionality grows. Belief-state methods (Chen et al. 2021 Karamzade et al. 2024; Wu et al. 2025) compress histories into latent beliefs, yet are prone to compounding prediction errors and distribution mismatch when deployed. In parallel, offine RL (Levine et al. 2020) trains policies from static datasets without online interaction, but struggles to balance conservatism against effective generalization, often yielding cautious or suboptimal behavior. When delay handling is combined with offline training, these difficulties intensify. Naive augmentation produces artificial delay distributions that poorly covered by static data, leading to extreme sample inefficiency and dimensional blow- up. Mcanwhile, policies and belief models trained separately offline lack corrective feedback: at deployment, even minor distribution shifts force the policy to query unseen date-action pairs, where the frozen belief module must extrapolate. Errors then accumulate step by step, compounding over long horizons and causing rapid performance collapse.
In this work, we propose DT-CORL (Delay-aware Transformer-belic Constrained Offline RL), a novel belief-based framework explicitly designed tackle the compounded challenges arising in the offline delayed RL setting. By employing a transformer to infer compact belief states, DT-CORL reformulates policy-constrained offline learning in delayed MDPs as standard MDP optimization, enabling rebust policies to be trained directly from delay-free data. This end-to-end formulation both improves sample efficiency and belief model expectation during training. thereby sidestepping the Empirically, comprehensive experiments on the D4RL suite (Fu et al. (2200) across various delay scenarios-inclue varying delay lengths, deterministic delays, and stochastic celays-consistently confirm the superiority of DT-CORL over baseline methods, encompassing SOTA online delayed RL methods, augmented-star approaches, and belief-bbsed methods (i.e., direct integration of pretrained belief models with offline RL algorithms).
and compounding-
issues that plague original belief-state pipelines.
In Sec. 2] we review related literature. In Sec. 3 we introduce the necessary background and assumptions on delayed Markov decision processes (MDPs). In[Eq. (1) we explicitly define the offline delayed RL problem and the corresponding policy-iteration constrained optimization formulation in the augmented delayed MDP. Sec. 4. we provide theoretical connection between the above policy optimization formulation and belief-based constrained policy iteration. Then, in Sec. 4.2] we present our proposed DT-CORL framework, illustrating practical implementation details of our algorithm. Comprehensive empirical evaluations are presented in Sec. 5] Finally, we conclude with a discussion of findings and implications in Seec. 6
2 RELATED WORK
Delayed RL. Delayed reinforcement learning arises in domains such as high-frequency trad-
ing (Hasbrouck & Saar 2013) and transportation (Cao et al. 2020). While reward delay has been extensively analyzed (Arjona-Medina Medina et al. 2019 Han al. 2022 Zhang al., 2023). we focus on the more challenging observation/action delay. Existing solutions follow two main strategies. Augmentation-bassed methods restore the Markov property by stacking the past A actions (and sometimes states), then learning policies in the enlarged state space. Examples include DIDA (Liotet) cet al., 2022), DC/AC (Bouteiller et al.) 2020), ADRL and BPQL (Kim et al. 2023 Wu et al. 2024b), which bootstrap from small-delay tasks, and VDPO Ww et al. 2024a),, which frames delayed control as a variational inference problem. Their weakness is structural: the augmented dimension grows linearly with A, causing sample inefficiency and poor scalability. Belief-based methods instead compress histories into latent states and act in the original space. DATS performs Gaussian filter- ing (Chen et al. 2021), D-Dreamer builds recurrent world models (Karamzade et al.) 2024), D-SAC
2

---

Published as a conference paper at ICLR 2026
uses causal- transformerr attention (Liotet et al. 2021). and DFBT applies sequence-tor -e-sueeece transformers to reduce compounding error (Wu et al. (025). These approaches are more compact, but still accumulate belief error over long rollouts and face distribution mismatch when deployed. All of the above methods assume online interaction with a delayed environment. Aside from some imitation-style studies that pre-train on delay-free logs and then fine-tune online (Liotet al. [022) Chen et al.) (202I), we are unaware of any work that learns delay-robust policies offline using only delay-frre data.
Offline RL and Online Adaptation. Offline reinforcement learning seeks to train high-performing policies from fixed datasets without further interaction (Shi et al., 2026), addressing safety and data- collection constraints in domains such as robotics, healthcare, and recommendation systems (Levine) ct al. 2020). Early approaches emphasized value conservatism instance, CQL (Kumar et al. 2020 and IQL (Kostrikov et al. 2021) penalize Q-values of out-of-distribution (OOD) actions to prevent overestimation. Complementary policy-constran methods such as BRAC (Wu et al. 2019) and TD3,+BC (Fujimoto & Gu (021) regularize the learned policy toward the behavior policy to remain within dataset support. While these strategies reduce OOD failure, they can still yield suboptimal policies if the dataset lacks trajectories near the optimum. To address this, offlinetoo- online (hybrid) methods bootstrap from an offline policy and fine-tune with limited online data, e.g.. AWAC (Nair et al. 2020), Conservative Fine-Tuning (Nakamoto al. 2023), and Hy-Q (Song et al. Other work seeks to bridge mismatches between simulators and the real world (Feng et al.
2022).
Nuu et al., 2022, Tiboni et al. 2023), but none address the equally important challenge delay gaps when offline data is delay-free while deployment involves delayed dynamics. This challenge echoes classic control theory, where time-delay compensation has been studied for decades. The
2023
Smith Predictor (Smith) 1957). for example, optimizes controller a delay-free internal model and predicts the "present" plant output to counteract runtime delays (Normey- Ricoo & Camacho 2007]
Grimholt & Skogestad 2018). Such techniques are widely used in industry precisely because they leverage delay-Iree models to operate reliably under delayed dynamics (Thomas et al., 2020 Mejia ett al., 2022] Moraes et al., 2024). In similar spirit, our setting seeks to learn from delay-free offline data while ensuring robustness at deployment in delayed environments.
PRELIMINARIES AND PROBLEM FORMULATION
MDP. An RL problem is typically formulated as a finite-horizon Markov Decision Process (MDP), defined by the tuple A, H, 7, ) (Stton et al. 1998). Here, S denotes the state space, A the action space, P:S A S > [o. 1] the probabilistic transition kernel, r S A - IR the reward function, H the horizon length, 7 E (0, 1) the discount factor, and Po the initial state distribution. At each timestep 1, given the current state 8, S, the agent selects an action d, ~ T (-(8xt) according to policy S x A + [0, 1J. Subsequently, the MDP transitions to the next state Sc+1 P(-|st, az). and the agent receives a scalar reward r; ;= r(st. We further introduce several mild assumptions
commonly adopted in the RL literature (Liotet et al. 2022 Rachelson & Lagoudakis
2010):
Definition 3.1 (Lipschitz Continuous Policy (Rachelson & Lagoudakis 2010)). A stationary Markovian policy ι is LLC if for all $1, 82 E.S.
Wi (T(-1s1), [(-ss2)) S L ds(s1, 82).
Definition 3.2 (Lipschitz Continuous MDP (Rachelson Lagoudakis 2010)). An MDP is (Ip, T.x) Lipschitz Continuous (L.C) if for all (s1,a1), (82, a2) E $ A.
Wi (P(-)s,, a1), P(-102 a2)) < Lp (s), 82) + d (a),a))),
|r(1) a1) - r(22, 0z) S LR (ds 52) + du az)),
where Wi is LI-Wasserstein distance. Definition 3.3 (Lipschitz Continuous Q-function (Rachelson & Lagoudakis] 2010)). Consider an (Lp, -L) MDP and L -LC policy If the discount factor y satisfies 7Lp (1 + Ln) < 1, then the action-value function QF is Lq-Lisschitz continuous for some finite constant Lq > 0.
Delayed MDP. A delayed RL problem can be reformulated delayed MDP with Markov prop- erty based on the augmentation approaches (Altman & Nain 1992). Assuming the delay being A, the delayed MDP is denoted as tuple (L, A, PA. PA),, where the augmented state
3

---

Published as a conference paper at ICLR 2026
space is defined X' := S A (e.g., augmented state 2; {S--A. aL-A.
am-1}
X), A is the action space, the delayed transition function is defined as Pa (2 := P(s--A+1 at-a)6,, (a/) I Sa._, (a/_.) where 8 is the Dirac distribution, the delayed reward function is defined as r Ft, a+) = (e,))) [r(s,, az)] where is the belief function de- fined as 6 (5t == JS+ I P( P(sr-a+i+||s. A+i. O2- A (---++))s((A+iie the initial augmented state distribution defined as PA - p II :- Noted that delayed RI. is not necessarily only observational delay, there could be action delay well. However, it has been proved that action- delay formulated delay RL problem is a subset of observation-delay RL problem (Katsikopoulos & Engelbrecht] 2003). Thus, without loss of generality, we only consider the general observation delay in the remainder of the paper.
Problem Formulation. We consider the offline delayed RL setting in which the agent learns solely from a pre-collected static dataset, D {(s}, 0i, ri. s -..... ,K}, consisting of K trajectories collected by the behavior policy y in delay-free environment (e-g., a simulator or other idealized demonstration system). At deployment, however, the policy operates under delayed feedback. where observations and rewards arrive after either a fixed latency of Á steps, or variable latency bounded by A, which we treat as an effective fixed delay-a common simplification in control settings. In offlin learning under this delayed MDP formulation, the agent must construct augmented atatenaction pairs from the delay-free dataset DD enabling training in the delayed setting without new environment interaction. However, standard offline RL methods suffer from out-of- distribution (OOD) generalization issues when the learned policy queries actions not well-covered in D (Levine et al. 2020). To address this, policy-constrained approaches such as BRAC (Wu al. 2019) and ReBRAC (Tarasov et al., (023a) enforce similarity between the learned policy and the dataset's behavior policy, often by bounding divergence measure D(T,, jt). This motivates the policy-constrained learning objective under the delayed MDP:
Q arg min E (x,,,,')~D [(QA - (r) + 7 Qa
Qs(r"
(1)
k++ arg max E (BNTT (+1) (z, a]] s.t. D(TA,PA)
(2)
na
Here, Q3 is the action-value function defined over augmented states, and μr is the dataset's behavior policy lifted to the augmented space. The constraint margin controls the allowable divergence between the learned policy Ta and μa, mitigating OOD queries in the offline setting.
OUR APPROACH
We now present the DTTCORL, a novel offline RL framework for adapting online delayed feedback. DT-CORL integrates transformer- based belief modeling with policy-constraieed offline learning to address the challenges outlined previous sections. This section describes how we construct delay- compensated belief representations, train policy using belief prediction, and incorporate policy regularization to ensure effective deployment under delay. Specifically, we introduce a belief-based policy-iteration framework that infers latent, selyy-compensated states through a belief function, providing a compact and semantically grounded alternative. Then, we provide detailed algorithmic implementations. We further support this approach with learning efficiency discussion to explain its efficiency benefits compared with the augmented approach.
4.1 BELIEF-BASED POLICY ITERATION
While the augmented-sten formulation restores the Markov property in delayed MDPs, applying policy iteration in this space introduces key drawbacks, particularly under the offline setting. First, the effective state dimension grows from (S to ||||A|,, increasing sample complexity and demanding significantly more data for reliable learning. Second, augmented trajectories reconstructed from delay-free datasets may not reflect the true delayed trajectories possibly happened online, leading to distribution mismatch and unstable value estimates. Lastly, treating action-history sequences as part of distinct augmented state inputs potentially discards some possible temporal order info, reducing sample efficiency and increasing overfitting risk. Therefore. trying to mitigate above problems, propose our belief-basee policy iteration framework, which essentially cast original augmented state space problem back into original state space via belief estimation preserve temporal alignment
4

---

Published as a conference paper at ICLR 2026
introduce by potential online delay. Following the BRAC framework (Wu et al. 2019), first convert previously-dete constrained optimization to the unconstrained formulation for ease of optimization, where an and 02 are regularized constant.
arg min B (' Q2
[ (z, a) (a, + 7 o (112) (2', -a1 P(aA.
k+1 arg max EEWD (awna()))) ( (x,
- (oI a)]
PA)
Moving forward, we need to map the delayed policy Ta and its Q function Q% back to the delay-free counterparts and Q via the belief distribution ba (s z) introduced Sec. 3 This requires (i) quantifying the performance gap between Ta and its belief-induced policy ग, and (ii) relating the augmented value Q% (ar, a) to the (*, a) with å ~ ba (-1z). Noted, throughout the remainder of this section we measure these discrepancies with the 1-Wasserstein distance (Villani et al.) 2008).
Lemma 4.1 (Delayed Performance Difference Bound (Wu et al., (024b)). For policies wa and n with A < A. Given any x E X, Qf Arr is LQ-LC, the performance difference between policies can be bounded as follows:
E (
[Var (x-) Qu. (27, a)) S LQ
E (1mba(112)
( (12)))
Lemma 4.2 (Delayed Q-value Difference Bound Wu et al. A < A. Given any X, Qa= is Lo-LC, the corresponding Q-value difference can be bounded as follows:
2024b)). For policies T and n. && with
L2Q
E Glke)
12- -2s(z,a) 1-7
E 2 (obbb())))
(
a'mPallza
(12)
Above two lemmas provide the exact quantification required. Moving along, we can convert iT and Q back to π and Q². And, we can arrive the new policy-iteration framework in the original state space bridging by the belief function ba as follows, where λ₁ and X are constants and μs is the behavior policy after data augmentation. Specific derivation from augmented offline PI to belief-based in Eq. (3) and Eq. (4) can be found in App. B.I
Q + arg min E ((,,)>>>
E (Es (8,
Q
E (a (3a-ba(1o(/((s.0))) + 2 E - -A Wi(T,, Wi(( a'wa(113')
(3)
(5, X2
arg max BRDD
Ba-bal-ja) aw7(-18)
(4)
Proposition 4.3. Let the policy before and after the update in Eqg. (4) be wold and nnew- Then after each policy evaluation Eqq. (3) we have Ragnsnew (Q Rnew (s, a) 2 BaNoios (Qrate (8, (s,a).
Above proposition proves that iteratively applying Eq. (3) and Eq. (4) will monotonically increase Q value after update. Detailed proof can be found in App. B.2
Remark 4.4. For deterministic MDP, ba is also deterministic, meaning that Eg-da (-1x) [r(8, a)]
r' (1t a) and (1))))))))) IQ" (8, a)] = Q ax. Since ba becomes an injection mapping under this setting, Eg- (-Ia) [r(8, a)] = r(s, a) should also hold. Thus, we can directly leverage existing offline delay-free data tuples for update, except for the policy related terms.
A seemingly simpler alternative is two-stage pipeline: (i) train belief model offline; (ii) freeze and apply any delay-free offline-RL algorithm to the original delay-free samples, deploying the resulting policy with that fixed belief function. This strategy, however, suffers from several drawbacks that DT-CORL avoids. (i) Biased value targets. In DT-CORL, the Bellman target in Eq (3) is computed through the belief ba ( (-12r), so the critic learns exactly the latent states the policy will later encounter. By contrast, the stagstgee method treats the filter as error- error-free; residual belief error becomes unmodelled noise, forcing the critic to fit moving target and biasing the learned Q-values.
5

---

Published as a conference paper at ICLR 2026
Offline Training
Ballef Pretrain
Online Adaption
Ballef Pl
S
Delayed Feedback
Agent
Transformer Belief
Prediction
Delayed Environment
Trajectory Buffer
c-+a+
Actor
Critic
O Dog
at
$
Policy Inference
Figure I:
Overall pipeline of DT-CORL. In the Offline Training phase, trajectory data are augmented to train the transformer belief, and with the trained transformer belief, we conduct belicf-based PI in the offline setting. In the Online Adaptation, we utilize the transformer belief to predict the current state from delayed observation, and adapt with offline-traine policy.
(ii) Distribution mismatch. Because the critic of the two-stage baseline is trained under (s, ~D while the policy, evaluated through the frozen belief, quickly drifts to new action choices, the training and deployment distributions diverge. This mismatch is magnified in delayed environments and leads to sub-optimal returns, phenomenon confirmed in our experiments (see Sec. 5). (iii) Inferior sample efficiency in stochastic MDPs. For stochastic dynamics, each offline tuple (s, a, r, s') reveals only a single next state, whereas DTTCORI reuses the same transition to generate A latent states along the helief rollout, extracting richer supervision. Similar gains from "multiple synthetic samples per transition" have been observed in model-base offline RI. (Kidambi et al. [2000] Yu et al., (2020). In short, embedding belief prediction inside the policy-iteration
loop both contains model error and keeps the critic, policy, and belief model aligned-benefite benefits frozen two-stage pipeline cannot match, in addition to avoiding the dimensional explosion of raw state augmentation.
Remark 4.5. Existing work has compared the sample-complexin of augmentation- and belief-based methods in the online delayed-RL setting (Wu al., 22024 b). The same intuition should carry over to offline learning, but a formal analysis would require assumptions-4 e.g., uniform data-coverage, linear value realizability, and bounded rewards- that are standard in statistical offline-RL. theory (Bai) et al., 2022 Shi et al.) [022] Xie et al. (202T) yet do not hold in our setting. Establishing tight offline bounds without these restrictive conditions therefore an open problem that we leave for future.
4.2 PRACTICAL IMPLEMENTATION
Belief Prediction. The belief prediction can be taken dynamic modeling problem. With a given offline trajectory ((s,. dz,, Tt, 1t) ) we can manually create the augmented state by stacking states and actions in A steps from start to end, where H. (,t-,,,,- (,-a;,A, A, 0t- A+1.".. at-1) and the true state is st. Thus. the problem becomes training a belief function ba which takes in 2, and predict s, To reduce the potential compounding loss from belief prediction, we employ transformer structure (Vaswani et al. 2017:
{ Wu et al. (025), where with the given augmented state transformer predicts sequence from & & A+1 to St. Based on the deterministic or stochastic nature of MDP, the transformer belief is updated with either MSE or MLE objectives. We validate the choice of transformer architecture (See. 5.3
Belief-based Policy Optimization. Although many offline RL methods regularize the policy with KI., MMD, or Wasserstein distances (Levine et al., 2020 Wu et al., 2019), computing these divergences exactly is costly. Following the pragmatic approach of TD3 +BC and ReBRAC (Fujimoto) & Gul 20221; Tarasov et al. 2222a),, we approximate the policy-behavior divergence by the mean- squared error between actions sampled from the learned policy and the augmented behavior policy. The policy-improvemmnt step therefore, becomes
k+1 arg max E(Z,))D Es (122)
[Ô=* (8, - -alla -alla-0f -
(
where 0 trades off exploitation and conservatism. This surrogate avoids training an explicit delay-augmentee behavior model while retaining a simple quadratic penalty that is trivial to compute in continuous control. Besides, for deterministic MDPs the immediate reward r(s,, 0r) can be

---

Published as a conference paper at ICLR 2026
Table 1: Normalized returns (%) on D4RL AntMaze tasks under deterministic and stochastic observa- tion delays A [4. 8, 16). Results are averaged over 3 seeds. Best per column (including ties) is shown in bold with a light blue background.
Setting
Method
umaze
umaze deeeee-iiveee diverse
mediumm mediuu-ppeyy
large-play
4
8
16
4
8
16
4
8
16
4
16
DEPT-SAC
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Augmented-Bo
60.7
62.3
31.0
69.0
58.7
30.0
0.0
1.67
1.0
0.0
0.3
0.0
Deterministic
Augmented-COU
72.3
44.0
12.7
27.7
23.7
22.0
0.33
1.67
4.67
0.0
1.33
0.67
Augmented-COMMe COMBO
76.0
37.0
13.3
26.0
19.0
23.0
6.33
5.67
3.0
0.0
1.33
1.0
DT-CORL
83.3
76.7
40.0
65.3
62.0
32.0
1.33
2.33
2.33
0.0
0.33
0.67
DBPT-SAC
0.0
0.0
0.0
0.0
Augmented- BC
60.3
55.7
24.7
59.7
51.7
40.7
1.33
2.33
2.0
0.0
0.67
0.0
Stochastic
Augmented-CC
61.3
37.0
12.7
17.3
9.67
1.33
0.33
Augmented COMBO
64.7
36.7
13.3
32.7
15.0
12.7
8.67
7.67
3.67
0.67
1.33
1.0
DT-CORLL
88.3
67.3
58.3
56.0
2.00
2.67
3.33
0.0
1.67
read directly from the dataset, so separate reward model is required (cf. [gq. (3) Remark 4.4).
Eq. (4)
and
Online Adaptation. At deployment time, we maintain circular action buffer of length A, initial- ized with random actions drawn from A. At timestep & the agent receives the delayed observation or from the plant, appends the most recent action a;-1 to the buffer, and forms the augmented input It for, ae-2...., Q--1)- This sequence is passed to the trained belief transformer, which returns a
state estimate $,. The policy (- {y) then selects the next action a. During the first A steps (and symmetrically near episode termination), the buffer is not yet full. We handle these boundary conditions by inserting a special [MASK] token for missing actions and enabling the transformer's built-in masking mechanism, ensuring consistent state prediction throughout the episode. Detailed neural network structures and hyper-parameters can be found in App. D
5 EXPERIMENTS
To evaluate the effectiveness of our approach, we conduct experiments on the standard D4RL benchmark suite, including both MuJoCo locomotion and AntMaze goal-conditioned tasks. Our experimental analysis highlights two primary advantages of DT-CORL. First, we demonstrate that our method significantly outperforms both traditional offline RL algorithms with state augmentation adaptation and the SOTA Online delay-robust RL algorithm. Second, show that DT-CORL surpasses hybrid approaches that combine separately trained delay-belief models with existing offline RL algorithms, underscoring the benefit of belief-involved policy evaluation. In our ablation studies, we examine how trajectory availability impacts performance and compare alternative belief architectures, including ensemble MLPs and diffusion--ased predictors, validate the choice of the transformer-basee belief model.
5.1 ANTMAZE
We benchmark D'TCOORL on the AntMaze goal-come coal-conditioned tasks from the D4RL offline RL suite Fu et al. 2020 Todorov al. 2012). Since existing method is designed to learn delay-robust policy solely from delay-free offline data, we construct two baseline methods for comparison: (i) Augmented- BC, which applies standard behavioral cloning Torabi al. 2018) in a A-step augmented state space; and (ii) Augmented-CC which runs Conservative Q-Learing (Kumar et al. (2020) on the same augmented state representation. Additionally, we compare DT-CORL against model-based offline (Yu et al. (022)) and the state-of-the-an art delay-robust RL algorithm DBPT-SAC (Wu et al., 2025). We evaluate all methods on four standard AntMaze environments: medium large-play, stochastic delay settings, we test on 4, 8, and 16 steps of delay respectively. For stochastic delay, it means that the delay at each step follows a uniform distribution, A E U(11 k), k E {4,8, 16). From Table I, online delayed RL methods such as DBPT-SAC collapse under the offline setting, confirming online methods' incompatibility with offline setting. In umaze and umaze-di seee-iiverre se, DT-CORL consistently outperforms all baselines and shows far less degradation as delay length increases, whereas augmentation-basee methods deteriorate sharply. On the harder medi i ammple play ay
RL baseline, Augmented-COMPE
umaze-diverse
and umaze. For both deterministic and
7

---

Published as a conference paper at ICLR 2026
Table 2: DT-CORL vs. belief-based baselines (Belicf-CQL, Belief-IQL) on D4RL MuJoCo tasks.
Normalized returns (%). Delays: deterministic A E f4, 8, 16), stochastic A ~ U(1, k), & {4. 8, 16). Best results per column are shown in bold with light blue background.
Task
Method
medi um
med expert
med med-replay 8
expert
16
4
16
16
4
8
16
Deterministic Delays
IQL
5.3
5.8
4.8
5.5
6.4
5.3
5.8
6.4
2.4
4.5
9.0
5.6
Hopper
CQL
8.2
4.3
7.3
5.9
4.4
6.1
7.9
3.9
Belief-IQL
27.7
25.4
26.7
24.7
24.7
25.5
23.4
18.7
17.4
Relief-CQL.
75.4
56.8
92.9
39.5
35.2
110.1
99.7
96.6
43.3
45.9
DT-CORL
79.4
85.0
71.8
113.0
112.2
109.9
99.4
100.8
100.
112.9
113.1
112.2
Belief-IQL
30.8
10.6
5.3
24.8
6.1
3.3
23.3
13.8
9.7
6.8
4.8
3.6
HalfCheetah
Belief-CQL
49.2
22.7
1.5
36.1
14.4
6,4
1.5
1.5
DT-CORLL
47.4
27.8
6.4
44.7
8.7
43.6
27.1
7.9
20.6
5.1
Belicf-IQL
33.4
25.7
24.6
49.6
17.3
16.4
28.2
20.6
18.3
25.5
19.9
16.7
Walker2d
Belief-CQL
87.0
64.1
39.2
105.8
99.5
51.0
93.3
93.5
61.0
111.1
110.8
97.7
DT-CORL
87.4
86.8
112.1
112.0 118.1
93.6
90.5
88.1
110.9
111.2
110.5
Stochastic Delays
IQL
9.4
10.4
4.0
2.9
11.4
3.1
Hopper
CQL
8.9
4.3
11.1
7.4
6.0
11.5
6.1
5.3
9.1
6.9
3.4
4.2
Belief-IQL
27.4
27.4
28.4
28.3
27.9
23.8
25.0
24.5
26.0
18.6
16.7
16.9
Belief-CQL
80.8
67.8
74.3
91.8
48.9
57.5
100.8
99.8
99.2
72.3
35.4
20.1
DT-CORI.
78.5
72.1
79.3
113.6
112.7
85.4
99.4
100.1
98.8
113.2
112.8
113.1
Belief-IQL
33.4
31.2
21.6
31.1
16.3
8.2
24.9
20.0
15.3
13.1
9.2
4.3
HalfCheetah
Belief-CQL
52.6
46.6
16.2
48.0
22.1
3.6
47.1
41.4
19.7
6.5
2.8
1.7
DT-CORI
48.2
47.5
38.4
70.0
44.3
31.7
47.7
43.3
30.4
85.1
12.7
5.8
Belief-IQL
34.6
36.7
49.4
25.0
20.4
31.4
25.4
24.2
43.1
27.7
20.7
Walker2d
Belief-CQL
78.9
112.1
85.1
95.3
111.1
109.6
DT-CORL 86.8 87.4 87.0
114.1
113.6
111.5
93.0
90.9
91.8
110.9
110.9
110.5
and large-play tasks, overall performance for all methods remains low; the poor results across methods suggest that additional goal-conditioned modifications may be required. Overall, DT- CORL demonstrates stronger robustness to increasing delays than augmentation-base
baselines. We also observe that augmentation-bass based methods degrade more under stochastic delays, as the additional randomness introduces variance that destabilizes policy learning. In contrast, our belief- based approaches exhibit the opposite trend: the reduced sample complexity and improved temporal prediction allow them benefit from the effectively shorter delays under stochastic settings. We further justify our method's superiority under various environments in additional MuJoCo tasks. Detailed results can be found in Table and Table
5.2 BELIEF-BE -BSSED COMPARISON
To justify the effectiveness of incorporating belief estimation in offline policy optimization, we compare method DT-CORI with the following basclines: (i) Offline RL, naive implementations of CQL and IQL without any delayed adaption. (ii) Belief-CQL, which feeds the CQL algorithm the transformer belief used by DT-CORL instead of raw augmentation. (iii) Belief-IQL, which runs Implicit Q-Learning Kostrikov et al. (2021) with the same delayed belief above. We test all methods on D4RL MuJoCo suite following the medi um, medi um temluummxiii weeeerr-epeett medi um- rmm-eelay. and expert trajectory setting. For both deterministic and stochastic delay, we adopt the setting described in the previous subsection. From Table 2] we can tell the clear performance gap between other naive aalief-based methods and DT-CORL, and the ineffectiveness of naive CQL and IQL under our delayed setting. Belief-IQL. relies heavily on implicit Q-learning updates, which are sensitive to inaccuracies in the latent belief state. Specifically, IQL's actor update weights actions by w = exp( exp(A/A), and under delayed/partial observations the same belief state carries noisy Q V; tiny errors explode after the exponential, producing unstable, off-support policy updates. Thus, without alignment between belief learning and policy training, small belief prediction errors can lead large value overestimation
underestimation. which end up with poor performance of Belief-IQL across all the scenarios. Belief-CQL. the Q-function is trained on the precomputed belief embeddings. This decoupling leads to suboptimal value estimates, especially under long delays
8

---

Published as a conference paper at ICLR 2026
Figure 2: Step-by-step detailed comparison of prediction accuracy for different models.
Model
Total Parameters (M)
Inference (ms)
Ensemble
1.80
69. 09.50:22.0 86
Transformer Dittusion
7.87
22.27:19.02
3.96
665. 665.21+35.5
Dillusion (fine:: 0.157
Transtormer ( (63)
Table 3: Comparison total parameters (M) and inference speed (ms) for each model up 16 steps.
Data
DI-CORL
Belief-CQL
Aug-CQL
MIG
25%
9.8
3.08
2.47
50%
12.0
3.80
2.80
75%
15.3
4.05
3.89
100%
27.8
8.90
3.80
Table 4:
Performance DT-CORL un-
der different data availability levels for the Ha Selffheettah-med um-v2.
Timestep
where belief inaccuracies compound. This tendency can be spotted across various tasks with different trajectory settings. DT-CORL aligns the belief prediction with downstream policy evaluation and optimization, allowing both the transformer and Q-function to co-adapt under delayed feedback. Besides, by conditioning the policy and critic on belief-derived latent states, DT-CORL mitigates the delay-induced distribution shift more effectively than purely augmenting state spaces or pretraining belief models.
5.3 ABLATION STUDIES
Why Transformer? In this subsection, we justify our choice of a transformer- belief model by comparing it against two common enseembl MI.ss and diffusion models. We evaluate all models in the Hopper-medde medium setting, measuring (i) multi-step prediction accuracy (MSE up to 16 steps; Figure 2) and (ii) inference speed and parameter count (Table 3). All models trained for 100 epochs on the same medi um dataset and evaluated in the environment; implementation details appear in AAp.. D Results (See Figuure and [Table 3) highlight clear trade-off: ensemble MLPs are lightweight but accumulate eтror rapidly over long horizons, degrading belief quality; diffusion models achieve strong accuracy but incur high computational cost due iterative denoising. In contrast, the transformer model provides the best balance, offering strong multi-step accuracy, stable long-horizon predictions, and fast inference suitable for online deployment. Additional qualitative prediction visualizations are included in App. C.I In addition, we further analyze the impact of transformer size on sample efficiency and prediction accuracy. Detailed experiments can also be found in App. and Table 9
Trajectories Availability. We next examine how offline data availability influences the performance of DT-CORL and its basclines. Using the Hal CCheetal ah aa-madiumme diumvv2 environment with fixed delay of 8 steps, vary the proportion of the offline dataset available for training: 25As shown in Table 4 all methods improve with more data, but DT-CORL consistently achieves the highest returns across all data levels. Notably, the performance gap between DT-CORL and the belief-based or augmentation-basee baselines widens as more data becomes available, while those baselines show only mild gains. These results indicate that DT-CORL not only maintains superior performance in low-data regimes but also scales more effectively with additional highhighhig the benefit of jointly learning the belief model and policy within a unified framework.
Joint Training Benefits. To empirically validate this coupling, we compare DT-CORL against variant that uses a separately pretrained belief model with no further adaptation during policy learning. As shown Table 5] joint training consistently outperforms separate pretraining across all delay settings in the Hopper suite siiee-yeeling significantly higher returns at both small and large delays. This result confirms that joint optimization reduces fifline-to-oningg distribution mismatch and enables the belicf model to specialize to the value function's 's error landscape, providing substantially more stable delayed-polie policy learning.
Delay Robustness.
To assess DT-CORL's reliability under challenging latency conditions, we evaluate two robustness settings: (i) scaling long-horizon delays (32-64 steps), and (ii) testing

---

Published as a conference paper at ICLR 2026
Table 5: Joint vs. Separate Belief Training
Average Performance Across Tasks
Beliekol BulieFCQL
Training Mode
16
60
Separate DI-CORL
91.7
814//774
6 73.1
Nomaazoo Bemum
102.8/99)
/
40
Table 6: DT-CORI, Across Delay Distributions
20
Method
Uniform
Gauss
Exp
Binom
DT-CORL
79.3
82.1
85.8
77.4
16
32
64
Delay Steps
Hond Pettrraace plou
US00t..
(a) Average performance
(b) Pen rotation
(c) Open door
(d) Hammer smash
Figure 3: (a) Describes the average performance of Aug-CQL, Belief-CQL, and DT-CORL across
three dexterous hand manipulation tasks (a)-(c) under various delay setting ranging from 4 to 16.
under alternative delay distributions that share the same expected delay but differ in variance and temporal structure. All distributional experiments use Hopper-medil um-v2, with maximum delay fixed at 16 and Gaussian, exponential, binomial, and uniform delays parameterized to have mean for fair comparison. Top right figure shows average normalized return across the three MuJoCo locomotion tasks delays increase, averaged over deterministic and stochastic variants. As expected, performance decreases with larger delays, highlighting the increased difficulty; full results appear in Table 10 and App. C.I From Table 6 we also observe that DT-CORL performs robustly across different delay distributions-eee without prior knowledge of the true online delay process demonstrating strong generalization of our belief estimation module.
5.4 DEXTEROUS MANIPULATION
To evaluate delay robustness in high-dimensional, contact-rich manipulation, benchmark all methods on the Dexterous Hand Manipulation suite (Zhu et al.) 2019), using the expert demonstrations provided and applying standard delay setting before. These tasks involve discontinuous contacts, multi-finger coordination, and highly sensitive state-action coupling, making delayed observation particularly challenging. As shown in Fig. 3 DTT-CORL consistently attains the highest performance across all tested delay levels and exhibits substantially more graceful degradation compared to Aug- COL and Belief-CQU This indicates that our belief model captures greegrained temporal structure that is critical for manipulation under latency. Full per-task results for Pen, Door, and Hammer appear Table I1] further highlighting D'T-CORL's robustness, particularly under for the Hammer task where both augmented and belief-based baselines failed.
CONCLUSION
We presented DT-CORL, the first framework for offline-to-online delay adaptation that learns delay- robust policies solely from delay-free data. By combining transformer based belief predictor with conservative, reguuaii-ee policy iteration, D'T-CORL avoids the dimensional blow-up of history augmentation while mitigating out-of-distribution errors common in offline RL. Experiments on various tasks show consistent gains over both augmented-statt and belief-based baselines, under deterministic and stochastic delays. Looking forward, several extensions remain open. First, DT- CORL currently assumes known, fixed delays; extending it to handle unknown, time-varying, or heterogeneous delays is important next step. Notably, such cases remain largely unaddressed in both online and offline delayed RL. Second, scaling the approach to high-dimensionee perceptual domains such as vision-basse based manipulation may require spatial attention structured world models.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
Simon Sinong Zhan, Xiangyu Shi, and Qi Zhu acknowledge the support from National Science Foundation grants 2324936, 2328973, 2328032, and 2133630. Qingyuan Wu and Chao Huang supported by the grant EP/Y002644/1 under the EPSRC ECR International Collaboration Grants program, funded by the International Science Partnerships Fund (ISPF) and the UK Research and Innovation.
ETHICS STATEMENT
We affirm that all authors have read and adhere to the ICLR Code of Ethics. Our work does not involve human or animal subjects, sensitive personal data, or privacy risks. The experiments are fully based on public benchmarks (D4RL), with no proprietary or private datasets, ensuring transparency and reproducibility. We used Large Language Models only for manuscript writing, formatting, and routine data- cata-processing all algorithmic design, theoretical derivations, and experimental evaluations solely the work of the authors. There no known immediate risks of misuse from method; however, we recognize that deployment in safety-critical systems under delays might require careful calibration.
REPRODUCIBLE STATEMENT
To ensure reproducibility of all experimental results, we provide the following supporting mate- rials and practices. The code implementing DT-CORL, including training scripts, belief model architectures, and evaluation pipelines, has been made available in the following repo (https //github. com/ S Simon2han-coooeecc DD- DD-CC All algorithmic assumptions, hyperparame- ters (learning rates, network architectures, regularization weights, transformer depth, etc.), and training settings are described in detail in Secc. of the main paper, and additional implementation details are given in App. D The datasets used are standard benchmarks (D4RL locomotion and AntMaze tasks). We report performance averaged over multiple random seeds. All experiment results listed in Sec.. and App.
REFERENCES
Eitan Altman and Philippe Nain. Closed-loop control with delayed information. ACM sigmetrics performance evaluation review, 20(1):193-204, 1992.
Jose A Medina, Michael Gillhofer, Michael Widrich, Thomas Unterthiner, Johannes Brand- stetter, and Sepp Hochreiter. Rudder: Return decomposition for delayed rewards. Advances in Neural Information Processing Systems, 32, 2019.
Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhihong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for mncertainty-driver offline reinforcement learning. arXiv preprint arXiv:220.111566, 2022.
Alfredo Bellen and Marino Zennaro. Numerical methods for delay differential equations. Numerical Mathematics and Scie, 2013.
Yann Bouteiller, Simon Ramstedt, Giovanni Beltrame, Christopher Pal, and Jonathan Binas. Rein- forcement learning with random delays. In International conference on learning representations, 2020.
Zhiguang Cao, Hongliang Guo, Wen Song, Kaizhou Gao, Zhenghua Chen, Le Zhang, and Xuexi Zhang. Using reinforcement learning minimize the probability of delay occurrence in trans- portation. IEEE transactions on vehicular technology, 69(3):2424-2436, 2020.
Baiming Chen, Mengdi Xu, Liang Li, and Ding Zhao. Delay-aware model-based reinforcement learning for continuous control. Neurocomputing, 450::19-1111 2021.
Yunhai Feng, Nicklas Hansen, Ziyan Xiong, Chandramouli Rajagopalan, and Xiaolong Wang. Finetuning offline world models in the real world. arXiv preprint arXiv:2310 10. 16029, 2023.
11

---

Published as a conference paper at ICLR 2026
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
Chriss Grimholt and Sigurd Skogestad. Should we forget the smith predictor? In IFAC-PupersOnLiie (PID 2018), volume 51, pp. 769-774, Ghent, Belgium, 2018.
Rachana Ashok Gupta and Mo- Yuen Chow. Networked control system: Overview and research trends. IEEE transactions industrial electronics, 57(7):2527-2533 2009.
Beining Han, Zhizhou Ren, Zuofan Wu, Yuan Zhou, and Jian Peng. Off-policy reinforcement learning with delayed rewards. In International conference on machine learning, pp. 8280-8303. PMLR, 2022.
Joel Hasbrouck and Gideon Saar. Low-latency trading. Journal of Financial Markets, 16(4):646-679, 2013.
Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and JoAGo GM Araāšjo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274)) 2022.
Ruochen Jiao, Yixuan Wang, Xiangguo Liu, Simon Sinong Zhan, Chao Huang, and Qi Zhu. Kinematics-aware trajectory generation and prediction with latent stochastic differential mod- cling. In 2024 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 565-572. IEEE, 2024.
Armin Karamzade, Kyungmin Kim, Montek Kalsi, and Roy Fox. Reinforcement learning from delayed observations via world models. arXiv preprint arXiv:2403. 12309, 2024.
Konstantinos V Katsikopoulos and Sascha E Engelbrecht. Markov decision processes with delays and asynchronous cost collection. IEEE transactions on automatic control, 48(4):568-577 2003.
Rahul Kidambi, Aravind Rajeswaran, Pranceth Netrapalli, and Thorsten Joachims, Morel: Model- based offline reinforcement learning. Advances in neural information processing systems, 33: 21810-21823, 2020.
Jangwon Kim, Hangyeol Kim, Jiwook Kang, Jongchan Baek, and Soohee Han. Belief projection- based reinforcement learning for environments with delayed feedback. Advances in Neural Information Processing Systems, 36:678-696, 2023.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learming. arXiv preprint arXiv:2110.06169 2021.
Srivatsan Krishnan, Behzad Boroujerdian, William Fu, Aleksandra Faust, and Vijay Janapa Reddi Air learning: a deep reinforcement learning gym for autonomous aerial robot visual navigation. Machine Learning, 11009):5501-5553 2021.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in neural information processing systems, 33:1 1111119919999 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.
Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín- Martín, Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, et al. Behavior-Ik: A benchmark for embodied ai with 1, 1,000 everyday activities and realistic simulation. In Conference on Robot Learning. pp. 80-93. PMLR, 2023.
Pierre Liotet, Erick Venneri, and Marcello Restelli. Learning belief representation for delayed reinforcement learning. In 202/ International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2021.
12

---

Published as a conference paper at ICLR 2026
Pierre Liotet, Davide Maran, Lorenzo Bisi, and Marcello Restelli. Delayed reinforcement learning by imitation. In International Conference on Machine Learning, pp. 13528-13556. PMLR, 2022.
A Rupam Mahmood, Dmytro Korenkevych, Brent J Komer, and James Bergstra. Setting up reinforcement learning task with a real-world robot. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4635-4644 IEEE, 2018.
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108.10470, 2021.
Carlos Mejía, Estefanía Salazar, and Oscar Camacho. A comparative experimental evaluation of various smith predictor approaches for a thermal process with large dead time. Alexandria Engi- neering Journal, (1(12)93377-939) 2022. ISSN 1110-0168. doi: https://bbl.or//1..0016....
2022.03.047. URL https:: htpp:://www.. www. scwencediedcccco rect. com/ wwwoo//ccenne//aeeeemmem science//rticcle/pii.
S1 001 6822002211
Tiago A. Moraes, Moisés T. da Silva, and Thiago A. M. Euzébio. Delay compensation in a feeder-conveyon system using the smith predictor: A case study in an iron ore processing
plant. Sensors, 24(12), 2024. ISSN 1424-8220. doi: 10.3390/524123870. URL https
www. www.mdpi... cww 1424 /220/24/12/33322
Tongzhou Mu, Zhan Ling, Fanbo Xiang, Derek Yang, Xuanlin Li, Stone Tao, Zhiao Huang, Zhi- wei Jia, and Hao Su. Maniskill: Generalizable manipulation skill benchmark with large-scale demonstrations. arXiv preprint arXiv:2107. 14483, 2021.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359 2020.
Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning. Advances in Neural Information Processing Systems, 36:62244-62261 2023.
Haoyi NIu, Yiwen Qiu, Ming Li, Guyue Zhou, Jianming Hu, Xianyuan Zhan, et al. When to trust your simulator: Dynamics- aware aware hybrid affline--addonline reinforcement learning. Advances in Neural Information Processing Systems, 35:36599-36612, 2022.
Julio E. Normey- Rico and Eduardo F. Camacho. Control of Dead-Time Processes. Springer, London, 2007.
Emmanuel Rachelson and Michail G Lagoudakis. On the locality of action domination in sequential decision making. 2010.
Laixi Shi, Gen Li, Yuting Wei, Yuxin Chen, and Yuejie Chi. Pessimistic q-learning for offline reinforcement learning: Towards optimal sample complexity. In International conference on machine learning, pp. 19967-20025.. PMLR, 2022.
Xiangyu Shi, Junyang Ding, Xu Zhao, Sinong Zhan, Payal Mohapatra, Daniel Quispe, Kojo Welbeck, Jian Cao, Wei Chen, Ping Guo, et al. Step-lim: Generating cad step models from natural language with large language models. arXiv preprini arXiv:2601. 12641, 2026.
O. J. M. Smith. Closer control of loops with dead time. Chemical Engineering Progress, 53(5): 217-219, 1957.
Yuda Song, Yifei Zhou, Ayush Sekhari, J Andrew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid Using both offline and online data can make rl efficient. arXiv preprint 1rXiv.2210.06718, 2022.
Wei Sun, Yuwen Chen, Yanjun Chen, Xiaopeng Zhang, Simon Zhan, Yixin Li, Jiecheng Wu, Teng Han, Haipeng Mi, Jingxian Wang, et al. Microfluid: A multi-chip rfid tag for interaction sensing based on microfluidic switches. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 6(3):1-23, 2022.
13

---

Published as a conference paper at ICLR 2026
Yihao Sun. Offlinerl-kit: An elegant pytorch offline reinforcement learning library. https
//githab. httph//.......caaaaaaaa////aaa//
2023.
Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
Jiaming Tang. Yufei Sun, Yilong Zhao, Shang Yang, Yujun Lin, Zhuoyang Zhang, James Hou, Yao Lu, Zhijian Liu, and Song Han. Vlash: Real-time vlas via future-state state-aware asynchronous inference. arXiv preprint arXiv:2512.01031, 2025.
Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems, 16911152-1,62, 2023a.
Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. Corl: Research-oriented deep offline reinforcement learning library. Advances in Neural Information Processing Systems, 36:30997-3100 2023b.
Christopher Thomas, Sun Yi, Shava Meadows, and Ryan Sherrill. Adaptive smith predictor for teleoperation of uavs with time-varying internet delay. International Journal of Control, Automation and Systems, (8(6):4455118 1473, 2020.
Gabriele Tiboni, Karol Arndt, and Ville Kyrki. Dropo: Sim-to-real transfer with offline domain randomization. Robotics and Autonomous Systems, 166:104432 2023.
Emanuel Todorov, Tom Frez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv preprint arXiv:1805.01954. 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017
Cédric Villani al. Optimal transport: old and new, volume 338. Springer, 2008
Yixuan Wang. Simon Zhan, Zhilu Wang, Chao Huang, Zhaoran Wang. Zhuoran Yang, and Zhu. Joint differentiable optimization and verification for certified reinforcement learning. In Proceed- ings of the ACM/EEE 14th International Conference on Cyber-Physical Systems (with CPS-IoT Week 2023), pp. 132-141, 2023a.
Yixuan Wang, Simon Sinong Zhan, Ruochen Jiao, Zhilu Wang, Wanxin Jin, Zhuoran Yang, Zhaoran Wang, Chao Huang, and Qi Zhu. Enforcing hard constraints with soft barriers: Safe reinforcement learning in unknown stochastic environments. In International Conference on Machine Learning. pp. 36593-36604, PMLR, 2023b.
Tianshu Wei, Yanzhi Wang, and Qi Zhu. Deep reinforcement learning for building hvac control. In Proceedings of the 54th annual design automation conference 2017, pp. 1-6, 2017.
Qingyuan Wu, Simon S Zhan, Yixuan Wang. Yuhui Wang, Chung Wei Lin, Chen Lv, Qi Zhu, and Chao Huang. Variational delayed policy optimization. Advances in Neural Information Processing Systems, 37:54330-55335 2024a.
Qingyuan Wu, Simon Sinong Zhan, Yixuan Wang, Yuhui Wang, Chung- Wei Lin, Chen Lv, Qi Zhu, Jürgen Schmidhuber, and Chao Huang. Boosting reinforcement learning with strongly delayed feedback through auxiliary short delays. In International Conference on Machine Learning, pp. 53973-53998. PMLR, 2024b.
Qingyuan Wu, Yuhui Wang, Simon Sinong Zhan, Yixuan Wang, Chung Wei Lin, Chen Lv, Qi Zhu, Jürgen Schmidhuber, and Chao Huang. Directly forecasting belief for reinforcement learning with delays. arXiv preprint vrXiv:2055000544 2025.
14

---

Published as a conference paper at ICLR 2026
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arX::: 2019.
Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman-consistent pessimism for offline reinforcement learning. Advances in neural information processing systems, 34:6683-6694, 2021.
Frank Yang, Sinong Simon Zhan, Yixuan Wang. Chao Huang, and Qi Zhu. Case study: Runtime safety verification of neural network controlled system. In International Conference on Runtime Verification, pp. 205-217. Springer, 2024.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129-14111 14142, 2020.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-basse policy optimization. Advances in neural information processing systems, 34:28954-28999 2999777-7967 2021.
Sinong Zhan, Yixuan Wang, Qingyuan Wu, Ruochen Jiao, Chao Huang, and Qi 7hu. State-wise safe reinforcement learning with pixel observations. In 6th Annual Learning for Dynamics & Control Conference, pp. 1187-1201. PMLR, 2024.
Yuyang Zhang, Runyu Zhang, Yuantao Gu, and Na Li. Multi-agent reinforcement learning with reward delays. In Learning for Dynamics and Control Conference, pp. 692-704. PMLR, 2023.
Henry Zhu, Abhishek Gupta, Aravind Rajeswaran, Sergey Levine, and Vikash Kumar. Dexterous ma- nipulation with deep reinforcement learning: Efficient, general, and low-cost. In 2019 International Conference on Robotics and Automation (ICRA), pp. 3651-3657. IEFE, 2019.
Qunxi Zhu, Yao Guo, and Wei Lin. Neural delay differential equations. arXiv preprint 2000:21022.0801, 2021.
Liang Zou, Martin Fränzle, Naijun Zhan, and Peter Nazier Mosaad. Automatic verification of stability and safety for delay differential equations. In International Conference on Computer Aided Verification, pp. 338-355. Springer, 2015.
15

---

Published as a conference paper at ICLR 2026
A LLM USAGE STATEMENT
The usage of LLMs in this work is limited to paper writing support. language refinement, and experimental data processing. Specifically, LLMs assisted in improving the clarity and coherence of the manuscript, generating LaTeX tables and formatting results for presentation. Importantly, LLMs were not involved in the design of algorithms, the development of theoretical results, the execution of experiments, ensuring that all core scientific contributions remain entirely the work of the authors.
B BELIEF-BASED POLICY ITERATION
B. POLICY ITERATION
To arrive the final derivation, need the following lemma.
Lemma B.1. (General Delayed Performance Difference (Wu et al. 2024b)) For policies a and μs
with any x E X, the performance difference is denoted as ((x)
[(x)
E (
[V(s)] Va,
1
[V($) Q($.a)
br-b(-12))
ami (1)m)
g
For next step, try cast the above optimization problem back to the ordinary state/action space with belief function b defined above. To begin with, we can convert Qa (z, a) back to Eg-d(-ja) (Q(s, a)] using Lem. 4.1
usingLem.
B(z,a,")~> -D [Q(3, a) Qa(x,a)] ind(-1ax)
8'mb(-la')
(r(8,a) a) + +-Bg [V]']]] R [r(x, a) + Va()))
= =>E(u,,x2 bnb(((2)
(gg (V -
< SL4 B (2, (M((a)
1 1-7
8mbb x)
Similarly, we can extend it to other term in the policy evaluation part.
E ADD
[Q(s', Q2(2', (z),
8'ab(-la')
a ('
E
s // a' d'axal-la')
(Q(5',a'))
Ea'nna((ie) (2axx'
x' x'rD
z'/D
L2Q E AND
[ 1 8'mb(-la')
Then, we can start to break down the policy evaluation defined above.
E (Q (z) - 7B ( ( + (
+ E (FENM) @ (8, The mi(T((3),MA(ttt))
1-2
- ((BANo(io(((8s + B
( (Q" 1- ((T((18);= mi(Ta.naa))
Lhe
16

---

Published as a conference paper at ICLR 2026
The next step is trying to sort out all the policy divergence term, and convert the expectation term with respect to na back R. For simplification, let's define b2 and hide the policy divergence term for now. We have the following:
((,,,aa)>D (BanM)i)) (3. (E [r(8 + s'ma(11z) 1Q (8"()",'')) )) c'mmt(((1
According to Lipchitz continuous assumptions on functior al. (2022), we can derive that Earoy (Q"(ss ai) Q" (s, a2))l I LQ.i(u. vs E S. Thus, we can derive the following:
02
(2) (8,( (Bs a)) + ByNat1e) (Q" a']l (1 (1-2)oWi (TAAT)T)))) a'wa((18")
(1-))
(5)
Now, we have the policy evaluation in the original state space format. Let's take close look the remaining policy divergence terms.
( mA) + (I cWi (m, + -1- 7 [C, FA) + W c
The triangle inequality holds for the Wasserstein distance. Specifically, Wp(pp p) S W)(,,,)
(p(,, p) for all μ., and in the same metric space and p > 1 (Villani et al. 2008). With the proper choice of a, is easy to unify all the I-Wasserstein terms to π and behavior policy. Besides, since all the I-Wasserstein terms are bounded here, it won't non't affect the convergence property of policy iteration.
Q arg min E (,,,,,) Q
[
(Es.0(()](((8aa) + Rx 10 (a)
a' a'~x(118")
Using the derivation above, we can easily reformulate the policy improvement back to the original state space.
ExDD
Fawna(l)))
[2 (x*((x;
(z: ()) - -0 (V2-D(TA,Na)
DEEDD D
imb((/a))
(8=(8)a)
--- Ta) - -22W()(AM))
LLQ
amit m)
+EEDD
E. Bgnd(-lo) a>a7(118)
(8,(
+ (1--))+1111))0 (1-7)cMi(T,TT) - (on
+EEDD
E Egnd(-la) [a] a-)((11)
Using similar trick mentioned above, with appropriate selection of a2 we can combine the above two policy divergence terms into one.
(8, (T,
Ex-D Bzab(le) a-)7(118)
B.2 POLICY IMPROVEMENT
Proof.
[ a..
Tnew arg max E EDDD
E (anb((aa) anst(-18)
17

---

Published as a conference paper at ICLR 2026
By following Eq. (4)
easily have have
(fie (5. -oWi (Tnen
B
E 8-b(//le)
an-ane((-16)
(Qāal8,a)] - -aWi(Tood
2 E
E snb((1at)
ansTotd(-18)
Quen (8, = B(
E Ba [r( + B s'-((1a))) Quew (s', - aWW (Tnew: H&N
a a'wanen(118)
2 ((
Bsmb((12) [)( +, s'mb(-la') ((-18)
[ata((", a')] - aWW (Told: pa)
= (8)
18

---

Published as a conference paper at ICLR 2026
C EXPERIMENTS DETAILS
C.I ADDITIONAL RESULTS
In this part, we provide visualization of the delay belief prediction over 16-step trajectory rollout in the HOPPER environment. Each frame corresponds to simulation step under delayed control setting. The agent executes actions conditioned on the predicted belief state rather than the true current observation. The sequence illustrates how the learned delayed belief model tracks and reconstructs the latent state dynamics despite observation-action delays, Accurate belief predictions enable agent maintain coherent behavior across all steps.
Step
Step
Step 2
Step 3
Step 4
Step 5
Step
Step
Step 8
Step 9
Step 10
Step 11
Step 12
Step 13
Step 14
Step 15
Figure 4: Ground Truth
19

---

Published as a conference paper at ICLR 2026
Step
Step
Step 2
Step 3
Step 4
Step 5
Step
Step 7
Step 8
Step 9
Step 10
Step 11
Step 12
Step 13
Step 14
Step 15
Figure Ensemble MLP
20

---

Published as a conference paper at ICLR 2026
Step
Step
Step 2
Step 3
Step 4
Step 5
Step
Step 7
Step 8
Step 9
Step 10
Step 11
Step 12
Step 13
Step 14
Step 15
Figure 6: Diffusion
21

---

Published as a conference paper at ICLR 2026
Step
Step
Step 2
Step 3
Step 4
Step 5
Step
Step 7
Step 8
Step 9
Step 10
Step 11
Step 12
Step 13
Step 14
Step 15
Figure Transformer
22

---

Published as a conference paper at ICLR 2026
Hupper
Method
16
16
Aug-BC
28.1
2..
14.0::2..
Aug
24.34.14
38.20.
37..007
67.7 1:3
06.2 18
21.1::20.
15.8 23
31.2 2.1
Belif-COO.
60.001
Belef-KQ
30.8
106:2:10
53..0
27.7 1:3
25:34:14
20.4:::1:
S144=10
25.7
21.6::11
DI-CORL
47.42 1.2
27.8
6.20.
$5.0 1.2
7188
$1. 1.0
$7.6 1,4
86.8
ospert
Aug
6.
50:00
422404
1036
18.7 1.5
0.8
$6.4
1211111
CQL
0.2
8.2.0.4
56.0033
0.2.0::0.4
50.5
6...:.088
0.5
34.8 1.0
0.:00.0
Belif-OOL.
15:
135
11081:0
Belef-
68200
68.00.
363.04
17.4
10.9-20.77
20.3=110
10.9 0.9
16.70.
DI-CORL
0.5.
51..0.4
523003
112.9::000
113.3
1122 0.44
1109:::00 0.4
111.2
100.5
Aug BC
205
5.
51:10
M9.5 1.7
40.2 2:2
106.7
53.3 2.2
122#19
7.
28.80..8
134.00
100.7
54.4 2.2
17.7
BeliefOQL.
26.8 1.4
6.1.1.1.
0.
26.7 1.3
26.6
1.3
24.7 2.22
40.5
17.3 1.2
16.41
DE-CORL
46.7 2.5
21.3::2
$7.1.9
113.0::00 0.8
112.3
1009:100 0.9
0.7
1120
1881111
Aug-SC
L.
4.20:1..0
5.2::12
25
25.0
21.7 2:2
28.1
19.5 2.5
1:5
Aug-CQ
9.
20:5:099
50.110
85.7 1.8
51411
4.44-144
08.5
840-1
Bellef CQL
BellefQQ.
233 1.3
138
67
24.7.1:1
20.5
20418:11
88.2
20.6 1.8
18338:1
DT-CORL
27.1
20.5
Table 7: Normalized returns (%) on D4RL MuloCo locomotion tasks with deterministic observation delays of 4, 8, and 16 steps. All results shown mean std over seeds.
Methal
Hupper
15
15
10.9
CQI
256.11 1.5
5:11:111
12:
67.8 1.7
63.8 2.0
20.5 2.2
73.5 2.5
30.8 3.2
9.2
74.3::2.6
BeliefOQL.
PE-CORL
4822 1.5
47.5
36.
76.61:18
72.1
70
86.8
ST.4 1.4
67.0
Aug-BC
0.70077
1046
0.
Aug COL
0.2
89200
37.20033
112.5 0.5
17.5 1.2
6.200
1.8
36.5 1.1
0.4
Belcf-COL
1:11::0
111:1:006
Brlef-Q0
0.0
43:80.
1869
16.9
907.400
PT-CORL
80.1 1.2
127:100
38..0
11288
1:
1009
110. 2:
110.0 0.5
Aug-BK
1.5
54:11
48:09
01.6::
23
8.5
50.6 0:00:00
11441
Aug COL
65
0.2.0.66
0.:00.444
132.9 0.0
61.3
1.00
28.5 2.0
1.8
Belif-CQO.
4.0
Belief
11.1
16.31:1
82200
25.3
27.9
20.5 1.
40.4
26.0 1.8
20.4
DI-CORI
200:5:23
31331:2.1
S8
113.8
112.7 0.7
S5.4
1158 a: 0.6
111.5
Aug-CC
17.02.
46:5:
4.40
23.9
27.2 2.6
21.7 222
24.822
14.0
9.1
Aug-
9.6
2.30:100
18::0.3
2.00
50.9
4.1.112
41:52:
7:1.1 18
1::::00
BeliefOOL.
249214
20.0.:11
15.8
25.02:12
115-21111
26.0 2000::1.
SL4
26.4 1.8
2422112
DI-CORI
47.75:2
453312
035:1:21
90.6 2:00
100.
98.
00.0
90.9 1.7
91881:1
Table 8: Normalized returns (%) on D4RL MuJoCo locomotion tasks with stochastic observation delays A U(1, k), k (4, 8, 16). Results shown mean + std over seeds.
Transformer Ablation. The ablation in Table 9 reveals clear relationship between the capacity of the transformer belief model and its predictive performance under delayed observations. As the num- ber of parameters decreases both sample efficiency and prediction accuracy degrade substantially. In particular, smaller models (e.g., latent dimension 64 or shallow 4-layer variants) require significantly more epochs to fit the offline trajectories and still exhibit higher prediction error (Performance t), indicating difficulty in capturing the temporal dependencies required for belief reconstruction under long delays.
Conversely, larger models (256-dim with 8-10 layers) achieve the best predictive performance while converging in far fewer epochs, demonstrating superior representational power and optimization stability. Notably, the 256x10 model provides the best overall performance and sample efficiency, confirming that a moderately sized transformer a favorable balance between predictive accuracy and computational cost.
23

---

Published as a conference paper at ICLR 2026
These results highlight crucial insight: belief estimation under delayed offline RL requires sufficient sequence modeling capacity, as delay-induced temporal misaligmment increases the effective horizon and dynamics complexity. Models that are too lightweight fail capture these dependencies, while overly large models provide diminishing returns relative to their computational cost. Thus, DT- CORL's chosen transformer configuration emerges the most robust and well-calibrated architecture for belief prediction under realistic delay settings.
Table Ablation on transformer belief model size in Hopper-medi um-v2 with delay A 16.
Results averaged over 3 seeds.
Tatent Dim
Layers
Params (M)
Memory (MB)
Epochs Converge < 0.01)
Performance (MSF)
10
7.87
17.5
14.3
0.32
7.08
14.3
0.63
11.1
0.95
256
5.49
7.96
1.24
128
10
4.64
4.56
61.3
1.44
64
10
3.82
1.28
2.68
Table 10: Long-delay robustness (32 and 64 steps) across HalfCheetah-, Hopper-, and Walker2d- medium-v2 tasks. DT-CORL consistently outperforms belief-based baselines in both deterministic and stochastic delay settings.
Method
HalfCheetah-med-y
Hopper-med-v2
Walker2d-med-12
32 (Det/Stoch)
64 (Det/Stoch)
32 (Det/Stoch)
64 (Det/Stoch)
32 (Det/Stoch)
64 (Det/Stoch)
Belief-IQL
1914/9.31
4.92/7.17
11.1 / 22.6
2.70/ 6.26
12.0/ 20.9
10.5 13,9
Belief-CQL.
5.61
4.09 / 4,32
9.05 / 20.4
2.71 2.71
7.45 18.3
5,30 8.71
DT-CORI.
6.36 11.1
5.81 / 7.63
13.8 26.1
2.77 6.48
60.0/ 85.4
21.4/41.3
Bellet-DL
Belle--CLL
DT-CORL
HalfChoctahh medium-v2
Hopper-medlum-22
Walkor2d-med um-
80
bormatess Retur (30
70
60
30
30
10
10
20
10
1.6
12
54
16
12
64
16
Delay Stope
Dctay Stops
Delay Stops
Figure Performance degradation under increasing delay across HalfCheetah-, Hopper-, and Walker2d-men -eeerm2d-deduu-22 tasks. Curves show the average return across deterministic and stochastic delay settings.
D IMPLEMENTATION AND REPRODUCIBILITY DETAILS
The implementation of Augmented-
Augmented BC and our DT-CORL is based on
CORL (Tarasov et al. 2023b) and CleanRL (Huang et al. 2022) The implementation of DBT-SAC and AD-SAC can be found in the original paper (Wu et al. 2025 2024b). And the implementation of Model- badel-based Offline RL(MOPO and COMBO) is based on the OriineLL-Kit Library (Sun) 2023).
We detail the hyperparameter settings of Transformer Belief and DT-CORL in Table 12 and Table 15 respectively. All the experiments are conduct the server equipped with NVIDIA A5000 GPU and AMD EPYC 7H12 64-Core CPU. The training time depends on state/action dimensions. offline data size, and max delay horizon, which determines the training time of belief predictions. To give approximate range, training on Hopper and training Adroit-Pen task with 16 delay steps take up around 7 hours.
roveer-ees -e-eedi mediumt -m--dium-22 with 4 delay steps take up around 2 hours,
24

---

Published as a conference paper at ICLR 2026
Table 11: Performance Adroit Hand tasks (Pen-expert-vl, Door-expert-vl, Hammer-expert-vl under deterministic and stochastic delays E {4,8, 16). Best per column is shown bold with light blue background.
Setting
Method
Pen-expert-vi
Door foor-expert-ve
Hammer-expert--i
8
16
16
16
Aug-COp IggCQQL
20.11
4.38
-0.65
78 .6
61.74
40.12
0.25
0.23
0.21
Deterministic
Belief-CQLL
50.43
26.57
19.43
97.61
83.35
67.91
0.27
0.29
0.22
DT-CORL
86.44
77.51
66.38
102.57
100.65
93.04
114.51
110.12
105.20
Aug-CQL CQL
15.15
6.17
-0.72
67.80
49.88
25.84
0.27
0.26
0.24
Stochastic
Belicf-CQL.
64.68
27.42
22.51
97.59
85.92
69.73
0.28
0.29
0.24
DT-CORL
90.56
79.33
74.85
102.97
101.47
97.43
122.13
112.22
112.33
Table 12: Hyper-parameters table Transformer
Table 13: Hyper-parameters table of Ensemble Belief.
Belief.
Hyper-parameter
Value
Jlyper-parameter
Value
Epoch
le2
Epoch
le2
Batch Size
256
Batch Size
256
Attention Heads Num
Attention Heads Num
Layers Num
10
Layers Num
10
Hidden Dim
256
Hidden Dim
256
Attention Dropout Rate
0.1
Attention Dropout Rate
0.1
Residual Dropout Rate
0.1
Residual Dropout Rate
0.1
Hidden Dropout Rate
0.1
Hidden Dropout Rate
0.1
Learning Rate
le-4
Learning Rate
le-4
Optimizer Weight Decay
AdamW
Optimizer Weight Decay
AdamW
le-4
le-4
Betas
(0.9, 0.999)
Betas
(0.9, 0.999)
Table 14: Hyper-parameters table of Diffusion
Table 15: Elyper-parametter table of DT-CORL.
Belief.
Hyper-parameter
Value
Parameter
Value
Learning Rate (Actor)
3e-4
Model Dimension
32
Learning Rate (Critic)
le-3
Embedding Dimension
32
Learning Rate (Entropy)
le-3
Dimension Multipliers
[1, 2, 2]
Train Frequency (Actor)
2
EMA Rate
0.9999
Train Frequency (Critic)
1
Diffusion Steps
20
Soft Update Factor (Critic)
5e-3
Predict Noise
True
Batch Size
256
State Loss Weight
10.0
Neurons
[256, 256]
Action Loss Weight
10.0
Action Noise
0.2
Classifier Guidance (w_eg)
0.3
Layers
3
Timestep Embedding
Positional
Hidden Dim
256
Kernel Size
Activation
ReLU
Solver
DDPM
Optimizer
Adam
Sampling Steps
20
Temperature
1.0
25


Reviews:
[{"review_id": "H5V0TAE7Qj", "reviewer": "Reviewer_zm9D", "summary": "This paper introduces DT-CORL, a method for learning delay-robust policies from purely delay-free offline data. The approach combines a transformer-based belief model to infer latent states from delayed observations with a constrained policy optimization objective, enabling effective deployment in environments with observation/action delays without requiring online interaction. Evaluations on D4RL benchmarks demonstrate strong performance against augmentation-based and belief-based baselines under varying delay settings.", "questions": "1. Have the authors considered testing in higher-dimensional observation spaces (e.g., pixel-based tasks) to assess the scalability of the belief model?\n2. Could the belief model be made more lightweight without sacrificing predictive accuracy?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Addresses a well-motivated and practical problem at the intersection of offline RL and robustness to delays, which is essentially important in real-world scenarios.\n\n2. The proposed integration of belief learning and policy optimization within an offline constraint framework is technically sound and justified by theoretical analysis.\n\n3. Empirical evaluation is comprehensive, covering multiple tasks, delay types, and delay lengths, with ablations validating design choices.\n\n4. The paper is clearly written, and the methodology is well-explained.", "weaknesses": "1. While the transformer belief model shows advantages, its computational overhead relative to simpler models is non-trivial.\n2. The experimental validation is confined to standard simulation benchmarks (D4RL). The absence of validation on a physical system or a high-fidelity simulator with realistic latency weakens the claims of practical contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PxBAHX5ecu", "reviewer": "Reviewer_Z5y8", "summary": "This paper presents offline RL methods for delay robust policy optimization. Towards this, the paper introduces a transformer based belief model to infer latent states from delayed observations and jointly learns this model while performing offline policy optimization to ensure the estimation doesnt suffer from distribution shift issues.", "questions": "See weakness above.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Interesting problem, well motivated and reasonably well written paper.", "weaknesses": "- The theory section is based on assumptions that aren't satisfied in practice even in locomotion tasks (defn 3.2), with discontinuities in the dynamics and / or rewards.\n- The theory doesnt sketch out sample complexity associated with estimation issues popping up due to delayed observations which, in the context of offline RL would be good to get a clear grasp of.\n- Intuitively, it feels like stochastic delays should be harder to estimate (at least, the estimators would have higher variance), but the methods appear to perform even better in this setup than the deterministic ones (e.g. table 1), which seems counterintuitive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rPCkxUaffM", "reviewer": "Reviewer_TYE4", "summary": "The authors propose an offline RL method that trains on delay-free offline data and is robust to online state-action delays. The proposed approach works by augmenting the state-action space via stacking a history trace and doing policy evaluation and optimization in the augmented space.", "questions": "1. Proposition 4.3: There are two moving parts in this equation, both the estimated Q value and the policy pi. Having a higher estimated Q value does not necessarily imply a better policy, right? Also, the notation here is a bit arbitrary - the current way of writing the equation seems to be comparing the Q value for the exact same action, even though the new and old policies might pick different actions.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem setting is meaningful, and the approach itself is sound and interesting. \n2. The empirical performance of the proposed approach is good across a wide range of tasks.", "weaknesses": "1. The authors assume a known delay window and construct an augmented dataset based on that delay window. However, assuming a known exact delay window for online deployment is rather unrealistic. If one uses a worst-case delay window for offline training, the proposed approach does not establish if training with a large delay window might still give reasonable online performance if the actual latency during deployment is smaller. \n\n2. The policy improvement statement they constructed seems insufficient; cf. my question below. \n\n3. The proposed approach heavily relies on the accuracy of the learned belief model. However, assuming being able to learn a good belief model offline without knowing the actual online latency seems to be a strong assumption - it essentially requires that the data distribution collected by the delay-free behavior policy is representative of the online closed-loop controller with delays. If this assumption is violated, everything proposed in this approach falls apart, and the authors do not have a formal statement established to address this point. As data coverage shift is a well-known open challenge in standard offline RL settings, the proposed approach seems to require even stronger assumptions on the dataset coverage. \n\n4. Section 4.1: Presentation can be improved: It is important to lead the presentation with the goal of each step. I personally got confused while reading this section and was trying to guess what the authors were trying to establish with each step. \n\n5. I feel that the ablation study can be strengthened: right now, the authors claim the benefit of jointly considering the delayed, augmented state belief model in policy evaluation and optimization steps by comparing to two customized baselines (CQL + IQL) equipped with a belief model, which does not seem to be an apples-to-apples comparison. I would appreciate a more direct ablation of the delay awareness from different components of the proposed approach. \n\n6. Minor:\n- Notation is vague at times: Lemmas 4.1 and 4.2, which value functions do the authors mean to bound? Any value functions?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZK2Jy0FlO9", "reviewer": "Reviewer_F3YW", "summary": "This paper addresses the problem of learning delay-robust policies from delay-free offline datasets for deployment in delayed environments. The authors propose DT-CORL (Delay-Transformer belief policy Constrained Offline RL), which combines a transformer-based belief model with constrained policy optimization. The key innovation is joint training of belief prediction and policy optimization to handle the sim-to-real latency gap without requiring delayed transitions during training. The method is evaluated on D4RL tasks under various delay scenarios.", "questions": "1. Performance gaps with COMBO: Can you provide deeper analysis of when and why Augmented-COMBO outperforms DT-CORL? Is there a fundamental trade-off between model-based and belief-based approaches that depends on environment characteristics?\n\n2. Belief model architecture: Why is the transformer significantly better than diffusion models despite similar prediction accuracy (Figure 2)? Is it purely computational efficiency or are there other factors?\n\n3. Distribution mismatch: You claim joint training avoids distribution mismatch, but the policy still uses the same offline dataset. Can you quantify the distribution shift between training and deployment more rigorously?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Solid theoretical framework: The reformulation from augmented-state delayed MDP to belief-based policy iteration is theoretically grounded with formal lemmas and a monotonic improvement guarantee (Proposition 4.3).\n\n2. Comprehensive experiments: Evaluation covers multiple D4RL environments with both deterministic and stochastic delays, multiple baseline comparisons, and thorough ablation studies on architecture and data availability.", "weaknesses": "1. Inconsistent baseline performance: DT-CORL sometimes underperforms Augmented-COMBO in AntMaze tasks (especially medium-play and large-play) and at smaller delays (∆=4, 8), without adequate explanation of when or why model-based approaches might be preferable.\n\n2. Limited experimental scope: Experiments are restricted to low-dimensional D4RL tasks without vision-based observations, comparisons with recent model-based offline RL methods beyond COMBO, or real-world validation of sim-to-real transfer claims.\n\n3. Incomplete computational analysis: Training time, memory requirements, and scalability to delays beyond ∆=16 are not evaluated, and the impact of the transformer's larger parameter count (7.87M vs 1.80M) on sample efficiency is unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)