Directly Forecasting Belief for Reinforcement Learning with Delays
Authors: Qingyuan Wu
Yuhui Wang
Simon Sinong Zhan
Yixuan Wang
Chung-Wei Lin
Chen Lv
Qi Zhu
Jürgen Schmidhuber
Chao Huang
Venue: ICML 2025 poster (2025)
Keywords: reinforcement learning
reinforcement learning with delays
belief representation
Primary area: reinforcement_learning
TL;DR: We present the directly forecasting belief method which can effectively reduce the compounding errors and improve performance.

Abstract: Reinforcement learning (RL) with delays is challenging as sensory perceptions lag behind the actual events: the RL agent needs to estimate the real state of its environment based on past observations. State-of-the-art (SOTA) methods typically employ recursive, step-by-step forecasting of states. This can cause the accumulation of compounding errors. To tackle this problem, our novel belief estimation method, named Directly Forecasting Belief Transformer (DFBT), directly forecasts states from observations without incrementally estimating intermediate states step-by-step. We theoretically demonstrate that DFBT greatly reduces compounding errors of existing recursively forecasting methods, yielding stronger performance guarantees. In experiments with D4RL offline datasets, DFBT reduces compounding errors with remarkable prediction accuracy. DFBT's capability to forecast state sequences also facilitates multi-step bootstrapping, thus greatly improving learning efficiency. On the MuJoCo benchmark, our DFBT-based method substantially outperforms SOTA baselines. Code is available at \href{https://github.com/QingyuanWuNothing/DFBT}{https://github.com/QingyuanWuNothing/DFBT}.

Directly Forecasting Belief for Reinforcement Learning with Delays
Qingyuan Wu
Yuhui Wang" *2 Simon Sinong Zhan *
Yixuan Wang² Chung- Wei Lin
Chen L
Qi Zhu Jürgen Schmidhuber 26 Chao Huang'
Abstract
world applications, particularly the delays that occur in the interaction between the agent and the environment due to physical distance, computational processing, and signal transmission. Otherwise, delays fundamentally affect the system's 's safety (Mahmood et al., 2018), performance (Cao et al., 2020) and efficiency (Hwangbo et al., 2017). Un- like gewardsdellas--aiisiie
Reinforcement learning (RL) with delays is chal- lenging as sensory perceptions lag behind the actual events: the RL agent needs to estimate the real state of its environment based past observations. State-of-the-art (SOTA) methods typically employ recursive, step-by-step fore- casting of states. This can cause the accumu- lation of compounding errors. To tackle this problem, our novel belief estimation method, named Directly Forecasting Belief Transformer (DFBT), directly forecasts states from obser- vations without incrementally estimating inter- mediate states step-by-step. We theoretically demonstrate that DFBT greatly reduces com- pounding errors of existing recursively forecast- ing methods, yielding stronger performance guar- antees. In experiments with D4RL offline datasets, DFBT reduces compounding errors with remark- able prediction accuracy. DFBT's capability to forecast state sequences also facilitates multi- step bootstrapping, thus greatly improving learn- ing efficiency. On the MuloCo benchmark, our DFBT-based method substantially outper- forms SOTA baselines. Code is available http:///gthub.com///nngg WFF.
credit assigmment issues that have been well studied (Arjona-Medina et al., 2019; Wang et al., 2024), observation-delay: and action-delays disrupt the Markovian property of the Markov Decision Process (MDP), posing more challenges in RL. Most of the lit- erature (Kim et al., 2023; Wang et al., 2023a; Wu et al., 2024b) mainly focuses on observation-delays, which has been proved as a superset of action-delays (Katsikopoulos Engelbrecht, 2003; Nath al., 2021). Therefore, this work mainly focuses on the RL with delays A in observation: at time step f, the agent can not observe the real environment state St but only the history state S A steps ago.
To enable RL in environments with delayed observations, it essential to restore the Markovian property (Alt- man & Nain, 1992; Katsikopoulos & Engelbrecht, 2003). Augmentation- methods (Bouteiller et al., 2020; Kim et al., 2023) are based on an observation that the infor- mation state 2; = {s:-A: 04-A:-1), augmented from the last observable state and the sequential actions, carries the equivalent information with &t (Bertsekas, 2012). Thus, the policy learning over the original state space S can be trans- ferred the policy learning over the augmented state space X, which is memoryless and can be addressed by nominal RL. However, delays A increase, the dimensionality of the augmented state space X' expands significantly, lead- ing to a drastic deterioration in learning efficiency due to the exponentially increased sample complexity (Wu et al., 2024b).
1. Introduction
Reinforcement learning (RL) has achieved impressive suc- cess in various scenarios, including board games (Silver et al., 2016; Schritwieser et al., 2020), video games (Mnih et al., 2013; Berner et al., 2019) and cyber-physical sys- tems (Wei et al., 2017; Wang et al., 2023b;c; Zhan et al., 2024). The timing factor is critical to enable RL in real-
To improve learning efficiency in augmentation based meth- ods, belief-based methods (Walsh et al., 2009; Chen et al., 2021 propose to conduct RL in the original state space S, by estimating the instant unobservable state s, from the information state 2t of which the mapping is referred to as belief. The belief function is commonly forecasted re- cursively (Chen et al., 2021a; Karamzade et al., 2024): for L....., A, St-A+: ; estimated by applying approximate dynamic function with the previous state 8t-A St-Ati1 and the
'Equal contribution Uniieesstty of Southampton GGnnA, King Abdullah University of Science and Technology 'Northwestern University 'National Taiwan University NNnyynng Technological University 'The Swiss Al Lab IDSIA/USISUPSI. Correspondence to: Chao Huang <chao. <chao.huange @ soton.ac.uk>.
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Directly Forecasting Belief for Reinforcement Learning with Delays
previous action dt-A+i-1- Obtaining the estimation of the instant state St, the delayed RL problem is effectively re- duced to a delay-frre RL problem without enlarging the state space, mitigating the curse of dimensionality. How- ever, this recursive process is evidently affected by the error accumulation of the approximate dynamic function across A steps: the compounding errors grow exponentially with the delays A. This fundamental limitation of such recur- sive methodology for belief forecasting leads to significant performance degradation, especially in environments with long-delayed signals.
contributions are summarized as follows:
We present Directly Forecasting Belief Transformer (DFBT), a novel directly forecasting belief method that effectively addresses compounding errors in recur- sively generated belief.
We propose DFBT-SAC, a novel delayed RI. method that further improves the learning efficiency via multi- step bootstrapping on the DFBT.
We theoretically demonstrate that our directly forecast- ing belief significantly reduces compounding errors compared to existing recursively forecasting belief ap- proaches, offering a more robust performance guaran- tee.
This work aims to squarely address the compounding er- rors in existing belief-basee techniques with recursive strat- egy, using direct strategy in sequence modeling. Specif- ically, we present the Directly Forecasting Belief Trans- former (DFBT), novel directly forecasting belief method by reformulating belief forecasting as sequence model- ing problem. DFBT first represents the information state I, and reward signals At- to A tokens in the form 14-1 Unlike recursively fore- casting belief methods that invoke the forward predic- tion process A times, DFBT simultaneously forecasts the states 11 without introducing accumulated er- rors, effectively addressing the issue of compounding errors. We then integrate DFBT with the Soft Actor-Crite (SAC) method (Haarnoja et al., 2018) in online learning, incorporating multi-step bootstrapping on accurate state predictions generated from DFBT, which further improves learning efficiency. Theoretically, we demonstrate how the compounding errors in recursively forecasting belief affect the RL performance and how it is alleviated by our direct strategy. Empirically, using the D4RL benchmark (Fu et al., 2020), we show that DFBT achieves significantly higher prediction accuracy compared to other belief methods. On the MuJoCo benchmark (Todorov et al., 2012), across var- ious delays settings, we demonstrate that our DFBT-SAC consistently outperforms SOTA augmentation-bast and belief-based methods in both learning efficiency and overall performance.
We empirically demonstrate that our DFBT method effectively forecasts state sequences with significantly higher prediction accuracy compared to baseline ap- proaches.
{st-A, at- A+: /1-
We empirically show that our DFBT-SAC outperforms SOTAs in terms of sample efficiency and performance on the MuJoCo benchmark.
2. Related Works
In classical control theory, delays are modelled with de- diy-eiffeeettial equations (DDEs) (Cooke, 1963). A rich toolbox is available for analysing their reachability (Frid- man & Shaked, 2003; Xue et al., 2020), stability (Feng et al., 2019), and safety (Xue et al., 2021). These techniques, how- ever, assume fully known dynamics and become impractical for high-dimensional systems. Delayed reinforcement learn- ing (RL) is far closer to real deployments than the delay-free setting that dominates the literature. Latencies are intrinsic to robotics (Mahmood et al., 2018; Hwangbo et al., 2017), high-frequency trading (Hasbrouck & Saar, 2013), intel- ligent transportation (Cao et al., 2020), and many other domains. Depending on the methodology of retrieving the Markovian property, existing delayed RL approaches can be mainly categorized as augmentation-ben based and belief-based approaches.
In Section 3, we introduce the delayed RL problem and the concept of belief representation. Section 4 presents our proposed DFBT, which directly forecasts states from delayed observations, avoiding the need for recursive single- step predictions. Additionally, we develop DFBT-SAC by leveraging multi-step bootstrapping on the states predicted by DFBT. Through theoretical analysis in Section 5, we show that DFBT effectively mitigates the compounding er- rors associated with recursively forecasting belief methods, ensuring superior performance. Finally, in Section 6, we empirically demonstrate the superior prediction accuracy of DFBT on the D4RL datasets and the remarkable efficacy of DFBT-SAC compared to state-of of the-art approaches MuJoCo across various delays settings. Overall, our key
Augmentation- Approaches. Augmentation- approaches retrieve the Markovian property by augmenting the original state with the sequence of actions within the delays window, optimizing the policy in the resulting aug- mented MDP to mitigate performance degradation caused by errors in belief representation approximation. Specifically, DIDA (Liotet et al., 2022) learns the delayed policy in the augmented MDP by imitating the behaviours from the delay- free expert policy in the original MDP; DC/AC (Bouteiller et al., 2020) suggest using the delays correction operator
2

---

Directly Forecasting Belief for Reinforcement Learning with Delays
to improve the learning efficiency in the augmented MDP; ADRL (Wu et al., 2024b) introduces the auxiliary delayed task with the smaller augmented MDP for bootstrapping the larger augmented MDP; VDPO (Wu et al., 2024a) proposes to learn the delayed policy through the lens of variational in- ference, improving the learning efficiency via extensive op- approaches
prehensive theoretical analysis of how compounding errors seriously degenerate performance delays are increased.
3. Preliminaries
3.1. From Delay-free RL to Delayed RL
timization tools. However, augmentation-bast still operate on an increasingly large augmented state space as delays grow, inevitably suffering from the curse of dimen- sionality and resulting in significant learning inefficiencies.
A conventional delay-free RL problem is usually formal- ized as a Markov Decision Process (MDP) represented as tuple (S, A. P, R. p. ?)), where S is the state space, A is the action space, P S A x S - [0, 1] is the dynamic function, R: S x A + R is the reward function, p is the initial state distribution, and y E (0, 1) is the discount factor. The agent selects an action O ~ (-|sst) according to the policy π. : :S x A + [0. 1] based on the current state 8t at time step t. The agent will then observe the next state 8t+1 ~ P(. at) and the reward r = R(s,, a,) from the environment. The objective of the agent is to find the optimal policy n¹ that maximizes the expected discounted return G := (r)) -- R'R(st, az)] where p (T) rep- resents the distribution of trajectories induced by policy æ.
Belief-based Approaches. The belief-based approach re- trieves the Markovian property by forecasting the unobserv- able state through belief representation, then optimizing policy within the original state space. Thus, correspond- ingly, the prediction accuracy of approximated belief rep- resentation is highly related to the overall performance. DATS (Chen et al., 2021a) is the first to propose approximat- ing the belief representation using Gaussian distribution. Inspired by the world models (Schmidhuber, 1990h;a; Ha & Schmidhuber, 2018), D-Dreamer (Karamzade et al., 2024) employ a recurrent neural network to capture temporal de- pendencies in the belief through forward prediction in the latent space, enabling adaptation to high-dimensional obser- vations. D-SAC (Liotet et al., 2021) utilizes the attention mechanism in the causal transformer to integrate the infor- mation across the delays. Unfortunately, the approximation errors of recursively forecasting belief accumulated at each step finally lead to compounding errors, which hinder pre- diction accuracy and performance.
A delayed RL problem can be formalized as an augmented MDP by applying the augmentation technique (Altman & Nain, 1992; Katsikopoulos & Engelbrecht, 2003) to re- trieve the Markovian property. For a delayed RL problem with constant delays A, the newly formed MDP is repre- sented as (X, A, Pa. where X == S x A is the augmented state space, A is the original action space, the delayed dynamic function is Pa (J0+, [Te, ar)
P P(st-A-1/St-A a1-2) (at) I1 (a)_()a)_)) where 8 is the Dirac distribution, the delayed reward function is defined as RA(1,, ax) = Ber-d(-1ze)) [R(s,, at)] where b : ::x' x S + [0, 1] is the belief representation mapping from the augmented state space X' to the original state space S, Pa = p II 5a the initial augmented state distribu- tion.
Time Series Forecasting in RL. Time series forecasting plays a critical role in predicting future states in RL, par- ticularly within the context of world model learning (Ha & Schmidhuber, 2018), which simulates the dynamics of the environment, predicting future states and planning effec- tively without directly interacting with the real environment. World model learning often leverages time series forecast- ing techniques to capture temporal dependencies, enhancing the agent's 's understanding of the environment. Applications of world models include roboties, autonomous driving, and gaming scenarios (Hafner et al., 2019). Additionally, the RL problem can be treated as a sequence modeling problem, which can be effectively solved by the transformer ( (Vaswani et al., 2017). Directly predicting the action from the desired outcomes (Schmidhuber, 2019), decision transformer (Chen et al., 2021b) predict future actions based on historical tra- jectories and desired outcomes. Similarly, trajectory trans- former (Janner et al., 2021) models trajectories as sequences, enabling policy optimization by leveraging autoregressive modeling. Therefore, in this paper, we investigate how to leverage the advanced time series forecasting approaches in addressing the issue of compounding errors existing in the recursively forecasting belief. We also present a com-
3.2. Belief Representation in Delayed RL
Delayed RL can be viewed special form of a partially observable RL problem where the observation is contami- nated by noise, instead of being delayed. Therefore, similar to partially observable RL, delayed RL also have the belief representation defined as follows:
b(sx (r)) =
A-1 / 1=0
(1)
The belief representation can retrieve the Markovian prop- crty via mapping the augmented state space X' to S, recast- ing the delayed RL problem in the original MDP without augmenting the state space. The belief representation can be viewed as the recursive forward prediction of the dynamics
3

---

Directly Forecasting Belief for Reinforcement Learning with Delays
P. With the belief representation, the agent can directly learn in the original state space S. In this work, and á represent the true state of the environment and the predicted state of the approximate belief, respectively.
Algorithm DFBT-SAC
Input: offline dataset D, DFBT by, critic Qu actor TTi # training DFBT on the offline dataset
for Epoch ..... do
Update by the D via Equation (2)
4. Our Approach
end for
# learning with DFBT on the online environment
In this section, we present the Directly Forecasting Be- lief Transformer (DFBT), new directly forecasting belief method. By framing belief forecasting as sequence model- ing problem, DFBT directly predicts the unobservable states. Instead of recursively predicting the next state step-by-step, DFBT can effectively capture the dependency across the de- lays via the attention mechanism of the transformer model. Therefore, our DFBT can effectively reduce the compound- ing errors of the recursively forecasting belief, especially in the long delays. Specifically, as summarized in Figure 1, we train the DFBT using a pre-collected trajectory dataset, latter deploying the trained DFBT in the environment with delays to reconstruct the delay-free environment for training. Furthermore, we integrate multi-step bootstrapping with the predicted states of DFBT to improve learning efficiency.
for Epoch = 1, do
Update Qo on via Equation (3)
Update T on via Equation (4)
end for Output: by and T
space X, thus maintaining the superior sample complexity in online learning.
Then, we present our practical delayed RL method, named DFBT-SAC by incorporating the trained DFBT be with Soft Actor-Critic (Haarnoja et al., 2018). To improve the learning efficiency with the DFBT, the critic of DFBTT- SAC is bootstrapped on the states predicted by the DFBT with the multi-step learning (Sutton & Barto, 2018; Hessel et al., 2018) and delay-frre training techniques (Wu et al., 2024b; Kim al., 2023). Specifically, given multi-step data (zrlokens 81-A+1a), the critic Qş parameterized by is updated via:
4.1. Directly Forecasting Belief
The belief representation learning problem can be viewed as a sequence modeling problem. Given a pre-collected delay-free trajectory {si, (i, ri) we sample the sub- trajectory with A timesteps { a4-A4:: Pt Tt-a: ) We model the past reward signals Ti-A:t-1 into the aug-
1 (Q0())-)) x)²
(3)
2
mented state z2- = (s:--.. Ot-Aa-1} for considering more dynamic information in belief. Then, we have reformed the representation of the augmented state to A tokens for se- Y=
where N N-step temporal difference target Y is defined as:
N-1
Σ
quence modeling: tokeess (8t-2, At-A+i: "+-A+11 1A-1 The Directly Forecasting Belief Transformer (DFBT) bo leverages the transformer architecture (Vaswani et al., 2017), utilizing the attention mechanism effectively capture dependencies across long delays. Inputting with zoookens DFBT simultaneously predicts the unobserved A states ( via autoregressive modeling with loss:
1=0
+2
E
[Q(=-A+NN + log ((lss=-A+)))
A-M Re + (-1ak
and N(S A) is the bootstrapping steps on the DBFT. In this work, we set = 8 as default, and we also conduct the ablation study (Section 6) investigate bootstrapping steps settings. The actor Fe parameterized by Φ is updated by:
log (/0² (st (sr-a+ilu (ikkene))) ]]
V
E
[log n (a|s: - Qu(s:-AtN,))
a- 4 ++-A+N (:Ipplen)
where (-(zr) represents the i-th prediction. In determin- istic environment with deterministic belief function, we typically replace Equation (2) with the Mean-Square-Error (MSE) loss.
(4) The pseudo-code of DFBT-SAC provided Algorithm 1.
5. Theoretical Analysis
4.2. Multi-step Bootstrapping with DFBT
In this section, we theoretically illustrate that performance degeneration is rooted in compounding errors of recursively forecasting belief (Section 5.1), which can be effectively addressed by our DFBT, directly forecasting belief, thus achieving a better performance guarantee (Section 5.2).
Next, we directly deploy the trained DFBT by in the online environment with delayed signals to reconstruct the delay- free environment where the agent can directly learn with the original state space S instead of the augmented state

---

Directly Forecasting Belief for Reinforcement Learning with Delays
(o) Training Dirvet Forsounting Belief
(b) Learning with Direct Forecasting Bellef
(c) Mulli-step Rootsropping with the Direct Forecasting Bellef
Freaa By a -, 0, a.)
Directly Forecastnng
Plant with
Ar3
Belief Transformer
Multi-step Return
Projactory Dataset
Agent
Deley-free Plant
Forecasted
Figure Pipeline DFBT-SAC. (a) Training DFBT on trajectory dataset. (b) The agent can interact and learn with the delay-free free environment recovered by the DFBT highlighted in the dashline box. (c) Multi-step bootstrapping on the forecasted states from DFBT.
Before starting the theoretical analysis, introduce the definition of the performance degeneration of the ground- truth belief b as follows.
pounding errors, which consist of the recursive and are exponentially increased with delays. We assume that the recursively forecasting belief Po has the approximated error bound as shown in Assumption 5.4.
Definition 5.1 (Performance Degeneration of Ground-truth Belief (Liotet et al., 2022)). For policies π and =λ with delays A. Given any 2; E X, the performance difference of the ground-truth belief b is denoted as /
Assumption 5.4 (Approximated Dynamic Difference Bound). The distance between the approximated dynamic function Pe parameterized by @ and the ground-truth dy- namic function p is bounded, it satisfies that v(s, a) S x A,
A (I4)
[VT(S) Q" (s,
8-
AOITA 8)
S epp
Then, we demonstrate that the performance difference of the recursively forecasting belief determined by the com- pounding errors (Theorem 5.5).
where d" is the augmented state distribution induced by policy x with the initial state Zt.
Furthermore, we also introduce the Lipschitz Continuity of MDP (Definition 5.2) and value function (Definition 5.3), which are common and mild assumptions in the literature. Definition 5.2 (Lipschitz Continuity of MDP (Rachel- & Lagoudakis, 2010)). An MDP is Lp-LCC, if a1),, (32, a2) S x A. dynamic function P satisfies: W (P(-1ss $1, a1) || S2, a2)) & (p(d(($1, +2)+ a2)),
Theorem 5.5 (Performance Difference of Recursively Fore- casting Belief, Proof in Theorem B.1), For the delay-free policy т and the delayed policy T&. Given any E X, the performance difference precusive (xx) of the recursively fore- casting belief be can be bounded as follows, respectively.
For deterministic delays A we have
Lp
precursive (x)) S 1 (x.) by 1 L. cp
where W is the I.1 -a--assersttein distance, ds and A are distrance measure of the S and A, repectively.
compounding errors
Definition 5.3 (Lipschitz Continuity of Value Func- tion (Rachelson Lagoudakis, 2010)). Consider a LQ-LC Q-function Q of the LLC policy ı,, value function V satisfies that
And for stochastic delays 8 ~ ds(.), we have
Lp
pecursiiv (x+)
115TH
+ Lv 1 Lp
drudla) (-)
E (si) - (sa)) #2
Lvl(ull)))
compounding errors
where Lv = (1 + Lx) and μ4, are two arbitrary distri- butions over
Theorem 5.5 tells that the compounding errors are exponen- tially increased with the delays, seriously degenerating the performance.
5.1. Recursively Forecasting Belief: Compounding Errors Analysis
5.2. Directly Forecasting Belief:
Errors Analysis
In this section, we show that the performance degenera- tion of recursively forecasting belief influenced by com-
Next, we theoretically show that our directly forecasting belief can effectively address the compounding errors. The
5

---

Directly Forecasting Belief for Reinforcement Learning with Delays
Beled Frror barm)
Bajuf Emeor (LL Marm))
64
140
00
20.0
26
64
160
68
24
12
22
Delays
Delays
Delays
(a) HalfChoctah-v2
(b) Hopper-v2
(c) Walker2d-v2
DATS
D-Dreamer
D-SAC
DFBT-SAC (ours)
Figure 2. Belief errors comparison on (a) HalfCheetahh-Ve -2, (b) 1lopper-v2, and (c) Walker2d-v2.
sequence-modeling method estimates the states +1: 1: based on the augmented state in parallel, instead of in- voking times iteratively, which effectively alleviates the source of compounding errors. Especially, the attention mechanism in the transformer can selectively capture the long-range relationships within the augmented state with long delays. We assume that the belief error between the directly forecasting belief and the ground-truth belief is bounded (Assumption 5.6).
for deterministic delays A, and
1 Lp
Edirect I
E 6rida(.)
EP
1 Lp
for stochastic delays 8 ds(-).
Remark 5.9 (Empirical Validation of Proposition 5.8). is obvious that the belief errors of recursively forecasting grows much faster than the directly forecasting, which is not strictly related with the delay length. We also empirically show that Proposition 5.8 is always held in Section 6.2.1. In the future, we will theoretically investigate the sample complexity of recursive and direct forecasting beliefs.
Assumption 5.6 (Directly Forecasting Belief Difference Bound). The distance between the directly forecasting belief belief be parameterized by and the ground-truth belief is bounded, it satisfies that Vx, E X. we have W(bb(lazz)lbbb((eea < Edirect, where Edirect := max W(b( (1xx)) (-(x2)).
From Theorem 5.5, we derive the performance degener- ation bound of directly forecasting belief as follows.
Remark 5.10 (General Error Distribution Case). In the con- text of time forecasting, our theoretical results of perfor- mance degeneration comparison have the potential to ex- tend to the variance analysis commonly discussed in related literature (Taieb Atiya, 2015; Clements & Hendry, 1996; Chevillon, 2007).
Proposition 5.7 (Performance Degeneration Bound of Di- rectly Forecasting Belief, Proof in Proposition B.2). For the delay-free policy and the delayed policy Th- Given any In X, the performance degeneration ddreect of the directly forecasting belief be can bounded as follows respectively.
6. Experiments
For deterministic delays A, we have
Iddrect (xx)) S 1h (xx)) + Lyvdiveat-
6.1. Experimental Setting
For stochastic delays d ~ (.).), we have piidd (xx) <
We adopt D4RL. (Fu al., 2020) and MuJoCo (Todorov et al., 2012) the offline dataset and the benchmark re- spectively to evaluate our DFBT-SAC. For the baselines, we choose the SOTA augmentation-baas SAC (Haarnoja et al., 2018), BPQL (Kim et al., 2023), and ADRL (Wu et al., 2024b)) and belief-based methods (DATS (Chen al., 2021a), D-Dreamer (Karamzade et al., 2024), and D-SAC (Liotet al., 2021)). All of the belief- based baselines and our DFBT-SAC are trained on the same D4RL dataset. We first investigate the prediction accu- racy of beliefs (Section 6.2.1), followed by the performance comparison with deterministic and stochastic delays (Sec- tion 6.2.2). Additionally, we conduct ablation studies on the multi-step bootstrapping of our DFBT-SAC (Section 6.2.3).
E (-)a())
11 [z + Edseree
methods (A-
Then, we have the performance degeneration comparison be- tween directly forecasting belief and recursively forecasting belief.
Proposition 5.8 (Performance Degeneration Comparison, Proof in Proposition B.3). Directly forecasting belief could achieve a better performance guarantee [ I
preurssve (xt)), . if we have
1 Ipp A
Edirect <
Ep
Lp
6

---

Directly Forecasting Belief for Reinforcement Learning with Delays
Table 1. Performance on MuJoCo with Deterministic Delays.
The best performance is underlined, the best helicf-hased method is in red.
Augmentation-
Belief-hased
Task
Delays
A-SAC
BPQL
ADRL
DATS
D-Dreamer
D-SAC
DFBT-SAC()) (ours)
DFBT-SAC (ours)
8
0.10
0.40
0.44
0.08
0.08
0.12=0.06
0.38
0.35
HallCheetab-v2
32
0.0220.02
0.40
0.2620004
2.00
0.08g0000
0.08
0.40
0.42
128
0.04
0.08
0.14g0.02
0.10
0.15
0.09
0.4020.06
0.41
8
+0.31
00.00
0.95
+0.31
0.1 =.10
0. 0.16:0.05
0.92
0.77=0.18
1lopper-V2
32
2.102
0.89
0.73
0.07
0.11 =0.0s
0.11=0.01
0.60
0 0.68
128
0.04gn.01
0.08
0.07
0.08
0.09
0.00ga.01
0.16=0.02
0.20
8
0.44
10000
1.
0.13
0.11
0.0920.0
0.95
0.99
Walker2d-v2
32
0.10
0.37
0.16
0.02
0.08
0.08_
0.
0.64
128
0.06
00.0s
0.08
0.02
0.08
0.105
0.38
0.40==0.8
Table 2. Performance on MuloCo with Stochastic Delays,
The best performance is underlined, and the best belicf-based based method is in red.
Augmentation--usedd
Belief-bused
Task
Delays
A-SAC
BPQL
ADRL
DATS
D-Dreamer
D-SAC
DFBT-SAC (ours)
()
0.09
0.21
0.17
0.09
0.02
0.03_o.01
0.37=0.12
HalfCheetah-vz v2
3(1
0.01
0.03
0.23
0.1 0.11
0.02
0.01_a.01
0.3
U(1, 128)
0.00
0.03
0.15
0. 0.1640.03
0. 0.160.000
0.02_0.00
0.39
U[1,8)
0.17+0.05
0.20+0.04
0.18
0.04
0.07
0.14_0.04
0.86
Hopper-v2
321)
0.05
0.07
0.03
0.05
0.04
0.03_0.01
0.43=a.21
(11, 128)
0.03
0.04ga.01
0.04g0002
0.05
0.03ga.01
0.03_a.00
0.14
U1,
0.
0. 1..02
1.
0.07
0.07
0.12_0.04
1.11
Walker2d-v2
3())32)
0.
0.1610.04
0.11
0.09
0.12
0.05_0.02
0.67
U11, 128)
0.06
0.06
0.04
0.10
0.15
0.03 04
0.30
6.2. Experimental Results
Deterministic Delays. The performance of DFBT-SAC and baselines are evaluated on MuJoCo with deterministic delays (8. 32, and 128) reported in Table 1, showing that our DFBT-SAC overall outperforms other belief-based methods significantly. Specifically, DFBT-SAC shows comparable performance with the SOTA augmentation- based methods (BPQL. and ADRL) on tasks with relatively short delays (8). However, our DFBT-SAC yields a leading performance on challenging tasks when the delays increase to long delays (32 and 128). Additionally, we also compare DFBT-SAC(I), the single-step bootstrapping version of DFBT-SAC. The results imply that the multi-step bootstrapping technique can further improve the performance effectively, validat- ing our statement in Section 4.2. We will investigate the bootstrapping steps in the later ablation study.
6.2.1. 1. BELIEF PREDICTION ACCURACY
We first evaluate the state prediction accuracy of our DFBT in D4RL offline datasets. Using the L1 norm as the belief error, We report the error curves increasing with delays of the belief of DATS, D-Dreamer, D-SAC and our DFBT- SAC in Figure 2. All methods are trained on the D4RL mixed dataset including random, medium and expert policy demonstrations. The implementation details are provided in Appendix A. From the results, we can tell that our DFBT can address the compounding errors effectively via directly forecasting delayed observations, thus maintaining the best prediction accuracy with increased delays, which is con- sistent with theoretical results. We further provide the belief qualitative comparison in Appendix E.
Stochastic Delays. We also evaluate DFBT-SAC on Mu- JoCo with stochastic delays which follow the uniform dis- tribution U. As shown in Table 2, our DFBT-SAC remark- ably outperforms all belief-based baselines on all tasks with all delays settings (U 8), (1,32), and (11 128)). Es- pecially for U11, 128) delays, DFBT-SAC also performs approximately 144%, 180%, and 100% better than the sec- ond best baselines on HalfCheetah-v2, Hopper-v2, and Walker2d-v2, respectively.
6.2.2. PERFORMANCE COMPARISON
We report the normalized return Rnomn := Rag Rundon Rg Roppens where Ralg. Reac Reandom are the return of the algorithm, delay-free SAC, and random returns, respectively. Each method is evaluated across 5 random seeds, and the imple- mentation details are provided in the Appendix A. Addi- tional experiment results and learning curves for different delays are provided in the Appendix C and Appendix D, respectively.
Learning Curves on MuJoCo with Challenging Delays. As summarized in Figure 3, we report the learning curves on MuloCo with 128 and U(1, 128) delays. Our DFBT-
7

---

Directly Forecasting Belief for Reinforcement Learning with Delays
0.5
0.20
Ratum
Batum
6:05
00
0.2
Global
1/
50
Global
36
0.
82
Global 633
30.
(a) HalfCheetah-v2 (128 Delays)
(b) Hopper-v2 (128 Delays)
(c) Walker2d-v2 (128 Delays)
0.4
0.4
M
0.3
0.12
0.3
Ratum
0.1
6:22
Global Steps
Glbal Steps
Global Steps
(d) HalfCheetah-v2 (U(1,, 128) Delays)
(c) Hopper-v2 (U(1, 128) Delays)
(f) Walker2d-v2 (1, 128) Delays)
A-SAC
BPQL
ADRL
DATS
D-Dreamer
D-SAC
DFBT-SAC (ours)
Figure 3. Learning Curves on MuloCo with 128 Delays and U(1, 128) Delays.
SAC exhibits a leading learning efficiency across all these challenging delays. Especially in the MuJoCo with 128 delays, DFBT-SAC can leam remarkably faster than all baselines, resulting in 173.3% (HalfChectah-v2), 122.2% (Hopper-v2), and 263.6% (Walker2d-v2) final performance improvement better than the second-best baselines.
6.2.3. ABLATION STUDY ON BOOTSTRAPPING STEPS
As mentioned in Section 4.2, we conduct the ablation study on the bootstrapping steps in DFBT-SAC. The result pre- sented in Table 3 tells us that DFBT-SAC with bootstrap- ping steps achieves the averaged best performance compared to other choices. It is also confirmed that bootstrapping with the states predicted by the trained DFBT can effectively improve performance.
Table 3. Final Performance on Walker2d-v2 DFBT-SAC with different bootstrapping steps. The best performance is underlined.
Results Analysis and Discussion. Based on the above experimental results, we can observe general trend in both deterministic and stochastic delays that DFBT-SAC shows comparable performance with SOTA augmentation- based methods under short delays scenarios, in which ingmentation-basel methods generally have better perfor- mance than other belief-based approaches. Since when delays are short, the dimensionality of the augmented state space is proper for learning, and without the approxima- tion errors from belief representation. However, as delays increase in both scenarios, DFBT-SAC unleashes its advan- tages. This empirical finding validates our theoretical anal- ysis in Section 5. For augmentation-bant delays increase, the augmented state in augmentation-baat methods grows too rapidly, making efficient policy optimiza- tion infeasible. For recursively forecasting belief methods, the compounding errors in belief function approximation grow exponentially with increasing delays, as demonstrated in Figure and Section 6.2.1. This results in inaccurate predictions of delayed observations and undermines subse- quent policy training. In contrast, DFBT's prediction error remains largely unaffected by the length of delays, enabling DFBT-SAC to maintain strong performance even in scenar- ios with long delays.
Bootstrapping Steps N
Delays
2
4
8
0.86
0.96
1.00_0.14
1.11
16
0.84
0.76
1.02_0.12
0.99
32
0.63
0.
0.67
0.67
64
0.34
0.28
0.29
0.41
128
0.24
0.29
0.27
0.30
Table 4. Performance comparison of DFBT-SAC with different belief training methods MuJoCo tasks with 32 delays. The best performance is underlined.
methods, as the
Task
Online
Offline
Offine Fine-luning
HalfChectah-v2
0.11 0.39
0.42
0.12
0.39 0.04
Hlopper-v2
0.10 0.58
0.68 1 0.20
0.04
Walker2d-v2
0.09 0.27
0.64 1 0.10
0.96 1 0.32
6.2.4. EVALUATION ON ADDITIONAL MuJoCo TASKS.
conducted experiments the Pusher-v2, Reacher-v2, and Swimmer-v2 v2 with deterministic 32 delays. The offline datasets (500k samples) are collected from a SAC policy, with other settings unchanged. The results are shown in Table 5, showing that DFBT-SAC achieves superior perfor- mance on these tasks.
8

---

Directly Forecasting Belief for Reinforcement Learning with Delays
Table 5. Performance comparison additional MuloCo tasks with 32 deterministic delays. The best performance is underlined.
Task
A-SAC
BPQL
ADRL
DATS
D-Dreamer
D-SAC
DFBT-SAC (ours)
Pusher-v2
0.05 + 0.00
0.93
0.58
0.95 + 0.24
0.92 + 0.10
0.87 + 0.18
0.94 + 0.04
1.04 = 0.24
Reacher-v2
0.89 + 0.08
0.08
0.83
0.06
0.85 + 0.01
0.82 + 0.13
0.84 0.02
0.88 + 0.07
0.93 + 0.06
Swimmer-v2
0.27 + 0.05
0.80 0.14
0.60 + 0.06
0.25 + 0.05
0.21 + 0.07
0.30 + 0.07
1.01 + 0.27
Table 6. Performance on stochastic MuJoCo with deterministic 128 delays. The best performance is underlined.
Task
A-SAC
BPQL
ADRL
DATS
D-Dreamer
D-SAC
DFBT-SAC (ours)
HallCheetah-v2
0.00 1 0.03
0.01 + 0.05
0.13 + 0.04
0.13 1 0.03
0.07 1 0.01
0.00 + 0.04
0.35 1 0.04
Hopper-v2
0.03 + 0.04
0.08 1 0.05
0.06 + 0.04
0.05 + 0.06
0.04 1 0.05
0.04 4: 0.05
0.13 + 0.22
Walker2d-v2
0.06 + 0.02
0.04 + 0.01
0.08 + 0.01
0.06 1 0.03
0.10 + 0.03
0.05 + 0.02
0.30 + 0.07
6.2.5. ABLATION RESULTS OF BELIEF TRAINING.
6.3. Limitations and Challenges
Some task-specific information may be missing if the belief is frozen in the RL process, leading to limited performance improvement. This issue can be mitigated by fine-tuning the belief within the RL process. The results, shown in Table 4, demonstrate that fine-tuning helps the DFBT capture the task task-specific information with better performance. Note that there are many potential methods for capturing task-specific information, not limited fine-tuning DFBT. Belief learn- ing from scratch in the online RL process always suffers from instability issues. Therefore, in this paper, we separate belief learning from the online RL process, which allows us to investigate the belief component solely, eliminate poten- tial influences from the RL. side.
In this work, we empirically validate that approach can address the compounding errors of recursively forecasting belief with significant performance improvement. However, there remain some limitations and challenges as follows.
Online Belief Learning. In this paper, we mainly con- sider learning DFBT from the offline dataset, which means the performance of the belief is subject to the quality and quantity of the dataset. However, direct learning belief in the online environment will introduce an auxiliary represen- tation task, destabilizing the learning process.
Sample Complexity of Directly Forecasting Belief. AI- though we have empirically demonstrated that directly fore- casting belief can effectively reduce the compounding errors of recursively forecasting belief. However, is worth theo- retically analysing the sample complexity of belief learning, especially in online scenarios.
6.2.6. STOCHASTIC MuJoCo.
We conducted additional experiments the stochastic Mu- JoCo tasks with a probability of 0.001 for the unaware noise and deterministic 128 delays. As shown in Table 6, the results demonstrate that our DFBT-SAC achieves superior performance in these stochastic MuJoCo tasks.
7. Conclusion
This work investigates the challenges of RL in environments where inherent delays exist between actions and their cor- responding outcomes. Existing belief-based approaches usually suffer from the compounding errors issue of the recursively forecasting belief as the delays are increased, seriously hindering the performance. To resolve this issue, we present DFBT, new directly forecasting belief repre- sentation method. Furthermore, we present DFBT-SAC, which facilitates multi-step bootstrapping in learning the value function via the state prediction from the DFBT, effec- tively improving the sample efficiency. We demonstrate that our DFBT greatly reduces compounding errors, yielding stronger performance guarantees. Our empirical results val- idate that DFBT has remarkable prediction accuracy in the D4RL datasets. We also empirically show that our DFBT- SAC not only effectively enhance the learning efficiency with superior performance in the MuJoCo benchmark.
6.2.7. INFERENCE SPEED COMPARISON.
We conducted additional experiments on computational ef- ficiency. As shown in Table 7, the results demonstrate that directly forecasting belief maintains a consistent and stable inference speed (around 4 ms) across different delays. In contrast, the recursively forecasting belief experiences in- ference speed issues as delays increase. In HalfChectah-v2 with 128 delays, the training times of DATS and D-Dreamer are around 10 hours and 15 hours, respectively, while those of D-SAC and DFBT-SAC are both around 6 hours.
Table 7. Inference speed (ms) comparison in HalfCheetab--2.
Delays
DATS
D-Dreamer
D-SAC
DFBT-SAC
8
1.10 1 0.02
1.85 0.01
4.03 0.03
4.18 1 0.04
32
3.85 : 0.06
6,80 +: 0.04
4.03 0.04
4.18 0.04
128
14.97 + 0.22
26.51 + 0.19
4.03 + 0.03
4.15 + 0.05
9

---

Directly Forecasting Belief for Reinforcement Learning with Delays
Acknowledgement
Chen, B., Xu, M., Li, L.,, and Zhao, D. Delay-aware model- based reinforcement learning for continuous control. Neu- 2021a.
We sincerely appreciate the insightful suggestions provided by ICML 2025 reviewers, which help us improve the quality of this paper. We sincerely acknowledge the support by the grant EP/Y002644/1 under the EPSRC ECR International Collaboration Grants program, funded by the International Science Partnerships Fund (ISPF) and the UK Research and Innovation. We also sincerely acknowledge the support by Taiwan NSTC under Grant Number NSTC-112-2221 E-002-168-MY and from King Abdullah University of Science and Technology (KAUST) Center of Excellence for Generative AI, under award number 5940. Simon Sinong Zhan and Qi Zhu are partially supported by the US National Science Foundation grants 2324936 and 2328973.
rocomputing,
Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. De- cision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-15099 2021b.
Chevillon, G. Direct multi-step estimation and forecasting. Journal of Economic Surveys, 21(4):746-785, 2007.
Clements, M. P. and Hendry, D. F. Multi-step estimation for forecasting. Oxford Bulletin of Economics and Statistics, 58(4):657-684, 1996.
Impact Statement
Cooke, K. L. Differential difference equations. In Inter- national symposium on nonlinear differential equations and nonlinear mechanics, pp. 155-171. Elsevier, 1963.
This paper aims to advance the field of Machine Learning. While acknowledging there are many potential societal con- sequences of our work, we believe that none need to be specially highlighted here.
Feng, S., Chen, M., Zhan, N., Fränzle, M., and Xue, B. Taming delays in dynamical systems: Unbounded veri- fication of delay differential equations. In International Conference on Computer Aided Verification, pp. 650-669. Springer, 2019.
References
Altman, E. and Nain, P. Closed-loop control with delayed information. ACM sigmetrics performance evaluation review, 20(1):193-204, 1992.
Fridman, E. and Shaked, U. On reachable sets for linear systems with delay and bounded peak inputs. Automatica, 2003.
Arjona- J. A., Gillhofer. M., Widrich, M., Un- terthiner, T., Brandstetter, J., and Hochreiter, S. Rudder: Return decomposition for delayed rewards. Advances in Neural Information Processing Systems, 32, 2019.
Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning, 2020.
Asadi, K., Misra, D., and Littman, M. Lipschitz continuity in model-based reinforcement learning. In International Conference on Machine Learning, pp. 264-273. PMLR, 2018.
Ha, D. and Schmidhuber, J. World models. arXiv preprint arXiv: 1803. J0122, 2018.
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05900 2018.
Berner, C., Brockman, G., Chan, B., Cheung, V., Debiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., Hesse, C., et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint 19Xiv:1912.06660 2019.
Hafner, D., Lillicrap, T., Ba, J., and Norouzi, M. Dream to control: Learning behaviors by latent imagination. arXiv 2019.
Bertsekas, D. Dynamic programming and optimal control: Volume 1, volume 4. Athena scientific, 2012.
preprint arXiv:1912.01660
Hasbrouck, J. and Saar, G. Low-latency trading. Journal of Financial Markets, 16(4):646-679, 2013.
Bouteiller, Y., Ramstedt, S., Beltrame, G., Pal, C., and Binas, J. Reinforcement learning with random delays. In International conference on learning representations, 2020.
Hessel, M., Modayil, J., Van Hasselt, H., Schaul, T., Os- trovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M.. and Silver, D. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Cao, 7., Guo, H., Song, W., Gao, K., Chen, Z.,, Zhang, I.., and Zhang, X. Using reinforeement learning to mini- mize the probability of delay occurrence in transportation. IEEE transactions on vehicular technology. 69(3)24424- 2436, 2020.
Huang, S., Dossa, R. F. J., Ye, C., Braga, J., Chakraborty, D., Mehta, K., and AraAšjo, G. Cleanrl: High-quality
10

---

Directly Forecasting Belief for Reinforcement Learning with Delays
single-file implementations of deep reinforcement learn- ing algorithms. Journal of Machine Learning Research, 23(274):1-18; 2022.
Schmidhuber, J. An on-line algorithm for dynamic rein- forcement learning and planning in reactive environments. In 1990 IJCNN international joint conference neural networks, pp. 253-258. IEEE, 1990a.
Hwangbo, J., Sa, I., Siegwart. R., and Hutter, M. Control of quadrotor with reinforcement learning. IEEE Robotics and Automation Letters, 2(4):2096-21)) 2103, 2017.
Schmidhuber, J. Making the world differentiable: on us- ing self supervised filly recurrent neural networks for dynamic reinforcement learning and planning in non- stationary environments, volume 126. Inst. für Informatik, 1990b.
Janner, M., Li, Q., and Levine, S. Offline reinforcement learning as one big sequence modeling problem. Ad- vances in neural information processing systems, 34: 1273-1286, 2021.
Schmidhuber, Reinforcement learning upside down: Don't t predict rewards-just map them to actions. arXiv preprint
Karamzade, A., Kim, K., Kalsi, M., and Fox, R. Rein- forcement learning from delayed observations via world models. arXiv preprint arXiv:2403.12300 2024.
2019.
Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmiu,, S., Guez, A., Lockhart, E., Hassabis, D., Graepel. T., al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(7839): (04-609,44 2020.
Katsikopoulos, K. V. and Engelbrecht, S. E. Markov de- cision processes with delays and asynchronous cost col- lection. IEEE transactions on automatic control, 48(4): 568-574, 2003.
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. Mastering the game of with deep neural networks and tree search. nature, 429(7587)4444-48,, 2016.
Kim, J., Kim, H., Kang, J., Back, J., and Han, S. Belief projection-bad based reinforcement learning for environments with delayed feedback. In "Thirty-seventl Conference Neural Information Processing Systems, 2023.
Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018.
Liotet, P., Venneri, E., and Restelli, M. Learning a belief representation for delayed reinforcement learning. In 2021 International Joint Conference on Neural Networks (IJCNN) pp. 1-8. IEFF, 2021.
Taieb, S. B. and Atiya, A. A bias and variance analysis for multistep-ahee ahead time series forecasting. IEEE trans- actions on neural networks and learning systems, 27(1):
Liotet, P., Maran, D., Bisi, L., and Restelli, M. Delayed reinforcement learning by imitation. In International Con- ference on Machine Learning, pp. 13528-13556. PMLR, 2022.
62-76, 2015.
Tarasov, D., Nikulin, A., Akimov, D., Kurenkov, V., and Kolesnikov, S. CORL: Research-oriented deep offline reinforcement learning library. In 3rd Offline RL. Work- shop: Offline RL a "Launchpad", 2022. URL https: 9bBcv.
Mahmood, A. R., Korenkevych, D., Komer, B. J., and Bergstra, J. Setting up a reinforcement learning task with a real-world robot. In 2018 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp- 4635-4640. IEEE, 2018.
//openreview.. net
Todorov, E., Erez, T., and Tassa, Y. Mujoco: physics engine for model-based control. In 2012 IEEE/RSI inter- national conference intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonogiou, I., Wierstra, D., and Riedmiller, M. Playing atari with deep reinforcement learning, arXiv preprinl arXiv:1312.5602. 2013.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
Nath, S., Baranwal, M., and Khadilkar, H. Revisiting state augmentation methods for reinforcement learning with stochastic delays. In Proceedings of the 30th ACM Inter- national Conference on Information & Knowledge Man- agement, pp. 3346-1355 2021.
Walsh, J., Nouri, A., Li, I.., and Littman, Learning and planning in environments with delayed feedback. Autonomous Agents and Multi-Agent Systems, 18:83-105. 2009.
Rachelson, E. and Lagoudakis, M. G. On the locality of action domination in sequential decision making. 2010.
Wang, W., Han, D., Luo, X., and Li, D. Addressing signal delay in deep reinforcement learning. In The Twelfth

---

Directly Forecasting Belief for Reinforcement Learning with Delays
International Conference Learning Representations, 2023a.
Wang, Y.. Zhan, S., Wang. Z., Huang, C., Wang, Z., Yang, Z., and Zhu, Q. Joint differentiable optimization and verification for certified reinforcement learning. In Pro- ceedings of the ACM/IEEE 14th International Conference ShysiclPysicaal Systems (with CPS-IoT Week 2023), pp. 132-141, 2023b.
Wang, Y., Zhan, S. S., Jiao, R., Wang, Z., Jin, W. Yang. Z., Wang, Z., Huang, C., and Zhu, Q. Enforcing hard con- straints with soft barriers: Safe reinforcement learning in unknown stochastic environments. In International Con- ference on Machine Learning, pp. 36593-36604. PMLR, 2023c.
Wang, Y., Strupl, M., Faccio, F., Wu, Q., Liu, H., Grudzieñ, M., Tan, X., and Schmidhuber, J. Highway reinforcement learning, arXiv preprint 1.Xi::24005.82888 2024.
Wei, T., Wang. Y., and Zhu, Q. Deep reinforcement learning for building hvac control. In 2017 54th ACM/EDAC/IEEE Design Automation Conference (DAC), pp. 1-6, June 2017. doi: 10.1 1411443061139930062222
Wu, Q. Zhan, S., Wang. Y., Wang, Y., Lin, C.-W., Lv, C., Zhu, Q., and Huang, C. Variational delayed policy optimization. Advances in neural information processing systems, 2024a.
Wu, Q. Zhan, S. S., Wang. Y., Wang, Y., Lin, C.-W,, Lv, C., Zhu, Q.. Schmidhuber, J., and Huang, C. Boosting reinforcement learning with strongly delayed feedback through auxiliary short delays. In Forty-first International Conference on Machine Learning (ICMI. 2024), 2024b.
Xue, B., Wang, Q., Feng, S., and Zhan, N. Over-and under- approximating reach sets for perturbed delay differential equations. IEEE Transactions on Automatic Control, 66 (1):283-290, 2020.
Xue, B., Bai, Y., Zhan, N., Liu, W., and Jiao, L. Reach-avoid analysis for delay differential equations. In 2021 60th IEEE Conference Decision and Control (CDC), pp. 1301-1307. IEEE, 2021.
Zhan, S., Wang, Y., Wu, Q., Jiao, R., Huang, C., and Zhu, Q. wise safe reinforcement learning with pixel observations. In 6th Annual Learning for Dynamics & Control Conference, pp. 1187-1201, PMLR, 2024.
12

---

Directly Forecasting Belief for Reinforcement Learning with Delays
A. Implementation Details
The implementation DFBT and DFBT-SAC is based on CORL (Tarasov et al., 2022) and CleanRL (Huang et al., 2022). The codebase for reproducing our experimental results also provided in the Supplementary Material. We detail the hyperparameter settings DFBT and DFBT-SAC in Table 8 and Table 9, respectively.
Table 8. Hyper- parameters table of DFBT.
Hyper-pan Slyper-parameter
Value
Epoch
le3
Batch Size
256
Attention Heads Num
4
Layers Num
10
Hidden Dim
256
Attention Dropout Rate
0.1
Residual Dropout Rate
0.1
Hidden Dropout Rate
0.1
Learning Rate
le-4
Optimizer
AdamW
Weight Decay
le-4
Betas
(0.9, 0.999)
Table 9. Soyper-paramtent table of DFBT-SAC.
saper-parameter
Value
Bootstrapping Steps N
8
Learning Rate (Actor)
3e-4
Learning Rate (Critic)
le-3
Learning Rate (Entropy)
le-3
Train Frequency (Actor)
2
Train Frequency (Critic)
Soft Update Factor (Critic)
5e-3
Batch Size
256
Neurons
[256, 256]
Layers
3
Hidden Dim
256
Activation
ReLU
Optimizer
Adam
13

---

Directly Forecasting Belief for Reinforcement Learning with Delays
B. Theoretical Analysis
Theorem B.1 (Performance Difference Recursively Forecasting Belief). For the delay-free policy n and the delayed policy Nλ. Given any 24 E X, the performance difference Irecussve (x+) of the recursively forecasting belief bo can be bounded as follows, respectively.
For deterministic delays A, we have
1- Lp
prewsivee (Ix) S (x)((()
(2t) + LV 1 Lp
compouidding errory
Proof. For deterministic delays A, the performance difference recursive can be written as: And for stochastic delays 5 ~ da(.), we have
1 Lp (xx) + Lv
preursive (x+)) S E 6-ddal))
1 Lp EP
compounding
rreuursiee (x:) = E
[V*(s:)]
V* (**(xx)
serxbu(-(xre)
11 (x+)
E (11
[v*()
IE
[V*(s:)
And recall that we have the Lipschitz continuity the value function V*:
E [st()
E [" & Ly W(o(((())))))
(embo(-12z))
For ((bo (V(be((lrz)]lb(((lz (x)),, we follow the proof sketch (Asadi et al., 2018).
We use (-)x;) A) note that the belief function with the specific delays i. For instance, (-2) 6(-12))
and ((0)(1()))))))))))))))))))) (-122) az).
Then, we have
(122))16(4((1(aa)) (-1zz)) ( wcsh
=
<
(11) a.-1))
Sep + LDM(6 (Har)=--
s(1 TpDEP Tp (100
...
(1 +
)cp Lp A-2 ccp Lp (1= (-(xe)
< (1I.....
Lp
A )pp
1 - I.p'
Ep.
1 Lp
Therefore, we have
1 Ipp 4
recursive (xz) 1 (Ire) Lv Lp
cp.
14

---

Directly Forecasting Belief for Reinforcement Learning with Delays
The above theoretical results be extended to the stochastic delays å dd(-) easily. The performance difference of the ground-truth belief ged is defined as:
1 y
(xxe) (xt)
E
[V' ($) (=, (s.a))
swhss (1)))
amw (-12)
B [2c]
Finally, we have
Lv 1 - T.p
Trecusive (rr)
da (8)
1 (3+e)
Lp cp
8=1
Proposition B.2 (Performance Degeneration Bound of Directly Forecasting Belief). For the delay-free policy a and the delayed policy nd. Given any V., & X, the performance degeneration Idirect of the directly forecasting belief by can bounded as follows respectively.
For deterministic delays A, we have
1 1 (ze) + LV Edirect..
For stochastic delays 8 ~ d-(.), we have
Idirect (xt)
5
E brddal))
1 (I,) Eyddreet
Proof. Applying Assumption 5.6 and the proof of Theorem B.I.
Proposition B.3 (Performance Degeneration Comparison). forecastin belief could achieve a better performance guarantee ((fiec(())) < mreuiiie (x:)). if we have
1 1- Lp
Edirect <
1 - Lip
for deterministic delays A, and
Lp 5 1 - Lp
Edtrect
E b.ndal.)
cp
for stochastic delays d.()).
Proof. For deterministic delays A, if we have
1 Lp A
Edirect
EP,
1 Lp
then it is obvious that we have
1 - Lp
True (x:) + LvEdirect
Iinue
Ly
cp.
1 Lp
[2c)
For stochastic delays 8 ~ d.)+), if we have
L&p 6 Lpp
Cairect
E (-da())
then it is obvious that have
Lp
naw (xt) + Ly
L (x+)) | + Lv B
Cp
8rda() (-)
1 - Lp
[imss (x+)
7 (sre)
15

---

Directly Forecasting Belief for Reinforcement Learning with Delays
C. Additional Results on MuJoCo.
We report additional experimental results on MuJoCo, including more different patterns of deterministic and stochastic delays in Table 10 and Table 11, respectively.
Table 10. Performance MuJoCo with Deterministic Delays. Augmentation-base
The best performance is underlined. the best belief-based method is in red.
Belief-based
Task
Delays
A-SAC
BPQL.
ADRL.
DATS
D-Dreamer
D-SAC
DFBT-SAC (ours)
0.10
0.4040.04
0.44
0.08
0.08
0.12+0.06
0.35+a.12
16
0.06
0.42
0.29
0.08
0.08
0.11
0.40+0.05
HalfChoctah-v2
32
0. 0.02+0.02
0.40
0.26
0.11
0.08
0.08
0.42
64
0.01 10.01
0.11
0.16 0..02
0.12
0. 10.05
0.1210.07
0.39
128
0.04
0. a.08
0.14
0.10
0.15
0.09
0.41
00.3
0.87=0.09
0.95
0..1
0.11
0. 0.16+0.05
0.77+0.18
16
0.17
0.9 0.92
0.94
0.24
0.19
0.1 0.11+0.01
0.89
Hopper-v2
32
0.10
0.89
0.73
0.00
0.10
0.11
0.68+0.20
64
0.05
0..300.30
0. 0.11
0.13
0.09
0.08
0.19
128
0.0410.01
0.08 0.002
0.07 10.01
0.08 10.01
0.09
0.06
0.20
0.44
1.07 +0.02
10
0. 10.05
0. 0.11
0.09
0.99
0.13
0.96
0.67+0.21
0.06
0.12+0.03
0.08+0.04
0.95+0.11
Walker2d-v2
32
0.10+0.02
0.37 20.25
0.16
0.02
0.08
0.08+0.02
0.64+0.10
64
0.07 #0.01
0.14
0.10
0.01
0.08
0.08
0.41+0.14
128
0.06
0.07 10.03
0.08
0.02
0.08
0.11
0.40
Table Performance on MuloCo with Stochastic Delays. The best performance is underlined, and best belief-based method is in red.
Augmentation-basee
Belief-basee
Task
Delays
A-SAC
BPQL
ADRL
DATS
D-Dreamer
D-SAC
DFBT-SAC (ours)
U(1 8)
0.09
0.21=0.07
0.17
0.09=0.03
0.02+0.01
0.03
0.37
U(L,
0.04
0.31 20.08
0.24
0.00
0.03
0.01
0.37
HalfCheetah-v2
3()
0.01
0.33
0.23
00.1
0.02
0.01
0.31
64)
0.06
0.23
10.02
0.16 _..0
0.04
0.01
0.40
U28)
0.01
2.03
0.15
0.16
0.16
0.02
0.39
8)1
0.17+0.05
0.20
0.18
0.04
0.07
0.14+0.01
0.86
U6)
0.08
0.11
0.07
0.04_a.01
0.03
0.04+0.02
0.89
Hopper-v2
3(1
0.05
0.07 20.09
0.05
0.05
0.04
0.03
0.43
641
0. 0.03
0.03
0.03
0.05
0.03
0.0340.01
0.17
128)
0.0310.01
1.04_
0.002
0.05
0.03
0.03
0. 0.141
((1
0.36
0.40
0.41
0.07
0.07
0. 0.1240.04
1.11
U61
0.19
0.27 0.2722.17 20.17
0,24g0.10
0.0820.02
0.13
0.07
0.99
Walker2d-v2
U(1,32)
0.12+0.03
0.1620.04
0.11
0.09
0.12+0.04
0.05
0.67
64)
0.08
0.09
0.06
0.08
0.15
0.06
0.41
U21)
0. 0.06
0.06
0.04+0.02
0.10=a.01
0.15
0.03
0.30
16

---

Directly Forecasting Belief for Reinforcement Learning with Delays
D. Learning Curves on MuJoCo.
We report learning curves on MuJoCo with different patterns deterministic and stochastic delays Figure and Figure respectively.
17

---

Directly Forecasting Belief for Reinforcement Learning with Delays
64
1.0
Beour
0.-
022
0.
Global
Glebal steps
Global
(a) HaliCheetahh-2 (8 Delays)
(b) Hopper-v2 (8 Delays)
(c) Walker2d-v2 (8 Delays)
8.3
0.6
92tum
3.4
Global Steps
Global
Global
(d) HalfCheetah-v2 (16 Delays)
(e) Hopper-v2 (16 Delays)
(1) Walker2d-v2 (16 Delays)
1.0
7.5
8
Return
2.4
0.1
1.2
3.0
Fiobal Steps
Global
Global
HalfCheetah--2 (32 Delays)
(h) Hopper-v2 (32 Delays)
(i) Walker2d-v2 (32 Delays)
0.3
Reuu
0.3
0.1
0.2
0.0
3.0
5.0
20
Liobal Steps
Global Steps
Global Staps
(i) HalfCheetah-v2 (64 Delays)
(k) Hopper-v2 (64 Delays)
(I) Walker2d-v2 (64 Delays)
0.4
Retum
Bettum
0.3
6:10
Global Steps
Global Steps
Global Stups
(m) HalfCheetah-v2 (128 Delays)
(n) Hopper-v2 (128 Delays)
(o) Walker2d-v2 (128 Delays)
A-SAC
BPQL
ADRL
DATS
D-Dreamer
D-SAC
DFBT-SAC (ours)
Figure 4. Learning Curves MuJoCo with Deterministic Delays.
18

---

Directly Forecasting Belief for Reinforcement Learning with Delays
Recum
0.4
8.1
12-
20
Global Stops
Gbbal Stops
Global Stops
(a) HalfCheetah-v2 (U(1,8 8) Delays)
(b) Hopper-v2 (L, Delays)
(c) Walker2d-v2 (V(18 Delays)
1.0
0.3
2.-
Mm
Rezum
2.4
12-
0.2
3.0
Global Steps
Glbaal Steps
Global Stups
(d) HalfCheetah-v2 (11 16) Delays)
(e) Hopper-v2 (U(1, 16) Delays)
(f) Walker2d-v2 (U11, 16) Delays)
M
Recum
Retum
0.4
0.1
0.1
2.0
Global Steps
Global
Global
(g) HalfCheetah-v2 (U,1332) Delays)
(h) Hopper-v2 (U,(132) 32) Delays)
Walker2d-v2 (2, Delays)
0.5
8.
6155
2.10
6:28
0.0
Llobal Steps
Global Steps
Global Staps
30
(j) HalfCheetah-v2 (U4)4) Delays)
(k) Hopper-v2 (U1(1, 64) Delays)
Walker2d-v2 (U((1,4) Delays)
0.4
Retum
0.2
6:00
20
Global
Global
Glabal
(m) HalfCheetah-r2 (U(1, 128) Delays)
(n) Hopper-v2 (U(1, 128) Delays)
(o) Walker2d-v2 (1,,,, 128) Delays)
A-SAC
BPQL
ADRL
DATS
D-Dreamer
D-SAC
DFBT-SAC (ours)
Figure Learning Curves on MuJoCo with Stochastic Delays.
19

---

Directly Forecasting Belief for Reinforcement Learning with Delays
E. Belief Qualitative Comparison
report the qualitative comparison of the beliefs on HalfCheetah-v2. Hopper-v2, and Walker2d-v2 in Figure 6, Figure and Figure 8, respectively.
20

---

Directly Forecasting Belief for Reinforcement Learning with Delays
(a) Truth (8 Delays)
(b) DATS Delays)
(c) D-Dreamer Delays)
(d) D-SAC Delays)
(c) DFBT Delays)
Truth Delays)
Delays)
Delays)
Delays)
Delays)
(k) Truth (32 Delays)
()) DATS (32 Delays)
(m) D-Dreamer (32 Delays)
(n) D-SAC (32 Delays)
(o) DFBT (32 Delays)
(p) Truth (64 Delays)
(q) DATS (64 Delays)
(r) D-Dammmer (64 Delays)
D-SAC (64 Delays)
(L) DFBT (64 Delays)
Truth (128 Delays)
(v) DATS (128 Delays)
(w) D. D.Draamer (128 Delays)
(x) D.SAC Delays)
(y) DFBT (128 Delays)
Figure Belief qualitative comparison on HalfCheetah-v2 with different delays.
21

---

Directly Forecasting Belief for Reinforcement Learning with Delays
(a) Truth (8 Delays)
(b) DATS Delays)
(c) D-Dreamer Delays)
(d) D-SAC Delays)
(c) DFBT Delays)
Truth Delays)
Delays)
Delays)
Delays)
Delays)
(k) Truth (32 Delays)
()) DATS (32 Delays)
(m) D-Dreamer (32 Delays)
(n) D-SAC (32 Delays)
(o) DFBT (32 Delays)
Truth (64 Delays)
(q) DATS (64 Delays)
D- D-Drramer (64 Delays)
(s) D-SAC (64 Delays)
(L) DFBT (64 Delays)
Truth (128 Delays)
(V) DATS (128 Delays)
(w) D. Dreraaeer (128 Delays)
(x) D-SAC Delays)
DFBT (128 Delays)
Figure Belief qualitative comparison on Hopper-v2 with di different delays.
22

---

Directly Forecasting Belief for Reinforcement Learning with Delays
(a) Truth (8 Delays)
(b) DATS Delays)
(c) D-Dreamer Delays)
(d) D-SAC Delays)
(c) DFBT Delays)
Delays)
Delays)
Delays)
Delays)
Delays)
(k) Truth (32 Delays)
()) DATS (32 Delays)
(m) D-Dreamer (32 Delays)
(n) D-SAC (32 Delays)
(o) DFBT (32 Delays)
(p) Truth (64 Delays)
(q) DATS (64 Delays)
(r) D- D-Dammrer (64 Delays)
D-SAC (64 Delays)
(L) DFBT (64 Delays)
Truth (128 Delays)
DATS (128 Delays)
(w) D. Dreraaeer (128 Delays)
(x) D.SAC Delays)
(y) DFBT (128 Delays)
Figure& 8, Belief qualitative comparison on Walker2d-v2 with different delays.
23


Reviews:
[{"review_id": "nBr420GJEn", "reviewer": "Reviewer_KEME", "summary": "The authors focus on reinforcement learning with delayed observations. To mitigate this issue, most prior work learns a dynamics model which, given a known delay time $\\Delta t$, rolls out a dynamics model from $t$ to $t + \\Delta t$. The policy then makes decisions based on $s_{t + \\Delta t}.\n\nWhile prior work tends to use recurrent models, the authors suggest using a transformer to generate all states between $s_t$ and $s_{t + \\Delta t}$ in one shot, sidestepping error propagation that might occur from calling a recurrent model sequentially.\n\nThe authors train their transformer to predict future states using the negative log likelihood. They pretrain their transformer on offline dataset, then freeze the parameters and utilize the transformer-produced states to train the policy and value function as one would in normal RL.\n\nThe authors provide some error bounds for belief forecasting, then run some experiments on MuJoCo tasks. First, they plot the belief forecasting error, and then they compare the corresponding trained policy performance", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate Reviewer KEME's thoughtful comments. Below, we give responses to your questions and concerns.\n>### Q1: Evaluation of other MuJoCo tasks.\n\nIn this work, we consider the MuJoCo benchmark. To ensure transparency and reproducibility in belief training, we utilize the open offline RL datasets (D4RL), including HalfCheetah-v2, Hopper-v2, and Walker2d-v2 only. We acknowledge that other tasks are also valuable. Therefore, we conducted experiments on the Pusher-v2, Reacher-v2, and Swimmer-v2 with deterministic 32 delays. The offline datasets (500k samples) are collected from a SAC policy, with other settings unchanged. The results are shown in Table R1, showing that DFBT-SAC achieves superior performance on these tasks.\n\nTable R1. Performance on additional tasks with deterministic 32 delays.\n|Task|A-SAC|BPQL|ADRL|DATS|D-Dreamer|D-SAC|DFBT-SAC|\n|--|--|--|--|--|--|--|--|\n|Pusher-v2|$0.05\\pm 0.00$|$0.93\\pm 0.58$|$0.95\\pm 0.24$|$0.92\\pm 0.10$|$0.87\\pm 0.18$|$0.94\\pm 0.04$|$\\mathbf{1.04\\pm 0.24}$|\n|Reacher-v2|$0.89\\pm 0.08$|$0.83\\pm 0.06$|$0.85\\pm 0.01$|$0.82\\pm 0.13$|$0.84\\pm 0.02$|$0.88\\pm 0.07$|$\\mathbf{0.93\\pm 0.06}$|\n|Swimmer-v2|$0.27\\pm 0.05$|$0.80\\pm 0.14$|$0.60\\pm 0.06$|$0.25\\pm 0.05$|$0.21\\pm 0.07$|$0.30\\pm 0.07$|$\\mathbf{1.01\\pm 0.27}$|\n\n>### Q2: Writing and grammar issues.\n\nThanks for the reviewer's helpful suggestions. We will revise the paper to improve clarity and ensure consistency in grammatical tense and voice.\n\n>### Q3: Necessity of treating \"delayed RL\" as a separate research problem.\n\nAs the reviewer mentioned, some robotics tasks incorporate RNN or Transformer to handle partial observability. We acknowledge that these models implicitly address delays by capturing sequential dependencies and retaining memory over time. However, we emphasize that **explicitly handling delayed observation-action interactions through delayed RL remains essential both technique-wise and application-wise**. The key reasons are as follows:\n\n(1) **Unique problem structure enables specialized algorithms.** Delayed MDP poses unique challenges due to entirely missing observations rather than partially missing ones, though it can be viewed as a specialized form of POMDP [1]. **Its explicit delay-induced structure enables specialized, efficient algorithms that outperform general-purpose POMDP methods**. For example, at timestep $t$, the agent can only access the historical state $s_{t-\\Delta}$. In this context, the agent has to explicitly handle delays $\\Delta$ using state augmentation [2] or belief representation [3] techniques to retrieve the Markovian property and enable efficient RL. For instance, in the state augmentation technique, the decision-making is based on the augmented state $x_t:=\\\\{s_{t-\\Delta},a_{t-\\Delta:t-1}\\\\}$. This paper aims to address the compounding errors issue within the belief representation method, thereby enhancing learning efficiency and performance. These distinctive properties and challenges in delayed RL necessitate treating it as a distinct research problem, separate from the conventional partially observable RL problem.\n\n(2) **Strong application-driven motivations.** Delayed RL aims to address the delayed feedback problem, which is practical and common in real-world control applications (e.g., transportation systems [4] and financial systems [5]). In robotics, several studies have demonstrated that delayed RL could improve the system's safety, agility, efficiency, and robustness [6, 7]. The practical demands of real-world scenarios also underscore the necessity to investigate delayed RL as a distinct research problem.\n\n>### Q4: $U(1, 128)$ delays explanation and Transformer for stochastic delays.\n\nFor $U(1, 128)$ delays, it does not mean learning the deterministic delays of $64$. Under this setting, in each timestep $t$, the probability of observed state $s_{t-1}$ equals that of every observed state till $s_{t-128}$. Therefore, the agent shall have the ability to address varying delays ranging from 1 to 128. As shown in Fig.2, the transformer can effectively address the compounding errors issue, maintaining superior and consistent prediction accuracy across varying delays. These accurate predictions further improve the learning efficiency and final performance of RL.\n\n>### Reference\n>\n>[1] Karamzade, Armin, et al. \"Reinforcement learning from delayed observations via world models\".\n>\n>[2] Bouteiller, Yann, et al. \"Reinforcement learning with random delays\".\n>\n>[3] Walsh, Thomas J., et al. \"Learning and planning in environments with delayed feedback\".\n>\n>[4] Cao, Zhiguang, et al. \"Using reinforcement learning to minimize the probability of delay occurrence in transportation\".\n>\n>[5] Deng, Yue, et al. \"Deep direct reinforcement learning for financial signal representation and trading\".\n>\n>[6] Mahmood, A. Rupam, et al. \"Setting up a reinforcement learning task with a real-world robot\".\n>\n>[7] Hwangbo, Jemin, et al. \"Control of a quadrotor with reinforcement learning\".", "claims_and_evidence": "The authors claim to:\n- Propose a transformer architecture to tackle the forecasting problem\n- Integrate the architecture in SAC\n- Demonstrate that their method reduces compounding forecasting errors\n- Demonstate that their method results in better policies\n\nI think they provide sufficient evidence to back their claims.", "methods_and_evaluation_criteria": "The authors select a standard MuJoCo baseline, however they only focus on three tasks. I think it could be helpful to evaluate on the entire MuJoCo suite, even if the other results are only written in the appendix.", "theoretical_claims": "The authors make theoretical claims but I did not check them closely.", "experimental_designs_or_analyses": "The experimental design consists of both deterministic and random delays, and performs ablations.\n\nI commend the authors for also demonstrating the DFBT-SAC(1) results. Initially, I was concerned that their improved returns could result primarily from the use of an n-step return. This ablation assuages my concerns.", "relation_to_broader_scientific_literature": "I an not familiar with the field of delayed RL.", "essential_references_not_discussed": "I an not familiar with the field of delayed RL.", "other_strengths_and_weaknesses": "The authors' method is well-founded and their experimental setup is well-done. The idea is fairly straightforward and provides strong results.\n\nI think the writing could be a bit clearer, and I suggest the authors go through the paper and try to minimize tense changes and stick to either passive or active voice.\n\nMy biggest concern is not necessarily on the authors work, but rather with the field of \"delayed RL\". I suspect that most realistic robotics tasks already integrate an RNN/Transformer to handle partial observability. Such a setup would be able to handle delayed observation-action interactions implicitly, removing the need to consider delayed RL as a separate problem.", "other_comments_or_suggestions": "As above, try and be more consistent with grammatical tense and voice to make the paper slightly nicer to read. More MuJoCo tasks could also strengthen the evidence for the authors' claims.", "questions_for_authors": "It is unclear to me why the transformer architecture works so much better for stochastic delays. For $U(1, 128)$, does this just mean learning a deterministic delay of $64$? If you cannot know the delay ahead of time, it seems like this is the best you can do. Can you explain this further?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h1cABYq1wr", "reviewer": "Reviewer_dQtK", "summary": "This paper introduces a method for directly predicting the current belief state in reinforcement learning with delays using a Transformer-based model. The main idea is to use Transformers for state forecasting to help mitigate the effects of observation delays in RL environments. The approach is simple, modular, and easy to implement.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate Reviewer dQtK's thoughtful comments. Below, we give responses to your questions and concerns.\n\n>### Q1: Contribution and Novelty Clarification\n\nFirst, we would like to express our gratitude for the reviewer's comments. We recognize that our current statement may have led to some misunderstandings, and we will refine the claim and clarify the statement in the revised version. Here, we want to clarify our contribution and the novelty. This paper aims to address the compounding errors issue in belief-based methods via directly forecasting belief. While we acknowledge that the directly forecasting approach has been used in delayed RL, and we have cited the related work and considered it as the baseline [1], we note that existing work with online belief learning always suffers from instability and inefficacy issues. To overcome these issues, we separate the belief training from the RL process to stabilize the online learning process (Lines 406-411). Specifically, as shown in Fig. 1, the DFBT is trained on the offline datasets, then later frozen and deployed in the environment with delays, enabling efficient RL training. To this end, \n1. We propose DFBT, which incorporates reward signals in tokens for capturing sufficient dynamic information. Empirical results (Fig. 2) demonstrate that DFBT achieves superior prediction accuracy, effectively addressing the compounding errors issue.\n2. By leveraging the accurate predictions from DFBT, we integrate the multi-step bootstrapping technique on the forecasted states to improve learning efficiency.\n3. We theoretically demonstrate that directly forecasting belief significantly mitigates compounding errors, providing a stronger performance guarantee.\n\n>### Q2: The separated belief is limited to task-specific information of RL.\n\nWe acknowledge that some task-specific information may be missing if the belief is frozen in the RL process, leading to limited performance improvement. This issue can be mitigated by fine-tuning the belief within the RL process. The results, shown in Table R2, demonstrate that fine-tuning helps the DFBT capture the task-specific information with better performance. Note that there are many potential methods for capturing task-specific information not limited to fine-tuning DFBT. However, they are orthogonal to this work's contribution. We will add related discussions for broader interests in the revised version. As mentioned in the limitations (Lines 406-411), belief learning from scratch in the online RL process always suffers from instability issues. Therefore, in this paper, we separate belief learning from the online RL process, which allows us to investigate the belief component solely, eliminate potential influences from the RL side.\n\nTable R2. Performance comparison of DFBT-SAC with different training methods on MuJoCo tasks with 32 delays.\n|Task|Online|Offline|Offline + Fine-tuning|\n|--|--|--|--|\n|HalfCheetah-v2|$0.11\\pm0.39$|$\\mathbf{0.42\\pm0.12}$|$0.39\\pm0.04$|\n|Hopper-v2|$0.10\\pm0.58$|$0.68\\pm0.20$|$\\mathbf{0.84\\pm0.04}$|\n|Walker2d-v2|$0.09\\pm0.27$|$0.64\\pm0.10$|$\\mathbf{0.96\\pm0.32}$|\n\n>### Q3: Inefficiencies in high-dimensional environments.\n\nIn this work, we mainly consider MuJoCo tasks, which have relatively low-dimensional state spaces, such as HalfCheetah (17), Hopper (11), and Walker2d (17). As discussed in the paper (Lines 47-51, 99-102), our approach belongs to the belief-based method, which can efficiently address the \"curse of dimensionality\" issue of the augmentation-based approach in facing long delays. For the high-dimensional state space (e.g., image-based RL tasks), it is essential to learn a low-dimensional and compact latent state space for efficient state prediction. However, this falls outside the scope of the current paper. We will include a related discussion in the revised version and plan to explore high-dimensional delayed RL tasks in future work.\n\n>### Q4: Experiments on computational efficiency.\n\nThanks for your helpful suggestion. Based on your advice, we conducted additional computational efficiency experiments. As shown in Table R3, the results demonstrate that directly forecasting belief maintains a consistent and stable inference speed (around 4 ms) across different delays. In contrast, the recursively forecasting belief experiences inference speed issues as delays increase. In HalfCheetah-v2 with 128 delays, the training times of DATS and D-Dreamer are around 10 hours and 15 hours, respectively, while that of D-SAC and DFBT-SAC both are around 6 hours.\n\nTable R3. Inference speed (ms) comparison in HalfCheetah-v2.\n|Delays|DATS|D-Dreamer|D-SAC|DFBT-SAC|\n|--|--|--|--|--|\n|8|$1.10\\pm0.02$|$1.85\\pm0.01$|$4.03\\pm0.03$|$4.18\\pm0.04$|\n|32|$3.85\\pm0.06$|$6.80\\pm0.04$|$4.03\\pm0.04$|$4.18\\pm0.04$|\n|128|$14.97\\pm0.22$|$26.51\\pm0.19$|$4.03\\pm0.03$|$4.15\\pm0.05$|\n\n>### Reference\n>\n>[1] Liotet, Pierre, et al. \"Learning a belief representation for delayed reinforcement learning\".", "claims_and_evidence": "1. Performance claims are well-supported by comprehensive experiments across delay settings.\n2. Ablation studies on multi-step bootstrapping provide convincing evidence of this technique's importance.\n3. Innovation claim somewhat overstated - direct prediction approach itself isn't new when handling delay problem in RL.", "methods_and_evaluation_criteria": "1. The proposed method is based on a Transformer architecture for belief forecasting, which is intuitive and straightforward. \n2. The experimental setup considers high delays (8 to 128 steps) and random delays (U(1,n) distribution), making the evaluation more aligned with real-world scenarios. \nHowever:\n1. The separation of the prediction module from the reinforcement learning process limits the ability of the prediction network to adapt to task-specific informations. This design choice could lead to inefficiencies in tasks where task-relevant information differs from state prediction accuracy.\n2. The choice to operate directly in the original state space may lead to inefficiencies in high-dimensional environments. The decision to predict directly in the original state space rather than learning a task-optimized lower-dimensional representation presents challenges in high-dimensional environments. This approach risks the \"curse of dimensionality\" when states contain redundant or task-irrelevant information. In complex environments, only a subset of state variables may be critical for decision-making, making direct prediction of the entire state inefficient. A comparison with approaches that learn low-dimensional task-specific representations could provide additional insights.", "theoretical_claims": "No special points to be mentioned.", "experimental_designs_or_analyses": "1. Methodology appropriate for research problem, using standard experiments for delays. It covers motion controlling, would be great if it can be extended to other areas. \n2. Good testing across environments with different delay characteristics such as randomness and time scale.\n3. Well-designed ablation studies, such as bootstrapping steps (N=1,2,4,8).\n\nTo be improved:\nThe direct prediction approach likely offers computational advantages over complex recursive forecasting methods. By avoiding intermediate steps, it reduces error accumulation and computational overhead. An explicit analysis of computational efficiency (training time, inference speed, memory usage) would strengthen the paper's practical value proposition.", "relation_to_broader_scientific_literature": "While the paper's conceptual innovation may be limited, its main contribution lies in providing an efficient, straightforward, and universally applicable solution to the delayed reinforcement learning problem using modern frameworks. Although the core ideas have appeared in previous methods, the paper's implementation using contemporary Transformer architecture brings practical improvements. The significance of this work is not in proposing entirely new concepts, but in effectively adapting modern deep learning techniques to create an \"out-of-the-box\" solution that outperforms existing approaches.", "essential_references_not_discussed": "N/A.", "other_strengths_and_weaknesses": "See above", "other_comments_or_suggestions": "See above", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RTzBzQhDM1", "reviewer": "Reviewer_4D3p", "summary": "This paper addresses reinforcement learning with delayed observations by proposing a Directly Forecasting Belief Transformer (DFBT). DFBT treats state estimation as a sequence modeling problem—predicting the current (and intermediate) states directly from past delayed observations instead of forecasting them iteratively. The authors combine DFBT with Soft Actor-Critic and introduce multi-step bootstrapping from the predicted states to improve learning efficiency. Empirical results on MuJoCo tasks with both fixed and random delays show significantly higher performance than prior augmentation-based or recursively predicted belief methods.\n\n## update after rebuttal\nBoth before and after the rebuttal phase, I believe that the work has a certain novelty, and therefore, I leave my current high assessment.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate Reviewer 4D3p's thoughtful comments. Our detailed responses to your questions and concerns are as follows:\n\n>### Q1: One potential weakness is that DFBT requires an offline dataset to pre-train the belief model. In scenarios where such data is not available, one would have to gather it (possibly by random exploration) or train the model online (which the paper did not explore). Augmentation-based methods, by contrast, learn everything online (though at a heavy sample cost). The authors did not explicitly discuss this trade-off.\n\nThanks for your insightful comment on the trade-off between augmentation-based and belief-based approaches. We will add the related discussion in the revised paper based on your suggestion. As mentioned in the paper (Lines 406-411), learning DFBT online from scratch always suffers from instability and inefficacy issues. Therefore, we separate the belief learning from the RL process for stabilizing the online learning process. Separating belief learning from the online RL process and freezing belief representation during RL allows us to investigate the belief component solely, eliminate potential influences from the RL side. To address the reviewer's concern, we report the experimental results of learning belief representation on MuJoCo tasks with deterministic 32 delays in Table R4. The results show that directly learning belief representation in the RL process suffers from instability and efficiency, thus leading to poor performance.\n\n>### Q2: How well does DFBT generalize if the policy visits out-of-distribution states not covered in the offline dataset? Could fine-tuning the belief model online further boost performance or stability?\n\nThanks for the thoughtful suggestions. In the online RL process, the agent indeed visits out-of-distribution (ood) states, which leads to a relatively limited performance gain. As mentioned by the reviewer, this issue can be addressed by fine-tuning the belief representation with the online RL process. To address this concern, we conduct the additional experiment: fine-tuning DFBT online on MuJoCo tasks with deterministic 32 delays. The experimental results show that fine-tuning the belief representation can gain better performance improvement with much longer training hours (increasing training hours from 6 to 12). Note that there are other methods to improve the DFBT's online performance. However, they are orthogonal to this work's contribution. Therefore, in this paper, we separate belief learning from the online learning process, which allows us to investigate the belief component solely, eliminate potential influences from the RL side. We will add related discussions in the revised version.\n\nTable R4. Performance comparison of fixed and fine-tuned DFBT on MuJoCo tasks with deterministic 32 delays.\n|Task|Online|Offline|Offline + Fine-tuning|\n|--|--|--|--|\n|HalfCheetah-v2|$0.11\\pm0.39$|$\\mathbf{0.42\\pm0.12}$|$0.39\\pm0.04$|\n|Hopper-v2|$0.10\\pm0.58$|$0.68\\pm0.20$|$\\mathbf{0.84\\pm0.04}$|\n|Walker2d-v2|$0.09\\pm0.27$|$0.64\\pm0.10$|$\\mathbf{0.96\\pm0.32}$|\n\n>### Q3: Have you considered outputting distributional beliefs for stochastic environments?\n\nThanks for the helpful suggestions. We conducted additional experiments on the stochastic MuJoCo tasks with a probability of 0.001 for the unaware noise (similar settings with related works [1, 2]) and deterministic 128 delays. As shown in Table R5, the results demonstrate that our DFBT-SAC with distributional belief achieves superior performance in these stochastic MuJoCo tasks.\n\nTable R5. Performance on stochastic MuJoCo with deterministic 128 delays.\n\n|Task|A-SAC|BPQL|ADRL|DATS|D-Dreamer|D-SAC|DFBT-SAC|\n|--|--|--|--|--|--|--|--|\n|HalfCheetah-v2|$0.00\\pm0.03$|$0.01\\pm0.05$|$0.13\\pm0.04$|$0.13\\pm0.03$|$0.07\\pm0.01$|$0.00\\pm0.04$|$\\mathbf{0.35\\pm 0.04}$|\n|Hopper-v2|$0.03\\pm0.04$|$0.08\\pm0.05$|$0.06\\pm0.04$|$0.05\\pm0.06$|$0.04\\pm0.05$|$0.04\\pm0.05$|$\\mathbf{0.13\\pm 0.22}$|\n|Walker2d-v2|$0.06\\pm0.02$|$0.04\\pm0.01$|$0.08\\pm0.01$|$0.06\\pm0.03$|$0.10\\pm0.03$|$0.05\\pm0.02$|$\\mathbf{0.30\\pm 0.07}$|\n\n>### Q4: Works like TransDreamer (Chen et al., 2022) also explore Transformers in partially observable RL but from a full model-based viewpoint. Fu et al. also discusses compounding error bound with the Lipschitz assumption.\n\nThanks for the suggestions on missing related works. We will discuss these related works and references in the revised paper.\n\n>### Q5: Minor issues: \"delays are fundamentally affect the system’s safety\" and \"cypher-physical systems\"\n\nThanks for your helpful comments. We will fix these typos in the revised version.\n\n>### Reference\n>[1] Kim, Jangwon, et al. \"Belief projection-based reinforcement learning for environments with delayed feedback.\" Advances in Neural Information Processing Systems.\n>\n>[2] Wu, Qingyuan, et al. \"Boosting Reinforcement Learning with Strongly Delayed Feedback Through Auxiliary Short Delays.\" International Conference on Machine Learning.", "claims_and_evidence": "The key claim is that direct (rather than recursive) forecasting mitigates compounding errors as delay increases, leading to superior policy returns. Their theoretical bounds show that the error does not scale exponentially with delays, and empirical comparisons on MuJoCo (e.g., HalfCheetah, Hopper, Walker2d) confirm much better performance at long delays. This is well-supported by both error metrics (on offline data) and final returns in the RL tasks.\n\nOne potential weakness is that DFBT requires an offline dataset to pre-train the belief model. In scenarios where such data is not available, one would have to gather it (possibly by random exploration) or train the model online (which the paper did not explore). Augmentation-based methods, by contrast, learn everything online (though at a heavy sample cost). The authors did not explicitly discuss this trade-off.", "methods_and_evaluation_criteria": "The authors pre-train the Transformer-based belief model using offline trajectories (D4RL) and then use it in online learning with a standard SAC agent. They measure belief prediction accuracy (L1/MSE error) and normalized returns on MuJoCo. Baselines include recent augmentation-based (BPQL, ADRL) and belief-based (D-SAC, D-Dreamer) methods, making the comparisons fair and thorough.", "theoretical_claims": "They provide bounds showing that recursive belief estimation accumulates errors exponentially in the worst case, whereas direct prediction has a linear bound in terms of overall model error.", "experimental_designs_or_analyses": "Experiments are methodically done on MuJoCo tasks with delays ranging from 8 to 128, as well as random delays. \n\nAdditional experiments that might make the paper stronger: 1. analyzing the impact of the algorithm for ood states 2. stochastic environment 3. Results for online finetuning the belief", "relation_to_broader_scientific_literature": "This work builds on both augmentation-based and belief-based delay handling methods, connecting to model-based RL (e.g., Dreamer) by learning a forward model but differs by predicting states in one shot with a Transformer.", "essential_references_not_discussed": "Works like TransDreamer (Chen et al., 2022) also explore Transformers in partially observable RL but from a full model-based viewpoint. Fu et al. also discusses compounding error bound with the Lipschitz assumption.\n\n1. Reinforcement Learning with Transformer World Models. Chen et al.\n2. Performance Bounds for Model and Policy Transfer in Hidden-parameter MDPs. Fu et al.", "other_strengths_and_weaknesses": "No", "other_comments_or_suggestions": "Minor issues:\n1. “delays are fundamentally affect the system’s safety”\n2. “cypher-physical systems”​", "questions_for_authors": "1. How well does DFBT generalize if the policy visits out-of-distribution states not covered in the offline dataset?\n2. Could fine-tuning the belief model online further boost performance or stability?\n3. Have you considered outputting distributional beliefs for stochastic environments?*", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The manuscript introduces DFBT which directly forecasts states from observations without estimating intermediate states incrementally.
Reviewers generally expressed positive views of the paper.
The rebuttal was very thorough and provided important clarifications. I recommend acceptance.
Decision: Accept (poster)