Variational Delayed Policy Optimization
Authors: Qingyuan Wu
Simon Sinong Zhan
Yixuan Wang
Yuhui Wang
Chung-Wei Lin
Chen Lv
Qi Zhu
Chao Huang
Venue: NeurIPS 2024 spotlight (2024)
Keywords: reinforcement learning
time-delay system
Primary area: reinforcement_learning
TL;DR: We propose variational delayed policy optimization, a novel delayed reinforcement learning method from the perspective of variational inference for effectively improving the sample efficiency without compromising the performance.

Abstract: In environments with delayed observation, state augmentation by including actions within the delay window is adopted to retrieve Markovian property to enable reinforcement learning (RL). Whereas, state-of-the-art (SOTA) RL techniques with Temporal-Difference (TD) learning frameworks commonly suffer from learning inefficiency, due to the significant expansion of the augmented state space with the delay. To improve the learning efficiency without sacrificing performance, this work novelly introduces Variational Delayed Policy Optimization (VDPO), reforming delayed RL as a variational inference problem. This problem is further modelled as a two-step iterative optimization problem, where the first step is TD learning in the delay-free environment with a small state space, and the second step is behaviour cloning which can be addressed much more efficiently than TD learning. We not only provide a theoretical analysis of VDPO in terms of sample complexity and performance, but also empirically demonstrate that VDPO can achieve consistent performance with SOTA methods, with a significant enhancement of sample efficiency (approximately 50\% less amount of samples) in the MuJoCo benchmark.

Variational Delayed Policy Optimization
Qingyuan Wu University Southampton
Simon Sinong Zhan Northwestern University
Yixuan Wang Northwestern University
Yuhui Wang
Chung- Wei Lin
King Abdullah University of Science and Technology
National Taiwan University
Chen Lv Nanyang Technological University
Qi Zhu Northwestern University
Chao Huang University of Southampton
Abstract
In environments with delayed observation, state augmentation by including tions within the delay window adopted to retrieve Markovian property to enable reinforcement learning (RL). However, state-of-the-ard (SOTA) RL techniques (TD) learning frameworks often suffer from learning
with Temporal-Differee inefficiency, due to the significant expansion of the augmented state space with the delay. To improve learning efficiency without sacrificing performance, this work introduces novel framework called Variational Delayed Policy Optimiza- tion (VDPO), which reformulates delayed RL as a variational inference problem. This problem is further modelled a two-step iterative optimization problem, where the first step is TD learning in the delay-free environment with a small state space, and the second step behaviour cloning which can be addressed much more efficiently than TD learning. We not only provide theoretical anal- ysis VDPO in terms of sample complexity and performance, but also empir- ically demonstrate that VDPO can achieve consistent performance with SOTA methods, with a significant enhancement of sample efficiency (approximately 50% less amount of samples) in the MuloCo benchmark. Code is available at https//giihub.com/innggaa WuNothing/VDDP..
Introduction
Reinforcement learning (RI.) has achieved considerable success across various domains, including board game [32], video game [27], cyher-physical systems [40, 41, 43]. Most of these achievements lack stringent timing constraints, and, therefore, overlook delays in agent-environment interaction. However, delays are prevalent in many real-world applications stemming from various sensors, computation, etc, and significantly affect learning efficiency [17], performance [6], and safety [26]. While observation-delay, action-delay, and reward-delay [10] are all crucial, observation-delay receives the most attention [7, 33, 42]. Unlike reward-delay, observation-delay. which is proved to be superset of action-delay [19, 29], disrupts the Markovian property inherent to the environments. In this work, we focus on the reinforcement learning with a constant observation-dela A: at any time step the agent can only observe the state s without access to states from A + to t.
"Equal Contribution t Correspondence to: Chao Huang, chao. www.gdsoton.acc.....oooo
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Lugmentation-baset approach is one of the promising methodologies [4. 19]. It retrieves the Markovian property by augmenting the state along with the actions within the window of delays to a new state zt, i.e., It {st A; Ot-A,". -
yielding delayed MDP. However, the underlying sample complexity issue remains a central challenge. Pioneering works [5. 29] directly conduct classical temporal-difference (TD) learning methods, e.g., Deep Q Network [28] and Soft Actor- Critic [13], over the delayed MDP. However, due to the significant growth of the dimensionality, the sample complexity of these techniques increases tremendously. State-of-the-art (SOTA) methods [20, 39, 42] mitigate this issue by introducing an auxiliary delayed task with shorter delays to help learning the original longer delayed task (e.g., improving the long-delayer policy based the short-delayed value function). However, sample inefficiency is not addressed sufficiently due to the TD learning paradigm still being affected significantly by the increased delays. The memory-less approach [33] improves the learning efficiency by ignoring the absence of the Markovian property of observation-delay RL. and learning over the original state space with a cost of serious performance drop. Therefore, the critical challenge still remains: how to improve learning efficiency without compromising performance in the delayed setting.
To overcome such a challenge, we propose Variational Delayed Policy Optimization (VDPO), a novel delayed RL framework. Inspired by existing variational RL methods [1, 2. 25J, VDPO can utilize extensive optimization tools to resolve the sample complexity issue effectively via formulating the delayed RL problem as variational inference problem. Specifically, VDPO operates alternatively: (1) learning a reference policy over the delay-frree MDP via TD learning and (2) imitating the behaviour of the learned reference policy over the delayed MDP via behaviour cloning. In the high dimensional delayed MDP, VDPO replaces the TD learning paradigm with the behaviour cloning paradigm, which considerably reduces the sample complexity. Furthermore, we demonstrate that VDPÓ not only effectively improves the sample complexity, but also achieves consistent theoretical performance with SOTAs. Empirical results show that compared to the SOTA approach [42], our VDPO has significant improvement in sample efficiency (approximately 50% less amount of samples) along with comparable performance at most MuJoCo benchmarks.
This paper first introduces notations related to delayed RL and variational RL (Sec. 2). In Sec. 3, present how to formulate the delayed RL problem by our approach VDPO. Through theoretical analysis, we show that VDPO can effectively reduce the sample complexity without degrading the performance in Sec. 3.2. Practical implementation of VDPO is presented in Sec. 3.3. In Sec. 4, the experimental results over various MuloCo benchmarks under diverse delay settings validate our theoretical observations. Overall, our contributions summarized as follows:
variational inference problem followed
We propose Variational Delayed Policy Optimization (VDPO), a novel framework of delayed RL algorithms emerging from the perspective of variational RL.
We demonstrate that VDPO enhances sample efficiency, by minimizing the KL divergence between the reference delay-free policy and delayed policy in a behaviour cloning fashion.
We illustrate that VDPO shares the same theoretical performance as SOTA techniques, by showing that VDPO converges to the same fixed point.
We empirically show that VDPO not only exhibit superior sample efficiency but also achieves competitive performance comparable to SOTAS across various MuloCo benchmarks.
2 Preliminaries
MDP. delay-free RL problem can be formalized as Markov Decision Process (MDP), denoted by a tuple (S, A, P. R.7,)), p), where S, A represent state space and action space respectively, P : S x A S + [0, represents the transition function; the reward function is denoted as R S x A + R: 7 E 10 is the discount factor, and p(so) is the initial state distribution. At each time step t, the agent takes the action a, ~ T(-[s:) based on the current observed state s; and the policy : S x AA- + [0, 1], and then observes the next state St+1 P(-Ist, a.) and reward signal r: = R(s,, a.). The objective of an RL problem is to find the policy क which can maximize the expected return E,-p,, (r) [T(r) = Er-p, (r) IE 7 R a,)] where (T) is the trajectory distribution induced by policy æ. We use d (st) to denote the visited state distribution of policy л.

---

Delayed MDP. A delayed RL problem with a constant delay is originally not an MDP, but can be reformulated as delayed MDP with Markov property based on the augmentation approaches [4, 19]. Assuming the constant delay being A, the delayed MDP is denoted as tuple (X, A, Pa. R... 7. PA), where the augmented state space is defined as == x A (e.g., an augmented state it (5c-A.0--2: 0x-1) X), A is the action space, the delayed transition function is defined as Pa [2,, (+) P(s A+1 ( A, ar- a (a/) - 8a, (d)_1) where 8 is the Dirac distri- bution, the delayed reward function is defined as R& (Tz, a1) == E (ms) [R(se, az)l where b is
the belief function defined as ((st := SS II (( ( (s Ati. ar- A+i ds - A+++1 the initial augmented state distribution is defined Pa p pll 8a..
Variational RL. Formulating the RL problem as probabilistic inference problem [21] allows us to use extensive optimization tools in solving the RL problem. From the existing variational RL literature [30, 36], we usually define O as the optimality of the task (e.g,, the trajectory obtains the maximum return). Then the probability of trajectory optimality can be represented as p(O = l|r). Then, the objective of variational RI. becomes finding policy и with highest log evidence: max, log Pr((O = 1). Then, we can derive the lower bound of log p0 = 1) by introducing a prior knowledge of trajectory distribution q(T).
log Px (0 = 1) 2 Brwa(r) [log p(O 1|r)] - (L(((T))lp = ELBOO(T,))
(1)
where KI. is the Sullback-Leiblee (KI.) divergence and FL.BO q) is the evidence lower bound (ELBO) [2, 30]. The objective of variational RL maximizing the ELBBO, which can be achieved by various optimization techniques [1, 9, 30].
3 Our Approach: Variational Delayed Policy Optimization
In this section, we present new delayed RI. approach, Variational Delayed Policy Optimization (VDPO) from the perspective of variational inference. By viewing the delayed RL. problem as a variational inference problem, VDPO can utilize extensive optimization tools to address sample complexity and performance issues properly. We first illustrate how to formulate delayed RL as the probabilistic inference problem with an elaborated optimization objective. Subsequently, we theoretically show that the inference problem is equivalent to two-step iterative optimization problem. Then, present the framework of VDPO along with the practical implementation.
3.1 Delayed RL as Variational Inference
Delayed RI. can be treated as an inference problem: given the desired goal O, and starting from a prior distribution over trajectory т, the objective estimate posterior distribution
consistent with O. The posterior can be formulated by Boltzman like distribution p(O = 1/r) exp ((T) [2, 31]
where a the temperature factor. Based on the above definition, the optimization objective delayed RL can be defined as follows.
1)= max log / pl0 1170=117)pma(T)))
max log
P P
(2)
where P=a (0 -0=11 is the probability of the optimality of the delayed policy Tı and Pra (7) is the
trajectory distribution induced by TA. Based on Eq. (1) and Eq. (2), we can also show that the ELBO for optimization purpose is follows (derivation of Eq. (3) can be found in Appendix B).
log Pra (0 = 2 Erop, [logp(O = 1|7)] (L(p (T)) = EEBOO T TA)))
(3)
B4
where Pa (T) is the trajectory distribution induced by an newly-introduced reference policy и. As shown in Eq. (3), we transform the original optimization problem as two-step iterative optimization problem: maximizing term while minimizing term B. Next, we detail how our VDPO optimizes objectives A and B separately.
3

---

3.1.1 Maximizing the performance of reference policy by TD Learning
In this section, we discuss the treatment of term A in Eq. (3) and investigate the performance and sample complexity reference policy under different MDP settings. Maximizing term in Eq. (3) is equivalent to maximizing the performance of π as follows.
max (r) [(ogp(O 1|7) = max (rrp,(r) [J(r)]-
(4)
For Eq. (4), we can train the reference policy я in various MDPs with different delays or even delay-free settings. We show that the performance (Lem. 3.1) and sample complexity (Lem. 3.2) of reference policy я are correlated to the specific MDP setting. Based on existing literature [12, 22] and motivated by existing works [42]. VDPO chooses training the reference policy in the delay-free MDP for gaining the edge in terms performance and sample complexity.
Lem. 3.1 indicates that the performance of the optimal policy is likely decreased by increasing delays. This motivates us to learn the reference policy the delay-free MDP for proper performance.
Performance:
Lemma 3.1 (Performance in delayed MDP, Theorem 4.3.1 in [22]). Let M1, M2 be two constant delayed MDPs with respective delays A,, (A(1 < 42). For the optimal policies in M1, M2. we have J 2 Tz.
Sample Complexity: Furthermore, for a specific TD-based delayed RL method (e.g., model-based policy iteration), delays also affect its sample efficiency stated in Lem. 3.2 that stronger delays will lead to much higher sample complexity, resulting in relative learning inefficiency. Therefore, learning the delay-fres reference policy makes VDPO superior in sample complexity compared to learning under delay settings.
Lemma 3.2 (Sample complexity of model-based policy iteration, Theorem 2 in [12)). Let M be the constant delayed MDP with delays A. Model-based policy iteration finds an c-optimal policy with probability a using sample size O (IIIIA 1 where |X| = ||A|AA|
Based on the above analysis and inspired by the existing work [23, 42], VDPO adopts delay-free policy as the reference policy. More rigorous analyses are presented in Sec. 3.2, and we will detail the practical implementation in Sec. 3.3.
3.1.2 Minimizing the behaviour differrnce by Behaviour Cloning
With fixed reference policy minimizing term B in Eq. (3) can be treated as behaviour cloning the trajectory level. However, behaviour cloning at the trajectory level is relatively inefficient compared with training at the state level as we have to collect an entire trajectory before training. We next show that we can directly minimize the level KI. divergence KI.(T (I.(((a.|s))) [e]]))]] (atlar)) as presented in Proposition 3.3.
Proposition 3.3 (State-level KL divergence, proof in Proposition C.1). For a fixed reference policy m. the trajectory-leve KL divergence can be reformulated to state-level KL divergence as follows.
f
(L(P=(T)lP=
Σ
d'
+Const.,
(5)
State-level KL divergence
where Const. =KL(p(s))llpazz))
Σ f / < ((aass))KLL(((sseeeee.l)ll.
at) )da,ds
Since transition dynamics, initial state distributions, and reference policy are all fixed at this point, we can minimize the state-level KI. divergence instead of the trajectory-Ievel KI. divergence for efficient training, and then the optimization objective becomes follows.
min (I) (T)) = min KL ( (a.la))))
In this way. VDPO divides the delayed RL problem into two separate optimization problems including Eq. (4) and Eq. (6). How to practically implement VDPO solve these optimization problems will be presented in Sec. 3.3.

---

3.2 Theoretical Property Analysis
Next, we explain why our VDPO achieves better sample efficiency compared with conventional delayed RL methods, followed by performance analysis of VDPO.
Sample Complexity Analysis. In fact, VDPO can use any delay-free RL method to improve the performance of the reference policy (maximizing A). Here, we assume that VDPO maximizes A by the model-based policy iteration, and the sample complexity of maximizing A is SS ( In )
as described in Lem. 3.2. And minimizing B in VDPO is equivalent state-level behaviour cloning which has the sample complexity of O
[2') [A]
(1-7)420) as stated in Lem. 3.4.
Lemma 3.4 (Sample complexity of behaviour cloning, Theorem 15,3 in [3]), Given the demonstration
from the optimal policy, behaviour cloning finds E-optimal policy with probability 1 using
sample size O
|n LA| (1-7)-20)
Based on Lem. 3.2 and Lem. 3.4, we can drive the sample complexity of VDPO (Lem. 3.5).
Lemma 3.5 (Sample complexity of VDPO, proof in Lem. C.2). Assumed that maximizing A in Eq. (3) by model-based policy iteration while minimizing B in Eq. (3) by behaviour cloning. VDPO
finds an e-optimal policy with probability 1 - using sample size
|S||A (1 71-
|n [A] In 0'(1 (1-1)) ))
maxx
Then, based on Lem. 3.5, we show that our VDPO has better sample complexity than most TD-only methods (e.g., model-based policy iteration [12]. Soft Actor-Critec [5, 20, 42)) follows.
Proposition 3.6 (Sample complexity comparison, proof in Proposition C.3). In the delayed MDP, as a + the sample complexity of VDPO (Lem. 3.5) is less or equal to the sample complexity of model-based policy iteration (Lem. 3.2):
|S||A O max (1 (1-7)3
1 |A|
)) s So
IIIII ( (11 In 2
o (1-7)4200
Proposition 3.6 tells us that VDPO can reduce the sample complexity effectively, reaching the same performance but requiring fewer samples compared to model-based policy iteration.
Performance Analysis. We investigate the convergence of the delayed policy VDPO (Lem. 3.7) and show that VDPO can also achieve the same performance existing SOTAs (Proposition 3.8). As mentioned above, Eq. (4) in VDPO can be solved by existing delay-free RI. method (e.g., model- based policy iteration) to learn an optimal reference policy x². Then, we get the convergence of the delayed policy = via Eq. (6).
Lemma 3.7 (Convergence of delayed policy in VDPO, proof in Lem. C.4). Let nt be the optimal reference policy which is trained by a delay-free RL algorithm. The delayed policy nr converges to A satisfying that
T (3 = Eg. [ (r*((a,
(ar[s)),, Vx, E X
Based on Lem. 3.7, we show that the convergence of VDPO is consistent with that of existing SOTA methods (Proposition 3,8).
Proposition 3.8 (Consistent fixed point, proof in Proposition C.5). VDPO shares the same fixed point (Eq. (7)) with DIDA [23], BPQL [20] and AD-SAC [42] for the same delayed MDP.
Proposition 3.6 and Proposition 3.8 together illustrate that VDPO effectively improve the sample efficiency while guaranteeing consistent performance with SOTAs [20, 23, 42].
3.3 VDPO Implementation
In this section, we detail the implementations of VDPO, specifically the maximization Eq. (4) and the minimization Eq. (6) respectively. The pseudocode of VDPO is summarized in Alg. 1, and the training pipeline of VDPO is presented in Fig. I.
5

---

Minimizing term in Eq.(3)
Maximizing term A Eq.(3)
nef
Actor loss Eq.())
(q.(11)
Delaved policy
Transformer
Reference policy
Reference critic
Belief estimator
Belief loss
Critic loss
A1
8
Eq,(10)
9t-
8t-1
8+
Eq.(8)
Figure 1: The training pipeline of VDPO.
Algorithm 1 Variational Delayed Policy Optimization
Input:
the reference policy T. and critic Qs: transformer with belief head bø and policy head mii for each update step do
# A of Eq. (3): Maximizing the performance of the reference policy
Updating critic Qo via Eq. (8) # Soft policy evaluation
Updating policy Ty via Eq. (9) # Soft policy improvement
# B of Eq. (3): Minimizing the state-level KL between и and Na
Updating belief head by via Eq. (10) # Belief representation learning
Updating policy head Tip via Eq. (11) # Behaviour cloning end for Output: by and Tip
Eq. (4) aims to maximize the performance of the reference policy in the delay-free setting, which VDPO addresses using Soft Actor- Critic [13]. Specifically, given transition data (st, at at, St-1). SAC updates the critic Qo parameterized by 0 via minimizing the soft TD error:
V
(1)()))))) -v)- v)
(8)
2
where Y = rt + Ea m. (110e)))))) [20 ac+1) - log To (ac-1 (s==1)) where Ty. is the reference policy parameterized by v. And the reference policy π. optimized by the gradient update:
E
[log T. (āls.) - Qo(s,, a)]-
Eq. (6) aims to minimize the state-level KL divergence between the reference policy π and delayed policy TA. Note that the true state s; under the delayed environment is inaccessible. Thus VDPO adopts a two-head transformer [37] to approximate not only the delayed policy πA but also the belief estimator that predicts the state $,, as transformer shows a superior representation performance in behaviour cloning [8, 18]. We also discuss how different neural representations influence the RL performance later Sec. 4.2.3. similar transformer architecture proposed [24] is adopted, which serializes the augmented state input. Based on the information bottleneck principle [34], the encoder needs to encode the input as embeddings with sufficient information related to the true states. Thus, the belief decoder and the policy decoder share a common encoder which only trained while training the belief decoder, and freeze the gradient backward of the encoder in training the policy decoder.
{Sr-A, At.A:.... (r-1) } to {(8:-A, a, ) as the
Specifically, for given augmented state Ie and true states { the belief decoder be parameterized by é aims reconstruct the states - based the xg. Therefore, the belief decoder by is optimized by the reconstruction loss:
0 (1r).
(10)
where (100 (arz) is the i-th reconstructed state of the belief decoder by and MSE is the mean square error loss. Given the reference policy मy and the pair of augmented state and states (æ,, {s--A++)A)) the policy decoder Te parameterized by is optimized by minimizing the KL loss:
KL (-1z)) (+1(((+))))
(11)
where m/62 (-12s) is the i-th output of the policy decoder T..

---

Experimental Results
4.1 Experiment Settings
We evaluate our VDPO in the MuJoCo benchmark [35]. For the selection of baselines, we choose the existing SOTAs including Augmented SAC (A-SAC) [13]. DC/AC [5], DIDA [23]. BPQL [20] and AD-SAC [42]. The setting of pyper--praamttess is presented in Appendix A. We investigate the sample efficiency (Sec. 4.2.1) followed by performance comparison under different settings of delays (Sec. 4.2.2). We also conduct the ablation study on the representation of VDPO (Sec. 4.2.3). Each method was run over 10 random seeds. The training curves can be found in the Appendix E.
4.2 Experimental Results
4.2.1 Sample Efficiency
We first evaluate the sample efficiency in the MuJoCo with 5 constant delays. Using the performance of delay-free policy trained by SAC, Relay. as the threshold, report the required steps reach this threshold within 1M global steps in Table From the results, we can tell that VDPO shows strong superiority in terms of sample efficiency, successfully reaching the threshold in all tasks and achieving the best sample efficiency in 7 out of 9 tasks. Specifically, VDPO only requires 0.42M and 0.67M steps to reach the threshold in Ant-v4 and Humanoid-v4 respectively, while none of the baselines can reach the threshold within 1M steps. In Halfcheetah-r4 Hopper-v4, Pusher-v4, Swimmer-vv 14 and Walker2d-v4. the steps taken by our VDPO is around 51% (ranging from 25% to 78%) of that required by AD-SAC, SOTA baseline. Based on these results, we can conclude that VDPO shows a significant advantage in sample complexity compared to other baselines. Additional experimental results of 25 and 50 constant delays are presented in Appendix D.
Table 1: Amount of steps required to reach the threshold Retar in MuJoCo tasks with 5 constant delays within 1M global steps, where x denotes that failed to hit the threshold within 1M global steps. The best result is in blue.
Task (Delays=S)
A-SAC
DCIAC
DIDA
BPQL
AD-SAC
VDPO (ours)
Ant-v4
x
x
X
X
X
0.42M
HaliCheetah-v4
0.99M
0.56M
0.44M
Hopper-v4
0.83M
0.35M
X
0.29M
0.12M
0.07M
Humanoid-v4
X
X
X
X
X
0.67M
HumanoidStandup
0.64M
0.35M
0.I0M
0.09M
0.14M
0.14M
Pusher-v4 v4
0.17M
0.02M
0.10M
0.27M
0.04M
0.01M
Reacher-v4
X
0.61M
0.
0.90M
0.44M
0.77M
Swimmer-v4
X
0.94M
0.10M
X
0.13M
0.07M
Walker2d-v4
X
0.52M
0.67M
0.25M
4.2.2 Performance Comparison
The performance of VDPO and baselines are evaluated on MuJoCo with various settings and normalized indicator [39, 42] Retnor Reta Refrand" where Retaig and Retrand are the performance of the algorithm and random policy, respectively. The results of MuJoCo benchmarks with 5, 25, and 50 constant delays are shown in the Table 2, showing that VDPO and AD-SAC outperform other basclines significantly in most tasks. Overall, VDPO and AD-SAC (SOTA) show a comparable performance, which is consistent with the theoretical observation in Sec. 3.2.
4.2.3 Additional Discussions
In this section, we conduct the ablation study to investigate the performance VDPO using different neural representations. Furthermore, we explore whether VDPO is robust under stochastic delays.
Ablation Study on Representations. As mentioned Sec. 3.3, we investigate how the choice of neural representations for belief and policy influences the performance of VDPO. Baselines include multiple-layer perceptron (MIP) and Transformer without belief decoder. The results presented in Table 3 show that the two-head transformer used by our approach yields the best performance
7

---

Table 2: Normalized Performance Retnor in MuJoCo tasks with 5, 25, and 50 constant delays for 1M global steps, where + denotes the standard deviation. The best performance is in blue.
Task
Deluys
A-SAC
DCAAC
DIDA
BPQL
AD-SAC
VDPO (ours)
0.18=0.01
0.25=0.05
0.89 20.03
0.90-2003
0.7220.20
1.1120.04
Ant-v4
25
0.07=0.07
0.19
0.29
=.11
0. 0.664a.04
0.56
0.02-0.04
0.19-a.02
0.19+0.05
0.38-0.07
0.48+0.06
0.46+0.07
0.35_.o.1 10.15
0.40 L0.23
10.01
10.00
L.07 10.00
1.03
HalfChcetah-v4
25
0.04
g..07
0.10
0.87 20.04
0.71
0.70
0.12=0.17
0.12
0.15
0.73=0.17
0.74
0.72+0.21
1.02=0.28
1.16
0.40
1 1.25=0.09
4.07
1.22+0.08
Hopper-v4
25
0.13
0.19
0.27
1.21
0.86
0..4
50
0.04
0.04
0.09+0.01
0.71 =0.13
0.72+0.03
0.22
5
0.13=0.02
0.59
0,08+0.04
0,96=0.05
0.98+0.07
1.1540.07
Humanoid-y4
25
0.05=0.01
0.04=0.01
0.07
0.12=z0.0
0.25
0.1240.02
50
0.04_0.01
0.03 0.03
0..00
0.08 =0.8
0. 0.10
0.1240.00
5
1.02=0.08
1. 1.16
1.00
1.13_0.07
1.22+0.03
1.29
HumanoidStandup-v4
25
0.97_10.00
1.030
0.97
1 1.09
1.1510.08
1.13
50
0..02
1.02
0.89
1 1.06
1.1210.02
1.04
5
1.1
=0.02
1.29
1.01
1.06-0.0s
1.36
1.17
Pusher-v4
25
0. 0.49_0.32
1.1220.02
1.0440.01
1.07
1.29
1.30
50
0.00
1.13
1.0440.02
1.00
1.23+a.0z
1.33
5
0.97
1.02
1.03
1.00
L033
1.02.0.03
Reacher-v4
25
0.9610.02
1o000
1..81
10.05
0.98
1.02 1.02
50
0.86=0.02
0.89
0.93
0.90
0.91
1.02+0.03
5
0.88
1.11 20.30
1.05
20.02
1.82
2.30
Swimmer-v4
25
0.72
0.78
0.93
1.36
1.36
2.52+4.40
2.35
50
0.69 =0.04
0.68
0..03
2.23
2.71
2.42+0.22
0.76
0.85=0.12
0.61
1.20-20.11
L.12+0.09
1.2740.04
Walker2d-v4
25
0.12
0. 0.26
0.10
0.50
0.72+a.11
0.27+0.11
50
0.11
0. 0.11
0.08
0.23
0.23
0.11
compared to other candidates. The results also confirm that an explicit belief estimator implemented by a belief decoder can effectively improve performance.
Table 3:
Normalized Performance Retnor of VDPO using different representations in MuJoCo tasks
with constant delays, where denotes the standard deviation. The best result is in blue.
Tasks (Delays=5)
MLP
Transformer w/o belief
Transformer w/ belief (ours)
Ant-v4
0.8640.20
1.09
1.11
HalfCheetah-ve
0.95
1.37
1.03
Hopper-v4
1.1140.19
1.13
1.22
Humanoid-ve v4
0. 0.7844.11
0.8940.43
1.1544.00
HumanoidStandup- v4
1.2844.005
1.2840.08
1.29
Pusher-v4
1.35440.04
1.34
1.17 2-0.06
Reacher-v4
1.02+0.05
1.02+0.04
1.02+0005
Swimmer-v4
202940.37
2.11
2.30
Walker2d-v4
1.1340.20
1.1540.14
1.27 +0.04
Stochastic Delays. We compare the performance in the MuJoCo with 5 stochastic delays where A 5 is with a probability of 0.9 and A E [1, 5] is with a probability of 0.1. The results in the Table 4 demonstrate that VDPO outperforms other baselines at most tasks under stochastic delays. Especially in the Ant-r4 and Walker2d-14, VDPO performs approximately 62% and 49% better than the second best approach, respectively. the Reacher-v4 and Swimmer-v4, VDPO achieves a comparative performance with the best baseline. We will conduct theoretical analysis of VDPO under stochastic delays in the future.
Limitations and Future Works. We mainly consider deterministic benchmarks in this paper, which are commonly adopted the SOTAs [5, 23, 39]. However, the recent work ADRL [42] illustrates that existing approaches may have performance degeneration in stochastic environments, which can be mitigated by learning an auxiliary delayed task concomitantly. We will investigate in the future to integrate VDPO with ADRL to address stochastic applications.
8

---

Table Normalized Performance Retnor in MuJoCo tasks with 5 stochastic delays for IM global steps, where + denotes the standard deviation. The best result is in blue.
Tasks
A-SAC
DC/AC
DIDA
BPQL
AD-SAC
VDPO (ours)
Ant-v4
0.18
0.27
0.55
0.58
0.69=0.17
1.12
HaliCheetah-v4
1.
0.36
0.75
0.76.0.16
1.03
1.07
Hopper-v4
0.85
4.94
0.31
0.68
L.05
1.3
Humanoid v4
0.15
0.
0.07
4.40
0.97
1.0640.00
HumanoidStandup.vv4
1.03
1.20
1.00
1.10
1.26=40.07
1.27
Pusher-v4
0.
1.1720.02
1.022a.01
1.07
1.2220001
1.34
Reacher-v4
0.98
1.0240.01
1. 1.02+20.00
0.85
1.05
1.01
Swimmer-v4
4.
1.47
1.03+0.02
1.53
2.36=0.64
2. 2.1340.18
Walker2d-v4
0.
0.8940.08
0.54
0.59
0.63
1..3
5
Related Works
Compared to the common delay-free setting, delayed RL with disrupted Markovian property [4. 19] is closer to real-world complex applications, such robotics [17, 26], transportation systems [6] and financial market trading [14]. Existing delayed RL techniques conduct learning over either original state space (referred to direct approach) augmented state space (referred to augmentation- based approach). Direct approaches enjoy high learning efficiency by learning in the original small state space. However, early approaches simply ignore the absence of Markovian property caused by delay and directly conduct classical RL techniques based on delayed observations, which distinctly suffer from serious performance drops. The subsequential improvement
train based on unobserved instant observations, which are predicted by various generative models, e.g., deterministic generative models [38]. Gaussian distributions [7], and transformers [24]. However, the inherent approximation errors in these learned models introduce prediction inaccuracy and result in sub- optimal performance issues [24]. To summarize, direct approaches achieve high learning efficiency,
but commonly with a cost of performance degeneration.
The augmentation- based approach is notably more promising it retrieves Markovian property via augmenting the state with the actions related to delays and thus legitimately enables RL techniques over the yielded delayed MDP [4, 19]. However, the augmentation--asse approach works in significantly larger state space, which is thus plagued by the curse of dimensionality, resulting in learning inefficiency. To mitigate this issue, DC/AC [5] leverages the multi-step off- policy technique to develop a partial trajectory resampling operator to accelerate the learning process. Based on the dataset aggregation technique, DIDA [23] generalizes the pre-trained delay-free policy into an augmented policy. Recent attempts [20, 39] evaluate the augmented policy by a non-augmented Q- function for improving learning efficiency. ADRL [42] suggests introducing an auxiliary delayed task with changeable auxiliary delays for the trade-off between the learning efficiency and performance degeneration in the stochastic MDP. However, these approaches still suffer from the sample complexity issue due to the fundamental challenge of TD learning in high dimensional state space.
The conceptualization of RL inference problem has gained attraction recently, allowing the adaption of various optimization tricks to enhance RL efficiency [11, 15, 21, 31]. For instance, VIP [30] integrates different projection techniques into the policy search approach based on variational inference. Virel [9] introduces variational inference framework that reduces the actor-critic method to the Sispetttion-Maxxnizzze regularized methods in the EM fashion. CVPO [25] extends MPO to the safety-critical settings. The novel trial in this work of viewing the delayed RL as a variational inference problem allows us to use extensive optimization tools to address the sample complexity issue in delayed RL.
(EM) algorithm. MPO [1. 2] is a family of off off-policy entropy-
6
Conclusion
This work explores the challenges of RL problems in environments with inherent delays between agent interactions and their consequences. Existing delayed RL methods often suffer from learning inefficiency as lemporal-differeen
learning in the delayed MDP with high dimensional augmented state space demands an increased sample size. To address this limitation, we present VDPO, new delayed RI. approach rooted in the variational inference principle. VDPO redefines the delayed RI. problem into a two-step iterative optimization problem. It alternates between (I) maximizing the

---

performance of the reference policy by temporal -iffereenc learning in the delay-free setting and (2) minimizing the KL divergence between the reference and delayed policies by behaviour cloning. Furthermore, our theoretical analysis and the empirical results in the MuloCo benchmark validate that VDPO not only effectively improves the sample efficiency but also maintains robust performance level.
Acknowledgments and Disclosure of Funding
We sincerely acknowledge the support by the grant EP/Y002644/1 under the EPSRC ECR Interna- tional Collaboration Grants program, funded by the International Science Partnerships Fund (ISPF) and the UK Research and Innovation, and the grant 2324936 by the US National Science Foundation. This work is also supported by Taiwan NSTC under Grant Number NSTC-112-222 1---1---02-------- 168-MY..
10

---

References
[1] A. Abdolmaleki, J. T. Springenberg, J. Degrave, S. Bohez, Y. Tassa, D. Belov, N. Heess, and
M. Riedmiller. Relative entropy regularized policy iteration. arXiv preprint 2018.
[2] A. Abdolmaleki, J. T. Springenberg, Y. Tassa, R. Munos, N. Heess, and M. Riedmiller. Maxi- mum posteriori policy optimisation. arXiv preprint / 1806006900 2018.
[3] A. Agarwal, N. Jiang, and S. M. Kakade. Reinforcement learning: Theory and algorithms. 2019.
[4] E. Altman and P. Nain. Closed-loop control with delayed information. ACM sigmetrics performance evaluation review, 20(1):193-22) 1992.
[5] Y. Bouteiller, S. Ramstedt, G. Beltrame, C. Pal, and J. Binas. Reinforcement learning with random delays. In International conference on learning representations, 2020.
[6] Z. Cao, H. Guo, W. Song, Gao, Z. Chen, L. Zhang, and X. Zhang. Using reinforcement
learning to minimize the probability of delay occurrence in transportation. IEEE transactions on vehicular technology, 69(3):2424-22) 2436, 2020.
[7] B. Chen, M. Xu, L. Li, and D. Zhao. Delay-aware model-based reinforcement learning for continuous control. Neurocomputing, 1550:1991220 2021.
[8] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and
I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:1 5084-150977 2021.
[9] M. Fellows, A. Mahajan, T. G. Rudner, and S. Whiteson. Virel: A variational inference framework for reinforcement learning. Advances in neural information processing systems, 32, 2019.
[10] V. Firoiu, T. Ju, and Tenenbaum. At human speed: Deep reinforcement learning with action delay. arXiv preprint arXiv: 2018.
[11] J. Fu, K. Luo, and S. Levine. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint arXiv:1710.11245, 2017.
[12] M. Gheshlaghi Azar, Munos, and H. J. Kappen. Minimax pac bounds on the sample complexity of reinforcement learning with generative model. Machine learning, 91:325-349, 2013.
[13] Haarnoja, Zhou, P. Abbeel, and Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference machine learning, pages 1861-1870. PMLR, 2018
[14] Hasbrouck and G. Saar. Low-latency trading. Journal of Financial Markets, 16(4):646-679, 2013.
[15] J. Ho and S. Ermon. Generative adversarial imitation learning, Advances in neural information processing systems, 29, 2016.
[16] S. Huang, R. F. J. Dossa, C. Ye, J. Braga, D. Chakraborty, K. Mehta, and G. Araāšjo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research. 23(274):1-18, 2022.
[17] Hwangbo, I. Sa, R. Siegwart, and M. Hutter. Control of quadrotor with reinforcement learning. IEEE Robotics and Automation Letters, 2(4):2096-2103, 2017.
[18] M. Janner, Q. Li, and S. Levine. Offline reinforcement learning as one big sequence modeling problem. Advances neural information processing systems, 34: 14:3-12-, 2021.
[19] K. V. Katsikopoulos and S. E. Engelbrecht. Markov decision processes with delays and asynchronous cost collection. IEEE transactions on automatic control, 48(4):568-57) 5744, 2003.
[20] Kim, H. Kim, J. Kang, Baek, and S. Han. Belief projection- reinforcement learning for environments with delayed feedback. In seirth- Conference on Neural Information Processing Systems, 2023.
[21] Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. 2018
arXiv preprint

---

[22] P. Liotet. Delays in reinforcement learning. arXiv preprint arXiv:2309.11099 2023.
[23] P. Liotet, D. Maran, L. Bisi, and M. Restelli. Delayed reinforcement learning by imitation. In International Conference on Machine Learning, pages 13528- PMLR, 2022.
[24] P. Liotet, E. Venneri, and M. Restelli. Learning a belief representation for delayed reinforcement learning. In 2021 International Joint Conference Neural Networks (LJCNV), pages 1-8.
IEFE, 2021.
[25] Z. Liu, Z. Cen, V. Isenbaev, W. Liu, S. Wu, B. Li, and D. Zhao. Constrained variational policy optimization for safe reinforcement learning. In International Conference on Machine Learning, pages 13644-13668. PMLR, 2022.
[26] A. R. Mahmood, D. Korenkevych, B. J. Komer, and J. Bergstra. Setting up reinforcement learning task with real-world robot. 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4635-4640 IEEE, 2018.
[27] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
[28] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep rein- forcement learning. nature, 51 518(7540):529-53) 2015.
[29] S. Nath, M. Baranwal, and H. Khadilkar. Revisiting state augmentation methods for reinforce- ment learning with stochastic delays. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 1346-1355, 2021.
[30] G. Neumann. Variational inference for policy search in changing situations. 2011.
[31] D. Ramachandran and E. Amir. Bayesian inverse reinforcement learning. In IJCAI, volume 7, pages 2586-2591, 2007.
[32] J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lock- hart, D. Hassabis, T. Graepel, al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(7839)6004-609 2020.
[33] E. Schuitema, L. Buşoniu. R. Babuška, and P. Jonker. Control delay in reinforcement learning for real-time dynamic systems: memoryless approach. In 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 3226-3231. IEEE, 2010.
[34] N. Tishby and N. Zaslavsky. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (itw), pages 1-5, IEEE, 2015.
[35] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033 IEEE, 2012.
[36] M. Toussaint. Robot trajectory optimization using approximate inference. In Proceedings of the 26th annual international conference on machine learning, pages 1049-1056, 2009.
[37] A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and
I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[38] J. Walsh, A. Nouri, L. Li, and M. L. Littman. Learning and planning in environments with delayed feedback. Autonomous Agents and Multi-Agent Systems, 18:83-105, 2009.
[39] W. Wang, D. Han, X. Luo, and D. Li. Addressing signal delay in deep reinforcement learning.
In The Twelfth International Conference on Learning Representations, 2023.
[40] Y. Wang, S. Zhan, Z. Wang, C. Huang, Z. Wang, Z. Yang, and Q. Zhu. Joint differentiable opti- mization and verification for certified reinforcement learning. In Proceedings of the ACM/IEEE 14th International Conference on Cyber-Physiceh Systems (with CPS-IoT Week 2023), pages 141, 2023.
[41] Y. Wang, S. S. Zhan, R. Jiao, Z. Wang, W. Jin, Z. Yang, Z. Wang, C. Huang, and Q Zhu. Enforcing hard constraints with soft barriers: Safe reinforcement learning in unknown stochastic environments. In International Conference on Machine Learning, pages 36593-36604. PMI.R 2023.
12

---

[42] Q. Wu, S. S. Zhan, Y. Wang, C.-W. Lin, C. Lv, Q. Zhu, and C. Huang. Boosting long-delayed reinforcement learning with auxiliary short-delayed task. arXiv preprint arXiv:2402.03141, 2024.
[43] S. S. Zhan, Y. Wang, Q. Wu, R. Jiao, C. Huang, and Q. Zhu. State-wise safe reinforcement learning with pixel observations, arXiv preprint arXiv:231 1. 1.02227, 2023.
13

---

A
Implementation Details
The
setting of VDPO is presented in Table 5. For baselines, we adopt similar hyper-parameters settings as suggested by original works, including A-SAC. DC/AC [5]. DIDA [23], BPQL [20] and AD-SAC [42]. The implementation of VDPO based on CleanRL [16], and we also provide the code and guidelines to reproduce our results in the supplemental material. Each run of VDPO takes approximately hours on NVIDIA A100 GPU and 8 Intel Xeon CPUs.
Table 5: Hyper-parameterr table of VDPO.
Hyper-pane parameerr
Value
Buffer Size
1,000,000
Batch Size
256
Global Timesteps
1,000,000
Discount Factor
0.99
Reference Policy
Learning Rate for Actor
3e-4
Learning Rate for Critic
1e-3
Network Layers
Network Neurons
[256, 256]
Activation
ReLU
Optimizer
Adam
Initial Entropy
-|A
Learning Rate for Entropy
le-3
Train Frequency for Actor
Train Frequency for Critic
Soft Update Factor for Critic
5e-3
Delayed Policy (Transformer)
Sequence Length
Embedding Dimension
256
Attention Heads
Layers Num
Attention Dropout Rate
0.1
Residual Dropout Rate
0.1
Embedding Dropout Rate
0.1
Learning Rate Optimizer
3e-4
Adam
Train Frequency for Belief Decoder
Train Frequency for Policy Decoder
1,000
14

---

B Evidence Lower BOund (ELBO) of Delayed RL
Following the similar derivation sketch with [2, 25], we provide the derivation of Eq. (2) follows.
P P (O = 1) - -log / p = (
log / (T((T)
pl0 1/T) (T)
dT
P+ (T)
pp0 P (T)
Pr log
dr
(T(
(r) (Oo
1/7)] - (L((p(rr)le (T))
C Theoretical Analysis
Proposition C.1 (State-level KL divergence). For a fixed reference policy n, the trajectory-level KL divergence can transform into the formulation of state-level KI. divergence as follows
f d' ((66))LLirllllllllaaaall))) +Const +Cont...
(L)p(()))P Σ
(12)
State-level KT. divergence
where Const. =KL(p(so)llss(zo)
Σ (8t) / x Patp://sseeel.aaalllaaaaaaaaaaaa at) 1=
Proof. The trajectory distribution Pa (T) induced by a is given by:
(T( p(su) II
t=
Similarly, the trajectory distribution Pra (T) induced by na given by:
Pra (T) = Pa) (m))
II (s44)
ar (a
Therefore, the trajectory-level KI. divergence can be written as
(L(px(T)) (T))
= =B (r) [log (T)) - logPra PTa (T)
E Errop (T) logg(so)
log [P(s:+1|s;, (a)s(l)))))]
log (p(2))b((0)
[log ((s))((4110)))) + log Ps (1141/1) + log
(a)lx))))))
=E, -Erop, (T)
log P(sct1ls1, 0z) + log T(ac
log b(s xx) + log ar) + log a (arlz1))
1=0
p(so)
P(8 18, (z)
+ log (at(8)))
= =Brope(T)
log
log b(s, 2z P Pa a (#141
PA (Io)
TT (ag (rr)
p(su)
P (s) b(st|2,, P ( at)
π ( (s)) log
(T-pp(t)
log
log
+ E ETopat
PA (M()
F (a))
15

---

For C, we have
P P(s 3e, ar)
C' = (L(o(s))lleaz0)) +
Ps [T) log b(s,l1z)) P T [c)
dt
P(s Se, (c, (5() / x(ar1s.) / P(5 at) log b(st|1,) (2+41l0e, dss da,ds;
(L(p(()))lAA(o))
(L(D()))lls Σ f (s() /
2 0.
Therefore, C is a constant determined by the transition function of the dynamics and the fixed reference policy x.
Then, for D, we have
2 D =
(url(t)) dr (a(a)) d'(ss) / a(aals)) log ((eels))
Σ t=
(r(T) log
daids;
(f6s:KKlattaassssaaaaaaaaaaa 1=
Lemma C.2 (Sample complexity of VDPO). Assumed that maximizing A in Eq. (3) by model-based policy iteration while minimizing in Eq. (3) by behaviour cloning, VDPO finds an e-optimal policy with probability using sample size
|
|n [A] In 0' (1 (1-7)'²2")) ))
max
(1 332
Proof. Applying Lem. 3.2 and Lem. 3.4.
Proposition C.3 (Sample complexity comparison). In the delayed MDP, as a + O, the sample complexity of VDPO (Lem. 3.5) is less equal the sample complexity of model-based policy iteration (Lem. 3.2):
|S|A
In 1
18 IAI )) S0 (1-
IA 73
In 1n-
O max
(1 7)-7)322
Proof. It is obvious that
|
I 1
In <
In
3e2 EE2
(1 322 σ
have |S| |x| |
Then, we show that which is equivalent
X In IA
|A
<
In 0.
422
(1 - 3
In A
< - (1 7) lno
A
This inequality always holds when -
lno
>> 1 |A
lim -(1 - 7)
+o0
σ
A
16

---

Lemma C.4 (Convergence of delayed policy in VDPO). Let πT" be the optimal reference policy which is trained by a delay-free RL algorithmm The delayed policy nu converges v satisfying that
T (0) (..-b(()a.)) (m (ai]ss)]; Vay E X.
(13)
Proof. We can derive the result from the solution of Eq. (6).
Proposition C.5 (Consistent fixed point). VDPO shares the same fixed point (Eq. (7) with DIDA [23], BPQL [20] and AD-SAC [42) for the same delayed MDP.
Proof. The fixed points DIDA (Eq. (3) in [23]), BPQL (Eq. (23) in [20]) and AD-SAC (Theorem 5.9 in [42]) all are
a (x) = bl) ( (a:ls))) Vo, E X.
which is consistent with the fixed point of VDPO.
17

---

D Additional Experimental Results
In MuJoCo with 25 and 50 constant delays, We report the required steps hit this threshold within IM global steps in Table and Table 7 respectively.
Table 6: Amount of steps required hit the threshold Ret, in MuJoCo tasks with 25 constant delays within 1M global steps, where x denotes that failed to hit the threshold within 1M global steps. The best result in blue.
Task (Delays=25)
A-SAC
DCIAC
DIDA
BPQL
AD-SAC
VDPO (ours)
Ant-v4
X
X
HaliCheetah-v4
X
X
X
Hopper-v4
X
0.69M
X
X
Humanoid-v4
X
X
X
HumanoidStandup-vk
X
0.38M
0.09M
0.09M
0.48M
Pusher-v4
X
0,09M
0.10M
0.02M
0.03M
0.01M
Reacher-v4
X
0.83M
X
X
0.22M
Swimmer-v4
X
0.39M
0.12M
0.09M
Walker2d-v4
X
X
X
X
X
X
Table 7: Amount of steps required to hit the threshold Rety in MuJoCo tasks with 50 constant delays within 1M global steps, where x denotes that failed to hit the threshold within 1M global steps. The best result blue.
Task (Delays=50)
A-SAC
DCIAC
DIDA
BPQL
AD-SAC
VDPO (ours)
Ant-v4
X
X
X
X
HalfCheetah-v4
X
X
X
X
X
Hopper-v4
X
X
X
Humanoid-v4
X
X
X
HumanoidStandupp-44
X
0.68M
0.21M
0.08M
0.84M
Pusher-v4
X
0.14M
0.10M
0.18M
0.02M
0.0TM
Reacher-v4
X
X
X
0.39M
Swimmer-v4
X
0.29M
0.11M
0.15M
Walker2d-v4
X
18

---

E Learning Curves
6300
6000
3500
5000
3000
4000
2500
Return
Return
Return
2000
2000
2000
2000
1030
1000
Global Steps
Global Steps
Global Steps
(a) Ant-v4
(b) HalfChectah-v4
(c) Hopper-v4
160000
6000
140000
-50
Retum
6000
Retur
100
3000
100000
80000
150
60000
3.0
0.2
0.4
0.6
0.8
1.0
0.4
0.6
0.2
0.0
1.0
Global Steps
Global Steps
Global Steps
(d) Humanoid-v4
(e) HumanoidSSanduu--44
(l) Pusher-v4
100
4000
Return
Return
Return
2000
40
-12
1000
0.0
0.2
0.4
0.6
0.0
1.0
0.0
0.2
0.4
0.6
0.8
1.0
044
0.6
Global
Steps
Global Steps
Global
Steps
Reacher-v4
Swimmer-yd
Walker2d-v4
A-SAC
DC/AC
DIDA
BPQL
AD-SAC
VDPO (ours)
Figure Learning curves in MuJoCo tasks with constant delays where the shaded areas represented the standard deviation.
19

---

5000
3500
3000
3000
4000
AWh Me ma
2500
Return
Return
Return
2000
2000
1500
1.00
1000
1000
500
3.0
0.6
0.6
0.6
Global Steps
Global Steps
Global Steps
(a) Ant-v4
(b) HalfChectah-y4
(c) Hopper-v4
2000
1500
Return
M
Retur
120000
Return
100
300000
-150
200
0.0
0.6
0.8
1.5
0.0
0.6
1.0
Global Steps
Global Steps
Global Steps
(d) Humanoid-wy v4
(c) HumanoidStandup-v4
(f) Pusher-v4
150
3500
-5.0
125
7.5
2500
Return
10.0
Return
75
Return
2000
12..5
1.000
1000
500
3.0
02
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
3.0
Global Steps
Global
Steps
Global Steps
Reacher-v4
(h) Swimmer-v4
Walker2d-v4
A-SAC
DC/AC
DIDA
BPQL
AD AD-SAC
VDPO (ours)
Figure 3: Learning curves in MuJoCo tasks with 25 constant delays where the shaded areas repre- sented the standard deviation.
20

---

5000
Return
Return
3000
1000
1000
-1000
0.0
0.6
0.6
Global Steps
Global Steps
Global Steps
(a) Ant-v4
(b) HalfChectah-v4
(c) Hopper-v4
700
140000
120000
Retum
Retur
Return
a00000
-150
400
300
200
60000
250
8.0
0.6
0.8
1.5
0.6
0.8
1.0
1.0
Global Steps
Global Steps
Global Steps
(d) Humanoid-vy
(c) HumanoidStandup-v4
(f) Pusher-v4
130
100
1000
Return
Return
Return
800-
600
-30
400
-10
200
3.0
0.2
0.4
0.6
0.8
1.0
-25
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Global Steps
Global
Steps
Global Steps
Reacher-v4
(h) Swimmer-v4
Walker2d-v4
A-SAC
DC/AC
DIDA
BPQL
AD AD-SAC
VDPO (ours)
Figure 4: Learning curves in MuJoCo tasks with 50 constant delays where the shaded areas repre- sented the standard deviation.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
We have clearly claimed our contribution and scope in the abstract and introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: In Section 4.2.3, we have discussed the limitation that this work suffers from performance degeneration issues in stochastic environments.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
22

---

Answer: [Yes]
Justification: We have provided the full set of assumptions and complete and correct proof in the main text and appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided all the information needed to reproduce our experimental results, including implementation details in the main text and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
23

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We have provided our code to reproduce the experimental results in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / httppsscccppbpcccccceecce ceroolicggiideee CodeeSbbm ssi aasonnooicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have provided the implementation details the main text and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: seeds.
We have reported the experimental results averaged over multiple random
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
We have provided the computer resource in the implementation detail of the appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http:/::////uiseeeeemee cc/public/Eeteeeccceecceeeeeeeee d ines?
Answer: [Yes]
Justification: The research conducted in this paper conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper aims advance the field of Machine Learning. We believe there are no potential societal impacts of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
25

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi wwwode.hho....c. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
26

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "iHRe8L855M", "reviewer": "Reviewer_FFJG", "summary": "The paper proposes a novel delayed rl algorithm called variational delayed policy optimization, which reformulates delayed RL as a variational inference problem and solves it with a two-step iterative optimization. Both theoretical and empirical results show that VDPO achieves better performance in sample efficiency in the mujoco benchmark.", "questions": "1.\tIt seems that the proposed method must be trained in a delayed-free environment. So the reviewer wonders whether the baseline methods like A-SAC are conducted in such setting. If they are not, the comparison is not quite fair from the reviewer’s perspective.\n2.\tCould the author present the comparison of inference time among VDPO and other baseline methods?", "limitations": "The authors have clearly presented the limitations in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe proposed method bridges the gap between delayed RL and variational RL, which shows its novelty.\n2.\tThis paper presents both theoretical analysis and empirical results on the sample complexity of VDPO.\n3.\tThe paper is overall well-written, and the idea is straightforward.", "weaknesses": "1.\tIt seems that the proposed method must be trained in a delayed-free environment. It may result in limited application scenarios.\n2.\tThe proposed method utilizes transformer as the policy. Given the computational cost of transformer, it may result in longer delay in real-world applications. However, the authors do not take it into account in the experiments.\n3.\tEquation 10 seems based on the underlying assumption that the subsequence state obeys Gaussian distribution, which may not hold in some real-world scenarios.\n4.\tAccording to Figure 3, when the number of the constant delays is large, it looks that VDPO doesn’t work very well. Moreover, given the computational cost of transformer, the real performance of VDPO is doubtful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer FFJG for the comment. Before replying to all the concerns and questions in detail, we want to clarify that this work adopts commonly used evaluation metrics, sample efficiency and performance (return), and conducts fair comparison with existing works [1, 2, 3]. Our detailed responses are as follows:\n# Weakness 1 and Question 1: Training in the delay-free environment and baseline selection.\nWe appreciate the reviewer for the interesting questions. For **Weakness 1**, we clarify that **training in a delay-free environment performs no restriction on VDPO's applicability** and the reason is as below.\n\nLet $\\{(s_{t-\\Delta},a_t,r_t)\\}$ be a sampled trajectory in an environment with a constant delay $\\Delta$. At time step $t$, though the agent can only observe $s_{t-\\Delta}$, the true instant state $s_{t}$ will be observed at $t+\\Delta$. Thus we can easily synthesize a delay-free trajectory for training based on a time-delay trajectory. It illustrates why VDPO, the same as SOTAs [1, 2, 3] that also leverage the training in a delay-free environment, can be applied in any constant-delay applications.\n\nFor **Question 1**, the SOTA DIDA [1], BPQL [2] and AD-SAC [3] all include delay-free training in their frameworks, which are the baseline methods in this paper. Sample efficiency and performance are the common evaluation metrics used in delay RL materials [1, 2, 3]. Therefore, we believe we conduct **fair comparison under common setting**, following the standard manner in delayed RL research.\n\n# Weakness 2 and Question 2: The inference delays of transformer in VDPO.\nWe clarify that existing delayed RL works [1, 2, 3], including this paper, all use **sample efficiency and performance** as the evaluation metrics. The reason for not considering inference time is as below.\n\nThe delays caused by the inference of the neural networks with common architectures are usually much less than the time delays in real-world applications (see Table R1). Specifically, in this work, the inference time of baselines (MLP) is around 0.5 ms, and the inference time of VDPO (transformer) is approximately 1.8 ms (all run on one NVIDIA A100 GPU), while in the teleoperation of robotics [4], especially in space robotics [5, 6], the delays are ranging from 5 s to 10 s. The inference time of the neural network is approximately **three orders of magnitude less than the delays**. Therefore, the inference delays can be safely neglected.\n\nTable R1: Inference time of different neural network architectures.\n|Delays=5|MLP|Transformer|\n|-|-|-|\n|Inference time (ms)|$0.529 (\\pm 0.005)$|$1.858 (\\pm 0.039)$|\n\nAdditionally, if a significantly complex neural architecture is adopted in the future, the inference delay issue can be addressed naively by adding extra delay step in the training process in advance. For example, the policy can be trained in the delayed environment with 6 delays for deploying in the real environment with 5 delays, if the inference delay takes more or less 1 control cycle, since the inference time should be constant for a specific neural architecture under the same hardware setting.\n\nTo sum up, we believe the inference delay issue is not a critical issue and can be easily addressed if necessary. However, we appreciate the reviewer for raising it and will add the discussion in the revised version to avoid any potential confusion.\n\n# Weakness 3: Assumption of Equation 10.\nWe clarify that Equation 10 **does not** rely on the assumption that the subsequence state obeys Gaussian distribution. Equation 10 is the loss function of the belief function $b$ which aims to learn a representation for the delayed policy $\\pi_\\Delta$ via reconstructing the states $\\\\{s_{t-\\Delta+i}\\\\}_{i=1}^\\Delta$ from the augmented state $x_t$, inspired by existing works [7, 8].\n\n# Weakness 4: Performance of VDPO.\nIn this work, VDPO aims to effectively improve the sample efficiency without compromising the performance. Therefore, we focus on two objectives: whether VDPO can achieve better sample efficiency, and whether VDPO can provide comparable performance with SOTA techniques.\n\nIn terms of the **sample efficiency**, our VDPO can successfully hit the threshold in 4 out of the 9 tasks (HumanoidStandup-v4, Pusher-v4, Reacher-v4 and Swimmer-v4) when delays increase to 25 and 50, while the SOTA baseline AD-SAC hit the threshold in 3 out of the 9 tasks while requiring more samples in majority tasks. \n\nIn terms of **performance**, when delays are increased to 50, the SOTA baseline AD-SAC achieves the best performance in 5 out of the 9 tasks (Ant-v4, HalfCheetah-v4, Hopper-v4, HumanoidStandup-v4, and Swimmer-v4), and our VDPO achieves the best performance in 3 out of the 9 tasks (Humanoid-v4, Pusher-v4 and Reacher-v4), showing a comparable performance with the SOTA baseline. \nWe believe our empirical results based on **common experiment setting and fair comparison** demonstrate that our VDPO achieves better sample efficiency than SOTA baselines and **does provide comparable performance** with the SOTAs.\n\n[1] Liotet, Pierre, et al. \"Delayed reinforcement learning by imitation.\", 2022.\n\n[2] Kim, Jangwon, et al. \"Belief projection-based reinforcement learning for environments with delayed feedback.\", 2023.\n\n[3] Wu, Qingyuan, et al. \"Boosting Reinforcement Learning with Strongly Delayed Feedback Through Auxiliary Short Delays.\", 2024.\n\n[4] Du, Jing, et al. \"Sensory manipulation as a countermeasure to robot teleoperation delays: system and evidence.\", 2024.\n\n[5] Penin, Luis F. \"Teleoperation with time delay-a survey and its use in space robotics.\", 2002.\n\n[6] Sheridan, Thomas B. \"Space teleoperation through time delay: Review and prognosis.\", 1993.\n\n[7] Liotet, Pierre, et al. \"Learning a belief representation for delayed reinforcement learning.\", 2021.\n\n[8] Xia, Bo, et al. \"DEER: A Delay-Resilient Framework for Reinforcement Learning with Variable Delays.\", 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E86Na3eAZD", "reviewer": "Reviewer_wuNr", "summary": "The paper proposes a new algorithm (VDPO) for delayed RL which first learns a reference policy in the delay-free MDP using standard RL tools and then uses behavior cloning to encourage visitation of optimal trajectories in the delayed MDP.\nThe paper shows theoretical guarantees on sample complexity and performance, and conducts an extensive empirical evaluation where VDPO improves sample-efficiency upon SOTA.", "questions": "None.", "limitations": "See weaknesses above.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The proposed algorithm is well-motivated under the inference interpretation of RL and can be expressed succinctly in a natural objective.\n* I find the approach of VDPO to address the sample complexity issue of baselines by first optimizing the reward in the delay-free MDP and then using behavior cloning very neat.\n* The paper derives guarantees on sample complexity and performance of VDPO which compare favorably to those of naïve approaches using augmented state spaces.\n* The experimental evaluation is extensive and shows a clear improvement on sample complexity.\n* I find that the manuscript reads very well and is nicely structured.", "weaknesses": "* It would be interesting to see an extension of this approach to stochastic delays.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the positive comments provided by Reviewer wuNr, and our responses are as follows.\n\n# Weakness 1: It would be interesting to see an extension of this approach to stochastic delays.\n\nWe appreciate that the importance of addressing stochastic delays is recognized. Though this paper mainly focuses on constant delays, we also conduct experiments to explore the robustness of VDPO under stochastic delays, as shown in Table 4. However, the neat theoretical results under constant delays no longer hold when the delays are stochastic. \nFor instance, when delays become stochastic, Lemma 3.1 does not hold as it is difficult to compare the performance between two stochastic delayed MDPs with different distributions of delays.\nIntuitively, learning in the delayed MDP with stochastic delays is challenging because the agent needs to learn the policy in the augmented state space with varying delays.\nTherefore, considerable effort is needed to derive theoretical results in this setting. \n\nWe believe a significant modification is needed to adapt VDPO to stochastic delays and we will address it in our future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XiWJLoUbK3", "reviewer": "Reviewer_io6Z", "summary": "The paper re-examines the delayed RL framework in terms of an equivalent variational inference problem. The authors theoretically motivate maximising the return of a reference policy that does not feature delays using an argument of improved sample complexity and performance that improves with smaller delays. They then derive  an objective for behavioural cloning to learn this desired policy. These two optimisation problems amount to a variational EM style algorithm on the ELBO. \n\nThe authors then provide a theoretical analysis of their approach, deriving SOTA sample complexity and performance bounds. The authors then introduce their VDPO algorithm to practically carry out the VEM optimisation before evaluating in several domains where it achieves SOTA sample complexity and similar performance to the SOTA algorithm AD-SAC.", "questions": "I have no questions about the work", "limitations": "There are no major limitations and all negative effects are addressed.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "**Presentation and Clarity**\n\nThe writing style is clear, scientific and presentation is excellent. The authors do a great job summarising related work, the theoretical framework (I particularly liked how Eq 3 is presented and referred back to) and explaining their theoretical results carefully. The material is not simple and could become unwieldily in the wrong hands, so its a credit to the authors that they can present their work this way.  \n\n**Contribution**\n\nThe paper combines both insightful theoretical results with a solid empirical evaluation, making valid contributions to both. Their algorithm should be a reference benchmark for delayed RL and I can imagine their research having high impact in the field.\n\n**Soundness**\n\nI've checked the maths and everything follows clearly to me.", "weaknesses": "There are no major weaknesses of the paper. \n\nThere are a few typos that I found:\n\nLine 74: $p\\rightarrow \\rho$", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We genuinely appreciate the positive feedback provided by Reviewer io6Z.\nThe typos will be corrected in the revised version.\nThis work aims to present a new delayed RL method from the perspective of variational inference which can effectively improve the sample complexity without compromising the performance, supported by comprehensive theoretical analysis.\nSpecifically, Equation 3 indeed plays a critical role in our framework. It bridges delayed RL and variational RL, which provides us the opportunity to use extensive optimization tools to solve the delayed RL problem effectively.\nFurthermore, Equation 3 inspires our delayed RL method, VDPO and its practical implementation.\nThe experimental results demonstrate that our VDPO can effectively improve the sample efficiency while achieving comparable performance with the SOTA baseline.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response
We sincerely appreciate the insightful comments and feedback from all the reviewers. 
The main contribution of this work is to address the sample efficiency issue in delayed RL by first introducing variational inference to reformulate the original problem and then solving the high-dimensional policy learning based on behaviour cloning. We are pleased that our approach, VDPO can effectively **improve the sample complexity without compromising the performance**, supported by both theoretical analysis and experimental results.

Regarding the main concern on inference time from Reviewer FFJG, we clarify that existing delayed RL works [1, 2, 3], including this paper, all use **sample efficiency and performance** as the evaluation metrics. The reason for not considering inference time is that commonly adopted neural architectures, e.g., MLP, transformer, etc., for policy representation share similar inference times, which are all significantly smaller (at least three orders of magnitude less) than one normal control cycle. Such inference times can be safely neglected considering the typical delay (multiple control cycles) in real-world applications. The detailed comparison between MLP inference time, transformer inference time, and the real-world delays can be found in answer to Weakness 2 by Reviewer FFJG. 
We appreciate Reviewer FFJG's comment and will add related discussion in the revised version to avoid any potential confusion.


[1] Liotet, Pierre, et al. "Delayed reinforcement learning by imitation." International Conference on Machine Learning. PMLR, 2022.

[2] Kim, Jangwon, et al. "Belief projection-based reinforcement learning for environments with delayed feedback." Advances in Neural Information Processing Systems 36 (2023): 678-696.

[3] Wu, Qingyuan, et al. "Boosting Reinforcement Learning with Strongly Delayed Feedback Through Auxiliary Short Delays." Forty-first International Conference on Machine Learning. PMLR, 2024.

Meta-review: This paper tackles the challenging problem of delayed observation and proposes Variational Delayed Policy Optimization (VDPO), which reformulates delayed RL as a variational inference problem. I think the idea is novel, and it receives the unanimously acceptance by the reviewers. The impressive part is that VDPO is able to improve the sample efficiency significantly with 50% less amount of samples and this is strong. I recommend the paper be accepted. I also recommend the authors to consider how to expand the existing Mujoco experiments to more realistic settings such as real robotics.
Decision: Accept (spotlight)