Addressing Signal Delay in Deep Reinforcement Learning
Authors: Wei Wang
Dongqi Han
Xufang Luo
Dongsheng Li
Venue: ICLR 2024 spotlight (2024)
Keywords: Deep Reinforcement Learning
Signal Delay
Robotic Control
Continuous Control
Primary area: reinforcement learning
TL;DR: This paper formalizes and addresses signal delay in deep reinforcement learning, introducing effective strategies that maintain high performance in robotic control tasks despite substantial delays.

Abstract: Despite the notable advancements in deep reinforcement learning (DRL) in recent years, a prevalent issue that is often overlooked is the impact of signal delay. Signal delay occurs when there is a lag between an agent's perception of the environment and its corresponding actions. In this paper, we first formalize delayed-observation Markov decision processes (DOMDP) by extending the standard MDP framework to incorporate signal delays. Next, we elucidate the challenges posed by the presence of signal delay in DRL, showing that trivial DRL algorithms and generic methods for partially observable tasks suffer greatly from delays. Lastly, we propose effective strategies to overcome these challenges. Our methods achieve remarkable performance in continuous robotic control tasks with large delays, yielding results comparable to those in non-delayed cases. Overall, our work contributes to a deeper understanding of DRL in the presence of signal delays and introduces novel approaches to address the associated challenges.

Published as a conference paper at ICLR 2024
ADDRESSING SIGNAL DELAY IN DEEP REINFORCEMENT LEARNING
Wei Wang' Dongqi Han Western University, Canada
Xufang Luo² Microsoff Research Asia
Dongsheng Li
wa waybaba22w@@gaiz2 com,
(donggihan, xufluo, dongsli) htw.sliimmmcrootttccoom com
ABSTRACT
Despite the notable advancements in deep reinforcement learning (DRL.) in re- cent years, a prevalent issue that is often overlooked is the impact of signal de- lay. Signal delay occurs when there is a lag between an agent's perception of the environment and its corresponding actions. this paper, we first formalize delayed-observation Markov decision processes (DOMDP) by extending the stan- dard MDP framework to incorporate signal delays. Next, we elucidate the chal- lenges posed by the presence of signal delay in DRL, showing that trivial DRL algorithms and generic methods for partially observable tasks suffer greatly from delays. Lastly, propose effective strategies overcome these challenges. Our methods achieve remarkable performance in continuous robotic control tasks with large delays, yielding results comparable those in non-delayed cases. Overall, our work contributes to a deeper understanding of DRL in the presence of signal delays and introduces novel approaches to address the associated challenges.
INTRODUCTION
Deep reinforcement learning (DRL.) and its applications have undergone rapid development in recent years (Sutton & Barto, 1998). The success DRL has been witnessed not only in virtual tasks like videos games (Vinyals et al., 2019) and simulated robotic environments (Haarnoja et al., 2018a), but also in many challenging real-world tasks such as controlling tokamaks (Degrave et al., 2022) and tuning language models with human feedback (Schulman et al., 2017; Brown et al., 2020).
However, an often existed problem has long been ignored deep RL studies, that is the delay of sig- nals, i.e., the agent may not immediately observe current environmental state, or the agent's action cannot immediately act on the environment. Signal delay exists widely in various practical appli- cations. For example, in autonomous vehicle navigation (lafariporniincaaii et al., 2022), delayed feedback can occur due to real-world constraints, such as network latency or sensor processing time. In finance (Fang et al., 2021), high-frequency trading algorithms may experience delays in receiv- ing information about market conditions due to network congestion or data processing bottlenecks. In robotics, the communication latency between robot's sensors, actuators, and control systems can lead to delayed responses (Abadfa et al., 2021). Additionally, in medical applications, such as telemedicine remote surgery (Meng al., 2004), there can be delay between receiving patient data and the actual execution of the required actions. Last but not least, sometimes even the delay short (c.g., ms due to neural network inference), the environment may have already changed a lot, c.g., when controlling a Tokamak (characteristic timescale of 0. ms) (Degrave et al., 2022). These delays pose a significant impact on the effectiveness of deep RL-based solutions, necessitating urgent research to address this challenge.
On the other hand, signal delay is also a critical issue in biological systems. For instance, neural signals in humans take approximately 1550 ms (Gerwig et al., 2005) to propagate from the brain to the museles (efferent delay) and from sensors to the brain (afferent delay). This delay can be significant in motor control (Bastian, 2006), considering that Usain Bolt moves for more than 10 meters and pianist plays dozens of notes every second. As DRL is well recognized as a biologically plausible
This work was done during Wei Wang's internship MSRA. Correspondence to: Dongqi Han (dongqi- han @ micrroooft.comm Dongsheng Li (dongsli@mierross) miiiiissttcoccoooooommmm.

---

Published as a conference paper at ICLR 2024
Exceution
Transmil-out
Environment
Ant-v4
Infervnce
AT
AT
Roward (Normelized)
HaltCheetah--v
AT
Hopper-v4
Walker2d-v4
Inference
Execution
Agent
ATR AT!
+Ar AT ATP
0.2
AT
AT
Perception
Trammit
Perception
Trammil-in
Delay
(b)
(c)
The process of inference incurs time costs (e.g.. human brain, GPU). Execution: Time lags occur when signals travel to their point of execution (e.g., hand, robotic arm). Perception: Processing raw signals involves processing time (e.g., from eye to brain, sensor to processor). Transmission: Implementing actions often entails transmission delays to the intended environment (c.g., online gaming, remote control scenarios). (c) Even few steps of delay significantly deteriorate the performance of SAC.
Figure I: (a-b) Examples and illustration of delays in real-world scenarios. Inference:
framework for modeling control and behavior (Botvinick et al., 2020), addressing signal delay in DRI. shed light on the corresponding neural mechanisms animals and humans (Stein, 2009).
However, given the wide existence of signal delay and its importance, there is surprisingly few studies try address this problem in DRL while there have been extensive studies on delayed feedback in control theory and bandit problem, the only study, to our knowledge, explicitly touched this problem deep reinforcement learning is from Chen et. al. (Chen et al., 2021). However, they make a strong assumption of knowing the reward function, which is often unknown in practice, and only discuss the case of fixed time of delay. The problem of signal delay in DRL in more general cases urges to be studied.
The current study aims to address the problem of deep RL with delay (DRLwD), particularly in continuous control tasks since most applications require continuous action space, such as robotics, autonomous driving, remote surgery, and modeling biological motor actions. The primary contribu- tions of this work summarized follows. First, investigate the impact of delay on perfor- mance and provide empirical evidence demonstrating its significant effect through comprehensive experiments. Next, we provide mathematical formulation for the delay problem, encompassing both action and observation delays, and derive theoretical insights to devise an effective solution. Then, building on our suggested insights, we examine a range of ideas to mitigate or overcome the key challenges and provide an empirical evaluation of the effectiveness of each idea. Finally, we suggest simple and general approaches for actor-critic critic architectures that effectively address the im- pacts of signal delay in DRL. Overall, our work contributes to a deeper understanding of DRL in presence of signal delays and presents novel model architecture to overcome the associated challenges.
PROBLEM DEFINITION
2.1 BACKGROUND: PARTIALLY OBSERVABLE MDP
A POMDP is defined as tuple (S, A. T, T,X.O.7). where S and A are the state and action spaces, respectively. Po specifies the initial state distribution such that Po(s) is the probability of state E S being an initial state. T specifies the state transition probability such that T(s', fr,rrss.o the probability of reaching to a new state E with an immediate reward r E R after taking an action A state E S. X denotes the observation space. O specifies the observation probability such that (z)ss) is the probability of an observation x E X at a state s S. S.y E 10, the discount factor.
2

---

Published as a conference paper at ICLR 2024
2.2 DEFINING DELAYED-OBSERVATION MDP
First we consider a standard MDP without delay (referred to as the original MDP), in which state is denoted by s. We extend the original MDP to incorporate observation delay (delay steps can be non-fixed). We define a Delayed observation Markov decision processes (DOMDP)', as a special case of POMDP, by considering its state o = (s(-2), s(-=)),
8(-1),s), where T is the max-
imum delay, and 8 is the state of the original MDP. Intuitively, s(-)) is the state of the original MDP from i steps ago, and the superscript (-t) indicates the relative timestep shift so as to main- tain the Markovian property. The transition probability function of a DOMDP is then defined as T(a', = To(s', ross",r(ssu a) II"-, I(s"(=0) = s(-(+1)), where I denotes the indicator function, and To(s', r|s, a) is the transition probability of the original MDP. The term I I(s"((=)) = s(-++)) transfers sf g(-++1) at current step to sf s(-t) at next step, therefore explaining s as the delayed state from steps ago, without the necessity of introducing absolute time step as in a time-dependent MDP (Boyan & Littman, 2000).
Then define the observation probability as O(ālo) s) s(T++)),
E (())((S = s(-1),, where is the observation (delayed state), and P(t) is the probability that the signal delays for t steps. A simple case is that P(t) = Il(t = AT), which means the signal delay is fixed as AT steps, and thus = s(-A7), So far, we have defined a DOMDP by specifying elements of a POMDP.
action delay
Idcal Execution
Agent
Agent
Ideaal Infermne
Eav
Env
Re
Ard
obs
Mc-an-a
Oc-ar
0c-1
A2"
(b)
(<)
Figure 2: (a) Illustration of action and observation shifting in DOMDP. All kinds of delays (b) Delays examples, Inference Delay: the model inference time cost, Observation Delay: time cost to transition observation to the agent. Action delay: time to excute action. (c) Equivalent to the single delays with the same total durat.
The complexity and diversity of environments can lead to various kinds of delays, such as inference delay AT" action delay AT", and observation delay AT as showed in Fig. 2(b). largely adding to the intricacy of the problem. Thankfully, all these types of delays are equivalent. Only the sum of them, denoted by AT, impacts the decision-making. This realization paves the way for a streamlined approach to modeling the problem. We proceed to establish this through the following theorem (Katsikopoulos & Engelbrecht, 2003):
Theorem 2.1 (Delay Equivalence). For an agent employing policy =<al-) at time step i, the impact of state delay A AT' and AT on the observation transition P( is equivalent. In other words, long as AT is the same, the agent perceives different variants of AT*, AT; and ATa values as identical. This conclusion hold for both the fixed and unfixed delay. (Proof in Appendix).
Consequently, focus the DOMDP in right side of Fig. 2b for the sake of notational simplicity and comprehension. Specifically, in Fig. 2, the system is modeled by an ideal agent with zero inference time and zero execution time. The only delay between the state of the environment and the decision-making input of the agent is AT = AT + AT + AT'. Therefore, we consider observation delay in this work without loss of generality.
Note that Chen et al. (2021) has defined awlay- MDP. However, they assume the delay is fixed. By contrast, our definition considers more general case where the delay could he variant and probabilistic.
3

---

Published as a conference paper at ICLR 2024
3 CHALLENGE OF DRL WITH DELAYED SIGNALS
Standard DRL and general POMDP algorithms exhibit catastrophic failure with signal delay Now we empirically examine the performance of existing RL algorithms straightforwardly applying to environments subjected to various delays on four MuloCo environments. The algorithms include deep deterministic policy gradient (DDPG) (Lillicrap et al., 2015), twin- sin-deeayed DDPG (TD3) (Fu- jimoto et al., 2018), soft actor-criti (Haarnoja et al., 2018b) and RNN- -based approach for general POMDP tasks (RNN Strong) (Ni et al., 2022). Fig. shows the normalized results of the exper- iments, in which the reported scores proportionate environment of SAC without delay. It reveals that even a modest delay of one environment step substantially undermines the performance of the all these algorithms. Delays surpassing 4 steps lead to catastrophic failure. These results underscore the considerable adverse impact delays can have on the efficacy of DRL algorithms when being directly applied. In the follows, we try figure out where the difficulties of DRLwD stem from.
the maximum attainable score for each
RNN Strong
Vanilla DDPG
Vanilia SAC
Vanilla TD3
SAC (No delay)
Ours
7000
7000
1200
1000
1200
1000
1000
1000
97.4%
3200
1000
1000
3000
Stop
Step
Step
Step
(a) Delay
(b) 0 Delay 4
(c) Delay 8
(d) Delay 12
Figure 3: Task-averaged performance in MuJoCo environments (Ant-v4, HallCheetah-v4, Hopper- v4, Walker2d-v4) under various delay steps for state-of-the-ard algorithms. SAC (No delay) acts as the benchmark for optimal performance. The red dashed line and shaded area represent the mean and standard deviation of our method's 's rest-performing variants, serving as benchmark.
Misalignment between observed and true state Fig. 4 illustrates that when observation signal delays occur, the agent's actual state can differ from its observed state, leading to varying optimal actions. Addressing this mis- alignment between the current observation (delayed sig- nal) and true state (non-delayed) is a central challenge in DRLwD for optimal decisions. One straightforward solu-
8t
Current State
tion is to estimate the true state based on available infor- mation, such as the delayed signal and previous actions. This would transform the problem to standard MDP task using the estimated true state. However, accurate estima- tion be difficult due several factors, including plex state transitions, stochastic environmental state tran- sitions, and data distribution shifts (Pan & Yang, 2010) during the process of online RL as the agent improves its policy (Quinonero-Candlla needed to address the problem. The following section will discuss various methods, including our proposed model, and empirically evaluate their effectiveness handling the delay issue.
8t-AT Delayed Signal
Varying optimal actions be- tween actual state and observed state.
Figure 4:
al., 2008). In light of these challenges, alternative approaches
4 DEVELOPING ALGORITHMIC FRAMEWORK FOR DOMDP
Building on the definition and discussion presented earlier, goal is to develop algorithm that effectively addresses the DOMDP problem. However, our objective extends beyond merely creating standalone algorithm specifically for DOMDP, Instead, aim to explore versatile framework and components that can be integrated with current or future algorithms to handle DOMDP problems effectively. This approach will lay foundation that facilitates the seamless integration of future
4

---

Published as a conference paper at ICLR 2024
developments in reinforcement learning with existing strategies. Considering the current state-of- the-art actor-critic methods has demonstrated success in continuous control tasks (Lillicrap et al,, 2015; Haarnoja et al., 201 8a; Fujimoto et al., 2018), we ground our work actor-critic architecture.
The rest of this section is organized as follows: We first investigate the input design of the critic (Sec. 4.1) and the actor (Sec. 4.2). Subsequently, explore other potential training techniques that show promise in improving performance under DOMDP scenarios (Sec. 4.3).
Critic
8t
A4-1
04
8g
Time Stop
Actor
MLP
Deooder
Decoder
Decoder
Deooder
Defash
Reconcliing Precess
A--di-1
4t A7
Enceder
Ennoder
Encodor
R
O1-3
Actor
Actor
Memory
Onlse Offine
Butter
Transtormar
Prediction
Prediction
Encoding
Encodingt
(b)
Figure 5: Diagrams of the Proposed Methods: (a) Delay-Reconciee Training for Critic: Utilizes only the online side during the deployment stage. The critic would use non-delayed information after recoiling offline stage. (b) State Augmentation for Actor: Considers three methods for input 8, with historical information MLP, RNN, and Transformer. For MIP, historical actions 0r-A7:1-1 are concatenated with the state dimension. In RNN, the input consists the current state and last- step action. In the case of the Transformer, the input includes state and action {ss, h- information spanning AT steps. (c) Complementary Techniques for DOMDP Resolution: Employs prediction/encoding auxiliary loss to aid in representation learning. An encoder first encodes the 3, into the hidden feature 2,, and then a decoder decodes it to $t. Prediction refers to using the predicted results & the input of the actor, while the Encoding method utilizes features 2, in the hidden space. For both methods, two variants are considered; Detach indicates stop of gradients backpropagation (with the notation of f).
4.1 PPLAY-RECONCIA TRAINING FOR CRITIC
Estimating value functions is the most basic requirement for actor-critic methods. With time-shifted observations, a question naturally arises: how can we accurately estimate the state value, i.e., train the critic, under such circumstances? Contrary to standard POMDP problems, crucial insight for DOMDPs that full observation and reward available during offline training once the delay is reconciled following real-time inference. In addition, as depicted in Fig. 5a, the forward pass of the critic is not required during the inference stage when interacting with the environment.
These facts motivate to design post-processing recovery pipeline that first recovers the histor- ical information without delay, and then uses it for critic training. For instance, in delay-affected scenarios such online gaming trading, trajectory without delay. While this information is not available during inference, it can still be employed for critic training, thanks to the actor-critic structure. Similar techniques can be found in
time-calibrate the historical data recover the
Although previous studies, such as those by Vinyals et al. (2017); Baisero et al. (2022); Foerster al. (2018), have demonstrated the efficacy of asymmetric observation actor-critic paradigm. Research also indicates that an asymmetric input design may lead to non-synchronous problem and gradient bias, as discussed by Baisero & Amato (2021). To further explore this issue, we include a comparison with a symmetric design in our study. In this alternative approach, the critic receives the same input as the actor, named with "Symmetric" as the prefix the following context.
4.2 STATE AUGMENTATION FOR ACTOR
In the preceding section, introduced historical recovery pipeline for critic training. A similar approach, however, is not applicable to the actor, as the oracle s, is unavailable during the inference stage. This section explores potential ways augment the actor's input facilitate training.
5

---

Published as a conference paper at ICLR 2024
Reviewing established POMDP algorithms, common strategy, given the actor's inability to access the oracle state s, during inference, is to provide the actor with supplementary imput to aid in the recovery of <c. These inputs typically consist of historical environmental observations such as op- ponents' behavior. However, it's evident that the historical state 8<x is not informative in DOMDPs, as [ 8,) 0, assuming Markovian environment, where ((; denotes mutual information.
So, aside from the historical state, what other information could aid in recovering the oracle state s;? Intuitively, in DOMDP, the oracle state 8, is beyond 5z, the most significant transition influence is likely the actions taken during 8 and 8t, denoted 0r- AT:-- The following theorem (Katsikopoulos & Engelbrecht, 2003) proves the necessity of incorporating aT- into the agent's s input.
Theorem 4.1 (Markovian Property). Integrating historical actions act into the observation trans- forms it into MDP with state transition probability (85 0e), where = (5t, At-AT:t-1).
The proof is deferred to Appendix D.2. The theoretical analysis suggests that in DOMDPs, the acle state Stg is independent of &tct, given St,, i.e., Stp 1 Secto (tg Additionally, it discloses a strong association between the oracle state 8, and historical action a AT--- In conclusion, incorpo- rating historical actions into the observation allows us to convert a DOMDP to a MDP, which is a fundamental requirement for the convergence of many methods.
Next, we investigate two strategies to incorporate historical information into the actor, as in Fig. 5b: 1) MLP Encoder: Concatenating historical action data aT-AT;: with &to 2) RNN Encoder: Adding only the last action a;-1, more efficient for handling large action spaces or vast histories.
4.3 INVESTIGATION OF COMPLEMENTARY TECHNIQUES FOR DOMDP SOLUTION
Previous research (Igl et al., 2018; Subramanian et al., 2022; Lambrechts et al., 2023) shows that auxiliary loss can improve RL agent performance. Expanding on this and the Delay-Reconciled Training for Critic, our work introduces techniques beyond input design to enhance our framework. We aim enable the actor "imagining" the true state s; from delayed observations &, and actions a- Currently, the actor's learning relies on the critic's 's value estimations. To improve this, we propose two strategies using oracle observations for additional supervision.
Prediction Refer to Fig. 5c left side, prediction network is trained, using 5n as input and š, as output. The loss between the predicted state & and the oracle state 8t is minimized. The prediction results are then utilized as actor input. Two variants were tested: one detaches (indicated with f) the output of the observation prediction network before inputting to the policy network ot (r)), and the other does not. Detaching the variables can stabilize the interaction between prediction and policy networks while maintaining the connection potentially enhances supervision learning.
Encoding As depicted in Fig. Sc Right, prediction network is trained to generate hidden features 2t- These features, εg, serve as the input to the policy network (-)) This approach may be particularly effective when the original observation is hard to accurately predict, or when information is sparse.
EXPERIMENTAL RESULTS
We performed our experimental evaluations across MuJoCo environments (Todorov et al., 2012) with signal delay. We compare to popular RL algorithms for continuous including DDPG (Lillicrap et al., 2015), TD3 (Fujimoto et al., 2018), SAC (Haarnoja et al., 201 (0);;;; and DRL. algorithms for POMDP including RNN-based (Ni al., 2022) and belief-state-
(Han et al., 2020) (VRM); as well as DATS (Chen et al., 2021), model-based algorithm for DRLwD (Tab. 1). We examine four distinct environment settings: (1) Fixed Delay, (2) Unfixed Delay, (3) Probabilistic State Transition (4) Large State Space. (1-3) include Ant-v4, Walker2d-v4, 44 and HalfCheetah-vé:
Implementation Details are provided in the appendix, covering the environment setup (Sec. E.1), implementation of baselines from other research (Sec. E.2), our network architecture (Sec. E.3), hyperparameter selection (Sec. E.4), implementation of delayed environments (Sec. E.5), our code framework (Sec. E.6), and additional details (Sec. F.7).

---

Published as a conference paper at ICLR 2024
5.1 EMPIRICAL EVALUATION OF DESIGNS
Delay-Reconciled Training for Critic The learning curve associated with the traditional method exhibits a plateau (Fig. 3), indicating that the learning process is not effectively progressing. How- ever, when we apply the new design for the critic, the learning curves for both fixed (Fig. 6c) and unfixed delay scenarios (Fig. 6е) display a more dynamic and promising trend. This indicates that our delay-Reconeiled Training design contributes to a more progressive learning process. Further- more, as demonstrated in Fig. 6a,b, the performance of Delay-Reconcille consistently surpasses that of the Vanilla SAC when delay is introduced, and this performance gap widens as the delay increases. This result underscores the robustness of our design for the critic.
Training for the Critic
State Augmentation for Actor The aforementioned results demonstrate the effectiveness of our critic input design. Building upon this, we further apply state augmentation for the actor, based on Training for Critic. As shown in Fig. 6c,e, the learning curve of the model
the Delay-Reconcin Gelay-Reconciled Training for Critic plateaus when the delay exceeds 8 steps. trained solely with Delay- However, after incorporating state augmentation for the actor, the learning curve becomes more positive, depicted in Fig. 6d.f. Moreover, the performance of the state augmentation for the actor consistently exceeds that of the model trained solely with Pelay-Reconciled Training for Critic (Fig. 6a.b). As shown in Tab. the final average improvement is 1.2% 50.8% 75.9% fixed delay and 8.4% + 48. 1% - 77.0% on unfixed delay. These results suggest that both techniques contribute consistent performance gain and can be combined for further enhancement.
Complementary Techniques. As evident from Tab. 1, the most effective performance for each de- lay predominantly arises from algorithms employing Prediction and Encoding techniques. Specif- ically, when the delay is fixed and the delay step exceeds 1, the top-performing algorithms con- sistently involve Encodingt and Prediction On average, Encoding and Prediction 1 enhance the performance of State Augmentation for Actor from 75.9% to 84.5% and 83.6%, respectively. How- ever, the efficacy of explicit supervision significantly diminishes when the delay is unfixed. The average performance of Encoding! (77.9% approximates that of State Augmentation (77.0%) No- tably, the performance of Prediction (72.5%) even drops by 4.5% compared to State Augmentation. This observation underscores that the explicit introduction of Prediction doesn't 't always guarantee performance improvement. Whether the delay is fixed substantially influences the effectiveness of techniques introducing additional prediction supervision.
(a)
(b)
(c)
7300
Dekay
Daler Resomole Catle
6300
4300
3300
Celor
Deloce
Step
(d) )roce
(e)
7000
Detay
5008
0000
0300
$800
Raward
Rewardd
2008
3000
3300
2006
2300
0.0
105
Figure 6: Consistent performance improvement critic and actor input design. (a) Fixed Delay (b) Unfixed Delay (c) Reconceecde Training for Critic with fixed delay (d) Delay- Delay-Reconciled Training for Critic with fixed delay (c) Actor State Augmentation- MLP with unfixed delay (f) Actor State Augmentation- MLP with unfixed delay. Rewards are averaged over four MuJoCo tasks.
7

---

Published as a conference paper at ICLR 2024
Table 1: Performance (%) of algorithms in fixed and unfixed delay environments. The shaded values represent results from unfixed delay environments. All values are normalized based on the environment-speciin
performance of Vanilla SAC with no delay". Data are presented as mean + standard error of the mean (S.E.M) The best performing methods, including those within the range of S.E.M of the best, are highlighted in bold. Here, 8, denotes the non-delayed state, & = 8T-A7 represents the delayed state, and 8n- (5e, (c-AT:t-) signifies the state with historical actions. & is the predicted state, while 24 is the hidden feature of the prediction network. In. and Net. denote the input and network of the critic/actor, respectively.
Cftie Net.
Acter Nec.
Delayed Time Steps
Mothed
AVe. F24
Exising Methoch
Vanilla DOPG (1 (Lilleraa el,, 2015)
MLP
MLP
112.2
49.5
18.2
13.3
110.1
15.7
16.0
5.9
112.0
2.3
12.
-13
10.5
18.3
11.
22
11.9
-12
11.0
-15
11.3
-1.
11.3
-1.
132
26
Vanilla (Pujinooo 2015)
MLP
36.5
416
6.6
8.1
73
68
10.5
Vasilla SAC (Haamejj et 20180)
%
MLP
MLP
108.0"
+9.4
+12.3
40.3
+10.5
52.5
20.5
+5.4
=15.2
64.5
83.0
533
+7.5
18.1
+3.3
-1.0
83.
5.0
+1.1
-0.7
+2.0
2.2
=2.3
1.2
+4.3
84
RNN Suoee Brscine (Si 2022)
RNN
RNN
+12.1
52.9
$7.1
18.9
2:5.2
34.5
1:577
32.4
+:411
24.7
15.1
34.5
16.1
16.1
11.
44.5
13.6
21.7
VRM (Elan all, 2000)
%
RNN
RNN
91.1
16.7
+: 12.3
66.8
1.207
71.5
1:7.6
54.9
8:1.9
60.3
13.2
12.0
8:16.
41.4
2:5.1
21.4
70.7
11.00
8: 4.2
14.79
82.0
16.4
26.4
12.7
19.5
701
DATS (Chen al, 2001)
A
RNN
RNN
46.7
73.
113.
7119
112.7
60.7
19.7
55.0
1.8.2
51.3
1.8.1
34.7
22.9
28.6
4.2
19.1
18:3
10.
25.8
C.Rc Iapsr
Delay Rooonsiled Trining
MLP
MLP
25.0
99.2
210.0
92.9
29.5
M
#10.4
14.5
=10.3
15.0
=11.1
75.5
=13.7
65.1
$9.7
66.7
=10.5
16.9
29.1
97.1
20.3
11.
290
90.8
=103
45.1
Actoc Input Design
Swen Argrantztionn MLP
MI P
MIPP
111.2
100.
92.8
27.6
26.6
96.6
15.1
813.8
102.1
19.2
19.6
85.0
19.9
19.9
54.0
64.7
18.6
18.4
65.5
19.2
75.4
19.3
77.0
State Argmerzaiion RMN
MLP
RMN
111.5
84.4
110.4
71.0
112.1
72.3
113.0
66.6
60.4
111.3
110.2
37.5
110.
43.7
24.6
13.2
22.4
14.8
21.3
15.1
15.0
27.8
17.5
27.0
16%
Aegmentationn
Trandommer
MLP
Tron.
+12.2
412.3
76.9
10.5
57.2
4-7.6
46.3
32.0
1.5
19.5
35.1
25.0
Exploring Esterdal Techeriqaass
Prodiction
MLP
96.1
85.6
21.0
73,0
58.1
73.5
Eranding!
MLP
MLP
612.1
97.6
101.4
49.0
113.0
49.6
$10.0
103.2
92.0
49.5
90.4
49.5
86.6
+8.2
+12.4
89.9
+10.8
77.9
73.3
19.9
49.7
60.2
+10.6
$4.5
71.9
+9.6
Synmeeie MLP
MLP
MLP
106.3
+9.4
29.1
75.1
38.9
72.5
63.7
45.7
61.2
+9.0
-11.7
$1.4
+9.6
$2.0
+13.
$7.5
27.0
31.0
+120
20.0
25.0
28.3
+123
38,6
+7.
3F.1
5.2 PERFORMANCE ACROSS DIVERSE ENVIRONMENTS
Performance on Basic Environments. As shown in Tab. 1, DDPG, TD3, and SAC experience a significant performance drop of over 79.6% when the 2 exceeds four time steps. Even a single-step delay can cause DDPG's performance to plummet from 49.6% to 13.3% (fixed delay) and 15.7% (unfixed delay). TD3 and SAC exhibit smaller yet substantial performance reduction, greater than 29.6%. While the RNN Strong (Ni et al., 2022) and VRM (Han et al., 2020) can somewhat func- tion in POMDP and enhance SAC's performance, the improvement is relatively mild. Our simple ReconnReee Training for Critic can considerably boost SAC's performance, consistently out- performing RNN Strong. The actor design also contributes to consistent performance improvement. For the techniques, we observe that the detached versions always outperform their non-detached counterparts; due space constraints, we defer these results Tab. 4 in the Appendix. For the sym- metric design of the actor and critic which both use 5, as input. The results reveal that this symmetric design (with an average performance of 37.1%) significantly underperforms compared to the asym- metric design with delay-reconcill critic. Additional investigations delve into the possible reasons of the baseline failures are provided, including issues related to representation learning (Sec. F.1) and redundant historical information (Sec. F.2). The learning curve can be found in Sec. F.7.
Fixed v.s. Unfixed delay As shown in Tab. 1, the stability of the delay most significantly influ- ences prediction and encoding-based methods. Under fixed delay settings, prediction- based meth- ods maintain 79% performance even with a 12-step delay, outperforming the second-best Actor State Augmentation-MLP at 01%.1o However, in an unfixed delay setting, the performance of predic- tion drops to 59%. falling below that of Actor State Augmentation-MIL explicitly introducing prediction supervision can have a detrimental effect performance, influ- enced by whether the delay is fixed. This aligns with the intuition that future states become more unpredictable when delay steps vary. Additional results can be found in Sec. F.3 in the appendix.
(79%). This implies that
8

---

Published as a conference paper at ICLR 2024
Performance under Large Observation Space and Probabilistic Environments. The size of the observation space impacts the performance of algorithms that explicitly introduce supervised learning supervision. As depicted in Fig. 7a-d and Fig. 7f, the performance of all algorithms drops significantly as the number of delay steps increases. Meanwhile, the performance of the Actor State Augmentation- ML. is less severely affected (Fig. 7e), experiencing only around 20% performance decrease when the number of delay steps reaches 12. Interestingly, when the prediction network is detached (Fig. 7b,d), the performance deteriorates even more drastically. This could possibly be attributed to the complexity of predicting true observations, leading the prediction network to gener- ate less useful information for the policy network. Additional insights and analyses are provided in Sec. F,4 of the appendix. Probabilistic environments could also contribute to the inder-performance of prediction methods, with more detailed findings available in Sec. F.5 and Sec. F.6.
(a)
(b)
6603
(c)
7890
7683
12
5003
2003
0.4
0.4
0.8
0.8
Stop
Stop
Step
(c)
(e)
(f)
Detay
1006
Daiyy
68:00
6000
50.00
4000
4000
2000
3000
2000
0.2
1.0
Step
Shep
166
Figure 7: Performance when observation space is large. (a) Prediction (b) Prediction (c) Encod-
ing (d) Encoding! (e) Actor State Augmentation- MLP (f) Performance summary
5.3 TAKE-AWAY MESSAGE
We now summarize the main findings of our experiments: (1) The time recovery process presents consistent and impactful enhancement to the performance of the Actor-Critic framework in DOMDP (Tab. 1). (2) Augmenting the observation to can also consistently contribute positively towards overall performance (Tab. 1). (3) Introducing a prediction network (detached) can yield promising performance when the environment is simple (fixed part in Tab. 1). (4) Explicit prediction model- based methods can lead to performance decrease (Tab. 1.Fig. 7) in scenarios where the environment highly complex (large observation spaces Fig. 7) and difficult model (probabilistic- Fig. 7).
CONCLUSION
This work aims address signal delay, a commonly overlooked issue in DRL (see Appendix B for a discussion about related works). We formulate DRLwD and propose effective approaches to alleviate the challenge of misalignment between observed and true states. Our research has certain limitations due to its focus on simulated robotic control environments. Dealing with signal delay in real-world applications could potentially encounter greater levels of uncertainty and additional challenges. Future work should explore real-world scenarios and transition from simulated to real environments (James et al., 2017; Kadian et al., 2020), for more discussions see Sec. C in Appendix.
LEGNONTES
This work was supported by Microsoft Research. The authors thank Yifei Shen for insightful con- tributions to the discussions that greatly enhanced this research.
9

---

Published as a conference paper at ICLR 2024
REFERENCES
Ignacio Abadía, Francisco Naveros, Eduardo Ros, Richard R Carrillo, and Niceto R Luque. A cerebellar- solution to the nondeterministie time delay problem in robotic control. Science Robotics, 6(58)xcab12756, 2021.
Andrea Baisero and Christopher Amato. Unbiased asymmetric reinforcement learning under partial observability, arXiv preprint arXiv:2105.11674, 2021.
Andrea Baisero, Brett Daley, and Christopher Amato. Asymmetric dqn for partially observable reinforcement learning. In Uncertainty in Artificial Intelligence, pp. 107-117. PMLR, 2022.
James L Bander and CC White. Markov decision processes with coise-corrupted and delayed state observations. Journal of the Operational Research Society, 50:660-668, 1999.
Amy J Bastian. Learning predict the future: the cerebellum adapts feedforward movement control.
Current opinion in neurobiology. 16(6):645-649, 2006.
Jose Blanchet, Renyuan Xu, and Zhengyuan Zhou. adlayvad learning in generalized linear contextual bandits. Mathematics of Operations Research, 2023.
Matthew Botvinick, Jane X Wang, Will Dabney, Kevin Miller, and Zeb Kurth-Nelson. Deep reinforcement learning and its neuroscientific implications. Neuron, 107(4))603-616, 2020.
Justin Boyan and Michael Littman. Exact solutions to time-dependent mdps. Advances in Neural Information Processing Systems, 13, 2000.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 2016.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 200.
Baiming Chen, Mengdi Xu, Liang Li, and Ding Zhao Delay-aware model-based reinforcement learning for continuous control. Neurocomputing, 450: 2021.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Ben- gio. A recurrent latent variable model for sequential data. In Advances in Neural Information Processing Systems, pp. 2980-2988, 2015.
Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Mag- netic control of Tokamak plasmas through deep reinforcement learning. Nature, 602(7897):414- 419, 2022.
Yuchen Fang, Kan Ren, Weiging Liu, Dong Zhou, Weinan Zhang, Jiang Bian, Yong Yu, and Tie- Yan Liu. Universal trading for order execution with oracle policy distillation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 107-115, 2021.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning, pp. 1587-1596. PMLR, 2018.
Mareus Gerwig, Karim Hajjar, Albena Dimitrova, Matthias Maschke, Florian Kolb, Markus Frings, Alfred F Thilmann, Michael Forsting, Hans Christoph Diener, and Dagmar Timmann. Timing of conditioned eyeblink responses impaired in cerebellar patients. Journal of Neuro- science, (5(15):3919-3931 2005.
Tuomas Haarojja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- ence on Machine Learning, pp. 1856-1865, 2018a.
10

---

Published as a conference paper at ICLR 2024
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv: arXiv:1812.05905. 2018b.
Dongqi Han, Kenji Doya, and Jun Tani. Variational recurrent models for solving partially observable control tasks. In International Conference Learning Representations, 2020.
Maximilian Igl, Luisa Zintgraf, Tuan Anh Le, Frank Wood, and Shimon Whiteson. Deep variational reinforcement learning for pomdps. In International Conference on Machine Learning, pp. 2117- 2126. PMLR, 2018.
Max Jaderberg, Wojciech Czarnecki, lain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, et al. Human- level performance in 3D multiplayer games with population- based reinforcement learning. Sci- ence, 364(6443):655----- 865, 2019.
Ammar Jafaripournimchahi Yingfeng Cai, Hai Wang, Lu Sun, and Biao Yang. Stability analysis of delayed-feedbace control effect in the continuum traffic flow of autonomous vehicles without v2i communication. Physica A: Statistical Mechanics and its Applications, 105:127975, 2022.
Stephen James, Andrew J Davison, and Edward Johns. Transferring end-to-end visuomotor control from simulation real world for multi-stage task. In Conference on Robot Learning, pp. 334- 343. PMLR, 2017.
Abhishek Kadian, Joanne Truong, Aaron Gokaslan, Alexander Clegg, Erik Wijmans, Stefan Lee, Manolis Savva, Sonia Chernova, and Dhruv Batra. Sim2real predictivity: Does evaluation in simulation predict real-world performance? IEEE Robotics and Automation Letters, 5(4):6670- 6677, 2020.
Konstantinos V Katsikopoulos and Sascha E Engelbrecht. Markov decision processes with delays and asynchronous cost collection. IEEE transactions on automatic control, 48(4))568-57) 2003.
Jangwon Kim, Hangyeol Kim, Jiwook Kang, Jongchan Back, and Soohee Han. Belief projection- based reinforcement learning for environments with delayed feedback. Advances in Neural Infor- mation Processing Systems, 36, 2023.
Gaspard Lambrechts, Adrien Bolland, and Damien Ernst. Informed pomdp: Leveraging additional information in model-based arXiv preprint arXiv:2306. / 1488, 2023.
Bingcong Li, Tianyi Chen, and Georgios B Giannakis. Bandit online learning with unknown delays. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 993-1002.
PMLR, 2019.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint 15090029999 2015.
Cai Meng, Tianmiao Wang, Wusheng Chou, Sheng Luan, Yuru Zhang, and Zengmin Tian. Remote surgery case: robot-assisted teleneurosurgery. In IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA' '04. 2004, volume pp. 819-823. IEEE, 2004.
Tianwei Ni, Benjamin Eysenbach, and Ruslan Salakhutdinov. Recurrent model-free RL can be strong baseline for many POMDPs. In International Conference on Machine Learning (ICML), pp. 16691-16723, PMLR, 2022.
Sinno Jialin Pan and Qiang Yang. A survey transfer learning. IEEE Transactions knowledge and data engineering, 22110):1355-135 2010.
Christos H Papadimitriou and John N Tsitsiklis. The complexity of Markov decision processes. Mathematics of Operations Research, 12(3):441-455 1987.
Joaquin Quinonero-Candela. Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset shift in machine learning. Mit Press, 2008.

---

Published as a conference paper at ICLR 2024
J Schmidhuber. Making the world differentiable: On using fully recurrent self-supervised neural net- works for dynamic reinforcement learning and plamning in non-stationary environments. Institur für Informatik, Technische Universität München. Technical Report FKI-126, 90, 1990.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
John Stein. Cerebellar forward models to control movement. The Journal of physiology, 587 2): 299, 2009.
Jayakumar Subramanian, Amit Sinha, Raihan Seraj, and Aditya Mahajan. Approximate information state for approximate planning and reinforcement learning in partially observed systems. The Journal of Machine Learning Research, 23(1) 538555883-56, 2022.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 TEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEFE, 2012. doi: 10010/00..2002.633888
Arun Verma, Zhongxiang Dai, and Bryan Kian Hsiang Low. Bayesian optimization under stochastic delayed feedback. In International Conference Machine Learning, pp. 2022.
PMLR,
Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, et al. Starcraft ii: A new challenge for reinforcement learning. arXiv preprint
2017.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czamecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350-33) 2019.
Nikos Vlassis, Michael L Littman, and David Barber. On the computational complexity of stochastic controller optimization in POMDPs. ACM Transactions on Computation Theory (TOCT), 4(4): 1-8, 2012.
Thomas Walsh, Ali Nouri, Lihong Li, and Michael L. Littman. Learning and planning in environ- ments with delayed feedback. Autonomous Agents and Multi-Agent Systems, 18:83-105, 2009.
Yuanyu Wan, Wei-Wei Tu, and Lijun Zhang. Online strongly convex optimization with unknown delays. Machine Learning, 111(3)877--99 2022.
12

---

Published as a conference paper at ICLR 2024
A ETHICAL STATEMENT AND BROADER IMPACT
This work addresses signal delay in DRI. with a commitment to transparency, reproducibility, and responsible application of our findings. The potential for misuse of our methods in harmful applica- tions is acknowledged, and we advocate for their use within ethical and legal guidelines.
Our work has potential for substantial impact, enhancing efficiency and safety of autonomous sys- tems reliant on DRL. However, the possibility of misuse in ethically ambiguous applications exists. Our standardization of DOMDP could guide future DRL research, but diverse focus within the field necessary to avoid research biases. We encourage the use of this work responsibly, considering its ethical implications.
B RELATED WORK
Signal delay in decision-making Several previous studies have dealt with signal delay in MDPs (Bander & White, 1999; Katsikopoulos & Engellhrecht, 2003; Walsh et al., 2009), however, their results have not been extended to deep RL. Chen et al. (2021) is the only work we found explicitly discuss DRLwD. There is also related work on traditional control and bandit problems with feedback delay, Recent works like Blanchet et al. (2023) and Wan et al. (2022) have tailored upper confidence bounds and Thompson sampling algorithms handle unknown and stochastic delays generalized linear contextual bandits. Li et al. (2019) explored the complexity of non-stochastic bandit and bandit convex optimization settings with unknown delays. Verma et al. (2022) provided solutions for stochastic delayed feedback in Bayesian optimization, presenting novel algorithms with sub-linear regret guarantees for efficient new function query selection amidst random delays. A concurrent study (Kim et al., 2023) proposes actor-critic algorithm which shows great perfor- mance gain in DRLwD by compressing the dimension of the augmented state. Our work considers architecture design for DRLwD, which is orthogonal these studies.
POMDP methods Theoretical complexity analysis reveals that solving POMDPs can generally be both statistically and computationally challenging (Papadimitriou & Tsitsiklis, 1987; Vlassis et al., 2012). A common solution for POMDP is to model the function with additional context by em- ploying RNNs, which used to distill the characteristics of prior observations and actions into the states of the RNN (Schmidhuber, 1990; Jaderberg et al., 2019; Vinyals et al., 2019). While a DOMDP is an instance of the broader POMDPs, have shown that by incorporating contextual actions into observation turns DOMDP to MDP problem. Nonetheless, we examined the perfor- mance of an -fate-of-the-ar RNN-based POMDP algorithm that performs good for many partially observable tasks (Ni et al., 2022), which is shown to be not suitable for DRLwD.
C FUTURE WORK AND DISCUSSIONS
In this section, we focus on the potential integration of timestamp data into our model, concept that presents promising avenues for future research but also comes with its own set of challenges.
C.I ENIIANCING MODEL ROBUSTNESS WITH TIMESTAMP DATA
The integration of timestamp data into our model is primarily aimed at optimizing performance in environments with variable delays and managing asynchronous data from multiple sources. This approach is particularly relevant when the historical number H aligns with the delay AT, as it can significantly improve the model's adaptability to changing delay patterns and enhance decision- making in systems with multiple, asynchronous data streams.
However, it's important to note that the effectiveness of this method may vary depending the specific environment. For instance, certain sensors might not support timestamping, or network messages may lack time-related data. This makes the trick not a general solution for all cases
These challenges are the primary reason why we have earmarked the integration of timestamp data subject for future work rather than including it in the main body of current paper. Future research will need to address these environmental variabilities and develop adaptable models that can effectively leverage timestamp data where available.
13

---

Published as a conference paper at ICLR 2024
C.2 BROADENING THE APPLICATION BEYOND THE ACTOR-CRITIC FRAMEWORK
The issue of delayed signal can arise in a broad range of applications. Hence, it crucial to develop methodologies that sufficiently versatile to integrated into various frameworks. In this project, we introduce general approach that can be seamlessly incorporated into any actor-critic framework. While some algorithms do not adhere the actor-critic framework. Expanding our methods to be entirely algorithm- and model-agnostic would significantly enhance their applicability and utility in a wider array of RI. contexts.
D THEOREM PROOF
D.1 THEOREM 2.1 DERIVATION
Proof. Let action delay be denoted as A and observation delay as AT We denote the actual action acting on the environment at time step a,, and the action given by the agent step as ā. With action delay AT", we have included in either of the action and observation delays. From the agent's view, the state transition probability under policy π (al-) with delay can be written as:
âT- The inference delay AT can be considered
(882-1188)
AT mm
Σ
=
AT m
AT*=0" 0e GT- AT
ATM an
ATM ne
P(AT" A
AT O
Here, P(AT') and P(AT") denote the probability of state and action signals delay for certain steps. It -A-TT=-A denotes the information available to the agent for selecting action ā AT -AT which until time step / A AT
Since the state transition probability function P(s++1-AT- AT-AT*, āt AT -TT- is Markovian (time-independent), have
((5)+118:)
AT
ATH
[ PAA...rrPPlsssoffaaarrrr.r...//
Σ
Σ
AT 0 AT
AT mar
+A7 mas
Σ
P(AT t
AAT
From the outcome, observe that the delay between decision-making and environmental response A and the delay between environmental output and decision makingg AT have the same effect on state transition, and what matters is merely A + a Therefore, can simply model the MDP with delay problem with only observation delay AT = AT + A as in DOMDP.
D.2 THEOREM 4.1 DERIVATION
Proof. To show the Markovian property when considering the augmented state (AT is max delay) (5t, 04-A4-11);
14

---

Published as a conference paper at ICLR 2024
need to show the transition probability function P(5 (1:4111;;000 (c: is independent of the time step t (Markovian):
P(5441|5e,00 at)
= B
AT
AT
P 8T 6T=0
Σ
Σ
AT
Af
Σ
P(8T"))((1))(P))11168 Σ
ST=0
where P(TT) is the probability of delay for 8T steps for &t, and P(8T*) is the probability of delay for 8T steps for St+1. Since P(8T") and P(6T) do not depends according to the definition of DOMDP, we only need to prove that the last term P( 18-118888 8T, at AT-a) ) is also independent of t (with 0 S 87, 87 AT). Let us discuss case by case:
When 8T > 8T + we have P($++1-877) (8-5T, 5T, at- (T-AT:) = because t + 1 8T <t t 5T and the previous state should not depend the future one.
When 87" = 57 + 1, we have P( ( 88-88 ar AT:)) = P(( =1
When 8T" 87, have P(s++1 -8 s0r111888-- 18-87; ae- AT-:)) 8( (t-8)) which is Markovian since is the state transition function of the original MDP.
When 8T 8T 1, we have
P(s1+1-5T/ 18--8 (T-AT:)
-P(S ( ag ST, at 0t-87+1)
-f
87+ 0t Po 88 a0---8 -r
Sc-aT+1
where each of the terms is the state transition function of the original MDP, thus (1841-881((668)) (T) is Markovian.
Similarly, when 8T < 8T 1, P(s411-88)) ss- sT, ar-AT:) can be written as accumulated integra- tion of state transitions of the original MDP the different steps, therefore remaining independent t.
So far, have shown that P(s (8- 01- AT:) is Markovian and thus P( 0z) is also Markovian.
Note that a similar theorem was also derived in (Chen et al., 2021). However, they considered fixed delay step while we consider the more general case of unfixed delay.
E IMPLEMENTATION DETAILS
E. ENVIRONMENT SETUP
This category includes four fundamental Mujoco environments: Walker2D, Ant, HalfCheetah, and Hopper. These environments are generally well-handled by existing methods, avoiding issues such as local optima (e.g.. in Reacher) or insurmountable challenges (e.g., in Hu- manStandup). Thus, they provide suitable baselines for evaluating the impact of domain delay. We modify these environments return the state from AT steps in the past, thereby simulating fixed delay. For implementation, we use a FIFO queue with length as AT and update the queue with the new state always return with the one the end of the queue the agent.
(1) Fixed Delay:
This simulation reflects real-world scenarios with random delays (like network latency). We implement it using FIFO queue of length AT that updates with states. Each step samples from the queue to give the agent its observation. A straightforward normal sampling isn t used real-world observations preserve order. Instead, we unique sampling strategy, where
(2) Unfixed Delay:
15

---

Published as a conference paper at ICLR 2024
sample from 1, shifts compared to the previous step. This maintains order and causes delays to range from to AT with an average delay proportional to AT.
(3) Large State Space: The dimensionality of the state space significantly influence the feasibil- ity of recovering the oracle state from delays. To study this, we utilized the Humanoid environment in Mujoco, which has state dimension of 376, far larger than that of Ant (27), HalfCheetah (17), Hopper (11), or Walker (17).
(4) Probabilistie: To approximate the inherent uncertainties of real-world control systems, ap- plied a probabilistic adaptation to the deterministic Mujoco environments (Todorov et al., 2012). We employed two methods to introduce randomness: (1) Noise action, where Gaussian noise, (0, 0.3), is added to the input action implemented by the environment; and (2) Sticky action, where there's 's 30% chance that the environment will execute the previous action instead of the current input.
E.2 IMPLEMENTATION OF BASELINES FROM OTHER RESEARCHES
This study specifically examines the solving of problem of delayed signal, a characteristic the en- vironment. Consequently, modify only the environmental aspects while maintaining the integrity of the baseline algorithms. To ensure accuracy, we strietly utilize the original authors' source code for baseline implementations, altering only the environmental parameters to prevent implementa- tion errors. Additionally, considering that all the baselines (Han et al., 2020; Ni et al., 2022; Chen al., 2021) have previously been tested in the Mujoco environment as reported in their papers, it is reasonable for us to employ the default parameters established in those studies. This consistency in environmental settings enables more direct and valid comparison of our findings with existing research.
E.3 NETWORK ARCHITECTURE
In this study, maintain consistent architecture design across all investigated variants, with focus on training paradigms and frameworks rather than network specifics. For both actor and critic networks in standard algorithms like SAC, DDPG, and TD3, we tures per layer, employing tanh as the activation function. In the case of RNN networks, we employ two layers of recurrent cells, also with 256 hidden features, align the number of parameters. For the transformer structure, we use single-head attention mechanism where key and value share the same tensor, and the query has separate tensor, each with feature size of 256. This attention block methods, including
two-layer MLP with 256 fea-
is applied twice, mirroring the two-layer MLP approach. For prediction-basee those involving encoding, the process involves first encoding inputs into a 256-feature space using two-layer MLP, followed by another two-layer MLP for making predictions. This uniform ar- chitecture across various models allows for a focused analysis on training methodologies, ensuring that observed performance differences are attributable the training paradigm rather than structural variations.
E.4 HYPERPARAMETERS
For a fair and consistent evaluation, we adhere to default hyperparameters as outlined in founda- tional studies for each algorithm. In the case of SAC, follow parameters from Haarnoja et al. (2018b). Our approach for -echniqu-speccffcc parameters begins with standard settings, followed by adjustments within a practical range. This includes tuning the prediction loss weight in methods involving prediction and encoding, testing values in the set (0.005, 0.01, 0.05, 0.1). Additionally, we explore auto KI. weight tuning, setting target KI. loss ranges, and sweeping through values (5. 20, 50, 200}. For input format consistency, especially in RNN-based models, use 32 sequences of length 64. The memory buffer size matched to the number of environment steps.
In scenarios involving multiple parameters, average performance across each (algorithm, envi- ronment, parameter) combination, subsequently identifying parameters that yield the best average performance for each ( (algorithm, environment) pair. Then, calculate the average performance of all the environments. This method ensures that performance is not compromised by suboptimal hyperparameter selection.
16

---

Published as a conference paper at ICLR 2024
E.5 DELYED IMPLEMENTATION
We developed accessible plug-in environment wrapper for delayed environments, utilizing the gym. Wrappez from the OpenAI Gymnasium library (Brockman et al., 2016). This wrapper seam- lessly integrates with pre-existing environments. In the fixed delay setting, the environment consis- tently returns observations from 7 time steps prior. For the more complex and practical unfixed delay setting, the delay can range from 0 T timesteps. To make more realistic, imposed strict condition ensuring that the incoming observation does not arrive later than previous observations. Furthermore, our implementation accommodates action delay, in accordance with the equivalence established in Sec. 2.2.
E.6 CODE IMPLEMENTATION
We designed a modular and configurable implementation of all methods in this paper with seamless compatibility with popular state-of-the-art offline DRL algorithms including DDPG, TD3, SAC. Through which all the designs mentioned in this paper can be easily load or unload for efficient application and further studies. For the hyperparameters of the algorithm, we align them with the values in the original paper where each algorithm was proposed.
E.7 ADDITIONAL DETAILS
Our performance analysis is derived from eight independent runs, ensuring statistical reliability. For the "Fixed/Basic" and "Unfixed" environments, evaluations conducted after 2 million vironmental steps, while assessments in other environments were completed after I million steps. This approach is based on the understanding that varying environments require different levels of interaction for effective learning.
F SUPPLEMENTARY EXPERIMENTAL RESULTS
F.1 Is REPRESENTATION LEARNING A BOTTLENECK?
In this section, endeavor to design experiment to ascertain whether the failure of certain baseline models can be attributed to deficiencies in representation learning. Specifically, we examine study. VRM is
the Variational Recurrent Model (VRM) as proposed by Han al. (2020) a method for solving POMDP problems by using a variational recurrent neural network (Chung al., 2015) which predict next observation from contextual observation and actions. Table I shows VRM's dramatic suffering from observation delay. To investigate whether this failure is due to unsuccessful representation learning. We present the representation learning loss (negative evidence lower bound) of the variational recurrent neural network of VRM in each tasks (Table 2). The results reflect no significant difference of the representation learning loss for delay steps and 4. However, VRM's performance (Table 1) with delay=4 is only around one-third of the case with delay=0. Thus, it appears implausible that the difficulties VRM encounters handling delayed tasks are caused by ineffective representation learning. This suggests that other factors may be contributing to the observed performance degradation.
Delay
Walker2d
HalfCheetah
Hopper
Ant
Average
0
-1.87 + 0.31
-1.60 + 0.15
-2.50 0.07
-1.68 = 0.21
-1.91
1
-1.90 +:000000 0.20
-1.74 + 0.10
2.54
-1.99 0.43
-2.04
2
-2.25 0.24
1.700:0... 0.20
-2.62
-1.89 + 0.21
-2.10
4
-2.26 0.06
-1.65
+ 0.22
-2.49 + 0.09
-1.81 + 0.24 0.24
-2.55
8
-1.92 0.18
-1.57 + 0.08
-1.81 + 0.08
-1677 = 0.20
-174
12
-1.86
0.14
-1.99 + 0.20
-156 0.14
-131 - 0.10
-150
Table 2: Representation learning loss (lower the better) of VRM in the end of learning for various environments with different delay steps (fixed delay). Data are mean + standard deviation.
17

---

Published as a conference paper at ICLR 2024
F.2 DOES REDUNDANT HISTORICAL INFORMATION HAMPER THE PERFORMANCE?
In addressing concerns about the potential redundancy of historical information in our model, we conducted experiments to determine the optimal quantity of historical data for enhancing perfor- mance. Specifically, we varied the number of past actions (H) considered by the model and assessed its impact under fixed delay conditions (Ar (0, 1,4)).
Our findings, detailed in Tab. 3, reveal that excluding past actions (H = 0) results in suboptimal performance. For instance, with At 4, the absence of historical actions led to a low performance score of 47.6. Conversely, incorporating a moderate history (H) significantly improves performance, as shown by a score of 84.3 for H = 2. However, excessively large histories (H 2 8) negatively impact performance, evident from decrease to 55.0 for H 12. This trend is consistent across different At values, indicating that a balanced approach to historical data is crucial.
Our experiments employed both "Delay Reconciled Training for Critic" and "State Augmentation MLP" methods, maintaining consistency across all variables except H. The results also shed light on the limited success of RNN structures in experiments. RNNs inherently process extensive historical information, leading to redundancy, particularly for state and action histories beyond the Al window. This aligns with our theoretical finding (Theorem 4.1) that maintaining the Markovian property requires only historical actions within At.
Table 3: Impact of Historical Information (H) Model Performance under Different Delays (At). The under-performing methods are marked with 4.
# Delay At
Historical Timesteps H 4
4
|f =
97.7+24.0
87.8+22.3
47.6+20.34
H
104.5=18.4
100.2+16.5
77.7+15.4
98.8+23.4
93.7+18.9
84.3+16.3
86.8+18.6
81.6+23.4
80.3+17.8
68.3+12.5.
73.0#13.6
58.6+15.5
H = 12
62.6+14.3
60.9#12.8
55.0+8.3|
F.3 COMPARING FIXED AND UNFIXED DELAY
Here, report additional results of algorithms fixed and unfixed delay in Tab. 7. As most algorithm performances are already reported Tab. 1, we primarily focus on the comparison of Prediction and Encoding methods' variations. Our findings reveal that detached methods consis- tently outperform non-detached methods. Specifically, for prediction-based methods, the average performance increases from 68.0% to 83.6% under fixed delay, and 58.8% to 72.5% under unfixed delay. For encoding-basee methods, we observe an average performance increase from 76.1% to 84.5% under fixed delay and 70.2% to 77.9% under unfixed delay. These results suggest that the detached design is preferable.
F.4 LARGE OBSERVATION SPACE
The performance of various algorithms in environments with large observation spaces is presented in Tab. 5. Consistently, both Delay-Reconciled Training and State Augmentation ML. yield pos- itive performance enhancements. Notably, State Augmentation MLP consistently offers the best performance. Conversely, the performances of Prediction and Encoding are substantially inferior, implying that explicitly introducing prediction supervision may adversely impact performance when the observaton space is large.
F.5 PROBABILISTIC SETTINGS GAUSSIAN MUJoCO
In this subsection, explore the impact of probabilistic state transitions by introducing diagonal Gaussian noise (0, o²) to the Mujoco environment. This approach tests our algorithm's 's robustness under varying degrees of uncertainty.
18

---

Published as a conference paper at ICLR 2024
Table 4: Performance (%) of algorithms in fixed and unfixed delay environments (additional comparation of detach and non-detach). The best performing methods of prediction and encoding are highlighted in bold.
Delayed Time Steps
Method
Avg. #24
Prediction
Prediction
103.1
77.1
80.2
73.2
70.4
59.6
68.0
+7.8
+11.3
+9.5
#11.5
+10.2
+13.2
+11.6
+8.6
+10.1
+8.3
+11.4
+10.0
#11.1
102.1
101.5
96.1
100.5
99.0
92.2
88.6
85.6
71.0
73.0
58.1
83.6
72.5
Prediction
+9.7
#13.9
#12.2
#14.4
#12.5
#16.2
+14,4
#10.8
+12.8
#10.3
#14.1
#12.5
#13.8
Encoding
98.5
91.7
102.5
93.1
83.4
81.8
77.9
80.1
70.2
66.0
62.5
76.1
70.2
Encoding
+10.9
+9.0
+8.5
#11.2
+9.7
+8.9
+8.7
101.4
103.2
92.0
90.4
86.6
77.9
73.3
69.2
84.5
Encoding
+12.1
29.0
#9.6
#10.0
+9.5
+9.5
+8.22
+12.4
#10.8
29.9
+9.7
=10.6
+9.6
Table 5: Performance (%) of algorithms in environment with large observation space. The top- performing methods are highlighted in bold.
Delayed Time Steps
Method
2
8
12
Vanilla SAC
24.7+13.6
11.8
12.3+2.8
2.5+1.2
2.6+1.7
2.9+1.8
Becyy-Rccoeeildd Training
23.9#6.4
26.6+7.2
21.8+8.1
17.3+4.7
13.2+2.5
14.6t3.1
State Augmentation MLP
83.8+22.9
103.4*1..5
94.6+11.9
102.7*6.4
101.3+7.8
89.7=14.5
State Augmentation RNN
48.3+30.8
67.7439.4
47.8438.6
47.5+43.7
53.6+45.8
55.4+42.6
Encoding
87.5+24.8
97.3+15.7
67.9421.8
20.444.7
18.243.0
14.5+2.9
Prediction
87.7432.6
22.3+6.2
16.643.7
18.543.9
16.942.4
14.4+2.9
Experiments were conducted with noise variances = 0.1,0,, 0.4), comparing our modi- fied algorithm (Ours") against the SAC. Notably, at delay=0, Vanilla SAC and Ours* performances are equivalent.
The performance metrics presented "Velue(Relativeee)" The "Value" is consistent with the normalization Tab. 1. The "Relative%" indicates the relative reward compared to the scenario when delay is O. Key observations include:
Vanilla SAC shows marked performance decrease with delay, dropping to 23.78%, 26.67%, 32.39%, and 44.25% for values of 0.05, 0.1, 0.2, and 0.4, respectively.
Our method significantly enhances performance delayed scenarios, with improvements from 23.78% 79.57%, 26.67% to 83.14%, 32.39% to 92.25%, and 44.25% to 98.23% for the respective a values.
These results demonstrate the effectiveness of our algorithm in environments with probabilistic state transitions, underscoring its robustness and adaptability.
F.6 PROBABILISTIC SETTINGS NOISY AND STICKY ACTIONS
This section reveals additional numerical results for algorithms within probabilistic state transition environments. Performance measures for two probabilistic environment implementations, "sticky action" and "noisy action", are outlined in Tab. and Tab. respectively.
Notably, both the "noisy action" and "sticky action" settings intensify the environment's complex- ity, causing performance decrease from 100.0% to 72.3% and 64.3%, respectively. For both the "noisy action" and "sticky action" environments, Delay- recynceeeodeiled training and State Augmentation MLP exhibit consistent performance improvement over vanilla SAC in all settings where the de- lay exceeds one step. This suggests that our design is effective in probabilistic environments as well. However, prediction and encoding-basee based methods fail perform satisfactorily within these settings.
19

---

Published as a conference paper at ICLR 2024
Table 6: Performance (%) of algorithms in probabilistic (Gaussian State) environment. The top- performing methods are highlighted in bold.
# Delayed Time Steps
Scale of Randomness
Method
8
12
0.05
Ours*
32.8 7.4 (100%)
35.0 +: 7.9
(106.71%)
32.9 +: 5.7
(100.30%)
31.7 :: 5.8 (96.65%)
27.1 7.2
(82.62%)
26.1 6.0
(79.57%)
0.05
Vanilla SAC
32.8 + 7.4 (100%)
12.4 +: 3.9 (37.80%)
11.0 + 1.9 (33.54%)
9.7 = 1.5 (29.57%)
8.1 1.1 (24.70%)
7.8 1.3
(23.78%)
0.1
Ours*
25.5
+: 4.0 (90.59%%
+: 5.3 (95.29%)
4.7 (83.14%)
21.9
(100%)
(85.88%)
(83.14%)
Vanilla SAC
2.9
11.0 14 (43.14%)
10.4 +: 1.7 (40.78%)
9.4 = 1.3 (36.86%)
7:4 1.2 (29.02%)
6.8 3.3
(100%)
(26.67%)
0.2
Ours*
14.2
13.3
13.3 + (93.66%)
13.4 + 2.1 (94.37%))
11.8 11.8.22.
13.1
(100%)
(93.66%)
(83.10%)
(92.25%)
0= 0.2
Vanilla SAC
14.2
+ 1.1 (71.13%)
+ (60.56%)
(100%)
(76.76%)
(56.34%)
(32.39%)
0.4
Ours*
11.3 (100%)
11.0 1.4 (97.35%)
10.0 + 2.3 (88.50%)
10.8 2.1
10.7 :: 1.7 (94.69%)
11.1
(95.58%)
(98.23%)
= 0.4
Vanilla SAC
11.3
10.0
10.1 + 1.2 (89.38%)
8.8 - 2.1 (77.88%)
8.0 1.8 (70.80%)
5.0
(100%)
(88.50%)
(44.25%)
Most notably, the prediction method's performance in a noisy action environment drastically drops from 72.8% 9.9% with just one-step delay. More importantly, prediction and encoding methods consistently underperform when compared to State Augmentation MLP, implying that their ap- plication does invariably yield positive performance effects on environments with probabilistic state transition.
Table 7: Performance (%) of algorithms in probabilistic (noise action) environment. The top- performing methods are highlighted in bold.
# Delayed Time Steps 4
Method
0
8
12
Vanilla SAC
72.3+12.4
48.1714.7
38.6#12.9
14.5+9.4
3.243.5
0.743.8
Reconceeddd Training
74.749.1
66.649.2
59.8=8.6
48.3+8.3
25.4+6.9
14.2+4.7
State Augmentation ML.
77.949.2
71.3+11.5
65.7+10.3
64.6#11.9
48.8+10.2
34.1+14.7
State Augmentation RNN
65.4+15.6
60.7+11.4
54.1=10.5
40.9+10.3
20.348.7
12.545.2
Predictionf
72.8+18.3
9,944.8
20.4+10.2
24.748.1
17.6#3.2
5.3+1.1
Encoding
35.7+19.8
41,2+19.4
43.8+12.7
39.5+11.6
25.945.7
25.4+1.9
Table 8: Performance (%) of algorithms in probabilistic (sticky action) environment. performing methods are highlighted in bold.
The top-
# Delayed Time Steps
Method
8
Vanilla SAC
64.3+18.7
44.6216.1
26.5+15.8
9.4+10.3
6.7+3,4
0.8+3.9
Delay-Reconciled Training
63.9+15.6
70.8+18.2
66.7+12.9
52.4+11.5
29.349.2
18.249.6
State Augmentation MI.P
68.7+11.2
62,4+13.5
62.6#18.3
57.8+21.9
38.9+13.6
32.
State Augmentation RNN
57.5+19.8
57.1413.4
51.9+14.8
39.3+18.6
16.648.7
15.748.4
Prediction!
52.9+14.6
27.343.9
27.8+6.1
21.447.2
13.5+4.3
12.6+4.9
Encoding!
42.744.8
37.242.7
37.422.4
34.8+12.6
29.3+7.2
26.9+4.8
20

---

Published as a conference paper at ICLR 2024
F.7 COMPREHENSIVE ANALYSIS OF LEARNING CURVES
This section provides an in-depth analysis of the learning curves exhibited by various algorithms across distinct environments. Figures 8 through 15 illustrate the progression of learning in environ- ments characterized by consistent, fixed delays. In contrast, figures 16 through 23 present the learn- ing trajectories within environments where delays are variable and unfixed. The results depicted here are intended to furnish a comprehensive benchmark for subsequent research in this domain. Additionally, we highlight several key insights derived from our analysis.
1. Even a 2-step delay can significantly impair the learning efficiency of standard algorithms, leading to nearly flat learning curves.
2. The impact of delay varies dramatically across different environments. For example, the Hopper environment is relatively less affected by delay, experiencing an overall perfor- mance drop of around 10% as illustrated in Fig. 13 and 21. On the other hand, the HalfCheetah environment sees a substantial performance decline of approximately 60% with 12-step delay.
3. The presence of delay affects both the speed of learning and optimal performance. As the number of delay steps increases, the learning curve usually shows steady rise in most cases, indicating the gradual adaptation of the learning process to the delay.
Deay
Deep
1903
280
1980
2:00
480
1000
0.2
(a) Ant
(b) HalfCheetah
(c) Hopper
(d) Walker2d
Figure 8: Learning curve of algorithms on environment with fixed delay: vanilla DDPG.
610
1400
4:08
3000
500
3138
4000
190/
2000
200
190
(a) Ant
(b) HalfCheetah
(c) Hopper
(d) Walker2d
Figure 9: Learning curve of algorithms environment with fixed delay; vanilla TD3.
21

---

Published as a conference paper at ICLR 2024
Dany
D1g
1200
180.0
1230
190
1038
100
138
(a) Ant
(b) HalfChectah
(c) Hopper
(d) Walker2d
Figure 10: Learning curve of algorithms on environment with fixed delay: vanilla SAC.
De
D447
Dclp
De
100
120/
2.58
2000
2:00
(a) Ant
(b) HalfCheetah
(c) Hopper
(d) Walker2d
Figure 11: Learning curve of algorithms on environment with fixed delay: RNN Strong.
1400
500
3158
280
195/
3158
8000
100
100
844
54
(a) Ant
(b) HalsCheetah
(c) Hopper
(d) Walker2d
Figure 12: Learning curve of algorithms on environment with fixed delay: delay critic training.
Oeer
O4
D4
190.
690
1900
200
150
1000
100
144
(a) Ant
(b) HalfCheetah
(c) Hopper
(d) Walker2d
Figure 13: Learning curve of algorithms on environment with fixed delay: state augmentation MLP.
22

---

Published as a conference paper at ICLR 2024
3000
5000
690
1990
3500
(a) Ant
(b) HalfChectah
(c) Hopper
(d) Walker2d
Figure 14: Learning curve of algorithms on environment with fixed delay: state augmentation RNN.
Pelay
Oeny
1300
100
108
0.3
(a) Ant
(b) HalfCheetah
(c) Hopper
(d) Walker2d
Figure 15: Learning curve of algorithms on environment with fixed delay: symmetric MI.P.
bady
O4dy
De
3800
1200
1985
3008
2.58
2:08
4000
02
(a) Ant
(b) HalfCheetah
(c) Hopper
(d) Walker2d
Figure 16: Learning curve of algorithms on environment with unfixed delay: vanilla DDPG.
Deny
Del
100
1300
400
04/0
2058
3000
1158
105
20/0
50
62
(a) Ant
(b) HalfChectah
(c) Hopper
(d) Walker2d
Figure 17: Learning curve of algorithms on environment with unfixed delay: vanilla TD3.
23

---

Published as a conference paper at ICLR 2024
Dany
1200
600
180.
2000
608
1088
100
(a) Ant
(b) HalfChectah
(c) Hopper
(d) Walker2d
Figure 18: Learning curve of algorithms on environment with unfixed delay: vanilla SAC.
De
D44y
Delp
De
100
125/
2.58
2000
2:00
(a) Ant
(b) HalfCheetah
(c) Hopper
(d) Walker2d
Figure 19: Learning curve of algorithms on environment with unfixed delay: RNN Strong.
6080
1402
3000
500
1250
3158
4.00
1900
100
1000
3.5
8.4
54
(a) Ant
(b) HalsCheetah
(c) Hopper
(d) Walker2d
Figure 20: Learning curve of algorithms on environment with unfixed delay: delay-reconciled critic training.
Onlr
O4
D4
120.
3138
630
1900
3058
200
1100
110
02
144
(a) Ant
(b) HalfCheetah
(c) Hopper
(d) Walker2d
Figure 21: Learning curve of algorithms on environment with unfixed delay: state augmentation MLP.
24

---

Published as a conference paper at ICLR 2024
1000
200
1980
1220
3000
100
(a) Ant
(b) HalfCheetah
(c) Hopper
(d) Walker2d
Figure 22: Learning curve of algorithms on environment with unfixed delay: state augmentation RNN.
D44
1300
2000
180
28/3
100
5.
02
(a) Ant
(b) HalfCheetah
(c) Hopper
(d) Walker2d
Figure 23: Learning curve of algorithms on the environment with unfixed delay: symmetric MLP.
25


Reviews:
[{"review_id": "dLxjzkSqyk", "reviewer": "Reviewer_eRXT", "summary": "The authors study the impact of signal delay in reinforcement learning problems. Their results show that even minor delays in receiving states can negatively impact the performance of various existing deep-learning algorithms. The authors formalize delay as the Delayed Observation Markov Decision Process and show theoretically grounded means to modify existing actor-critic algorithms to account for the delay. They propose augmenting the state with previous actions and consider conditioning on reconstructed observations at deployment. Their experiments suggest combining both techniques can mitigate the effects of signal delay.", "questions": "1.) Aren't delayed rewards already accounted for via the value function? Consider removing this wording in the introduction or clarify that your research looks at delays in states, as we could not find a discussion on addressing delays in received rewards. \n2.) The transition probability in section 2.2 is confusing. The next state only transitions if s’^{-t+ = s^{-t+1} and otherwise would return 0 if any states don't match. Is this correct, or a misunderstanding?\n3.) What were the parameters used in the experiments of Section 3? Including experimental details is reasonable for reproducibility. If they are in the appendix, please mention this in the section\n4.) What about including the time since the last received sample as a state feature?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Overall, the paper is well written. The authors succinctly characterize the issues of observation delay and motivate it thoroughly across various domains. Each section transitions well to the next and is accessible to readers. We appreciate the author's approach of formalizing the problem and then providing general solutions. Identifying a missing property in RL algorithms and generally applicable techniques is a good approach, and the performed experiments offer compelling evidence of their efficacy. We feel the authors do due diligence with their investigations to validate their solutions, and we appreciate them focusing on accessible solutions and ablating them in the paper.", "weaknesses": "One issue with the paper is the proposed use of previous actions in the augmented state representation. Some previous work suggests, particularly for sequence models [1], that this can be problematic, and it seems the results may mean this is the case as the Transformer and RNN results seem to benefit less from the state augmentation approach, especially as the horizons increase. We consider this a future work that the authors might consider discussing. Theorem 4.1 might require a few additional words to emphasize whether the provided proof is from previous work, mainly as the authors cite a paper when talking about this theorem.\n\nAlthough we overall find the paper accessible, some parts could be better written or expand on important details. Specifics below:\n\n1.)  Section 3 could benefit from clarifying the experimental setup or informing the reader where to find such information, for example. Additional comments below:\n\n2.) Including the % error cap for Figure 3 d. Consider including information on the calculations for these are done. \n\n3.) Figure 4 needs to be clarified what the robot is observing. Perhaps some thought bubble or other intermediate object to distinguish where the robot is versus where it THINKS it might help if, for example, if s_t - \\Delta T is supposed to be the information the robot is currently making decisions from, that was unclear. \n\n4.) Consider moving Table 1 to the appendix and use plots to highlight trends the authors observed from the results. It's difficult to process and follow all the observations. \n\n5.) Figure 6 is a bit difficult to dissect the details of the work. I suggest adding additional annotations to plots to make it more transparent what each subplot visualizes.\n\n\nCitation:\n[1] Shaking the foundations: delusions in sequence models for interaction and control. Ortega et al. 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iMLRdsURP8", "reviewer": "Reviewer_aPPk", "summary": "This study tackles a significant yet frequently underestimated issue in the integration of reinforcement learning agents into real-world applications: signal delay. The authors establish a formal framework for this problem by framing it as delayed-observation Markov decision processes. They then introduce a novel algorithm designed to effectively address this challenge. The empirical findings from experiments reveal that the proposed method consistently demonstrates strong performance, even in scenarios with multiple steps of signal delay. This research sheds light on a crucial aspect of reinforcement learning in practical settings, offering a promising solution for mitigating the impact of signal delays on agent performance.", "questions": "1. Could the performance of the method be further enhanced by considering advanced world models, like the one developed for model-based RL in Dreamer[1], as an alternative for the architecture of the state augmentation and prediction module?\n\n[1] Hafner, Danijar, et al. \"Mastering diverse domains through world models.\" arXiv preprint arXiv:2301.04104 (2023).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Clarity and Readability: The manuscript is well written, making it easy for readers to comprehend the presented concepts and findings. The clear and coherent writing style enhances the accessibility of the research.\n\n2. Significant Problem Addressed: This paper takes on an important and frequently underestimated issue, which has often been overlooked in prior research. \n\n3. Impressive Performance: The achieved performance, when compared to conventional vanilla algorithms, is notably robust and impressive. This showcases the effectiveness and practicality of the proposed approach, making it a noteworthy contribution to the field.", "weaknesses": "One potential weakness of the paper lies in its handling of experiments in a probabilistic state transition setting. While the study explores action space perturbation, such as sticky action and noisy action, it could benefit from a more comprehensive consideration of the stochasticity within the state space. This might involve scenarios with dynamic background images or robots with joint angular positions sampled from a normal distribution with high variance. Incorporating these additional sources of uncertainty would provide a more realistic and robust evaluation of the proposed algorithm's performance under varied conditions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xRguUdx8pw", "reviewer": "Reviewer_dZmE", "summary": "This paper considers the delay in MDP (e.g., inference delay, observation delay, action delay) and formulates Delayed-Observation MDP setting. Effective approach is proposed to deal with Delayed systems.", "questions": "No question.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The studied problem is interesting and meaningful. Delayed systems in widely and well studied in control community, see [1] [2] for example. There are thousands of works addressing classic concerns such as system stability and safety. I would suggest the authors to survey and (briefly) compare more control papers in this field because control theory and reinforcement learning are highly related and share similar concerns.\n\n2. The experimental results are encouraging and promising.\n\n3. I think this paper is well-written and easy to follow and understand.\n\n\n[1] Richard, J.P., 2003. Time-delay systems: an overview of some recent advances and open problems. automatica, 39(10), pp.1667-1694.\n[2] Cacace, F., Germani, A. and Manes, C., 2010. An observer for a class of nonlinear systems with time varying observation delay. Systems & Control Letters, 59(5), pp.305-312.", "weaknesses": "My main concern is the real-world application. Delayed systems investigation is motivated from the real systems, so I think the approach/theory developed for delayed systems should be examined through real-world experiments such as robotic, autonomous vehicles rather than just artificial simulations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1jvEqBYnU2", "reviewer": "Reviewer_6hPB", "summary": "This paper focuses on the problem of sequential decision making with signal delay. \n\nThat means observations on the current state do not arrive immediately after an action is taken, but arrive several steps after. \n\nThis delay in signal effectively makes the problem a POMDP, even when the agent always observes the underlying environment state.  \n\nTthe authors propose the formalization for such setting, called delayed osbervation MDP, and show that in fact it is a special case of POMDP. \n\nWhile we can potentially see this kind of problem as a POMDP and use standard DRL algorithms for POMDPs, the authors argue and show that these algorithms struggle in such setting. \n\nAs the main contribution of this paper, several techniques are proposed to improve the performance of DRL algorithms, in particular, actor critic methods, in this setting. \n\nThese techniques mainly rely on using priviledged information, e.g., access to the true current state, during offline training to improve the learning of actor and critic. \n\nSpecifically, for the critic, the authors propose to condition the critic on the true current state instead of the history of observations or the current observation which might be the true state a few steps earlier during training. Because the critic is not used during inference or deployment, this approach is possible as long as we have access to the true states during training. \nFor the actor, the authors propose to add actions taken in the past to the inputs of the actor as this will provide useful information to help the actor to predict the true current state. \n\nExperiments are performed to test the effectiveness of these proposed techniques. To do so, the authors take SAC, a strong model-free baseline, and build upon it with the proposed techniques. Results show that these techniques are indeed generally useful, especially when the delay is longer.", "questions": "In Figure 3, I find it weird to argue that baseline methods are failing by saying that there is a significant drop in performance from no delay to one-step delay. With different delay steps, we essentially have different POMDPs, which may inherently have different difficulty levels and different optimal performance. As such, the drop in performance I think is not sufficient to say the baseline methods are failing. However, a comparison to the optimal performance or performance from a better method would work. \n\nminor:\n- in Figure 3, colors correspond to delay lengths different in (d) than in other subplots. this can be confusing for readers.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "This is an interesting paper. \n\nIt looks at a problem that is somewhat ignored in the literature, indentifies the failure of existing algorithms, and proposes several techniques to improve the baselines, which are shown to be effective. \n\nMany baselines are considered and compared in the experiments.", "weaknesses": "My biggest concern on this paper is that many of the proposed techniques, if not all, are not new, and the paper misses a large body of closely relevant works. In Section 4.1, the authors propose to condition the critic on the true current state (without delay) to improve learning. The idea of using priviledged information during training by conditioning the critic on the ground-truth state in actor critic methods has been widely explored by both the POMDP and MARL communities. For example, see [1] and [2] for the former case and [3] for the latter case. Section 4.1 basically describes a trivial application of this idea to DOMDPs, which are special cases of POMDPs. Furthermore, the authors mention that the efficay of such asymmetric architecture has been demonstrated by past previous studies. However, in [2], it has been shown that conditioning the critic (only) on the true state is fundamentally questionable as it leads to biased gradients. \n\nThen in Section 4.2, the authors propose to give the actor past actions to help it infer the state at the current step. I don't understand why is this not done by default. In my understanding, DOMDPs are POMDPs and in POMDPs, past actions and observations should always be given to the policy for optimal control. I don't see how this is an innovation. \n\nThe method that is proposed in Section 4.3 has also been explored for the general POMDP settings. For example, in [4,5,6], people have explored auxiliary losses (to predict the next observation, state, reward) for improving the representation learning of RNNs when learning optimal policies in POMDPs. \n\nIn the beginning of Section 3, baseline DRL methods are tested in delay signal settings and they are demonstrated to struggle in such a setting. Reasons are guessed for why they fail. However, I do not think that the failure of these baselines methods are studied well enough. I would like to see more ablation study on understanding how exactly they fail, which I believe will greatly help readers understand in what way DOMDPs are different to POMDPs and what are their unique challenges. Is it really due to the signal delay? Or like in general POMDPs, the RNNs cannot come up with good representations for decision making? For example, if these methods struggle to learn representations that support effective decision making, then one can test the accuracy of using these learned representations to predict the true current state. There are possible causes mentioned in paper but I find them rather vague and are not enough to motivate the proposed algorithmic changes. \n\nI also find lots of experimental details are missing from the paper, such as the architectures of networks, and the hyperparameters. Also, for the baselines, such as DDPG, TD3, SAC, are they also RNN based? It would be strange to use a non-recurrent baseline in a POMDP environment. And are the past actions passed to the actors in these baseline methods? Regarding hyperparameters, it would also be useful to know how are the hyperparameters tuned for each baseline and the proposed method to ensure fair comparison. \n\n[1] Baisero, Andrea, Brett Daley, and Christopher Amato. \"Asymmetric DQN for partially observable reinforcement learning.\" Uncertainty in Artificial Intelligence. PMLR, 2022.\n\n[2] Baisero, Andrea, and Christopher Amato. \"Unbiased Asymmetric Reinforcement Learning under Partial Observability.\" Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems. 2022.\n\n[3] Foerster, Jakob, et al. \"Counterfactual multi-agent policy gradients.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 32. No. 1. 2018.\n\n[4] Igl, Maximilian, et al. \"Deep variational reinforcement learning for POMDPs.\" International Conference on Machine Learning. PMLR, 2018.\n\n[5] Subramanian, Jayakumar, et al. \"Approximate information state for approximate planning and reinforcement learning in partially observed systems.\" The Journal of Machine Learning Research 23.1 (2022): 483-565.\n\n[6] Lambrechts, Gaspard, Adrien Bolland, and Damien Ernst. \"Informed POMDP: Leveraging Additional Information in Model-Based RL.\" arXiv preprint arXiv:2306.11488 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)