Thinking While Moving: Deep Reinforcement Learning with Concurrent Control
Authors: Ted Xiao
Eric Jang
Dmitry Kalashnikov
Sergey Levine
Julian Ibarz
Karol Hausman
Alexander Herzog
Venue:  (2020)
Keywords: deep reinforcement learning
continuous-time
robotics

Abstract: We study reinforcement learning in settings where sampling an action from the policy must be done concurrently with the time evolution of the controlled system, such as when a robot must decide on the next action while still performing the previous action. Much like a person or an animal, the robot must think and move at the same time, deciding on its next action before the previous one has completed. In order to develop an algorithmic framework for such concurrent control problems, we start with a continuous-time formulation of the Bellman equations, and then discretize them in a way that is aware of system delays. We instantiate this new class of approximate dynamic programming methods via a simple architectural extension to existing value-based deep reinforcement learning algorithms. We evaluate our methods on simulated benchmark tasks and a large-scale robotic grasping task where the robot must "think while moving."

Published as a conference paper at ICLR 2020
THINKING WHILE MOVING: DEEP REINFORCEMENT LEARNING WITH CONCURRENT CONTROL
Ted Xiao', Eric Jang', Dmitry Kalashnikov', Sergey Levine Julian Ibarz!, Karol Hausman's Alexander Herzog
Google Brain, UC Berkeley, 3x
(tadxiao, ejang, dkclashnikov, slevine, julianibarz, (erolhoos:mann@ggoggge al alexherxogBx.Lean
ABSTRACT
We study reinforcement learning in settings where sampling an action from the policy must be done concurrently with the time evolution of the controlled system, such as when robot must decide on the next action while still performing the pre- vious action. Much like a person or an animal, the robot must think and move at the same time, deciding on its next action before the previous In order to develop an algorithmic framework for such concurrent control prob- lems, we start with continuous-ting formulation of the Bellman equations, and then discretize them in a way that is aware of system delays. We instantiate this new class of approximate dynamic programming methods via a simple architec- tural extension to existing value-based deep reinforcement learning algorithms. We evaluate our methods on simulated benchmark tasks and large-scale robotic grasping task where the robot must "think while moving". Videos are available at attts::/siite google. www/view////occoooo...o........o
has completed.
lemoving
INTRODUCTION
In recent years, Deep Reinforcement Learning (DRL) methods have achieved tremendous success a variety of diverse environments, including video games (Mnih et al.] (2015). zero-sum games (Sil- ver al. 2016), robotic grasping (Kalashnikov al. 2018), and in-hand manipulation tasks (Ope- nAl et al. 2018 While impressive, all of these examples ls use a blocking observe-think-act paradigm: the agent assumes that the environment will remain static while it thinks, so that its actions will be executed on the same states from which they were computed. This assumption breaks in the con- current real world, where the environment state evolves substantially as the agent processes obser- vations and plans its next actions. As an example, consider dynamic task such as catching a ball: it is not possible to pause the ball mid-air while waiting for the agent decide on the next control to command. In addition to solving dynamic tasks where blocking models would fail, thinking and acting concurrently can provide benefits such as smoother, human-like motions and the ability to seamlessly plan for next actions while executing the current one.
Despite these potential benefits, most DRL approaches are mainly evaluated in blocking simulation environments. Blocking environments make the assumption that the environment state will not change between when the environment state is observed and when the action is executed. This assumption holds true in most simulated environments, which encompass popular domains such as Atari (Mnih et al., 2013) and Gym control benchmarks (Brockman et al, (016). The system is treated in sequential manner: the agent observes state, freezes time while computing action, and finally applies the action and unfreezes time. However, in dynamic real-time environments such as real-world robotics, the synchronous environment assumption is no longer valid. After observing the state of the environment and computing an action, the agent often finds that when it executes an action, the environment state has evolved from what it had initially observed; we consider this environment a concurrent environment.
In this paper, we introduce an algorithmic framework that can handle concurrent environments in the context of DRI. In particular, we derive modified Bellman operator for concurrent MDPs and
Indicates equal contribution.

---

Published as a conference paper at ICLR 2020
present the minimal set of information that must augment state observations with in order to recover blocking performance with Q-learning. We introduce experiments on different simulated environments that incorporate concurrent actions, ranging from common simple control domains to vision-based robotic grasping tasks. Finally, we show an agent that acts concurrently in a real-world robotic grasping task is able to achieve comparable task success to blocking baseline while acting 49% faster.
2 RELATED WORK
Minimizing Concurrent Effects Although real-world robotics systems are inherently concurrent, it is sometimes possible to engineer them into approximately blocking systems. For example, using low-latency hardware (Abbeel et al., (006) and foooootttintt controllers (Cruz et al. 2017) mini- mizes the time spent during state capture and policy inference. Another option to design actions to be executed to completion via closed-loop feedback controllers and the system velocity is de- celerated to zero before a state is recorded (Kalashnikov al. 2018). In contrast to these works, tackle the concurrent action execution directly in the learning algorithm. Our approach can be applicd to tasks where it is not possible to wait for the system to come to rest between deciding new actions.
Algorithmic Approaches Other works utilize algorithmic modifications to directly overcome the challenges of concurrent control. Previous work in this area can be grouped into five approaches: (I) learning policies that are robust to variable latencies (Tan et al. 2018). (2) including past his- tory such as frame-stacking (Haarnoja et al. 2018). (3) learning dynamics models to predict the future state at which the action will be executed (Firoiu et al.) 2018; Amiranashvili et al., 2018). (4) using a time-delayed MDP framework (Walsh et al. 2007 Firoiu et 2018 Schuitema et al, 2010), and (5) temporally-aware architectures such as Spiking Neural Networks (Vasilaki et al. 2009 Frémaux et al. 2013), point processes (Upadhyay et al. 2018 Li et al. 2018). and adaptive approach able to (1) optimize
skip intervals (Neitz et al. 2018). In contrast to these works, for a specific latency regime as opposed to being robust to all of them, (2) consider the properties of the source of latency opposed to forcing the network to learn them from high-dimensional inputs, (3) avoid learning explicit forward dynamics models in high-dimensional spaces, which can be costly and challenging, (4) consider environments where actions are interrupted as opposed to discrete-time time-delayed environments where multiple actions are queued and each action is exe- cuted until completion. The approaches in (5) show promise enabling asynchronous agents, but are still active areas of research that have not yet been extended to high-dimensional, mmage-based robotic tasks.
Continuous-time Reinforcement Learning While previously mentioned related works largely operate in discrete-timg environments, framing concurrent environments as continuous-ting sys- tems is a natural framework apply. In the realm of continuous-time optimal control, path integral solutions (Kappen] 2005) Theodorou et al. 2010) are linked to different noise levels in system dy- namics, which could potentially include latency that results in concurrent properties. Finite differ- ences can approximate the Bellman update in continuous-ting stochastic control problems (Munos) & Bourgine 1998) and continuous-time temporal difference learning methods (Doyal 2000) can utilize neural networks as function approximators (Coulom) 2002). The effect of time-discretizatoo (converting continuous-
environments to discrete-time environments) is studied in [Tallec et al.) (2019), where the advantage update is scaled by the time discretization parameter. While these approaches are promising, is untested how these methods may apply to image-based DRL prob- lems. Nonetheless, we build on top of many of the theoretical formulations in these works, which motivate our applications of deep reinforcement learning methods to more complex, vision-based robotics tasks.
VALUE-BASHR REINFORCEMENT LEARNING IN CONCURRENT ENVIRONMENTS
In this section, we first introduce the concept of concurrent environments, and then describe the preliminaries necessary for discrete- and continuous-time RI. formulations. We then describe the
2

---

Published as a conference paper at ICLR 2020
MDP modifications sufficient to represent concurrent actions and finally, present value-baset RL algorithms that can cope with concurrent environments.
The main idea behind our method is simple and can be implemented using small modifications to standard value-based algorithms. centers around adding additional information to the learning algorithm (in our case, adding extra information about the previous action to Q-function) that allows to cope with concurrent actions. Hereby, we provide theoretical justification on why these modifications are necessary and we specify the details of the algorithm in Alg. [
While concurrent environments affect DRL methods beyond model-free value-based RL, we focus scope on model-free value-basee basee methods due to their attractive sample-efficiency and off-policy properties for real-world vision-based robotic tasks.
3.1 CONCURRENT ACTION ENVIRONMENTS
In blocking environments (Figure 4 in the Appendix), actions are executed in a sequential blocking fashion that assumes the environment state does not change between when state is observed and when actions are executed. This can be understood as state capture and policy inference being viewed as instantaneous from the perspective of the agent. In contrast, concurrent environments (Figure 4p in the Appendix) do not assume a fixed environment during state capture and policy inference, but instead allow the environment to evolve during these time segments.
3.2 DISCRETE-TING REINFORCEMENT LEARNING PRELIMINARIES
We use standard reinforcement learning formulations in both discrete-time and continuous- time settings (Sutton & Barto 1998). In the discrete-time case, at each time step i, the agent receives state s; from set of possible states and selects an action 0g from some set of possible actions A according to its policy и, where is mapping from S to A. The environment returns the next state 8:41 sampled from a transition distribution p(8 88; d:) and a reward r(5;, a;). The return for given trajectory of states and actions is the total discounted return from time step i with discount factor E (0, 1]: R2 k==0 r a(++). The goal of the agent is to maximize the expected return from each state s;. The Q-function for a given stationary policy и gives the expected return when selecting action at state s: (T a) = R2 8;
a). Similarly, the value function
gives expected return from state s: (*)
R. ( = s).
The default blocking environment formulation is detailed in Figure Ia.
3.3
VALUE FUNCTIONS AND POLICIES IN CONTINUOUS TIME
For the continuous-time case, we start by formalizing continuous-time MDP with the differential equation:
ds(t) = F(s(t), a(t))dt + (G(s(t),a(t)dl
(I)
where S = Rd is a set of states, A is a set of actions, F S x A - S and G S x A describe the stochastic dynamics of the environment, and / is a Wiener process Ross al. 1996). In the continuous-timg setting, ds(t) is analogous to the discrete-time p. defined in Section 3.2 Continuous-time functions stt) and (t( specify the state and i-th action taken by the agent. The agent interacts with the environment through state-dependent, deterministic policy function л and the return R of a trajectory T = (=(1), (t)) is given by (Doyal 2000):
S
R(T)
7 (s))ss)) a())dl,,
t=
which leads continuous-tine time value function Tallec et al. 2019):
(s())) = Er~z [s())
/
(3)
= E
t==
and similarly, continuous Q-function:
(=(s))),a,t H) = Ep f 1==1#
)² -t' (((r)((((), a(t'))d +~" H))
(4)
3

---

Published as a conference paper at ICLR 2020
Discrete-Time MDP
Cortinuous-Tmme MDP with Tima- Actions
EAS
EAS
Ls
s(L H LAS")
sit +10+LAS LAS
8(C
8
O 1lt + tAs"))
(1 + fAs)
1n actoon
7
1- II LAs"
+ LAs
+ II
Figure Shaded nodes represent observed variables and unshaded nodes represent unobserved random variables. (a): In "blocking" MDPs, the environment state does not change while the agent records the current state and selects an action. (b): "concurrent" MDPs, state and action dynamics are continuous time stochastic processes s(t) and (t). At time t, the agent observes the state of the world s(I), but by the time selects an action a; + LAs), the previous continuous-tine action function d,-1(1 H + lss") has "rolled over" to an unobserved state s(L + AAs) An agent that concurrently selects actions from old states while in motion may need to interrupt a previous action before it has finished executing its current trajectory.
where H is the constant sampling period between state captures (i.e. the duration of an action trajectory) and refers to the continuous action function that is applied between t and t + H. The expectations are computed with respect to stochastic process p defined in Eq.
3.4 CONCURRENT ACTION MARKOV DECISION PROCESSES
We consider Markov Decision Processes (MDPs) with concurrent actions, where actions are not exccuted to full completion. More specifically, concurrent action environments capture system state while the previous action still executed. After state capture, the policy selects an action that is executed in the environment regardless of whether the previous action has completed, Figure 4 in the Appendix. the continuous-ting time MDP case, concurrent actions can be considered horizontally translating the action along the time dimension (Walsh et al. 2007). and the effect of concurrent actions is illustrated in Figure Пb. Although we derive Bellman Equations for handling delays in both continuous and discrete-time RL, our experiments extend existing DRL implementa- tions that are based on discrete time.
shown in
3.5 VALUE-BASER CONCURRENT REINFORCEMENT LEARNING ALGORITHMS IN CONTINUOUS AND DISCRETE-TIME
We start our derivation from this continuous-tima reinforcement learning standpoint, allows us to easily characterize the concurrent nature of the system. We then demonstrate that the conclusions drawn for the continuous case also apply to the more commonly-uses discrete setting that we then use in all of our experiments.
Continuous Formulation In order to further analyze the concurrent setting, we introduce the following notation. As shown Figure an agent selects N action trajectories during an episode, a1...., AN. where each at(t) is continuous function generating controls as function of time t. Let tas be the time duration state capture, policy inference and any additional communication latencies. At time i, an agent begins computing the i-th trajectory a;(t) from state s(t), while concurrently executing the previous selected trajectory d;-1(t) over the time interval (t + tas, t.s). At time tas, where < t tas S t + H. the agent switches to executing actions from (t). The continuous-timg time Q-function for the concurrent case from Eq. can be expressed as following:

---

Published as a conference paper at ICLR 2020
Q" ("(s(1), (1, Bp
P=t
Executing action trajectory 0, 16) until tas
+ Ep
att)))
Epp 1= (s(t + H))
(5)
Value function
Executing action trajectory (t) until + H
The first two terms correspond expected discounted returns for executing the action trajectory at-1(t) from time (t, + fAs) and the trajectory (;( from time (t + tAs,t + tas + H). We can obtain a single-sample Monte Carlo estimator Q by sampling random functions values which simply correspond to policy rollouts:
- 1' r(s(t)) (((((t))aai(())dd'
Je=t
Mss f t a(s(t) - H))
-usttess
Next, for the continuous-time case, let us define new concurrent Bellman backup operator:
t'=tttss
1,,,,,,,,,,,
==t
As max E, R512
(1 (s(t+ a, a..1, + tAs, tAS).
In addition to expanding the Bellman operator to take into account concurrent actions, we demon- strate that this modified operator maintain its contraction properties that are crucial for Q-learning convergence.
Lemma 3.1. The concurrent continuous-tiin time Bellman operator is contraction.
Proof. See Appendix A.2
Discrete Formulation In order to simplify the notation for the discrete-time case where the dis- tinction between the action function (t(t) and the value of that function time step t, a,(t), is not necessary, refer to the current state, current action, and previous action St, a, a,-1 respec- tively, replacing subindex with Following this notation, we define the concurrent Q-function for the discrete-time case:
Q" ("tss Ot-1, 04, Lss, H)
r(si, 0(-1)011)+ '22 E (e (SitLAs (ftt (4, tas,tasstt tAs,tAs: H - -As)
(8)
Where Ass is the "spillover duration" for action a beginning execution at time + LAs (see Fig- ure [p). The concurrent Bellman operator, specified by subscript c, is as follows:
T: Q(e,, at- at, tss, H)
r(sc, 01-1
-42 maxE p pestigg 1² (stttass tAs, tAs).
0t+1
(9)
Similarly the continuous-time case, demonstrate that this Bellman operator is contraction.
5

---

Published as a conference paper at ICLR 2020
Lemma 3.2. The concurrent discrete-time Bellman operator is contraction.
Proof. See Appendix A.2
We refer the reader to Appendix A. for more detailed derivations of the Q-functions and Bellman operators. Crucially, Equation implies that we can extend a conventional discrete-time Q-learning framework to handle MDPs with concurrent actions by providing the Q function with values of t and a--1, in addition to the standard inputs $t, ag; t.
3.6 DEEP Q-LEARNING WITH CONCURRENT KNOWLEDGE
While we have shown that knowledge of the concurrent system properties (fAs and ar-1. as defined previously for the discrete-time case) is theoretically sufficient, is often hard accurately predict LAs during inference on complex robotics system. In order to allow practical implementation of our algorithm on wide range of RL agents, we consider three additional features encapsulating
concurrent knowledge used to condition the Q-function: (1) Previous action (az-1), (2) Action selection time (t,s), and (3) Vector-to-go (VTG), which we define as the remaining action to be executed at the instant the state is measured. We limit our analysis to environments where i
and VTG are all obtainable and H is held constant. See Appendix A.3 for details.
4 EXPERIMENTS
In our experimental evaluation we aim study the following questions: (1) Is concurrent knowledge defined in Section 3.6, both necessary and sufficient for Q-function to recover the performance of a blocking unconditioned Q-function, when acting a concurrent environment? (2) Which representations of concurrent knowledge are most useful for Q-function to act in concurrent environment? (3) Can concurrent models improve smoothness and execution speed of a real-robot policy in realistic, vision-based manipulation task?
4.1 TOY FIRST-ORDER CONTROL PROBLEMS
First, we illustrate the effects of a concurrent control paradigm on value-based DRL methods through an ablation study on concurrent versions of the standard Cartpole and Pendulum environments. We use 3D MuloCo based implementations in DeepMind Control Suite (Tassa et al. (018)) for both tasks. For the baseline learning algorithm implementations, we use the TF-Agents (Guadarramal al. 2018) implementations
Deep Q-Network agent, which utilizes a Feed-forward Neural Network (FNN), and Deep Q- Recurrent Neutral Network agent, which utilizes Long Short-Tens Memory (LSTM) network. To approximate different difficulty levels of latency in concurrent envi- ronments, utilize different parameter combinations for action execution steps and action selection steps (LAs). The number of action execution steps is selected from [0ms, 5ms, 25ms, or 50ms) once environment initialization. ÍAs selected from {0ms. 5ms, 10ms, 25ms, 50ms) either once at environment initialization or repeatedly at every episode reset. In addition to environment parame- ters, we allow trials to vary across model parameters: number of previous actions store, number of previous states to store, whether to use VTG, whether to use tAS, Q-network architecture, and number of discretized actions. Further details are described in Appendix A.4.I
To estimate the relative importance of different concurrent knowledge representations, we conduct an analysis of the sensitivity of each type of concurrent knowledge representations to combinations of the other hyperparameter values, shown in Figure 2a While all combinations of concurrent knowledge representations increase learning performance over baselines that do not leverage this information, the clearest difference stems from including VTG. In Figure 2b we conduct similar analysis but on a Pendulum environment where tas fixed every environment; thus, we do not focus on ts for this analysis but instead compare the importance of VTG with frame-stacking previous actions and observations. While Crame-stacking helps nominally, the majority of the performance increase results from utilizing information from VTG.

---

Published as a conference paper at ICLR 2020
MMh AG gree action
FRR,,
LSTM,
FMB., prec.che
LST unconditoned
FMB. inconditione
Slocking Lncond Caasline
Eladdngg Uncond Rioved Dasalins
(a) Cartpole
(b) Pendulum
Figure 2: In concurrent versions of Cartpole and Pendulum, observe that providing the critic with VTG leads to more robust performance across all hyperparameters. (a) Environment rewards achieved by DQN with different network architectures [either feedforward network (FNN) Long Short-Term Memory (LSTM) network] and different concurrent knowledge features [Uncon- ditioned, Vector-to-gu go (VTG), or previous action and fAs] on the concurrent Cartpole task for ev- hyperparameter in a sweep, sorted in decreasing order. (b) Environment rewards achieved by DQN with a FNN and different frame-stacking and concurrent knowledge parameters on the con- current Pendulum task for every hyperparameter in sweep, sorted in decreasing order. Larger area-under-cuuve implies more robustness hyperparameter choices. Enlarged figures provided in Appendix A.5
(a) Simulation
(b) Real
Figure 3: An overview of the robotic grasping task. A static manipulator arm attempts to grasp objects placed in bins front it. simulation, the objects are procedurally generated.

---

Published as a conference paper at ICLR 2020
Table Harge-Scals Simulated Robotic Grasping Results
Blocking
Timestep VTG
Previous Grasp Success Action
Episode Duration
Action Completion
Actions
Penalty
Yes
No
No
No
92.72% + 1.10%
132.09s +5.70s
92.33% + 1.476%
Yes
Yes
No
No
91.53% 1.04%
120.81s +9.13s
2
No
No
No
No
84.11% = 7.61%
122.15s #14.6s
43.4% 22.41%
No
Yes
No
No
83.77% 9.27%
97.16s
+6.28s
34.69% 16.80%
No
Yes
Yes
No
92.55% = 4.39%
82.98s
5.74s
47.28% 14.25%
No
Yes
No
Yes
92.70% = 1.42%
87.15s
#4.80s
50.09% = 14.25%
No
Yes
Yes
Yes
93.49% + 1.04%
90.75s +4.15s
49.19% + 14.98%
4.2 CONCURRENT QT-OPT ON LARGE-SCALE ROBOTIC GRASPING
Next, we evaluate scalability our approach to a practical robotic grasping task. We simulate 7 DoF with over-the-shoulder camera, where bin in front of the robot is filled with procedurally generated objects to be picked up by the robot. A binary reward assigned if an object lifted off a bin the end of episode. We train a policy with QT-Opt (Kalashnikov al. 2018), a deep Q-Learning method that utilizes the cross-entropy method (CEM) to support continuous actions. In the blocking mode, displacement action is executed until completion: the robot uses closed-loop controller to fully execute action, decelerating and coming to rest before observing next state. In the concurrent mode, an action is triggered and executed without waiting, which means that the next state is observed while the robot remains in motion. Further details of the algorithm and experimental setup are shown Figure 3 and explained in Appendix A.442
Table 1 summarizes the performance for blocking and concurrent modes comparing unconditioned models against the concurrent knowledge models described in Section 3.6) Our results indicate that the VTG model acting in concurrent mode is able recover baseline task performance of the blocking execution unconditioned baseline, while the unconditioned baseline acting in concurren model suffers some performance loss. In addition to the success rate of the grasping policy, also evaluate the speed and smoothness of the learned policy behavior. Concurrent knowledge models are able to learn faster trajectories: episode duration, which measures the total amount of wall-time used for an episode, is reduced by 31.3% when comparing concurrent knowledge models with blocking unconditioned models, even those that utilize shaped timestep penalty that reward faster policies. When switching from blocking execution mode to concurrent execution mode, we see a significantly lower action completion, measured as the ratio from executed gripper displacement commanded displacement, which expectedly indicates a switch to concurrent environment. The concurrent knowledge models have higher action completions than the unconditioned model the concurrent environment, which suggests that the concurrent knowledge models are able to utilize more efficient motions, resulting in smoother trajectories. The qualitative benefits of faster, smoother trajectories drastically apparent when viewing video playback of learned policies
Real robot results In addition, we evaluate qualitative policy behaviors of concurrent models compared to blocking models on real-world robot grasping task, which is shown Figure 3b. As seen in Table 2 the models achieve comparable grasp success, but the concurrent model is 49% faster than the blocking model in terms of policy duration, which measures the total exccution time of the policy (this excludes the infrastructure setup and teardown times accounted for in episode duration, which can not be optimized with concurrent actions). In addition, the concurrent VTG model is able to execute smoother and faster trajectories than the blocking unconditioned baseline, which is clear in video 2
5 DISCUSSION AND FUTURE WORK
We presented theoretical framework to analyze concurrent systems where agent must "think while movinng.Viewwin this formulation through the lens of continuous-time value-based reinforce-
https: httra:://sitess.ggglee.oomm http://www....ooooo.oooooooootta 1emoviggwwwiiiiiiiiimmmmmmm ng
8

---

Published as a conference paper at ICLR 2020
Table 2: Real-World Robotic Grasping Results.
Blocking Actions
VTG
Grasp Success
Policy Duration
Yes No
No
81.43%
22.60s +12.99s 11.52s + 7.272s
Yes
68.60%
ment learning, showed that by considering concurrent knowledge about the time delay tAs and the previous action, the concurrent continuous-time and discrete-time Bellman operators remained contractions and thus maintained Q-learning convergence guarantees. While more information than tAs and previous action may be helpful, we showed that tAs and previous action (and different representations of this information) are the sole theoretical requirements for good learning perfor- mance. In addition, we introduced Vector-to-go (VTG), which incorporates the remaining previous action be executed, as an alternative representation for information about the concurrent system that previous action and tss contain.
Our theoretical findings were supported by experimental results on Q-learning models acting in simulated control tasks that were engineered to support concurrent action execution. We conducted large-scale ablation studies on toy task concurrent 3D Cartpole and Pendulum environments, across model parameters as well concurrent environment parameters. Our results indicated that VTG the least peperpprramtte-sensttiiiii representation, and was able to recover blocking learning per- formance in concurrent settings. We extended these results to complex concurrent large-scale sienvironmentmulated robotic grasping task, where we showed that the concurrent models were able to recover blocking execution baseline model success while acting 31.3% faster. We analyzed the qualitative benefits of concurrent models through a real-world robotic grasping task, where we showed that concurrent model with comparable grasp success as blocking baseline was able to learn smoother trajectories that were 49% faster.
An interesting topic to explore in future work is the possibility of increased data efficiency when training on off-policy data from various latency regimes. Another natural extension of this work is evaluate DRL methods beyond value-based algorithms, such as on-policy learning and policy gra- dient approaches. Finally, concurrent methods may allow robotic control in dynamic environments where it is not possible for the robot to stop the environment before computing the action. these scenarios, robots must truly think and act the same time.
REFERENCES
Pieter Abbeel, Adam Coates, Morgan Quigley, and Andrew Ng. An application of rein- forcement learning to aerobatic helicopter flight. In Berhaard Schölkopf. John C. Platt, and Thomas Hofmann (eds.), NIPS, pp. 1-8. MIT Press, 2006. ISBN 0-262- 19568-2. URL http://ablp.miitiier..... wwwpiierreeddb/coooo http://///////////ss//2//22// htm html #Abbee CCQN06
Artemij Amiranashvili, Alexey Dosovitskiy, Vladlen Koltun, and Thomas Brox. Motion perception in reinforcement learning with dynamic objects. In CoRL, volume 87 of Proceedings of Machine Learning Research, pp. 156-168 PMLR, 2018. URL htthtt/tp//dlpp www.:hpp........ioom de db/
conf/ cor 12018 html Pamiraaaashii 1DK18
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and
Wojciech Zaremba. Openai gym, 2016. htt.htt:://aaxii.ooo/bb///////// 01540 cite arxiv:160.001400.
Rémi Coulom, Reinforcement learning using neural networks, with applications to motor control. PhD thesis, Institut National Polytechnique de Grenoble-INPGG 2002.
Nicolás Cruz, Kenzo Lobos-Tsunekaww: Tsunekawa, and Javier Ruiz del Solar. Using convolutional neural net- works in robots with limited computational resources: Detecting robots while playing cer. CoRR, abs/ 700 06702, 2017. URL htRthtt:// /tbpp..b...///////// uni- - -trier.re... de/ de/db/ db/ aat/////oaas////aa
coorr/oori 706 html w #CruzLR17

---

Published as a conference paper at ICLR 2020
Kenji Doya.
Reinforcement learning in continuous time and space. Neural Computation, 12(1): 2000. URL attp:://blin uni-trier. de db/ htt.//////oooaaaaaeeeeeeeeeeeeee html html (Doya00)
Vlad Firoiu, Tina Ju, and Joshua Tenenbaum. At Human Speed: Deep Reinforcement Learning with Action Delay. arXiv e-prints, October 2018.
Nicolas Frémaux, Henning Sprekeler, and Wulfram Gerstner. continuous time actor-critic framework with spiking neurons. PLoS computational biology, 9: e1003024, 04 2013. doi: po. 1003024.
Reinforcement learning using
Sergio Guadarrama, Anoop Korattikara, Oscar Ramirez, Pablo Castro, Ethan Holly, Sam Fishman, Ke Wang, Ekaterina Gonina, Chris Harris, Vincent Vanhoucke, et al. Tf-agents: A library for reinforcement learning in tensorflow, 2018.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algo- rithms and applications. CoRR, abs/1812.05905., 2018. URL de/ bee/db/oourraas/corttt cor.ie/coor11112 htmll #abs 1812 05905
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. Qt-opt Scal- able deep reinforcement learning for vision-based robotic manipulation. CoRR, abs/ 806. 10293, 2018. URL http:://tpppppp 10cs-1806-0 10293
ceniitrreer.e http://dddd////o/aaaeaaaaaaa corr www.o/ooooroooooooooooooooooeore horp://ooooaal///o/oooooororr 806 html
Kappen. Path integrals and symmetry breaking for optimal control theory. Journal of Sta- tistical Mechanics: Theory and Experiment, 2005(11):111011-P1000 nov 2005. doi: 10.
1088/1742- 1088/1742-5468/20000 54000////////0/////////11 2F200532F1192ppo
URL https://6b...rrg/0111883221
Shuang Li, Shuai Xiao, Shixiang Zhu, Nan Du, Yao Xie, and Song. Learning temporal point processes via reinforcement learning. In Proceedings of the 32Nd International Conference Neural Information Processing Systems, NIPS' 18, pp. 10804-10814 USA, 2018. Curran Asso- ciates Inc. URI.http://dll
acm wwarr//ciaaaeo 3tt.:::coooo.mmmmmmm333333
wff.
3327737
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. 2013. URL
http:://rxiv www.riiiv.rg/.oeeo wwwg/oor///////... ab/////s//111// 5602 cite arxiv: 1e..:311..222ommeee NIPS Deep Learn-
ing Workshop 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep rein- forcement learning. Nature, 518(7540):599553 533, February 2015. ISSN 00280836. URL http://dx doi. orr. 100g/ 1038 / htt./nauureeeee 12336.
Rémi Munos and Paul Bourgine. tic control problems.
Reinforcement learning for continuous stochas-
In M. I. Jordan, M. Kearns, and S. A. Solla (eds.), Advances in Newral Information Processing Systems 10, pp. 1029 URL httt:://pppers htpp.ccppppprr/
1035. MIT Press, 1998.
1404 wwinforrceentteeeee learning for continuous sonttimuous-stchasttae control problems................e pdf
Alexander Neitz, Giambattista Parascandolo, Stefan Bauer, and Bernhard Schölkopf. Adaptive skip intervals: Temporal abstraction for recurrent dynamical models. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems 31, pp. 9816 9826. Curran Associates. Inc., 2018. URL hatt::/ppapeze nips htt.psscccpaper//// htt.://eeeeiaiaaaa......oooooorr
8188
pdf
10

---

Published as a conference paper at ICLR 2020
OpenAI, Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Józefowicz, Bob McGrew, Jakub W. Pachocki, Jakub Pachocki, Arthur Petron, Matthias Plappert. Glenn Powell, Alex Ray, Jonas Schneider, Szymon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, and Wojciech Zaremba. Learning dexterous in-hand manipulation. CoRR, abs/ 0bbs/180000077, 2018. htthtt.p: //db: uni-trier de/ db/ journa htt//o/r//oorrr/ 808 ht.l #abs 1008-00077
Sheldon Ross, John Kelly, Roger Sullivan, William James Perry, Donald Mercer, Ruth M Davis, Thomas Dell Washburn, Earl V Sager, Joseph B Boyce, and Vincent I. Bristow. Stochastic processes, volume 2. Wiley New York, 1996.
Erik Schuitema, Lucian Busoniu, Robert Babuka, and Pieter P. Jonker. Control delay in reinforce- ment learning for real-time dynamic systems: memoryless approach. 2010 IEEE/RSJ Interna- tional Conference on Intelligent Robots and Systems, pp. 3226-3231, 2010.
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529:484-, January 2016. URL http://dx. doi org/10. 1038 1038/naturee696
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, March 1998. ISBN 0262193981. URL http::ttt//// //wp::///wwww/// aww.on. ca http:////w..//eee/////////o///// extc///icccc////////////////////
Correntin Tallec, Leonard Blier, and Yann Ollivier. Making Deep Q-learning Methods Robust to Time Discretization. arXiv e-prints, January 2019.
Jie Tan, Tingnan Zhang, Erwin Coumans, Atil Iscen, Yunfci Bai, Danijar Hafner, Steven Bohez, and Vincent Vanhoucke. Sim-to-Real: Learning Agile Locomotion For Quadruped Robots. arXiv e-prints, April 2018
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy P. Lillicrap, and Martin A.
Riedmiller. Deepmind control suite. CORR, abs/1801.00649 2018. URL http:://bllp html #abs 1801 00690
uni tiitttrie.com /wwdd////bbbbe httee//b/oooaraaaacooott///o///e
Evangelos Theodorou, Jonas Buchli, and Stefan Schaal. Reinforcement learning of motor skills in high dimensions: A path integral approach. pp. 2397 - 2403, 06 2010. doi: 10. 109//OOTT 2010.5509336.
Utkarsh Upadhyay, Abir De, and Manuel GodezzRodrizuez. Deep reinforcement learning of marked temporal point processes. In Proceedingy of the 32Nd International Conference Neu- ral Information Processing Systems, NIPS' 18, pp. 3172-3182, USA, 2018. Curran Associates Inc. BRRhhtt:://al acm. ottg e 33272211d-3324...11 3327144.3322238 3327238
Eleni Vasilaki, Nicolas Frémaux, Robert Urbanezik, Walter Senn, and Wulfram Gerstner. Spike- based reinforcement learning in continuous state and action space: When policy gradient methods fail. PLOS computational biology, 1:01006 12 2009. doi: 10.133//jorrnal.pct 1000586.
Thomas J. Walsh, Ali Nouri, Lihong Li, and Michael L. Littman. Planning and learning in envi- ronments with delayed feedhack. In Joost N. Kok, Jacek Koronacki, Ramón López de Mántaras, Stan Matwin, Dunja Mladenic, and Andrzej Skowron (eds.), ECML, volume 4701 of Lecture
Notes in Computer Science, pp. 442-453. Springer, 2007. ISBN 978-3-540-74957-8. URL
http:/ /ttp//dbbpppp... uni sai--treree trritrerrrreee. de /t/ http://cccooo///// http::///.....aaaa///ee///////// htm

---

Published as a conference paper at ICLR 2020
A APPENDIX
A.1 DEFINING BLOCKING BELLMAN OPERATORS
As introduced in Section 3.5 actions.
define continuous-time Q-function estimator with concurrent
0z, = d'attess
(10)
V=t
at 7d (("" + (
(11)
(7s))
(12)
="=tHH (asr((((((())))) (, {(""""") +1 Vs(( H))
(13)
1"=HHAS
(14)
1==
+ + -HS + H))
(15)
We observe that the second part of this equation (after *48) is itself a Q-function at time & + LAs- Since the future state, action, and reward values at l Las are not known at time i, we take the following expectation:
(4,,
(16)
Jezt
y (AS)Q((((), az ac+1, L + LAS, H - LAS)
(17)
which indicates that the Q-function in this setting not just the expected sum of discounted future rewards, but it corresponds to expected future Qffuctioon.
In order to show the discrete-time version of the problem, we parameterize the discrete-time con- current Q-function as:
O(5,,0111,000 a,, tAs, H) = r(st, 1-11) + " 1(8
(18)
x E Ep Sp(*c4nleee a.) V((+t())
(19)
which with tas = 0, corresponds to synchronous environment.
Using this parameterization, we can rewrite the discrete-time Q-function with concurrent actions as:
Q(s 01-1, t,s, H) r(st, a 111 + + '28" [ppreetss ( ( 0z)+
(20)
n
Epless V (81+1)]
(21)
r(st, a)- + '4 Eptnssss (n Q(5 Q(5,, a, a1,
t + tAs tao', , - tss
(22)
A.2 CONTRACTION PROOFS FOR THE BLOCKING BELLMAN OPERATORS
Proof of the Discrete-time Blocking Bellman Update
Lemma A.I. The traditional Bellman operator is contraction, i.e.: || Qoo - Qe a)ll < c||Qi a) - Q2(s.
(23)
where T Q(s, a) = r(s,a) a) max, Es, a²) and e 1 1.
12

---

Published as a conference paper at ICLR 2020
Proof. In the original formulation, show that this is the case following:
T'))(8, -2
(24)
=((s,)) + max (11 a')l r(s, max 1p122(s'
(25)
= Ey d') Q2(5' (2(ss'a'')) a')l
(26)
S ysupplu d') Q2 ,')},
(27)
with 0 S S and || sf
Similarly, we can show that the updated Bellman operators introduced in Section 3.5 are contractions well.
Proof of Lemma 3.2
Proof.
To Q1(st, as- 01, ti,t, H) - To 0,,,,, H)
(28)
=r(st; + #
max Bpter 1) QiQi a141, Las, Las,, H LAS)
(29)
act1
-r(st, (1s1) q
tA8
max E (a Q2(000222000000 ai+,, t LAs,AAs H tAs)
(30)
a++1
: max E ppssi han
kaa 1)1Q1 (sc. a.t, tas, tss H tAs) Q2( ai a:41, tts,ttss. H - -Ass
4:11
(31)
tAS
sup
Q1e ai a141, + Las, As, H Las) Q2(sn, ai ac41, Lss bAs, LAs)
41 1--As
(32)
Proof of Lemma 3.1
Proof. To prove that this the continuous-time Bellman operator is contraction, we can follow the discrete-time proof, from which it follows:
To
To 22(s(1), a;-1, tAs)
(33)
=ntss max (s)() ai, a.+1, tas, H tAS) Q2(s(t), ael(tt)),,aas ai-1, tas, H tAS)l
a++1
(34)
S Sae LAS
sup
(s)) a5, 0141 LAS, H H- LAS) - Q2(x(1)) ai 0:11, LAs, , (AS)
a()) -18
(35)
A.3 CONCURRENT KNOWLEDGE REPRESENTATION
We analyze different representations of concurrent knowledge in discrete-time concurrent envi- ronments, described in Section 3.6) Previous action o:-1 action that the agent executed at the previous timestep. Action selection time tAs measure of how long action selection takes, which can be represented as either a categorical or continuous variable; in our experiments, which take advantage of bounded latency regime, normalize action selection time using these known bounds. Vector-to. to-go VTG is a feature that combines Q,-1 and 8; by encoding the remaining amount of a,_ left to execute. See Figure 5 for visual comparison.
We note that 0,-1 is available across the vast majority of environments and is easy to obtain. Using LAs, which encompasses state capture, communication latency, and policy inference, relies
13

---

Published as a conference paper at ICLR 2020
Blocking Actions
Concurrent Actions
Capture
Policy Inference
= As
[= As
t=
= 2H
Action Execution before Selection
1=0
Action Executio during Action Selection
Associativity
= 2H
(a)
(b)
Figure 4: The execution order of different stages are shown relative the sampling period H as well the latency LAS- (a): In "blocking" environments, state capture and policy inference are assumed be instantaneous. (b): In "concurrent" environments, state capture and policy inference are assumed to proceed concurrently to action execution.
a
S + a
S-1
M
Vector-to-go (VTG)
Previcus Action (a
Figure 5: Concurrent knowledge representations can be visualized through an example of 2-D pointmass discrete-time toy task. Vector-to-go represents the remaining action that may be executed when the current state 8; is observed. Previous action represents the full commanded action from the previous timestep.
14

---

Published as a conference paper at ICLR 2020
on having some knowledge of the concurrent properties of the system. Calculating VTG requires having access to some measure of action completion at the exact moment when state is observed. When utilizing first-order control action space, such as joint angle or desired pose, VTG is easily computable proprioceptive state is measured and synchronized with state observation. In these cases, VTG is an alternate representation of the same information encapsulated by 0z-1 and the current state.
A.4 EXPERIMENT IMPLEMENTATION DETAILS
A.4.1 CARTPOLE AND PENDULUM ABLATION STUDIES
Here, we describe the implementation details of the toy task Cartpole and Pendulum experiments in Section [.]
For the environments, we use the 3D MuJoCo implementations of the Cartpole-Swingee and Pendul umm-Swl ngup tasks in DeepMind Control Suite (Tassa et al.] 2018). We use discretized action spaces for first-order control of joint position actuators. For the observation space of both tasks, use the default state space of ground truth positions and velocities.
For the baseline learning algorithms, we use the TensorFlow Agents (Guadarrama et al. 2018) implementations of Deep Q-Network agent, which utilizes Feed-forward Neural Network (FNN), and Deep Q Recurreett Neutral Network agent, which utilizes a Long Short-Term Memory (LSTM) network. Learning parameters such as learning.rate, lstm_size, and fc_layer.size were selected through hyperparameter sweeps.
To approximate different difficulty levels of latency concurrent environments, we utilize different parameter combinations for action execution steps and action selection steps (fAs). The number of action execution steps is selected from {0ms, 5ms, 25ms, or 50ms} once at environment initializa- tion. LAs is selected from (0ms, 5ms, 10ms, 25ms, or 50ms) either once at environment initializa- tion or repeatedly at every episode reset. The selected LAs is implemented in the environment as additional physics steps that update the system during simulated action selection.
Frame-stacking parameters affect the observation space by saving previous observations and actions. The number of previous actions store as well as the number of previous observations to store are independently selected from the range [0, 4]. Concurrent knowledge parameters, as described in Section 4 include whether to VTG and whether to use tas. Including the previous action is already feature implemented in the frame-stacking feature of including previous actions. Finally, the number of actions discretize the continuous space is selected from the range [3, 8).
A.4.2 LARGE SCALE ROBOTIC GRASPING
Simulated Environment We simulate a 7 DoF arm with an wver-the-soullde camera (see Figure 3.. A bin in front of the robot is filled with procedurally generated objects to be picked up by the robot and sparse binary reward is assigned if an object is lifted off a bin at the end of episode. States are represented in form of RGB images and actions are continuous Cartesian displacements of the gripper 3D positions and yaw. In addition, the policy commands discrete gripper open and close actions and may terminate an episode. In blocking mode, a displacement action is executed until completion: the robot a closed loop controller to fully execute an action, decelerating and coming to rest before observing the next state. In concurrent mode, an action is triggered and executed without waiting, which means that the next state is observed while the robot remains in motion. It should be noted that in blocking mode, action completion is close to 100% unless the gripper moves are blocked by contact with the environment or objects; this causes average blocking mode action completion to be lower than 100%, as seen in Table [
Real Environment Similar to the simulated setup, we use 7 DoF robotic arm with an over- the-shoulder camera (see Figure 3p). The main difference in the physical setup is that objects are selected from a set of common houschold objects.
Algorithm We train policy with QT-Opt (Kalashnikov et al. 2018), Deep Q-Learning method that utilizes the Cross-Entropy Method (CEM) to support continuous actions. A Convolutional Neural Network (CNN) is trained to learn the Q-functio conditioned image input along with
15

---

Published as a conference paper at ICLR 2020
CEM-samples continuous control action. At policy inference time, the agent sends image of the environment and batches CEM-sampled actions to the CNN Q-network. The highest-scoring action is then used as the policy's 's selected action. Compared to the formulation in Kalashnikov et al. 2018), we also add a concurrent knowledge feature VTG and/or previous action 04-1 as additional input the Q-network. [gooithmmTshar shows the modified QT-Opt procedure.
Algorithm 1: QT-Opt with Concurrent Knowledge
Initialize replay buffer D;
Initialize random start state and receive image op:
Initialize concurrent knowledge features
[VTGo 0, dz-1 t,t 0j:
Initialize environment state 8 = [o0, col:
Initialize oction-value function (s,, a) with random weights 0;
Initialize target action-value function (s,,a) with weights 0 = A:
while training do
for 1= 1, 7. do
Select random action o, with probability else a, CEM 81; Execute action in environment, receive ot+1, Tzi Process necessary concurrent knowledge features ct such as VTG ar-1, or tAst
Set = = =001=1 [0t+1,Cal]
Store transition (&t, at, st) in D;
if episode terminates then
Reset 8 to a random reset initialization state; Reset C++1 to O;
end
Sample batch of transitions from D; for each transition (8i, 0g rr)) in batch do terminal transition then
else
Select 0,+1 = CEM(Q. 8;;0;
y/ + (1++
end
Perform SGD on (y Q(si, ax; 0)2 with respect to 0;
end
Update target parameters Q with and 0 periodically:
end end
For simplicity, the algorithm described as if run synchronously on single machine. In practice, episode generation, Bellman updates and Q-fitting are distributed across many machines and done asynchronously; refer to (Kalashnikov al. 2018) for more details. Standard DRL hyperparameters such random exploration probability (e), reward discount (7), and learning rate are tuned through a hyperparameter sweep. For the time-penalizee baselines in Table I we manually tune a timestep penalty that returns fixed negative reward at every timestep. Empirically find that timestep penalty - 0.01, relative to a binary sparse reward of 1.0, encourages faster policies. For the non-penalized baselines, we set timestep penalty of -0.0.
A.S FIGURES
See Figure 6 and Figure 7
16

---

Published as a conference paper at ICLR 2020
1000
FNN, VTG
FNN. LS prev prev_action action
FNN, Unconditioned
530
LSTM, VTG
t. prev
LSTM, Unconditioned
Blocking Unconditianed Baseline
Reward
Percentage of Runs
Figure 6: Environment rewards achieved by DQN with different network architectures [either feedforward network (FNN) or a Long Short-Term Memory (LSTM) network] and different con- current knowledge features [Unconditioned, vector-to-ge (VTG), or previous action and Ass] on the concurrent Cartpole task for every hyperparameter in a sweep, sorted in decreasing order. Providing the critic with VTG information leads to more robust performance across all hyperparameters. This figure larger version of
630
Reward
420
FNN,
FNM, prev_a
FNN, prev. obs
FNN, Unconditioned
Blocking Unconditioned Baseline
Percentage of Runs
Figure 7: Environment rewards achieved by DQN with FNN and different Stame-stacking and concurrent knowledge parameters on the concurrent Pendulum task for every hyperparameter in sweep, sorted in decreasing order.
17


Reviews:
[{"review_id": "rygdpEwhKS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper tackles the problem of making decisions for the next action while still engaged in doing the previous actions. Such a delay could either be part of the design (like a robot deciding the next action before its actors and motors have come to full rest after the current action) or an artefact of the delays inherent in the system (i.e. latency induced by calculations or latency of sensors). The paper shows how to model such delays within the Q-learning framework, show that their modelling preserves desirable contraction property of the Bellman update operator, and put their model into practice by an extensive set of experiments: learning policies for several simulated and a real-world setting.\n\nThe authors claim that that addition of this \"delay\" does not hinder the performance much of the RL method is given sufficient \"context\" about the delay, i.e., given extra features as input in order to learn to compensate for it. The writing of the paper is lucid and sufficient background is provided to make the paper self-sufficient in its explanations.\n\nHowever, there are some reasons which do not allow me to fully support the paper's acceptance.\n\nThe changes made to the basic Q-learning setup, albeit novel and with desirable properties, in my opinion, are (i) theoretically relatively straight forward, (ii) are not expressive enough to capture the problem in its full generality (explained later), and (iii) need more empirical justification with problems where their modification is indeed indispensable. The authors touch on several different research areas cursorily (viz. continuous reinforcement learning, Bellman contractions, feature engineering) while providing grounds for their idea, but in the end return to the familiar domain of discrete Q-learning with semi-hand-crafted (though theoretically motivated) features where the latency of actions can take a set of fixed values and the state is sampled at fixed intervals. \n\nIf the actions are continuous, then could method from Doya (2000) directly be used to solve these problems? Can the value-based models which he describes be augmented and extensions developed which build on Lemma 3.1 instead of the well-trodden ground of Lemma 3.2? Especially, if one of the objectives which the authors claim their policies are better is \"policy duration\", then the absence of purely continuous policies is particularly egregious. Further, reducing the policy duration seems like an independent objective which perhaps can be used for reward shaping for the traditional policy methods, which will also lead to different baselines.\n\nThe authors explicitly say that their method focuses on \"optimizing for a specific latency regime as opposed to being robust to all of them;\" and that they explicitly avoid learning forward models by including additional features. However, the advantages of placing such restrictions on the design space are unclear at best. Would it be the case that the high-dimensional methods will fail in this setting? Are there theoretical advantages to working on limiting the attention to known latency regimes? I suspect that the authors have concrete reasons for making these design decisions, but these do not come across in the paper in the writing, or by means of additional baselines.\n\nAs an example of a different approach towards the problem, which the authors overlook in their related work section, is that of learning with spiking neurons and point processes. These areas of research have also been interested in problems of the \"thinking while moving\" nature: that of reinforcement learning in the context of neurons where the neurons act by means of spikes in response to the environment and other \"spikes\" [1, 2]. More recently, with point processes, methods have been developed to attain truly asynchronous action and state updates [3, 4]. A differently motivated work which ends up dealing with similar problems is in the direction of adaptive skip intervals [5], where the network also chooses the \"latency\" in the discrete sense. Adding such related work would help better contextualize this paper.\n\nSome other ways the authors can improve the paper are (in no particular order):\n\n - The description of the Vector-to-go is insufficient; some concrete examples will help.\n - The results of the simulated experiments are given in the form of distributions and it is very difficult to discern the effect of individual features in Figure 1. Additionally, due to missing error bars, or other measures of uncertainty, the claim that the performance of models with and without the delayed-actions is comparable to the blocking setting seems tenuous at best, just looking at the rewards.\n - In particular, for the real experiments, we need more details about the experiment runs to determine why the performance of the policies in the real world is so vastly different. Could the authors describe why the gap can be completely covered through simulations but not in the real world?\n\n[1]: Vasilaki, Eleni, et al. \"Spike-based reinforcement learning in continuous state and action space: when policy gradient methods fail.\" PLoS computational biology 5.12 (2009): e1000586.\n[2]: Frémaux, Nicolas, Henning Sprekeler, and Wulfram Gerstner. \"Reinforcement learning using a continuous time actor-critic framework with spiking neurons.\" PLoS computational biology 9.4 (2013): e1003024.\n[3]: Upadhyay, Utkarsh, Abir De, and Manuel Gomez Rodriguez. \"Deep reinforcement learning of marked temporal point processes.\" Advances in Neural Information Processing Systems. 2018.\n[4]: Li, Shuang, et al. \"Learning temporal point processes via reinforcement learning.\" Advances in Neural Information Processing Systems. 2018.\n[5]: Neitz, Alexander, et al. \"Adaptive skip intervals: Temporal abstraction for recurrent dynamical models.\" Advances in Neural Information Processing Systems. 2018.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "ryg6d_-MqB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers the theoretically interesting and practically important problem of concurrent deep reinforcement learning (DRL), i.e., DRL in which the agent has to decide the next action while performing the previous one. This introduces several significant challenges, including delays/latency and interruption of an on-going action. To address this issue, this paper proposes to consider the continuous time formulation of the concurrent control problem, derive a continuous-time Bellman equation for the concurrent control scenarios, and then derive its discrete-time counterpart. Contraction properties are shown for both the continuous-time and discrete-time concurrent Bellman equations, and a value-based DRL algorithm based on the concurrent Bellman equations is proposed and tested on a few tasks. \n\nThe high level idea of this paper is very interesting and attractive, and in particular, the introduction of continuous-time reinforcement learning is novel. In addition, the numerical experiments do show that a consistently improved performance of the proposed approach on both synthetic and more real-world robotic control tasks. However, there are several significant issues about technical clarity or even correctness in this paper, which I elaborate below:\n\n1. The settings in sections 3.1 and 3.2 are not clear. In particular, for 3.1, the author may want to specify the policy clearly, including whether it is stationary or non-stationary. And in addition, Q and V functions should either come with a \\pi superscript, indicating which policy they use, or a \\star superscript to indicate optimality. Section 3.2 does not make sense to me in general. It is not clear what the index i and the state value s_i(t) are. And it is not clear why we need to differentiate between values of states/actions and the functions themselves. The trajectory \\tau is also not clearly defined. The authors need to make these much more clear, and should clearly state the main setting/model that the paper is considering (which seems to be the concurrent discrete-time case, but also not very clear to me).\n\n2. The explanation of concurrent actions in continuous and discrete time is not clear. In particular, Section 3.3 only speaks of the settings on a high level, and only brief explanations are given in Figure 1b and the beginning of Section 3.4. Since the concurrent action setting is the central theme in this paper, I think a much more formal explanation should be given about how the system proceeds, instead of just a graphical example illustration. In addition, the concurrent actions in discrete-time setting part is not even clearly mentioned (but is stated in the title of Section 3.3 and discussed subsequently). The authors may also want to explain clearly what the episode is at the beginning of Section 3.4.\n\n3. The concurrent Bellman equation does not make much sense to me. In particular, I think to define the optimal Q function, the bellman equation (7) and (9) should have a \\max operator included. Otherwise, it is only for policy evaluation. Since the authors didn't clearly specify what the exact algorithm they are using (apart from a brief explanation by words in Section 3.5), I'm not sure whether I'm missing anything or not. But the authors should definitely include a algorithm frame at least in the appendix, to clearly specify which of and how the concurrent Bellman equations are applied in their algorithm.\n\nSo in sum, although I think the paper is interesting and novel on the high level, I don't think it's ready for publishing.\n\n############ post rebuttal comment ############\nAfter reading the authors' rebuttal and the modified version of the paper, I think most of my concerns have been correctly addressed. So I decide to improve my score to weak accept.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "ByxoiRtUcB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a first step in the direction of 'real-world relevant RL approaches' in the sense of considering environments that don't halt their execution until an agent has finished its optimal action computation and execution but actually just go on being an environment. For this, the notion of a concurrent action is introduced.\n\nThe paper focuses on value-based RL approaches. It introduces modifications to the classical MDP formulation such that concurrent actions can be handled. From a theoretical perspective the resulting Bellman operators (for both continuous and discrete time) remain contractions and thus maintain q-learning convergence guarantees. Qlearning models are adopted to support concurrent actions and the experiments  demonstrate that the suggested enhancements are working well.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the setting in reinforcement learning where the next action must be sampled while the current action is still executing. This refers to continuous time problems that are discretised to make them delay-aware in terms of the time taken for action execution. The paper presents adaptions of the Bellman operator and Q-learning to deal with this scenario.

This is a problem that is of theoretical interest and also has practical value in many real-world problems. The reviewers found both the problem setting and the proposed solution to be valuable, particularly after the greatly improved technical clarity in the rebuttals. As a result, this paper should be accepted.
Decision: Accept (Poster)