Thinking While Moving: Deep Reinforcement Learning with Concurrent Control

1 INTRODUCTION . In recent years , Deep Reinforcement Learning ( DRL ) methods have achieved tremendous success on a variety of diverse environments , including video games ( Mnih et al. , 2015 ) , zero-sum games ( Silver et al. , 2016 ) , robotic grasping ( Kalashnikov et al. , 2018 ) , and in-hand manipulation tasks ( OpenAI et al. , 2018 ) . While impressive , all of these examples use a blocking observe-think-act paradigm : the agent assumes that the environment will remain static while it thinks , so that its actions will be executed on the same states from which they were computed . This assumption breaks in the concurrent real world , where the environment state evolves substantially as the agent processes observations and plans its next actions . As an example , consider a dynamic task such as catching a ball : it is not possible to pause the ball mid-air while waiting for the agent to decide on the next control to command . In addition to solving dynamic tasks where blocking models would fail , thinking and acting concurrently can provide benefits such as smoother , human-like motions and the ability to seamlessly plan for next actions while executing the current one . Despite these potential benefits , most DRL approaches are mainly evaluated in blocking simulation environments . Blocking environments make the assumption that the environment state will not change between when the environment state is observed and when the action is executed . This assumption holds true in most simulated environments , which encompass popular domains such as Atari ( Mnih et al. , 2013 ) and Gym control benchmarks ( Brockman et al. , 2016 ) . The system is treated in a sequential manner : the agent observes a state , freezes time while computing an action , and finally applies the action and unfreezes time . However , in dynamic real-time environments such as real-world robotics , the synchronous environment assumption is no longer valid . After observing the state of the environment and computing an action , the agent often finds that when it executes an action , the environment state has evolved from what it had initially observed ; we consider this environment a concurrent environment . In this paper , we introduce an algorithmic framework that can handle concurrent environments in the context of DRL . In particular , we derive a modified Bellman operator for concurrent MDPs and ∗Indicates equal contribution . present the minimal set of information that we must augment state observations with in order to recover blocking performance with Q-learning . We introduce experiments on different simulated environments that incorporate concurrent actions , ranging from common simple control domains to vision-based robotic grasping tasks . Finally , we show an agent that acts concurrently in a real-world robotic grasping task is able to achieve comparable task success to a blocking baseline while acting 49 % faster . 2 RELATED WORK . Minimizing Concurrent Effects Although real-world robotics systems are inherently concurrent , it is sometimes possible to engineer them into approximately blocking systems . For example , using low-latency hardware ( Abbeel et al. , 2006 ) and low-footprint controllers ( Cruz et al. , 2017 ) minimizes the time spent during state capture and policy inference . Another option is to design actions to be executed to completion via closed-loop feedback controllers and the system velocity is decelerated to zero before a state is recorded ( Kalashnikov et al. , 2018 ) . In contrast to these works , we tackle the concurrent action execution directly in the learning algorithm . Our approach can be applied to tasks where it is not possible to wait for the system to come to rest between deciding new actions . Algorithmic Approaches Other works utilize algorithmic modifications to directly overcome the challenges of concurrent control . Previous work in this area can be grouped into five approaches : ( 1 ) learning policies that are robust to variable latencies ( Tan et al. , 2018 ) , ( 2 ) including past history such as frame-stacking ( Haarnoja et al. , 2018 ) , ( 3 ) learning dynamics models to predict the future state at which the action will be executed ( Firoiu et al. , 2018 ; Amiranashvili et al. , 2018 ) , ( 4 ) using a time-delayed MDP framework ( Walsh et al. , 2007 ; Firoiu et al. , 2018 ; Schuitema et al. , 2010 ) , and ( 5 ) temporally-aware architectures such as Spiking Neural Networks ( Vasilaki et al. , 2009 ; Frémaux et al. , 2013 ) , point processes ( Upadhyay et al. , 2018 ; Li et al. , 2018 ) , and adaptive skip intervals ( Neitz et al. , 2018 ) . In contrast to these works , our approach is able to ( 1 ) optimize for a specific latency regime as opposed to being robust to all of them , ( 2 ) consider the properties of the source of latency as opposed to forcing the network to learn them from high-dimensional inputs , ( 3 ) avoid learning explicit forward dynamics models in high-dimensional spaces , which can be costly and challenging , ( 4 ) consider environments where actions are interrupted as opposed to discrete-time time-delayed environments where multiple actions are queued and each action is executed until completion . The approaches in ( 5 ) show promise in enabling asynchronous agents , but are still active areas of research that have not yet been extended to high-dimensional , image-based robotic tasks . Continuous-time Reinforcement Learning While previously mentioned related works largely operate in discrete-time environments , framing concurrent environments as continuous-time systems is a natural framework to apply . In the realm of continuous-time optimal control , path integral solutions ( Kappen , 2005 ; Theodorou et al. , 2010 ) are linked to different noise levels in system dynamics , which could potentially include latency that results in concurrent properties . Finite differences can approximate the Bellman update in continuous-time stochastic control problems ( Munos & Bourgine , 1998 ) and continuous-time temporal difference learning methods ( Doya , 2000 ) can utilize neural networks as function approximators ( Coulom , 2002 ) . The effect of time-discretization ( converting continuous-time environments to discrete-time environments ) is studied in Tallec et al . ( 2019 ) , where the advantage update is scaled by the time discretization parameter . While these approaches are promising , it is untested how these methods may apply to image-based DRL problems . Nonetheless , we build on top of many of the theoretical formulations in these works , which motivate our applications of deep reinforcement learning methods to more complex , vision-based robotics tasks . 3 VALUE-BASED REINFORCEMENT LEARNING IN CONCURRENT ENVIRONMENTS . In this section , we first introduce the concept of concurrent environments , and then describe the preliminaries necessary for discrete- and continuous-time RL formulations . We then describe the MDP modifications sufficient to represent concurrent actions and finally , present value-based RL algorithms that can cope with concurrent environments . The main idea behind our method is simple and can be implemented using small modifications to standard value-based algorithms . It centers around adding additional information to the learning algorithm ( in our case , adding extra information about the previous action to a Q-function ) that allows it to cope with concurrent actions . Hereby , we provide theoretical justification on why these modifications are necessary and we specify the details of the algorithm in Alg . 1 . While concurrent environments affect DRL methods beyond model-free value-based RL , we focus our scope on model-free value-based methods due to their attractive sample-efficiency and off-policy properties for real-world vision-based robotic tasks . 3.1 CONCURRENT ACTION ENVIRONMENTS . In blocking environments ( Figure 4a in the Appendix ) , actions are executed in a sequential blocking fashion that assumes the environment state does not change between when state is observed and when actions are executed . This can be understood as state capture and policy inference being viewed as instantaneous from the perspective of the agent . In contrast , concurrent environments ( Figure 4b in the Appendix ) do not assume a fixed environment during state capture and policy inference , but instead allow the environment to evolve during these time segments . 3.2 DISCRETE-TIME REINFORCEMENT LEARNING PRELIMINARIES . We use standard reinforcement learning formulations in both discrete-time and continuous-time settings ( Sutton & Barto , 1998 ) . In the discrete-time case , at each time step i , the agent receives state si from a set of possible states S and selects an action ai from some set of possible actions A according to its policy π , where π is a mapping from S toA . The environment returns the next state si+1 sampled from a transition distribution p ( si+1|si , ai ) and a reward r ( si , ai ) . The return for a given trajectory of states and actions is the total discounted return from time step i with discount factor γ ∈ ( 0 , 1 ] : Ri = ∑∞ k=0 γ kr ( si+k , ai+k ) . The goal of the agent is to maximize the expected return from each state si . The Q-function for a given stationary policy π gives the expected return when selecting action a at state s : Qπ ( s , a ) = E [ Ri|si = s , ai = a ] . Similarly , the value function gives expected return from state s : V π ( s ) = E [ Ri|si = s ] . The default blocking environment formulation is detailed in Figure 1a . 3.3 VALUE FUNCTIONS AND POLICIES IN CONTINUOUS TIME . For the continuous-time case , we start by formalizing a continuous-time MDP with the differential equation : ds ( t ) = F ( s ( t ) , a ( t ) ) dt+G ( s ( t ) , a ( t ) ) dβ ( 1 ) where S = Rd is a set of states , A is a set of actions , F : S × A → S and G : S × A → S describe the stochastic dynamics of the environment , and β is a Wiener process ( Ross et al. , 1996 ) . In the continuous-time setting , ds ( t ) is analogous to the discrete-time p , defined in Section 3.2 . Continuous-time functions s ( t ) and ai ( t ) specify the state and i-th action taken by the agent . The agent interacts with the environment through a state-dependent , deterministic policy function π and the return R of a trajectory τ = ( s ( t ) , a ( t ) ) is given by ( Doya , 2000 ) : R ( τ ) = ∫ ∞ t=0 γtr ( s ( t ) , a ( t ) ) dt , ( 2 ) which leads to a continuous-time value function ( Tallec et al. , 2019 ) : V π ( s ( t ) ) = Eτ∼π [ R ( τ ) |s ( t ) ] = Eτ∼π [ ∫ ∞ t=0 γtr ( s ( t ) , a ( t ) ) dt ] , ( 3 ) and similarly , a continuous Q-function : Qπ ( s ( t ) , a , t , H ) = Ep [ ∫ t′=t+H t′=t γt ′−tr ( s ( t′ ) , a ( t′ ) ) dt′ + γHV π ( s ( t+H ) ) ] , ( 4 ) where H is the constant sampling period between state captures ( i.e . the duration of an action trajectory ) and a refers to the continuous action function that is applied between t and t +H . The expectations are computed with respect to stochastic process p defined in Eq . 1 .

Summary: This paper considers the theoretically interesting and practically important problem of concurrent deep reinforcement learning (DRL), i.e., DRL in which the agent has to decide the next action while performing the previous one. This introduces several significant challenges, including delays/latency and interruption of an on-going action. To address this issue, this paper proposes to consider the continuous time formulation of the concurrent control problem, derive a continuous-time Bellman equation for the concurrent control scenarios, and then derive its discrete-time counterpart. Contraction properties are shown for both the continuous-time and discrete-time concurrent Bellman equations, and a value-based DRL algorithm based on the concurrent Bellman equations is proposed and tested on a few tasks.