Dynamic Planning Networks

1 INTRODUCTION . The central focus of reinforcement learning ( RL ) is the selection of optimal actions to maximize the expected reward in an environment where the agent must rapidly adapt to new and varied scenarios . Various avenues of research have spent considerable efforts improving core axes of RL algorithms such as performance , stability , and sample efficiency . Significant progress on all fronts has been achieved by developing agents using deep neural networks with model-free RL ( Mnih et al. , 2015 ; 2016 ; Schulman et al. , 2015 ; 2017 ; OpenAI , 2018 ) ; showing model-free methods efficiently scale to high-dimensional state space and complex domains with increased compute . Unfortunately , modelfree policies are often unable to generalize to variances within an environment as the agent learns a policy which directly maps environment states to actions . A favorable approach to improving generalization is to combine an agent with a learned environment model , enabling it to reason about its environment . This approach , referred to as modelbased RL learns a model from past experience , where the model usually captures state-transitions , p ( st+1|st , at ) , and might also learn reward predictions p ( rt+1|st , at ) . Usage of learned statetransition models is especially valuable for planning , where the model predicts the outcome of proposed actions , avoiding expensive trial-and-error in the actual environment – improving performance and generalization . This contrasts with model-free methods which are explicitly trial-anderror learners ( Sutton & Barto , 2017 ) . Historically , applications have primarily focused on domains where a state-transition model can be easily learned , such as low dimensional observation spaces ( Peng & Williams , 1993 ; Deisenroth & Rasmussen , 2011 ; Levine & Abbeel , 2014 ) , or where a perfect model was provided ( Coulom , 2006 ; Silver et al. , 2016a ) – limiting usage . Furthermore , application to environments with complex dynamics and high dimensional observation spaces has proven difficult as state-transition models must learn from agent experience , suffer from compounding function approximation errors , and require significant amounts of samples and compute ( Oh et al. , 2015 ; Chiappa et al. , 2017 ; Guzdial et al. , 2017 ) . Fortunately , recent work has overcome the aforementioned difficulties by learning to interpret imperfect model predictions ( Weber et al. , 2017 ) and learning in a lower dimensional state space ( Farquhar et al. , 2017a ) . Planning in RL has used state-transition models to perform simulated trials with various styles of state traversal such as : recursively expanding all available actions per state for a fixed depth ( Farquhar et al. , 2017a ) , expanding all actions of the initial state and simulating forward for a fixed number of steps with a secondary policy ( Weber et al. , 2017 ) , or performing many simulated roll- outs with each stopping when a terminal state is encountered ( Silver et al. , 2016a ) . An issue arises within simulated trials when correcting errors in action selection , as actions can either be undone by wasting a simulation step , using the opposing action , or are irreversible , causing the remaining rollout steps to be sub-optimal in value . Ideally , the agent can step the rollout backwards in time thereby undoing the poor action and choosing a better one in its place . Additionally , during rollouts the agent is forced to either perform a fixed number of steps or continue until a terminal state has been reached ; when ideally a rollout can terminate early if the agent decides the path forward is of low value . In this paper , we propose an architecture that learns to dynamically control a state-transition model of the environment for planning . By doing so , our model has greater flexibility during planning allowing it to efficiently adjust previously simulated actions . We demonstrate improved performance against both model-free and planning baselines on varied environments . The paper is organized as follows : Section 2 covers our architecture and training procedure , Section 3 covers related work , Section 4 details the experimental design used to evaluate our architecture , and in Section 5 we analyze the experimental results of our architecture . 2 DYNAMIC PLANNING NETWORK . In this section , we describe DPN , a novel planning architecture for deep reinforcement learning ( DRL ) . We first discuss the architecture overview followed by the training procedure . Steps taken in the environment use subscript t and steps taken during planning use subscript τ . We provide additional motivation behind the architecture in Appendix A.1 . 2.1 DPN ARCHITECTURE . The architecture is comprised of an inner agent , an outer agent , a shared encoder , and a learned state-transition model . Figure 1 illustrates a high-level diagram of the DPN architecture . The outer agent ( OA ) is a feed-forward network and the inner agent ( IA ) is based on a recurrent neural network ( RNN ) . The architecture interacts with the environment by observing raw environment states st ∈ S and outputting actions at ∈ A via OA . However , before OA outputs an action at the IA performs τ = 1 , ... , T steps of planning by interacting with an internal simulated environment ; where this simulated environment is defined by the state-transition model and sub-section of OA ’ s network . Selection of an action at by OA uses the final hidden state hIT ∈ R1×h i of IA and an embedding of the current state zt ∈ R1×z . The objective of IA is to maximize the total utility provided to the OA ; where utility , given in Equation 1 , measures the “ value of information ” provided to OA if it were to have undergone a state transition from zτ to zτ+1 . Uτ ( hOτ+1 , hOτ , aτ , zτ ) = Q̂ ( zτ , aτ ) DKL ( hOτ+1||hOτ ) Q̂ ( zτ , aτ ) = π ( aτ |zτ ; θI ) Q ( zτ , aτ ; θO ) ( 1 ) where zτ+1 is the state transitioned to after performing an action aτ in state zτ , hOτ ∈ R1×h o and hOτ+1 are the hidden states of OA after perceiving the current state zτ and state transitioned to zτ+1 respectively , DKL is the KullbackLeibler ( KL ) distance measure , Q ( zτ , aτ ; θO ) is the value OA assigns to taking action aτ in the current state zτ , and π ( aτ |zτ ; θI ) is the learned policy of IA . The DKL is used to measure the distance in bits between hidden states of the OA . Computation of DKL between hidden states is provided in the Appendix B . The utility function captures the tension between choosing a state with high value , as described by the Q-function component , and a state that provides maximal information , described by the KL component . To maximize the utility the IA provides to the OA it must select states and actions that have both high value and provide information to the OA – maximizing the “ value of information ” . States with high value but low information or low value and high information will have low utility and therefore be less desirable . Therefore , to maximize utility for OA during each planning step τ , IA must select appropriate simulated-states z∗τ and actions a ∗ τ . A simulated-state z ∗ τ is selected from one of three embedded states tracked during planning : the previous zpτ , current z c τ , and root states z r τ ; with the triplet written as z { p , c , r } τ for convenience . Initially , z { p , c , r } τ=0 is set to an embedding zt produced by the encoder of the initial raw state st as zτ=0 = encoder ( st ) . The encoder is comprised of a series of convolutional layers specific to each environment and provided in the Appendix B . Before planning begins , OA ’ s hidden state is updated using zτ=0 : hOτ = W zhzτ ( 2 ) where W zh ∈ Rz×ho is a learnable parameter of OA with biases omitted . Within this work , we consider the intermediate activation from the OA , a feed-forward network , as a hidden state . The simulated-action a∗τ mirrors those available to OA in the environment , such that a ∗ τ ∈ A . 2.2 A PLANNING STEP At each planning step τ , shown in Figure 2 , the IA selects a simulated-state z∗τ and action a ∗ τ by considering the previous hidden state hIτ−1 , the triplet of embedded states z { p , c , r } τ , a scalar representing the current planning step τ/T , and OA ’ s hidden state hOτ given z c τ . The information is concatenated together forming a context and is fed into IA , a recurrent network producing an updated hidden state hIτ ∈ R1xh i . The updated hidden state is used to select the simulated-state z∗τ by multiplying z { p , c , r } τ with a 1-hot encoded weight wτ ∈ { 0 , 1 } 1×3 sampled from the Gumbel-Softmax distribution , G : wτ ∼ G ( W h3hIτ ) z∗τ = wτ [ z p τ , z c τ , z r τ ] ( 3 ) where W h3 ∈ Rhi×3 is a learnable parameter belonging to IA and G is the Gumbel-Softmax distribution ( GSD ) . Where the GSD is a continuous relaxation of the discrete categorical distribution giving access to differentiable discrete variables ( Jang et al. , 2016 ; Maddison et al. , 2016 ) . Empirically , we found that using a 1-hot encoding for the weight wτ gives greater performance than a softmax activation . Therefore , we used GSD in place of softmax activations throughout our architecture . Next , the simulated-action a∗τ ∈ { 0 , 1 } 1×A , is sampled as follows : a∗τ ∼ G ( W azh [ z∗τ , hIτ ] ) ( 4 ) where W azh ∈ Ra×z+hi is a learnable parameter of IA . In Equation 4 the selected simulated-state z∗τ and IA ’ s hidden state h I τ are concatenated , passed through a linear layer , and used as logits for GSD . Then , with the selected simulated-state z∗τ and simulated-action a ∗ τ , we produce the next state zτ+1 using the state-transition model , defined as : z′ = zτ + tanh ( W zzz∗τ ) z′′ = z′ + tanh ( ( a∗τW azz ) z′ ) z∗τ+1 = zτ + z ′′ ( 5 ) where W zz ∈ Rz×z and W azz ∈ RA×z×z are learnable parameters of the state-transition model . We parameterize each available action in A with a learned weight matrix that carries information about the effect of taking an action a∗τ ∈ R1×A . We use the same state-transition model presented by Farquhar et al . ( 2017a ) . Finally , the three embedded states are updated as : zpτ+1 = z c τ , z r τ+1 = zτ=0 , and zcτ+1 = z ∗ τ+1 . 2.3 ACTION SELECTION . The IA repeats the process defined in Section 2.2 of selecting z∗τ and a ∗ τ for T steps before finally emitting a final hidden state hIT summarizing the result of planning . The OA uses IA ’ s final hidden state hIT and its initial hidden state h O τ=0 to select an action at : at = W ahtanh ( W hhhIT + h O τ=0 ) ( 6 ) where W hh ∈ Rho×hi and W ah ∈ RA×ho are learnable parameters of OA . Finally , the hidden state of the IA is reset .

Summary: Quality: This paper proposes learning to plan approach that can learn to search with an inner agent; conditioned on the output of the inner agent (IA) the outer agent starts to learn a reactive policy in the environments. The inner agent, different from other searching agent, learns to decide what searching pattern to choose. The presented method shows better computation efficiency than competitive baselines. The main applications are on "box pushing" game and grid-world navigation.