Sequence-level Intrinsic Exploration Model for Partially Observable Domains

1 INTRODUCTION . Under the reinforcement learning formalism , the learning behavior of an agent is driven by the reward that the agent collects from the environment ( Sutton and Barto , 1998 ) . However , many real-world problems have sparse rewards and most existing algorithms struggle with such sparsity . One inherent reason that leads to the inferior performance of the conventional approaches in sparse reward domains is that initially , the agent trained with those approaches could hardly stumble into a reward/goal state by chance due to their simple exploration strategies ( Pathak et al. , 2017 ) . To tackle the sparse reward problems , it is crucial to incentivize the agent ’ s exploration behavior . One prominent line of solutions for encouraging agent ’ s exploration is via reward shaping ( Singh , 1992 ; Dorigo and Colombetti , 1994 ) , where the agent develops internal reward models to assign additional reward signals apart from the environment reward to encourage exploration . To model the internal reward signal , often , the agent ’ s curiosity-driven behaviors are formalized as intrinsic novelty models ( Schmidhuber , 1991 ; Singh et al. , 2004 ; Oudeyer et al. , 2007 ) , which characterize agent ’ s experience to compute the novelty scores . Our work belongs to the broad category of methods that solve the sparse reward problems with novelty models and reward shaping . Specifically , we consider the line of sparse reward problems that employ partially observable inputs , with the inputs scaling to high-dimensional state spaces , such as images . Such problems cover a range of important applications among AI research , e.g. , navigation , robotics control and video game playing . Even though the recently emerged intrinsic novelty models have demonstrate considerable efficiency in solving sparse reward problems with partial observability , we still face the following two major challenges . First , inferring the novelty for the true state given only the partial observations still remains an open problem . Most of today ’ s Action sequence with lengthObservation sequence with length state-of-the-art novelty models ( e.g. , ( Savinov et al. , 2019 ; Pathak et al. , 2017 ) ) only derive the novelty from local information , e.g. , concatenation of few recent frames . Second , though prediction error has been widely adopted as an effective metric to infer novelty , most of the existing approaches develop novelty model upon short-term prediction error such as 1-step look-ahead . Such short-term prediction task might be an inadequate proxy for representing the novelty over state space , i.e. , it might be too simple and thus result in inferior novelty scores . Our key motivations are as follows . First , sequence-level novelty models are desired to reason over the partially observable states with greater efficiency . Second , the novelty model should consider longer-term prediction than self-prediction or 1-step look-ahead , to infer more meaningful novelty scores . Based on the above intuitions , this work proposes a new sequence-level novelty model for partially observable domains with the following two distinct properties . First , we introduce a dual-LSTM architecture to reason over a sequence of past transitions to construct the novelty model . Second , we infer the novelty of a state from the prediction error of open-loop multi-step forward dynamics prediction , which is crucial to derive high quality novelty scores . 2 METHODOLOGY . Partially Observable Markov Decision Process ( POMDP ) generalizes MDPs by learning under partial observability . Formally , a POMDP is defined as a tuple 〈S , A , O , T , Z , R〉 , where S , A and O are the spaces for the state , action and observation , respectively . The transition function T ( s , a , s′ ) = p ( s′|s , a ) specifies the probability for transiting to state s′ after taking action a at state s. The observation function Z ( s , a , o ) = p ( o|s , a ) defines the probability of receiving observation o after taking action a at state s. The reward functionR ( s , a ) defines the real-valued environment reward issued to the agent after taking action a at state s. Under partial observability , the state space S is not accessible by the agent . Thus , the agent performs decision making by forming a belief state bt from its observation space O , which integrates the information from the entire past history , i.e. , ( o0 , a0 , o1 , a1 , ... , ot , at ) . The goal of reinforcement learning is to optimize a policy π ( bt ) that outputs an action distribution given each belief state bt , with the objective of maximizing the discounted cumulative rewards collected from each episode , i.e. , ∑∞ t=0 γ trt , where γ ∈ ( 0 , 1 ] is a real-valued discount factor . 2.1 INTRINSIC EXPLORATION FRAMEWORK . We now describe our proposed sequence-level intrinsic novelty model for partially observable domains with high-dimensional inputs ( i.e. , images ) . Our primary focuses are the tasks where the external rewards rt are sparse , i.e. , zero for most of the times . This motivates us to engage a novelty function to infer the novelty over the state space and assign reward bonus to encourage exploration . The novelty function is derived from a forward-inverse dynamics model . Figure 1 depicts a high-level overview of our proposed sequence-level novelty computation . To infer the novelty of a state at time t , we perform reasoning over a sequence of transitions with length L+H . Intuitively , we use a sequence of H consequent observation frames together with a sequence of actions with length L which are taken following the observation sequence , to predict the forward dynamics . As such , the novelty model performs open-loop multi-step forward prediction . By setting the length of the action sequence , i.e. , L , our proposed paradigm could lead to forward dynamics prediction tasks with varying difficulty . To process the input sequences , we propose a dual-LSTM architecture as shown in Figure 2 . Overall , each raw observation and action data are first projected by their corresponding embedding modules . Then LSTM modules are adopted over the sequences of observation/action embeddings to derive the sequential observation/action features . Then the sequential observation/action features are synthesized in a specific form of ht , which serves as the latent representation for the past transitions at time t and is employed as input to predict forward dynamics f ( ht ) . The error of the forward dynamics prediction is used to estimate the novelty r+t of the state at time t. Furthermore , to make the latent features over the past transitions more informative , we also incorporate an inverse dynamics prediction model finv to predict the action distributions . Overall , the proposed dual-LSTM architecture enables us to perform sequence-level reasoning and inferring novelty from the multi-step forward prediction . 2.2 SEQUENCE ENCODING WITH DUAL-LSTM ARCHITECTURE . The sequence encoding module accepts a sequence of observations with length H and a sequence of actions with length L as input . Formally , we denote the observation sequence and action sequence by Ot = ot−L−H−1 : t−L−1 and At = at−L−1 : t−1 , respectively . Specifically , each observation ot is represented as a 3D image frame with width m , height n and channel c , i.e. , ot ∈Rm×n×c . Each action is modeled as a 1-hot encoding vector at∈R|A| , where |A| denotes the size of the action space . Given the sequences Ot and At , the sequence encoding module first adopts an embedding module fe ( · ) parameterized by θE = { θEo , θEa } to process the observation sequence and the action sequence as follows , φOt = fe ( Ot ; θEo ) and φ A t = fe ( At ; θEa ) , ( 1 ) where θEo and θEa denote the parameters for the observation embedding function and the action embedding function , respectively . Next , LSTM encoders are applied to the output of the observation/action embedding modules as follows , [ hot , c o t ] = LSTMo ( φOt , h o t−1 , c o t−1 ) and [ hat , c a t ] = LSTMa ( φAt , h a t−1 , c a t−1 ) , ( 2 ) where hot ∈ Rl and hat ∈ Rl represent the latent features encoded from the observation sequence and action sequence . For simplicity , we assume hot and h a t have the same dimensionality . c o t and c a t denote the cell output for the two LSTM modules . Next , the sequence features for the observation/action hot and h a t are synthesized to derive latent features ht which describe the past transitions . Intuitively , the form of ht is proposed as follows : hitrt = h o t hat and ht = [ hot , hat , hitrt ] . ( 3 ) To compute ht , an multiplicative interaction is first performed over hot and h a t , which results in h itr t and denotes element-wise multiplication . Then ht is derived by concatenating the multiplicative interaction feature hitrt with the latent representations for the observation and action sequences , i.e. , hot and h a t . The reason for generating ht in this way is that the prediction task over the partial observation ot is related to both the local information conveyed in the two sequences themselves ( i.e. , hot and h a t ) , as well as the collaborative information derived via interacting the two sequence features in a form . The reason for performing multiplicative interaction is that the advancement of such operation in synthesizing different types of features has been validated in prior works ( Oh et al. , 2015 ; Ma et al. , 2019 ) . We demonstrate that generating ht in the proposed form is effective and crucial to derive a desirable policy learning performance in the ablation study ( Figure 7c ) of the experiment section . 2.3 COMPUTING NOVELTY . To compute the novelty , the latent features ht are first employed as input by a feedforward prediction function to predict the forward dynamics : ψ̂t = f ( ht ; θF ) and ψ∗t = f∗ ( ot ) , ( 4 ) where f ( · ) is the forward prediction function parameterized by θF , and ψ̂t denotes the prediction output . We use ψ∗t to denote the prediction target , which is computed from some target function f∗ ( · ) . Within the proposed novelty framework , the target function f∗ ( · ) could be derived in various forms , where the common choices include the representation of ot at its original feature space , i.g. , image pixels , and the learned embedding of ot , i.e. , fe ( · ; θEo ) . Apart from the conventional choices , in this work , we employ a target function computed from a random network distillation model ( Burda et al. , 2019 ) . Thus , f∗ ( · ) is represented by a fixed and randomly initialized target network . Intuitively , it forms a random mapping from each input observation to a point in a k-dimensional space , i.e. , f∗ : Rm×n×c → Rk . Hence the forward dynamics model is trained to distill the randomly drawn function from the prior . The prediction error inferred from such a model is related to the uncertainty quantification in predicting some constant zero function ( Osband et al. , 2018 ) . The novelty of a state is inferred from the uncertainty evaluated as the MSE loss for the forward model . Formally , at step t , a novelty score or reward bonus is computed in the following form : r+ ( Ot , At ) = β 2 ||ψ∗t − ψ̂t|| 2 2 , ( 5 ) where β ≥ 0 is a hyperparameter to scale the reward bonus . The reward bonus is issued to the agent in a step-wise manner . During the policy learning process , the agent maximizes the sum over the external rewards and the intrinsic rewards derived from the novelty model . Therefore , the overall reward term to be maximized as will be shown in ( 8 ) is computed as rt = ret + r + t , where r e t denotes the external rewards from the environment .

Summary: This paper extends the prediction-error based model by Pathak et. al., 2019 by learning a forward (and inverse) dynamics model for predicting a state feature multiple steps into the future (say, K-steps) given an open loop sequence of K actions as opposed to 1 step into the future, with the caveat that instead of using learnable state features, a random network is used for computing state features, similar to Random Network Distillation (RND) by Burda et. al., 2019. Also, their inverse dynamics models predicts the entire sequence of actions up to K steps. Experiments on VizDoom point-navigation tasks show that the proposed model does better than baselines as rewards get sparser. Ablations are provided to justify the choice of K in multi-step prediction, the choice of inverse dynamics and the choice of RND state features.