Event Discovery for History Representation in Reinforcement Learning

1 INTRODUCTION . Deep Reinforcement Learning ( RL ) algorithms have been successfully applied to a range of challenging domains , from computer games ( Mnih et al. , 2013 ) to robotic control ( OpenAI et al. , 2018 ) . These approaches use a Neural Network ( NN ) both to represent the current observation of the environment and to learn the agent ’ s optimal policy , used to choose the next action . For instance , the state observation can be the current game frame or an image of the robot camera , and a Convolutional Neural Network ( CNN ) may be used to obtain a compact feature vector from it . However , often RL environments are only partially observable and having a significant representation of the past may be crucial for the agent ( Kapturowski et al. , 2019 ) . For instance , in the Atari 2600 Pong game , the state can be represented as two consecutive frames . In this way , the agent can determine both the position of the ball and the direction of its movement . More complex partially observable domains require a longer state history input to the agent . For instance , when navigating inside a 1-st person-view 3D labyrinth , the agent obtains little information from the current scene observation and needs several previous frames to localise itself . A common solution to represent the observation history is based on the use of Recurrent Neural Networks ( RNNs ) , where the RNN hidden-layer activation vector is input to the agent , possibly together with the current state observation ( Mnih et al. , 2016 ) . However , RL is characterized by highly nonstationary data , which makes training unstable ( Schulman et al. , 2016 ) and this instability is exacerbated when a recurrent network needs to be simultaneously trained to extract the agent ’ s input representation . We show in Sec . 5.2 that in some common cases an RNN-based history representation struggles to improve the agent ’ s results over an input representation composed of only the instantaneous observations . In this paper we propose a different direction , in which the agent ’ s observation history is represented using a set of discrete events , which describe important changes in the state of the world ( Orr et al. , 2018 ) . Environment-specific events are automatically discovered during training by clustering past observations and are then used as landmarks in our history representation . Intuitively , this is inspired by the common human behaviour : when making decisions , humans do not keep detailed visual information of the previous steps . For instance , while navigating through the halls of a building , it is sufficient to recall a few significant landmarks seen during the walk , e.g . specific doors or furniture . Following this idea , in this paper we propose an Event Discovery History Representation ( EDHR ) approach , composed of 3 iterative stages : experience collection , policy optimisation and event discovery . We discover events by clustering past state observations . In more detail , we maximize the Mutual Information ( MI ) between the latent representations of temporally-close frames to cluster the frame semantics . This is a form of self-supervision , in which no additional annotation is needed for the event discovery : the higher the predictability of one frame with respect to the other , the larger the semantic information shared by the two frames ( van den Oord et al. , 2018 ; Hjelm et al. , 2019 ; Anand et al. , 2019 ; Ji et al. , 2019 ) . Once clusters have been formed , the probability distribution of a given frame Ft for the set of current clusters is used as the semantic representation of the state observation at time t and is recorded in a longer history . Finally , the history is input to the agent together with an instantaneous observation representation , obtained , following ( Mnih et al. , 2013 ) , as a stack of the last 4 frames . This information is now used for policy optimisation . Note that our proposed history representation is independent of the specific RL approach . However , in all our experiments we use the PPO algorithm ( Schulman et al. , 2017 ) for the policy and the value function optimisation . The 3 stages are iterated during training , and thus past clusters can be modified and adapted to address new observations , while the agent is progressing through the environment . We summarize the contribution of this paper below . First , we use a modified version of the Invariant Information Clustering ( IIC ) algorithm ( Ji et al. , 2019 ) to discover significant events in the agent ’ s past observations . Second , we propose to replace common RNN-based history representations with a time-dependent probability distribution of the observations for the events so far discovered . We evaluate our history representation ( EDHR ) using the PPO algorithm on several environments of the Atari-57 benchmark ( Bellemare et al. , 2012 ) and on the 3D environment Obstacle Tower ( Juliani et al. , 2019 ) , showing that it provides a significant boost with respect to the most common input representation methods . Specifically , we show that EDHR outperforms plain PPO in cases , where history can increase observability , and it outperforms RNN-based methods in several common cases , while simultaneously requiring 2× less wall-clock time for training . The source code of the method and all the experiments is publicly available at the anonymous link : https : //github.com/iclr2020anon/EDHR and will be published after acceptance . 2 RELATED WORK . Partially Observable Markov Decision Processes . We consider a setting in which an agent interacts with an environment to maximize a cumulative reward signal . The interaction process is defined as a Partially Observable Markov Decision Process ( POMDP ) < S , A , T , R , Ω , O , γ > ( Monahan , 1982 ; Kaelbling et al. , 1998 ) , where S is a finite state space , A is a finite action space , R ( s , a ) is the reward function , T ( ·|s , a ) is the transition function , Ω is a set of possible observations , O is a function which maps states to probability distributions over observations and γ ∈ [ 0 , 1 ) is a discount factor used to compute the cumulative discounted reward . Within this framework , at each step t , the environment is in some unobserved state s ∈ S. When the agent takes an action a , the environment transits to the new state s′ with probability T ( s′|s , a ) , providing the agent with a new observation o′ ∈ Ω with probability O ( o′|s′ ) and reward r ∼ R ( s , a ) . POMDPs are an active research area and several approaches have been recently proposed to address partial observability . DQN ( Mnih et al. , 2013 ) is one of the earliest deep RL models directly trained using the high-dimensional observation ot as input ( i.e. , the raw pixels of the current frame ) . To deal with partial observability , the input of the model is composed of a stack of the 4 last grayscale frames , processed as 4 channels using a 2D CNN . We also use this observation representation which we call instantaneous to highlight the difference from the past state history . Hausknecht & Stone ( 2015 ) replace the fully-connected layer in DQN with an LSTM ( Hochreiter & Schmidhuber , 1997 ) layer . At each step , the model receives only one frame of the environment together with the LSTM hidden-layer value which represents the past . The idea was extended in ( Igl et al. , 2018 ) by inferring latent belief states . To simulate reduced observability and demonstrate the robustness of the method , Hausknecht & Stone ( 2015 ) introduce the flickering Atari games , obscuring the observation with a certain probability at each step . These methods focus on shortterm partial observability , such as noisy observations , and approximate the state with an implicitly inferred continuous history . In contrast , our method explicitly represents each step of the past using a dictionary of discovered events , and concatenates a sequence of steps in an event-based history . Our final goal is to augment the current state representation with important past changes of the environment . Recurrent layers ( e.g . LSTM or GRU ( Cho et al. , 2014 ) ) , are a common choice for modern RL architectures . The A3C algorithm ( Mnih et al. , 2016 ) showed a high mean performance with an LSTM layer . R2D2 ( Kapturowski et al. , 2019 ) is specifically focused on processing long-history sequences ( 80 steps with additional 40 burn-in steps ) using an LSTM . However , despite its widespread , RNNs have disadvantages within an RL framework because they increase the training instability ( Sec . 1 ) . In this paper we propose an alternative history representation and we empirically show in Sec . 5.2 that our proposal outperforms RNN-based solutions in several common cases . Unsupervised and Self-supervised Learning . Unsupervised and self-supervised learning aim to extract useful representations from data ( e.g . images , videos ) without annotation and reward signal . While common approaches are covered in recent surveys ( Jing & Tian , 2019 ; Kolesnikov et al. , 2019 ) , below we analyse the most relevant methods to this work . A first class of unsupervised feature-extraction methods is based on a generative encoding-decoding scheme . The methods typically minimize the reconstruction error in the pixel space , thus the representation is encouraged to capture pixel-level details , which may differ from semantics . Ha & Schmidhuber ( 2018 ) combine Variational Autoencoders ( Kingma & Welling , 2014 ) ( VAEs ) with RNNs in their World Models . A VAE is trained to reconstruct the current frame and simultaneously produce a latent , compressed frame representation which is input to an RNN predicting the next latent feature vector . The input of the RL agent consists of the latent vector of the current observation and the hidden state of the RNN . An alternative to a reconstruction loss in the input space is clustering in the feature space . DeepCluster ( Caron et al. , 2018 ) iteratively alternates between k-means clustering of the CNN features and updating the network ’ s parameters with the pseudo-labels obtained by means of clustering . Another important self-supervised paradigm is based on maximizing the Mutual Information ( MI ) between latent representations . Typically these methods rely on collecting pairs of positive and negative samples . Two positive samples should share the same semantics , while negatives should be perceptually different . A discriminator is trained to distinguish between positive and negative pairs , so computing a lower-bound on the MI of the learned latent representations . In CPC ( van den Oord et al. , 2018 ) and ST-DIM ( Anand et al. , 2019 ) positives are temporally close frames , while in IIC ( Ji et al. , 2019 ) simple data-augmentation techniques are applied on still images . In DIM ( Hjelm et al. , 2019 ) this idea is extended including locality , and MI is computed also between global and local representations of the same image . DIM was applied to several games of the Atari 2600 suite ( Bellemare et al. , 2012 ) in ST-DIM , however , the reported evaluation is based only the disentanglement degree of the obtained representations and does not consider the possible benefit for an RL task . Our event discovery stage ( Sec . 3 ) is based on IIC ( Ji et al. , 2019 ) , but we use as positives temporallyclose frames extracted from an agent trajectory instead of data-augmentation . Note that the main difference between our proposal and IIC is that our goal is not clustering , but clusters are used as a set of discrete events to represent the agent ’ s history , in contrast to RNN and other continuous history representations .

Summary: This paper proposes a new way to represent past history as input to an RL agent, that consists in clustering states and providing the (soft) cluster assignment of past states in the input. The clustering algorithm comes from previous work based on mutual information, where close (in time) observations are assumed to be semantically similar. The proposed scheme, named EDHR (Event Discovery History Representation), is shown to perform better than PPO and an RNN variant of PPO on (most of) 7 representative Atari games, and better than PPO on the Obstacle Tower benchmark.