Event Discovery for History Representation in Reinforcement Learning
Authors: Aleksandr Ermolov
Enver Sangineto
Nicu Sebe
Venue:  (2020)
Keywords: reinforcement learning
self-supervision
POMDP

Abstract: Environments in Reinforcement Learning (RL) are usually only partially observable. To address this problem, a possible solution is to provide the agent with information about past  observations. While common methods represent this history using a Recurrent Neural Network (RNN), in this paper we propose an alternative representation which is based on the record of the past events observed in a given episode. Inspired by the human memory, these events describe only important changes in the environment and, in our approach, are automatically discovered using self-supervision.
 We evaluate our history representation method using two challenging RL benchmarks: some games of the Atari-57 suite and the 3D environment Obstacle Tower. Using these benchmarks we show the advantage of our solution with respect to common RNN-based approaches.

Under review as a conference paper at ICLR 2020
EVENT DISCOVERY FOR HISTORY REPRESENTATION IN REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Environments in Reinforcement Learning (RL) are usually only partially observ- able. To address this problem, a possible solution is to provide the agent with in- formation about past observations. While common methods represent this history using a Recurrent Neural Network (RNN), in this paper we propose an alterna- tive representation which is based on the record of the past events observed in a given episode. Inspired by human memory, these events describe only important changes in the environment and, in our approach, are automatically discovered using eeff-supervisio. We evaluate our history representation method using two challenging RL. benchmarks: some games of the Atari-57 suite and the 3D) envi- ronment Obstacle Tower. Using these benchmarks we show the advantage of our solution with respect to common approaches.
INTRODUCTION
Deep Reinforcement Learning (RL) algorithms have been successfully applied range of chal- lenging domains, from computer games (Mnih et (013) to robotic control (OpenAl et al. (218)). These approaches use a Neural Network (NN) both represent the current observation of the envi- ronment and to learn the agent's optimal policy, used to choose the next action. For instance, the state observation can be the current game frame or an image of the robot camera, and Convolutional Neural Network (CNN) may be used to obtain compact feature vector from it.
However, often RL environments are only partially observable and having a significant representa- tion of the past may be crucial for the agent (Kapturowski et al., 2019). For instance, in the Atari 2600 Pong game, the state can be represented as two consecutive frames. In this way, the agent can determine both the position of the ball and the direction of its movement. More complex partially observable domains require longer state history input to the agent. For instance, when navigating inside 1-st person-view 3D labyrinth, the agent obtains little information from the current scene observation and needs several previous frames to localise itself.
A common solution to represent the observation history is based on the use of Recurrent Neural Networks (RNNs), where the RNN hidden-layer activation vector is input to the agent, possibly together with the current state observation (Mnih al. 2016). However, RL is characterized by highly nonstationary data, which makes training unstable (Schulman et al. (016) and this instability exacerbated when recurrent network needs to be simultaneously trained extract the agent's input representation. We show in Sec. 5.2 that in some common cases an RNN-based history rep- resentation struggles improve the agent's is results over an input representation composed of only the instantaneous observations. In this paper we propose different direction, in which the agent's 's observation history is represented using set of discrete events, which describe important changes in the state of the world (Orr et al. 2018). Environment-specifie events are automatically discovered during training by clustering past observations and are then used as landmarks in our history rep- resentation. Intuitively, this is inspired by the common human behaviour: when making decisions, humans do not keep detailed visual information of the previous steps. For instance, while navigating through the halls of building, it is sufficient recall few significant landmarks seen during the walk, e.g. specific doors or furniture.
Following this idea, in this paper propose Event Discovery History Representation (EDHR) approach, composed of 3 iterative stages: experience collection, policy optimisation and event dis- covery. We discover events by clustering past state observations. In more detail, we maximize the

---

Under review as a conference paper at ICLR 2020
Mutual Information (MI) between the latent representations of temporally-close frames cluster the frame semantics. This is a form of self-supervision. in which no additional annotation is needed for the event discovery: the higher the predictability of one frame with respect to the other, the larger the semantic information shared by the two frames (van den Oord et al. 2018 Hjelm et al. 2019. Anand et al. 2019 ei al. 2019)).
Once clusters have been formed, the probability distribution of a given frame F; for the set of cur- rent clusters is used as the semantic representation of the state observation at time and is recorded in a longer history. Finally, the history is input to the agent together with an instantaneous obser- vation representation, obtained, following (Mnih al.] 2013), as a stack of the last 4 frames. This information is now used for policy optimisation. Note that our proposed history representation is independent of the specific RI. approach. However, in all our experiments we use the PPO algorithm (Schulman et al. 2017) for the policy and the value function optimisation. The 3 stages are iterated during training, and thus past clusters can be modified and adapted to address new observations, while the agent is progressing through the environment.
We summarize the contribution of this paper below. First, we use a modified version of the Invariant Information Clustering (IIC) algorithm (Ji et al.) 2019) to discover significant events in the agent's s past observations. Second, we propose to replace common time-dependent probability distribution of the observations for the events so far discovered. We evaluate our history representation (EDHR) using the PPO algorithm on several environments of the Atari-57 benchmark (Bellemare al. 2012) and the 3D environment Obstacle Tower (Juiaai iet al., 2019), showing that it provides a significant boost with respect to the most common input representation methods. Specifically, we show that EDHR outperforms plain PPO in cases, where history can increase observability, and it outperforms RNN-based methods in several common cases, while simultaneously requiring 2x less wall-clock time for training.
history representations with
The source code of the method and all the experiments is publicly available at the anonymous link: https///gitabb.comm////222/ooooo
and will be published after acceptance.
2 RELATED WORK
Partially Observable Markov Decision Processes. We consider a setting in which an agent inter- acts with an environment to maximize cumulative reward signal. The interaction process is defined as a Partially Observable Markov Decision Process (POMDP) < S, A, T, R, 92 0,7 > (Monahan 1982 Kaelbling et al. 1998). where S is a finite state space, A is finite action space, R(s a) is the reward function, T(1s, a) is the transition function, Ω is a set of possible observations, O is a function which maps states to probability distributions over observations and 7 E [0, 1) is a discount factor used to compute the cumulative discounted reward. Within this framework, at each step t. the environment is in some unobserved state E S. When the agent takes an action a, the environment transits to the new state s' with probability T(s' |s, a), providing the agent with a new observation of E Ω with probability O(d'ls') and reward r R(s,a).
POMDPs are an active research area and several approaches have been recently proposed to address partial observability. DQN (Mnih al. 2013) one of the earliest deep RL models directly trained using the high-dimensional observation o, as input (i.e., the raw pixels of the current frame). To deal with partial observability, the input of the model is composed of a stack of the 4 last grayscale frames, processed as 4 channels using 2D CNN. We also use this observation representation which we call instantaneous to highlight the difference from the past state history.
Hausknecht & Stone (2015) replace the fully-connected layer in DQN with an LSTM (Hochreiter & Schmidhuber] 1997) layer. each step, the model receives only one frame of the environment together with the LSTM hidden-layer value which represents the past. The idea was extended in (Igl et al. (0118) by inferring latent belief states. To simulate reduced observability and demonstrate the robustness of the method, Hauskkechh & Stone (2015) introduce the flickering Atari games, obscuring the observation with a certain probability at each step. These methods focus on short- term partial observability, such as noisy observations, and approximate the state with implicitly inferred continuous history. In contrast, our method explicitly represents each step of the past using dictionary of discovered events, and concatenates sequence of steps in an event-based history.
2

---

Under review as a conference paper at ICLR 2020
Our final goal is to augment the current state representation with important past changes of the environment.
Recurrent layers (c.g. LSTM GRU (Cho et al. 2014)), are common choice for modern RL chitectures, The A3C algorithm [Miih ct al. 2016) showed high mean performance with an LSTM layer. R2D2 (Kapturowski al. 2019) is specifically focused on processing long-history sequences (80 steps with additional 40 burn-in steps) using an LSTM. However, despite its widespread, RNNs have disadvantages within an RL framework because they increase the training instability (Sec. I In this paper we propose an alternative history representation and we empirically show in Sec. 5.2 that our proposal outperforms RNN-based solutions several common cases.
Unsupervised and Self-supervised Learning. Unsupervised and self-supervised learning aim to extract useful representations from data (e.g. images, videos) without annotation and reward signal. While common approaches are covered in recent surveys (Jing & Tian 2019; Kolesnikov et al., 2019). below we analyse the most relevant methods to this work.
A first class of unsupervised scheme. The methods typically minimize the reconstruction error the pixel space, thus the rep- resentation is encouraged to capture pixel-level details, which may differ from semantics. Ha & Schmidhuber (2018) combine Variational Autoencoders (Kingma & Welling (014)) (VAEs) with RNNs in their World Models. A VAE is trained to reconstruct the current frame and simultancously produce a latent, compressed frame representation which is input to an RNN predicting the next la- tent feature vector. The input of the RL agent consists of the latent vector of the current observation and the hidden state of the RNN.
methods is based on a generative encoding mecoding-eecodin
An alternative to reconstruction loss in the input space is clustering in the feature space. Deep- Cluster (Caron et al., 2018) iteratively alternates between k-means clustering of the CNN features and updating the network's 's parameters with the pseudo-labels obtained by means of clustering.
Another important self-supervised paradigm is based on maximizing the Mutual Information (MI) between latent representations. Typically these methods rely on collecting pairs of positive and negative samples. Two positive samples should share the same semantics, while negatives should be perceptually different. discriminator is trained distinguish between positive and negative pairs, so computing a lower-boume on the MI of the learned latent representations. In CPC (van den Oord et al., 2018) and ST-DIM (Anand al.) 2019) positives are temporally close frames, while in IIC (Ji et al.] 2019) simple data-augmentation techniques are applied on still images. In DIM (Hjelm et al.J 2019) this idea is extended including locality, and MI is computed also between global and local representations of the same image. DIM applied several games of the Atari 2600 suite (Bellemare et al. 2012) in ST-DIM, however, the reported evaluation is based only the disentanglement degree of the obtained representations and does not consider the possible benefit for an RL task.
Our event discovery stage (Sec. 3 is based on IIC (Ji al. 2019). but close frames extracted from an agent trajectory instead of data-augmentation. Note that the main difference between proposal and IIC is that goal is not clustering, but clusters are used as a set of discrete events to represent the agent's 's history, in contrast to RNN and other continuous history representations.
positives temporally-
3 EVENT DISCOVERY AND HISTORY REPRESENTATION
Event discovery stage. As mentioned in Sec. 12 we use IIC (Ji et al. 2019) for our unsuper- vised event discovery. While we refer the reader to the Appendix C and the original paper for the details, we show below how we use this clustering method to obtain a set of discrete events and simultaneously learn to encode our observations.
Let or be the observation of the environment at time step f, presented as an image. We want to learn a representation mapping & Ω - [0, _J where Y = [1, C} is set of C' events and ( is the probability distribution a discrete random variable 2 which foot-ins o, to the elements in Y and can be defined as: (o:) p(2 clo,) (Ji et al. 2019). (-)) is our unsupervised encoder and it is based on a CNN followed by a fully-connected layer and a softmax layer (Fig. [ parametrized by o. The clustering process in IIC is based on pairs of images randomly transformed which contain
3

---

Under review as a conference paper at ICLR 2020
softmax
CNN
FC
)(o))
max MI
softmax
Oc+
CNN
FC
((+++)
Figure I: Event discovery. This figure shows two replicas of our unsupervised encoder architecture (Q) which separately map two different observations in corresponding event probability distribu- tions. The latter are used to maximize the MI over the latent assignment variable 2, in this way obtaining of events Y and the supervision signal used to update the parameters (ő) of $.
similar semantics. In our case, we use two observations extracted from the same trajectory: or and Ot+k. where ~ U[L, L}. defines a temporal translation window in which observations are assumed to mostly like have the same semantic content. Small values of L (c.g. L 1) reduces the uncertainty and thus improves the stability during training. On the other hand, larger values of produce higher intra-cluster variability, in which the clustering process is forced to extract more abstract semantics to group similar frames. This parameter can vary for different RL environments, additionally we demonstrate the effect of adjusting
in our experiments in Sec. 5.2 this parameter in Appendix B
we use
Following (Ji et al. 2019), clustering is performed by computing C x C matrix whose elements represent the joint probability P(z = c c' oz+k) and then maximizing the MI directly using this matrix.
81
I(t)
1/2
CNN
FC
policy value
C
concat
S+1
(t''
(t)
A
Figure 2: The full agent model and our history representation.
History Representation and Policy Optimisation. Once & is updated (see Sec. we use p(or; to represent observation at time t and update the agent's policy. In more detail, we represent the agent with an Actor-Critic architecture approximated with NN я parametrised by We use the PPO algorithm to optimise the policy and the value function. At time step t, the agent concate- receives two types of data as input, an instantaneous representation I(t) consisting nated raw-pixel, grayscale frames I(t) for, o1-, 0r-2, 0t-3) and history H representing past events. Specifically, for each E {t.... S + 13 let o(or:; σ) be the probability distribu- tion of oy for the current of events Y. Our proposed history representation is simply given
4

---

Under review as a conference paper at ICLR 2020
Algorithm Training
I Initialize o and 0.
2 For iteration = =,...N do:
3
For actor 11...... do:
4
For f = I..... r do:
Observation collection stage:
5
Run no (((),, (()) in the environment and get O++1
6
Use ((0++1: Φ) to compute H(t + 1).
7
Collect [[t + 1), (t 1) and other PPO-specific information in buffer B.
Policy optimisation stage:
8 Use B update 0, and A in np epochs
Event discovery stage:
9 Use B update & in ne epochs
by the concatenation of these cluster soft-assignment vectors, computed the past S steps: (t) = ()) ((o ,)}, arranged as a S x matrix.
We empirically observed that the history matrix (t) is very sparse (Sec. 5.3 being most of the elements in ((+)) close to 0 (0() is usually very peaked). Thus, an even more compact history representation may be produced. Anyway, with small value of S (S 32 in all experiments) and small number of clusters C' (we use C = 16), (t) is a low-dimensional, sparse matrix which represents significant information about the recent agent's past trajectory. Importantly, in contrast with an RNN-based history representation, which may be prone to forgetting past information, in our proposed history representation (t), all the past events are stored, together with their time- dependent evolution.
T()) is input second CNN (V) which extracts vector 2 representing the instantaneous visual information (Vt is obtained by flattening the last CNN feature map). We call the parameters of v or, differentiate them from the agent's proper parameters 0 which control the policy and the value function (0 T0r; OA]). Finally, is concatenated with the (flattened) matrix (t) and input to the agent. Fig. shows the full architecture of our agent. We highlight that the CNN of the unsupervised encoder (γ) is different from V, and the two CNNs do not share parameters (ø * 81). Moreover, while the encoder is trained in an unsupervised fashion, V, used to extract visual information (v,) from T()), is trained using the RL reward. This emphasizes the complementary between the knowledge of the two networks, being the former based on clustering visual similarities in data which do not depend on the specific RL task, while the latter being trained in a standard RL framework.
4 TRAINING
We run the agent in several parallel environments for predefined number of steps (rollout size r). The actions are chosen according to the current policy and encoder (observation collection stage). Next, using the collected experience, we update 0 for epochs (policy optimisation stage) and for ng epochs (event discovery stage). The second and the third stages (described in Sec. BccB. are each other independent and can be performed in parallel, which removes the event discovery runtime overhead. These three stages are iterated, so to adapt +(.) (and Y) to the observation distribution change. The full algorithm is described in Alg.
4.1 IMPLEMENTATION DETAILS
The weights and 0 are initialized with a (semi-) orthogonal matrix, described in (Saxe et al. 2014), apart from the head (i.c., the fully-connected layer on top of the CNN, see Fig. l). which initialized randomly. All the biases are initialized with zeros.
Following (ii et al., 2019), to improve robustness, the head of is duplicated 5 times with differ- ent initial random parameters. Moreover, 4 includes an auxiliary overclustering head with C clusters. Each head produces a different representation with potentially different degree utility for the agent. A possible solution is the simple concatenation of the 5 different event probability
5

---

Under review as a conference paper at ICLR 2020
distributions 41 (P))...)))(()) However, this solution would increase the dimensionality of (t). We instead select the "best" head using the MI value as the selection criterion as follows. We split training in two phases. In the first phase (N Ppretrain iterations of the outmost loop in Alg. [ the agent receives only (() as input and 0 and & (and each head in b) are updated as usual (lines 8 and 9 in Alg. D Then, the head corresponding to the highest MI value (averaged over the last iterations) is selected and kept fixed during the second phase, in which the agent receives the full input (M (t) included).
In all our experiments we use the same set of hyperparameters and the same network architecture. We adopt the training procedure and the hyperparameters in (Schulman et al. (017). including the number of time steps equal to 10M. Our agent network architecture is similar to (Schulman et al.) 2017], except from the additional history input (H())). + composed of CNN and a fully- connected layer (Sec. (3). Specifically, its CNN configuration is similar to V except the first layer, which takes only one channel as input. Appendix A we list all other hyperparameters.
5 EXPERIMENTAL EVALUATION
We evaluate EDHR on two challenging deep RL benchmarks: the Arcade Learning Environment (Bellemare et al. 2012) (ALE) and the Obstacle Tower (Juliani et al.] 2019). Moreover, we compare the results of EDHR with PPO and PPO-RNN, where the latter described below.
PPO-RNN. V is the same as in EDHR and in PPO (Schulman et al. 2017). However, past observa- tions represented follows. The encoder + is replaced with similar network (T) but without the softmax layer and having 256 neurons as output of the fully-connected layer instead of C. For each /' E {l....( L- S + 1]. wi T(ov) is input a GRU (Cho al. 2014) with hidden layer of size 512 (re-initiated with zeros each S steps). Finally, the hidden layer activation vector represents the history and replaces (t) in Fig. 2 keeping all the rest of the agent unchanged. Note that no self culf-superviiion is used here to train the network components.
5.1 ENVIRONMENTS
In ALE we use 7 classic Atari 2600 video games: MsPacman, SpaceInvaders, Breakout, Gravitar, Obert, Seaquest and Enduro. Empirically, balanced complexity and diverse game dynamics make this subset of the benchmark representative for comparison. In all these environments, og is RGB image. The action space A is discrete and the number of actions varies in each game from 4 to 18. Following closely the experimental protocol in (Schulman et al. 2017), we apply the same observation pre-processing (frame downscaling, etc.).
Obstacle Tower a 3D 3-rd person game environment. It consists of a number of floors, each floor being composed of several rooms. The layout is procedurally generated based on an environment seed. The rooms can contain puzzles to solve, the complexity of which increases with the floor num- ber, creating natural curriculum learning for the agent. The environment has high visual fidelity, realistic lighting, a variety of textures and geometry, physics-driver interactions. Possible actions include movements in four directions, camera rotations and jumps. The agent receives reward of +1.0 after passing each floor and an auxiliary reward of +0.1 after solving an intermediate puzzle. An observation o, is 84 x 84 RGB image, converted to grayscale. The action space is discrete, with 54 actions, which are reduced to 12 after removing redundancy. This environment is especially challenging because of several reasons. Unlike the Atari games with simple sprite-hased graphics, this environment has diverse and realistic observations. The sparse reward signal complicates train- ing and favours a low-dimensional agent's input. The agent trained using environment seeds 0-7 and it is evaluated with seeds 50-57, which shows the generalisation abilities of the algorithm (see the Train and Test rows in Tab. respectively).
5.2 RESULTS
In all the experiments we report the final scores averaged over 100 episodes. Each experiment is performed 3 times with different random seed. We report the average value and the standard deviation. Tab. show the results. "PPO (Sch.)" refers to the results reported in (Schulman et al. 2017), while the results "PPO" are based on our reproduction of that algorithm. We compare the

---

Under review as a conference paper at ICLR 2020
Table 1: Scores the ALE benchmark
PPO (Sch.)
PPO
PPO-RNN
EDHR (Ours)
MsPacman
2096.5
1727.00 + 383.81
2040.63 + 104.88
2431.33 + 216.26
SpaceInvaders
942.5
912.63 5.20
952.30 + 126.78
1089.48 + 155.26
Breakout
274.8
413.07 = 22.12
410.40 2.21
368.24 = 36.54
Gravitar
737.2
581.17 = 23.01
600.33 + 197.36
587.50 = 46.22
Qbert
14293.3
11340.75 + 1463.59
9388.58 2408.55
11816.67 = 1952.70
Seaquest
1204.5
1227.73 481.22
1028.67 + 125.69
1342.13 1 339.85
Enduro
758.3
539.75 361.81
499.00 1 432.16
746.69 + 45.93
Table 2: Scores on the Obstacle Tower benchmark
PPO
PPO-RNN EDHR (Ours)
Train
1.54 + 0.07
3.34 + 0.45
2.24 + 0.18
Test
1.35 + 0.24
2.46 + 0.11
1.52 + 0.23
results with our reproduction remove the impact of minor technical implementation details, but in Tab. [ we also report the original results "PPO (Sch.)" as a reference. "PPO-RNN" is described above.
These results show the benefit of our history representation which is able to significantly boost the scores in most of the games, although more reactive games, like Breakout, do not show any EDHR ALE benchmark. Note that, in most of the games, PPO-RNN has similar or even worse performance of plain PPO. We believe this is due to the difficulty of training the training budget is limited. Specifically, the training budget, adopted in all our experiments and uniformly with all the tested methods, consists of 10M of time steps (Sec. 4. which is equivalent to the number of observations o, an agent receives from the environment, considering all the parallel actors and all the iterations. Moreover, if we consider the wall-clock time (and with the same training budget), in all our experiments PPO-RNN is on average about two times slower then our EDHR. In Appendix we include additional experiments with different hyperparameters of EDHR, an ablation model without self-supervision and another common RNN-based model.
benefit. PPO-RNN underperforms
RNN simultaneously with the agent in RL framework, especially when
The high score of PPO-RNN in Tab. 2 shows the importance of past information in the Obstacle Tower environment, where instantaneous observations are not sufficiently informative. EDHR suc- cessfully exploits the discovered events outperforming PPO. However, in this game our method un- derperforms PPO-RNN, indicating that in some cases the compact history representation useful information.
loose
The higher efficiency of our EDHR with respect to the training budget is shown in Fig. 3 where, for each game, we plot the scores reached by each method over the training iterations. These plots show that EDHR effectively makes use of the knowledge acquired in the unsupervised training stages and that, using an event-based history representation, most of the times training is casier. We believe this due to the fact that the agent, in making decision, can use the events, recorded in H ((), sort of "summary" of the important information in a past trajectory.
5.3 DISCOVERED EVENT ANALYSIS
What type of events does the method discover? To answer this question, we visually examined the observations and the corresponding events of a trained agent in two environments, MsPacman and Obstacle Tower. Fig. shows how the events are distributed during episode of MsPacman. Specifically, for each time step t. we plot the predominant event of corresponding to:
G arg max Do(ot) arg max P(z clo,)
cEY

---

Under review as a conference paper at ICLR 2020
MsPacman
Spacelnvaders
Breakout
EDHR
2000
PPO RNN
500
0
2000
4000
5000
8000
10000
2000
4000
6000
8000
10000
2000
4000
6000
8000
10000
iteration
Gravitar
Qbert
Seaquest
800
1500
10000
1000
400
5000
500
200
2000
4000
5000
8000
10000
2000
4000
6000
8000
10000
2000
4000
6000
8000
10000
Enduro
Obstacle Tower
EDMR
PPO
500
PPO-ANN
250
8000
10000
2000
4000
6000
8000
10000
Figure 3: Training dynamics, smoothed for readability
events
steps
Figure 4: The event dynamics during an episode of MsPacman. Each color corresponds a different predominant event computed using Eq. [ For simplicity, the y-axis of the figure contains only subset of the C 16 events.
This figure shows that the predominant event tend to be constant for a while during an episode, which is coherent with the expectation that important information about the past (an "event") changes with a low frequency. Moreover, we observed that these events correspond to specific changes of the environment. For instance, when the agent collects an object called "Power Pellet", which produces a switching of the color of the ghosts to blue and it allows these ghosts to be caten by the agent, then that moment there is predominant event change in (t).
Figure 5 shows some observations and the corresponding predominant events extracted from an episode of the Obstacle Tower environment. The observations have a sequential order (but they not each other adjacent).. New events correspond to the transition to a new room, or to different viewpoints of the same room. As shown in this figure, in the middle of the trajectory, the agent observes green door, and when it later returns to it, the event changes back to the correspond- ing value. In Appendix [ we plot another episode of the same environment, with high high-frequency selected observations and the corresponding predominant events, which more clearly shows the mantics behind the clusters discovered by the encoder and used by the agent.
Sparsity of our history representation. Using Obstacle Tower computed the probability that, in a trained agent, a random element in the S x C' matrix (t) has a value such that 0.1 < q < 0.9. This probability is 0.00146. Using the range of values 0.01 < q < 0.99, the probability is 0.0031. These empirical results show that our representation is very sparse and "binary", meaning that all the information is concentrated around the values 0 and 1.
8

---

Under review as a conference paper at ICLR 2020
11
8
4
4
Figure 5: Obstacle Tower predominant events associated with sequentially ordered observations of an episode.
CONCLUSION
We presented a method for history representation in RL which is an alternative common solu- tions based on RNNs. Our EDHR is based on the idea that important information about the past can "compressed" using events. Specifically, these events are automatically discovered using modification of the ICC clustering method (Ji et al. 2019), which is performed jointly with the agent training and iterated through time, to adapt the discovered events to the new observations. In EDHR, visual information is represented using two different networks: & and V. The latter is trained using a reward signal, so it presumably extracts task-speeific information from the observations. On the other hand, the encoder & is trained using self sufevvupervisinn and thus it focuses patterns which are repeated in the data stream, potentially leveraging a larger quantity of supervision signal. Al- though self-supervision has been explored in other RL and non-RL. works, this is the first work to show how the discovered information can be exploited in the form of discrete events used for history representation.
Our results, based on the challenging deep RL benchmarks, ALE and Obstacle Tower, show that EDHR can more effectively represent information about the past in comparison with task-oriented representations such as those used in PPO and PPO-RNN.
REFERENCES
Ankesh Anand, Evan Racah, Sherjil Ozair, Yoshua Bengio, Marc-Alexandre Côté, and R. Devon 2019.
Hjelm. Unsupervised state representation learning in atari. CoRR,
Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research 47, 47:253-279, 2012.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In European Conference on Computer Vision, 2018.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using encoder-decoder for statistical machine translation. In EMNLP, 2014.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems 31, pp. 2451-2463 Curran Associates, Inc., 2018.
Matthew J. Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. CoRR, 00115500765777 2015.
R. Devon Hjelm, Alex Fedorov, Samuel Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR, 2019.
Karan Grewal, Philip Bachman, Adam
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1785-1780, 1997.
Maximilian Igl, Luisa Zintgraf, Tuan Anh Le, Frank Wood, and Shimon Whiteson. Deep variational reinforcement learning for pomdps, 2018.

---

Under review as a conference paper at ICLR 2020
Xu Ji, João F. Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In ICCV, 2019.
Longlong Jing and Yingli Tian. Self-supervised visual feature learning with deep neural networks:
A survey. CoRR, (bss/1902.0666. 2019.
Arthur Juliani, Ahmed Khalifa, Vincent- Piceen Berges, Jonathan Harper, Ervin Teng, Hunter Henry, Adam Crespi, Julian Togelius, and Danny Lange. Obstacle tower: A generalization challenge in vision, control, and planning. In ICCAI, 2019.
Leslic Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101(1):99 134, 1998. ISSN 0004-3702.
Steven Kapturowski, Georg Ostrovski, Will Dabney, John Quan, and Remi Munos. Recurrent expe- rience replay in distributed reinforcement learning. In ICLR, 2019.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. ICLR, 2014.
Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting self-supervised visual represen- tation learning. CoRR, abs/1901.09003, 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. In NIPS Deep Learn- ing Workshop, 2013.
Volodymyr Mnih, Adri Puigdomnech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. ICML, 2016.
George E. Monahan. State the art-a survey of partially observable markov decision processes: Theory, models, and algorithms. Management Science, 28(1):1-16, January 1982.
OpenAl, Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafa Jzefowiez, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, Jonas Schneider, Szymon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, and Wojeiech Zaremba. Learning dex- terous in-hand manipulation. CoRR, 2018.
John Walker Orr, Prasad Tadepalli, and Xiaoli Z. Fern. Event detection with neural networks: A rigorous empirical evaluation. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 999-1004, 2018
Andrew M. Saxe, James L. McClelland, and Surya Ganguli. Exact solutions to the nonlinear dy- namics of learning in deep linear neural networks. In ICLR, 2014.
John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In ICLR, 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, Abs//700.00637, 2017.
Aäron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv: /001.6800..00 /807.03388 2018.
10

---

Under review as a conference paper at ICLR 2020
Table 3: Hyperparameters
Number of clusters C' History size
16
32
Temporal window L
3
Number of epochs 7g
10
Batch size (to update ())
512
Learning rate (to update ()
.0005
Pretraing iterations ratio Ppretrain
15%
Table Additional experiments
MsPacman
SpaceInvaders
Breakout
EDHR L = 3, 32 (default)
2431.33 4: 216.26
1089.48 155.26
368.24 1 36.54
EDHR L 1,S 32
2473.87 355.57
1048.72 128.57
375.54 = 31.90
EDHR L =8, -8,5 32
2038.33 + 435.01
910.85 + 44.65
389.07 = 6.31
EDHR L =3,S==
2148.03 247.37
864.05 + 122.53
411.09 = 11.89
EDHR L =3 64
1916.67 76.67
875.52 + 58.46
373.30 = 9,49
Without self-supervision
1897.70 90.54
981.20 + 171.63
411.58 + 2.86
PPO-RNN without (tt)
1317.47 + 136.67
746.80 + 33.82
189.40 + 29.84
A HYPERPARAMETERS
In Tab. we list all our hyperparameters which different from (Schulman al. 2017).
B ADDITIONAL EXPERIMENTS
In Tab. 4 we use three environments the ALE benehmark to show the impact of different EDHR hyperparameter choices together with an ablation study. Using minimum length temporal window 8 degrades the
(L. = 1) has similar performance to I 3, whilst a much larger value performance. This is probably due to the fact that large L value blurs out the borders between clusters, thus the event assignment is more ambiguous, less stable and less useful for the agent training.
As expected, a short history (S = 16) degrades the performance for the MsPacman and the SpaceIn- vaders environments. However, in this case Breakout performs slightly better, demonstrating that in a reactive environment the agent focuses on the instantaneous representation and the shorter history stimulates this. On the other hand, longer history (S 64) degrades the score for all the environ- ments: distant events are less useful for the training, whereas the doubled size of (t) makes the optimisation harder.
To show the impact of self-supervision, we include in Tab. 4 variant of EDHR without the event discovery stage. Specifically, "Without self-supervision" is equal EDHR, but & trained using the gradient of the reward signal, jointly with V. This experiment shows a dramatic degradation in the MsPacman and the SpaceInvaders environments, which highlights the gain which can be obtained using the proposed additional self-supervision. However, as above observed, in the more reactive environment Breakout the past information is less important, and the score achieved by "Without sueerssupeviiiinn" is equal the plain PPO (Tab. []. which indicates that the model ignores the history.
Finally, the last row in Tab, Acconcens variant of the PPO-RNN algorithm, where the input [t) and the corresponding CNN V are removed. this case the network receives the last S observations of the environment as images, each image processed using the CNN, the obtained features are input to the RNN, and the network is trained using the PPO algorithm. This model similar to the common RNN-based solutions (e.g. Mnih et al. (2016)). It is trained using the same hyperparameters of the

---

Under review as a conference paper at ICLR 2020
other experiments. Empirically observed that this solution is prone to higher variance in the CNN parameter updates, requiring larger training budget.
C INVARIANT INFORMATION CLUSTERING
In this section we briefly describe the self-supervieed method we used for clustering, referring the reader to the original paper (Ji al. 2019) for more details. Let x' E X be two different images sharing similar semantic information. In our case, 2; x' are two temporally-close frames, both con- tained in the temporal translation window (Sec. 3). while in (Ji et al. 2019) I, x represent the same image with some random perturbations (e.g. scaling, rotation, color adjustment, etc.).
is network mapping image into probability distribution & X - [0, 1jc, where C' is the number of clusters (in our case, events). ((x) is the probability distribution of a discrete random variable which (soft-)assigns x the elements in Y [1.... C?} and can be defined as: 2.(z) P(z c|z). If x and x' share the same semantic content, then they should be classified equally, otherwise they should be assigned different clusters. This objective obtained by maximizing the Mutual Information between the corresponding representations:
m ((x))). $
(2)
In more detail, pair (Ir, x²) corresponds the two variables (z, ='), with conditional joint distri- bution P(z c c' 22, x²) = (x &e (x'), where the equality holds when 2 and 2' are independent (Ji et al. 2019). After marginalization over the batch, the joint probability distribution
is given by the C' C matrix P, where each element at position c, cf is P P(z z' = e'):
P ((x,) ((x))T.
The marginals P. P(2 and P P(2' d') correspond to the sums the rows and the columns of P. rispectively. Finally, the optimisation objective in Eq. Dis computed directly using the definition of Mutual Information:
C Σ c=1
Pad P.P
1(z,z)= I(P)
P ce
In
(4)
D EVENT VISUALIZATION
Figure 6 shows detailed sequence of observations and the corresponding predominant events from an episode of Obstacle Tower. We observe that some events correspond to semantically general classes and do not focus only on visual appearance. For instance, event 15 indicates when the agent passes from one room to another, despite each transition (room) is visually different. Events 2 and 12 correspond to specific rooms, event detects the door, events and indicates specific views of room.
12

---

Under review as a conference paper at ICLR 2020
2
2
2
2
8
8
8
7
15
15
12
12
14
12
12
12
15
15
15
15
15
15
12
12
12
15
12
5
14
14
14
14
14
15
Figure 6: Detailed sequence of Obstacle Tower events
13


Reviews:
[{"review_id": "rye2pMZ0FS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new way to represent past history as input to an RL agent, that consists in clustering states and providing the (soft) cluster assignment of past states in the input. The clustering algorithm comes from previous work based on mutual information, where close (in time) observations are assumed to be semantically similar. The proposed scheme, named EDHR (Event Discovery History Representation), is shown to perform better than PPO and an RNN variant of PPO on (most of) 7 representative Atari games, and better than PPO on the Obstacle Tower benchmark.\n\nI would like to see this paper eventually published as I find the proposed technique original and quite relevant to current RL research, however I feel like its empirical evaluation is too weak at this time, which is why I am recommending rejection. I hope the results can be strengthened in a revised version so that I can increase my rating.\n\nThe main limitations of the current empirical evaluation are:\n•\tOnly 7 Atari games are used (vs 49 in the PPO paper the proposed technique is compared to), without justification for how they were chosen, and it seems like only 1 run is performed on each game (while RL algorithms are well known to exhibit high variance)\n•\tOn Obstacle Tower there seems to be also only one run of each algorithm (more runs could be done with different training & testing seeds in order to get an idea of the variance)\n•\tThere is no comparison to PPO+RNN on Obstacle Tower\n•\tI think a natural and important baseline to compare to is using the same architecture as in Fig. 2 but where the mapping Phi(o_t) is learned through regular backprop (using the same loss as when learning the mapping I(t)). This would validate that the advanced self-supervised clustering technique from Ji et al. (2018) is actually useful, and thus that the observed improvements are not simply due to providing 32 frames of history vs. 4 as in vanilla PPO.\n\nOther (more minor) remarks:\n•\tPlease explain better how the clustering technique from Ji et al. (2018) works, possibly in the Appendix if there is no room in the main body of the paper. This will make the paper more self-contained. \n•\tWithout fully understanding how this clustering technique works, it is difficult to me to get an intuition on how the clusters evolve during training, especially as new types of states are discovered by the agent. Some discussion on this topic would be appreciated.\n•\tIt would also be interesting to analyze the impact of varying the various new hyper-parameters (in particular S, C and L)\n•\tOn the first line of the last paragraph on p. 4, there is a missing reference « (Sec. ) »\n•\tOverall there are a bunch of typos throughout the paper that could easily be fixed\n\n\nFollow-up on author response: I remain inclined to stick to my rejection recommendation. I appreciate the efforts in providing more results, but I find them too limited and not entirely convincing: Table 4 is only done on 3 games, and the only one where the proposed method has a clear edge over \"Without self-supervision\" is MsPacMan. In addition, the fact that PPO+RNN shows much better performance than the proposed method on Obstacle Tower is also worrying.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "rJxzim-EcS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper focuses on the problem learning state representations that can effectively capture historical information\nin POMDPs. Specifically, the paper proposes an alternative approach to using RNN's for capturing such history - the authors adopt a variant of recently proposed Invariant Information Clustering approach to discover important events in past observation and then use these events to learn a probability distribution over observations to represent the state information. The goal here is to address the unstable and inefficient training issues associated with RNN based architectures. The authors validate their approach with experiments on seven tasks on Atari 57 benchmark and Obstacle Tower, both with discrete action space and compare their performance against both original and RNN versions of PPO. \n\nThe paper addresses an interesting problem to help learning better state representation in absence of  complete information about the environment. The approach of using IIC clustering (or any clustering approach) for learning state representations is novel. The overall goal of replacing RNN based methods in order to achieve stable, efficient learning in presence of budget constraints is very useful and hence this approach is potentially a good step in that direction. Although not adequate, the results in figure 3 provides good insight into effectiveness of the method in making training easier. \n\nHowever, I am inclining to reject this paper for the following reasons:\n(1) The motivation for using proposed clustering approach for history representation is neither clear  and nor well exposed. MI based methods are inherently difficult to learn and hence this approach needs  rigorous analysis on why it works when it does and how it fails.\n(2) The paper fails to position the new approach in comparison to related works both in discussions and experiments.\n(3) The experiments are very limited in nature and fails to demonstrate the efficacy of the proposed approach effectively. Further, the key contribution focusing on learning effective representations under constrained budget is not adequately tested.\n\nMajor concerns:\n\nMotivation\n-------------\n- It is not clear how the proposed clustering mechanism to discover events allows to successfully capture information that an RNN based approach does. For instance, RNN helps to capture long-term dependencies but it is very hard to interpret the proposed model from that aspect. Also, why is this particular clustering approach (Invariant Information Clustering) chosen? The motivation for using this approach is not clear and overall combination appears adhoc.\n- Further, RNN based methods can retain order information in sequential history of observations. However,  this method appears to not consider that information. Is this is true or am I mistaken here? If this is true, why would this not create issues in learning good representations for task where order is indeed important? \n- The paper discusses several articles that provide background on the methods used however it fails to position the exposition in comparison to existing RNN based approaches which is a big miss as the goal of the paper is to replace RNN based approaches [1,2,3,4].\n\nMethod\n------\n- In the event discovery stage, it is not clear why using consecutive observations is not useful. Also, if one does use L=1 (consecutive observations), can one reduce this method to RNN as you end of capturing all the previous history? Also, why L=3 is good across all different tasks? does it have any relation with the use of 4 frames in I(t)? \n- The authors mention tat H(t) matrix is highly sparse but also low-dimensional in all their experiments.  Would this is be the case for any other task? If not, is this a limitation of the method that you need S and C  to be small? \n- The authors attempt to use clustering based approach with the hope of recovering important information, however\nmention that H(t) stores all the past events. Isn't this contradictory? Also, why can RNN with an attention based mechanism not achieve similar effect?\n- It is also useful to analyse how will this method work in presence of long vs short history? Will the clustering itself and hence the learned representations get affected by length of available history?\n\nExperiments\n----------\n- Focusing only on PPO and designing an RNN version of PPO constrains the effectiveness of experiments in validating the approach. Authors mention difficulty of training with RNN as one reason for PPO with RNN's under performance but this is not convincing Could the performance of PPO with RNN be limited only due to specific RNN architecture used?\n- Authors must compare with other methods that use RNN approaches (e.g. [1]). Also, if methods such [2],[3],[4] are not  directly applicable, they must atleast, use their RNN based architectures to modify PPO and compare several baselines.\n- Why do the authors not report PPO with RNN for Obstacle Tower? \n- For the experiments, authors use a specific 10000 steps budget but this seems to be highly curated. The experiments\nwould be stronger if the authors show experiment over a range of budget. This will also give insights on when does RNN becomes better and is there a budget after which both methods perform equally well or RNN based approaches\nsurpass the current approaches.\n- Figure 3: Training dynamics seem to favor RNN approach for BreakOut, Gravitar. Do the authors have insight on why\nthis is the case?\n \nMinor points to improve submission not affecting the score:\n\n- The paper needs to be proofread for various typos and sentence construction issues\n- Table 1: For Qbert, original PPO (Sch.) is best performing, not EDHR\n- [5] talks about difficulties in MI based methods and I encourage the authors to look at the analysis in\nthis paper. As it is only an arXiv version and not published yet, I have not based my assessment on the\nexistence of this paper but still connection to such analysis will make this paper stronger.\n\n\n[1] Deep Variational Reinforcement Learning for POMDPs, Igl et. al.\n[2] On improving deep reinforcement learning for POMDPs, Zhu et. al.\n[3] Policy Learning with continuous memory states in partially observed robotic control, Zhang et. al.\n[4] Memory-based control with recurrent neural networks, Heess et. al. \n[5] On Mutual Information Maximization and Representation Learning, Tschannen et. al.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "S1e-0pOIuS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors study the problem of RL under partially observed settings. While most current (D)RL approaches use RNNs to tackle this problem, RNNS are trickier to optimise than FFNNs - in practice RNN-based DRL agents can perform well on partially observed problems, may require more effort to optimise, and may underperform FFNNs on domains with no/less partial observability. The proposed solution is to use a FFNN, but provide a \"history representation\", which is a set of feature vectors for previous timesteps that is extracted from a second network. The second network is trained separately using self-supervision (specifically, IIC, but adapted to use temporal consistency of observations rather than data augmentation). The proposed algorithm outperforms both PPO with a FFNN and PPO with an RNN on 5/7 Atari games (mainly games where partial observability is higher), as well as on the new and challenging Obstacle Tower benchmark - though PPO with RNN results are conspicuously missing on the latter! Given the promising approach (which also has a 2x better wall-clock training time than PPO with an RNN) and results, I would give this paper a weak accept. Some nice properties are that the instantaneous feature extractor is trained using RL, while the history feature extractor is trained using self-supervision at a high level (not pixel level), so that they are probably complementary; in addition it appears that in practice the resulting history features are sparse and usually binary, which is an intriguing and potentially useful property for future work.\n\nThere are several things to be done to improve the paper however. First and foremost, the results should be run over several seeds with standard deviation/error reported (it appears that this might be the case for Obstacle Tower, but no error is reported). I believe that the large improvements in some of the domains are significant, but it would be best to have this confirmed empirically. The authors could improve the presentation of background material by giving techniques names instead of using just author names. Although it would be expensive to show how changing L affects performance on all domains, some quantitative results on this hyperparameter would be useful - the same applies to C. The authors should also provide more clarity on choosing the history head - is the head identity fixed after pretraining? It would be useful to know how the more standard PPO with RNN architecture performs, but the authors' choice of architecture is a fitting comparison to their method, and does perform well in practice on the more partially observed domains, so the current setup is satisfactory.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose approaches to handle partial observability in reinforcement learning. The reviewers agree that the paper does not sufficiently justify the methods that are proposed and even the experimental performance shows that the proposed method is not always better than baselines.
Decision: Reject