Cross-State Self-Constraint for Feature Generalization in Deep Reinforcement Learning
Authors: Guan Ting Liu
Pu-Jen Cheng
GuanYu Lin
Venue:  (2021)
Keywords: reinforcement learning
generalization
regularization

Abstract: Representation learning on visualized input is an important yet challenging task for deep reinforcement learning (RL). The feature space learned from visualized input not only dominates the agent's generalization ability in new environments but also affect the data efficiency during training. To help the RL agent learn general and discriminative representation among various states, we present cross-state self-constraint(CSSC), a novel constraint that regularizes the representation feature space by comparing similarity of different pairs of representations. Based on the representation-behavior connection derived from the agent's experience, this constraint helps reinforce the general feature recognition during the learning process and thus enhance the generalization to unseen environment. We test our proposed method on the OpenAI ProcGen benchmark and see significant improvement on generalization performance across most of ProcGen games.

Under review as a conference paper at ICLR 2021
CROSS-STATE SELF-CONSTRAIN FOR FEATURE GEN- ERALIZATION IN DEEP REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Representation learning on visualized input is an important yet challenging task for deep reinforcement learning (RL). The feature space learned from visualized input not only dominates the agent's generalization ability in new environments but also affect the data efficiency during training. To help the RI. agent learn more general and discriminative representation among various states, present cross-state self-constraint((SSS) (CSSC), a novel technique that regularizes the represen- tation feature space by comparing representation similarity across different pairs of state. Based on the implicit feedback between state and action from the agent's s experience, this constraint helps reinforce the general feature recognition during the learning process and thus enhance the generalization to unseen environment. We test our proposed method on the OpenAl ProcGen benchmark and see signifi- cant improvement on generalization performance across most of Procgen games.
INTRODUCTION
Deep Reinforcement learning has achieved tremendous success on mastering video games(Mnih et al. 2015) and the game of GO(Silver al. (017). While training agent by using deep reinforce- ment learning algorithms, we usually assume that the agent could extract appropriate and effective features from different states and take actions accordingly. However, as more and more research works((hang et al. (2018), Song et al.] (2019). Dabney et al.] (2020) have pointed out, even well- trained RL agents that learns from visualized imput tend memorizing spurious pattern rather than understanding the essential generic features of given state. For example, an agent might pay more attention to the change of irrelevant background rather than noticing the obstacles or enemies(Song et al. 2019
To improve generalization in the new environment, various kinds of regularization method like dropout(Farebrook et al., 2018) and data augmentation((askin al.) (2200 has been proposed and tested in combination with reinforcement learning. Conventional methods like dropout and batch-norm has been proven to be effective in supervised-leaming and for self-supervised learning
like RL we see multiple related applications across various environments. Data augmentation like random crop(Laskin et al. 2020) random convolution recently and provide considerable generalization enhancement to the unseen levels of various tested environment et al. (2018), Cobbe et al. (2018), Cobbe et al. (2020)). The agent is acting on multiple augmented views of the same input and learn from these prior injected data. However, modifying state inforratiioninjecting prior to the data) may be risky or even detrimental for rep- resentation learning because vital features may be altered or lost (ex: flipping state image might change the corresponding behavioral meaning, cropping the input image might lose critical features like the enemy position in the game).
al.. 2019) have also been proposed
To avoid losing informative features of the visualized input, we choose different approach. As a human learner, we rarely depend on multiple augmented views of the same input to discriminate important or fictitious features. Instead, human learners try to recognize general patterns across mul- tiple states and act accordingly. In other words, if the same action(or behavior) has been conducted by a well-trained agent two different states, we would infer that the agent has conceived similar feature patterns in these states. For example, if one stops for ninety seconds at two different intersections, we would guess that the car might be stopped by the red light at both places(Figure 1b

---

Under review as a conference paper at ICLR 2021
From here we get the intuition about the relation between action(or behavior) and representation feature space learned by the agent: for a RL agent acting rationally across sequence of states, its behavior would be similar if it perceives similar critical patterns while acting differently if it per- ceives different patterns. Based on this intuition, we designed a novel constraint that performs regularization directly in the representation feature space of the learning agent. Our hypothesis for this constraint is simple: states in which the rational agent behaves identically should share more representation resemblance than states the agent behave differently.
We test this novel constraint in combination with Sainbow(Hessee et al. 2017), the state-of -ht art Q-learning method that combines numerous improvements with the original DQN model(Mnih et al. (0155). Inspired by the pair-wise structure used in BPR-opt(RendIe et al. (012), we design self-constraint based on the agent's behavior and utilize implicit feedback between positive and negative state pairs in the replay buffer. One thing worth noticing is that no change is needed for the underlying RI algorithm model adapt the proposed method, and can be easily applied to other models with minimal (Figure Ta)
Fainbew
Nokay
Nollyy
lapeet
Lupee
X
Rainbow
IMPALA
Acty Horey Layer
Add Novey
X1
sim sim(es (xa)) ex (x2)) stm(ee (x))
Layer
0.0
ru(t,) (x reftt.....
Cssc
X2
Q1
(a) Rainbow CSSC model structure
(b) Intuition for cross state feature matching
Figure 1: Model structure and CSSC concept
To measure the improvement on generalization to unseen environment levels, We test CSSC across 16 games on the OpenAl ProcGen benchmark et al. 2020). We significant improvement on generalization capability for most ProcGen games in comparison with base Rainbow model.
We highlight the main contributions of CSSC as follows:
Directly optimize feature generalization across various input states
Requires no additional modification on the input data and the base model
Enhance generalization and data efficiency on existing RL methods
2 RELATED WORK
2.1 AUXILIARY TASK FOR REPRESENTATION LEARNING IN RL
Representation Learning has been vital part for deep RL algorithms. Even though the main goal of RL is to find the optimal value function, Mccallum & Ballard (1996) and Li et al.] (2006) show that representation specialized this function may not be suitable for the sequence of value func- tions leading to it. On the other hand, Dabney et al. (2020) argue that the path toward optimal value function might be hindered by overfitting the representation to any intermediate value func- tion during training. One popular and effective method to address this challenge is by using auxiliary tasks(Jaderberg al. (2016). Bellemare al. (2019)). Dabney al. (2020) also propose new auxil- iary tasks by considering the value-improvement path holistically.
2.2
REGULARIZZTTIN ON RI.
2.2.1
ADDING STOCHASTICITY TO RL
Conventional practices like stochastic po Hausknecht Stone 2013), random starts Mnih et al.) 2015), sticky (etions(Hausknecht & Stone. 2015) and frame skipping wildly used for popular
2

---

Under review as a conference paper at ICLR 2021
tasks like Machadoo et al. 2017). By adding stochasticity to the environments during train- ing and testing, we would prevent simple algorithms like trajectory free(Keanns et al. 1999) and brute al. 2017) from optimizing
open-loop sequences of actions without considering the input states. However, as Zsann et al. (2018) has pointed out, injecting stochasticity the maze-like environments does not necessarily prevent the deep RI. agents from overfitting.
2.2.2 CONVENTIONAL REGULATIZATION ON RL
For deep supervised learning, regularizations has long been a major issue as generalization from training to test set is the major concern. Common regularization practices like L2 regularization, dropout, batch-normalization from supervised learning has also been tested in the reinforcement learning scenario. [oobee et al.] (20T8) and Farebrother et al. (2018) show that dropout and 1.2 regularization improve generalization to test environments in Coinrun et al. 2018 and Atari 2600(Bellemare et al., 2013) respectively.
2.2.3 DATA AUGMENTATION ON RL
For supervised learning related to images computer vision, data augmentation is popular way to enhance performance by injecting useful priors on the training data. In recent years, we see numerous data augmentation methods for RL that improve generalization
data efficiency. Lee et al. (2019) introduce a randomized convolution network that randomly perturbs input features. Srinivas et al. (2020) utilize contrastive learning and show considerable performance gains in terms of sample-eficiency. Kostrikov et al.] (2020) uses augmented data and weighted Q-functions to achieve state-of the-art data-efficiency on the DMControl (assa et al. 2018). [Laskin al.) (2020) investigate ten different data augmentation on RL and point out that random crop is the most effective on DMControl (Tassa et al. 2018
3 BACKGROUND
CSSC is a general framework of cross-state representation regularization for RL. In principle, one can apply CSSC to other variation of DQN or policy-based models for discrete action-space envi- ronments. In this work we pick Rainbow (Hessel et al. (017) and PPO (Schulman et al., 2017 as our base model to show that CSSC compatible with the original Nature DQN(Mnih et al. 2015). its multiple improvements and policy-based algorithms. In the following subsections will review Rainbow DQN and introduce the concept implicit feedback in Bayesian Personalized Ranking (BPR) proposed by Rendle et al. (2012).
3.1 RAINBOW
In combination with a convolutional neural network as visualized input encoder, Deep Q net- work(Mnih et al. 2015) demonstrates that it is possible to use neural network as function ap- proximator that maps raw pixels to Q values. Since then, multiple improvements such Double Q Learning Hasselt et al. 2015 Dueling Network Architecture Wang et al. 2015). Prioritized Experience (Schay et al. 2015) and Noisy Networks (Fortunato et al. 2017) have been pro- posed. In addition. Bellemare et al. (211)) proposed the method of predicting a distribution over possible value support through the C5I algorithm. By combining all the above techniques into a sin- gle off-policy algorithm, Rainbow DQN showcases the state-of-the-art sample-efficiency on Atari Benchmarks. The resulting loss function for Rainbow DQN is follows:
D (x.d("..")) (1))(de)
RRainbow
where .. is the projection onto fixed support 2. d(m)) and d, are the n-step target return distribution and the model-predicted return distribution respectively.
3.2 OPR-
As opposed to explicit feedback like user ratings, implicit feedback in recommendation systems focuses interactions like click view between users and items. In implicit feedback system only
3

---

Under review as a conference paper at ICLR 2021
positive observations available, and non-observed user-item pairs -e.g. user has not view item yet are considered as negative observations. Instead of rating prediction, BPR-opt(Rendle et a 2012) is designed for direct ranking optimization based the implicit feedback between users and items. BPR extends the user preference from observed interaction pairs to non-observed data by ranking the preference of positive observation and negatived observation across the training data. The formulation of the maximum posterior estimator for the personalized ranking optimization (BPR-opt) can be written as follows:
In 1np(O) >w)=
In (āris) - Xe||O|1 (u,is))
E
where represents the parameter vector of the base model. D, represent the batch samples from training data. p[O] >.) is the posterior probability conditioned on latent preference structure >u for user u, Xe is the model specific regularization parameter, a the sigmoid function, and Fusy(() is the rellev function model è which captures the preference relationship between user item i and item j. Rendle et al. (2012) use Fuij = Zui Laj to indicate that the user u prefers item over item j.
METHOD
The main idea of CSSC is to extend the similarity ranking across various state representation based on our hypothesis: representations motivating identical behavior should share more similarity than those motivating different behavior. We name this method as "self-constraint" emphasize that the state pairing is decided by the agent's behavior rather than pre-defined fixed labels. In the following subsection below, we introduce the following concepts to provide further explanation and insight: (i) definition of the behavior of agent at given state (ii) implicit feedback between state and behavior (iii) stoss-state self-constraint as an auxiliary loss for representation regularization
4.1 BEHAVIOR DEFINITION
We describe typical Markov Decision Process(MDP) as (X, A, R, P. 7) with state space as X, action space as A, reward function as R, state transition function as and discount factor as 7. The agent would take an action at state and then being transited I++1 by the environment with a given step reward r;. Here we define the behavior set B {(a,, a;+1..... ditn-1)la E A} as set of action series of length taken by the agent since state Z;. For b," BT and by By, DG = by only if 0;+p = ajtp for 0 < p S n 1. With this definition in mind, we can infer that the agent conduct the behavior be (left, fire) at state 2ú it would move left in state x; and fire in the next state 1- In the following paragraph we coin names like unigram, bigram and trigram for behaviors of length one, two and three respectively.
4.2 IMPLICIT FEEDBACK IN REINFORCEMENT LEARNING
In the traditional deep Q-learnig scenario, each state representation is highhl correlated to neighbor- ing states by the Bellman equation during the training process. However, observational overfitting could still happen as long as the representation of neighboring states can fit to specific sub-optimal version of Q-function indefinitely as mentioned in Dabney et al.) (2020). Besides the base RL al- gorithm, we can view the interaction between state and behavior as an implicit feedback system by regarding states sharing same action positive observation. The general feature among these positive observation would be reinforced if we compare their similarity with negative observation -e.g. states sharing different action. Through this "ranking" procedure the pair-wise relationship would be extended across non-neighboring states and therefore preventing observational overfitting specific Q-function.
4

---

Under review as a conference paper at ICLR 2021
4.3 IMPLEMENTATION OF PROSS-STATE SELF- COLSTRONNTT IN COMBINATION WITH RAINBOW AND PPO
For each sample of state triples(sp, Tg; Tg. X with (bg) bg b)) Ba and by = by # by, we decompose the estimator Epor and define it as:
pgr = pg fp co(p))) eo co(xp) cx(
(3)
where eg is the encoder function with weight 0 that maps the pixel input to 1-D array feature vector with shape like [element 0, element ..., element n] as the state representation. We directly perform inner-product on eg (xp) and ea ea(x)) to calculate the representation similarity & between &p and T... Please note that the encoder function is the same part of the original base model used for feature extraction on visualized input. For more details about neural network structures used this paper. please refer to Figure 7 and Figure [] To sample these state triplets without modifying the base DQN or PPO algorithm, we collect these state triples in the same batch transitions used to calculate Bellman loss policy loss. For policy- based algorithms and DQN without prioritized replay (Schaul et al. (201)) we pair up the state triples for each transition in the sample and calculate the CSSC loss as follows:
Lossc = ND. (p.grr)DD
1
Σ In (2pgrr)
(4)
where Dg represents the sample batch of size ND, from the replay buffer and a the sigmoid function. Then, we train the base algorithm in combination with the auxiliary CSSC loss as follows:
Liotal Lbase + Bessc Lossc
(5)
where Bease is the hyperparameter to control the contribution of CSSC during training. For pseu- docode of PPO with unigram CSSC, please refer to Algorithm 2 in the appendix.
In the case Rainbow with CSSC, we design the loss function in combination with the importance sampling (IP) weight as follows:
Liotal (Rariboow
ABesse LCssc)
WIP
where + and + are element-wise add and multiplication respectively. For pseudocode of Rainbow with unigram CSSC, please refer to Algorithm in the appendix.
We find that setting Bessc to 0.01 works quite well in most cases, so we stick with this setting for all the experiment conducted in the next section.
5 EXPERIMENT
Our primary goal for CSSC is to enhance the generalization capability of RL algorithm to unseen levels that share similar mechanism. Fortunately, OpenAl ProcGen(Cobbe et al. 2020) presents collection of game-like environments where the training and testing environments differs in visual appearance and structure. Therefore, we evaluate CSSC in three different ways:(i) generalization improvement 12 easy-mode games and 8 hard-mode games with Rainbow on OpenAI Proc- Gen (ii) generalization improvement on 16 easy-mode games with PPO on OpenAl ProcGen (iii) performance improvement on 32 games with Rainbow Gym Atari (iv) visualization the repre- sentation space learned with CSSC.
Table 1: Mean normalized score of Rainbow in OpenAl ProcGen Easy Mode
Mode
Rainbow
Rainbow uni-CSSC
Rainbow bi-CSSC
easy(trainn) @ 25M (12 Games)
0.2862
0.3832
0.4030
easy(test) 25M (12 Games)
0.1018
0.1600
0.1892
Improve. for (%) Improve. for test( %)
0.0
33.89
40.81
0.0
64.05
85.85
5

---

Under review as a conference paper at ICLR 2021
Raintow
011
Bambon CSSC
CSSC
0.4
Rambon ligann GB9C loo0
0072
sbow GB9C lool
Moan Normalized Score
Normalized Score
02-
02-
004
042
-0.
-015
-03
16
20
25
-0.0
10
20
30
Timesteps (M)
Timesteps (M)
(a) Easy mode
(b) Hard mode
Figure 2: Mean normalized score curve on OpenAI ProcGen environment. We normalized the episodic return of each game by the constants from Cobbe al. (2020) and report the mean score. Every curve is smoothed with an exponential moving average of 0.95 to improve readability.
Table 2: Mean normalized scores of Rainbow bigram-CSSC
Mode
Rainbow
Rainhow bi-CSSC
Improvement %
easy(train) @ 25M (12 Games)
0.2862
0.4030
40.81
easy(lest) @ 25M (12 Games)
0,1018
0812
85.85
hard(train) @ 30M (8 Games)
0.1085
0.1699
56.63
hard(test) @ 30M (8 Games)
0.0512
0.1199
133.94
5.1 ENVIRONMENT AND MODEL SETTING
For the following experiments, we use the IMPALA CNN architecture recommended bobCe et al. (2020) for the Rainbow and PPO model ProcGen benchmark. For the Gym Atari benchmark, stay with the Nature CNN as mentioned in Hessel et al. (2017) for the Rainbow model. All the experiments in this paper are conducted in the single-agent setting. For all hyperparameter used on ProcGen and Atari benchmark, please refer to Table in the Appendix. We use "unigram CSSC" or "bigram CSSC" to indicate the behavior length used to pair up the state triples for CSSC.
5.2 GENERALIZATION ON PROCGEN IN EASY MODE WITH RAINBOW
As Cobbe et al. 2020) have suggested, we conduct the training of 25 million timesteps on 200 training levels and evaluate the generalization improvement on the full distribution of testing levels across 12 ProcGen games. We test both the unigram and bigram CSSC in comparison with vanilla Rainbow and see significant improvement
testing performance. To evaluate the mean normalized score for bigram and unigram CSSC across 12 games, we follow the normalization method used in Cobbe et al. (2020) and show the result in Figure 2a and Figure T2a respectively. We also display
the learning curve of Digram-CSSC in Figure and unigram-CSS$ in Figure 13 In the following summarize the main findings below:
In supervised learning, posing regularization or constraint to the model would usually im- prove its testing performance unseen data at the expense of hurting training performance. However. we see that the proposed self-constraint improves both training and testing per- formance across most of the 12 game tested.

---

Under review as a conference paper at ICLR 2021
Stap at
Davelyer
Dodybbal
Pre Bot
Chooee
Jumper
Logger
Mape
Bightn
Bussight
Holl
Punder
Timestaps (M)
CDBC
Figure 3: Learning Curve of Rainbow with bigram CSSC in 12 ProcGen Games. We report the raw episodic return for both training and testing. All final scores are listed in Table Every curve is smoothed with an exponential moving average of 0.95 to improve readability.
For games like Starpilot, Chaser and Bigfish, we see that the testing performance is even better than the training performance vanilla Rainbow.
For the mean normalized score Table [ Bigram-CSSC and Unigram-CSSC bring 85% and 64% improvement respectively on the testing performance in comparison with base Rainbow model.
5.3 GENERALIZATION ON PROCGEN IN HARD MODE WITH RAINBOW
Coinrun
Starpilot
Bosslight
Bigfish
Score
Dodgeball
Fruibbot
Heist
Plunder
2.0-
1.0
0.5
30
-05
-1.0
0.0
10
10
Timesteps (M)
Rainbow (ran)
Rainbow Mgram CSSC (rann)
Ramnbow (we))
Rambow USSC
Figure 4: Learning Curve Rainbow with bigram CSSC in 8 ProcGen Games. We report the mean episodic return for training scores are listed in Table Every curve smoothed with an exponential moving average of 0.95 to improve readability.
and testing. Every mean return is shown across 3 seeds and all final
To further explore the generalization capability in more complicated and challenging tasks, we ex- amine the generalization improvement brought by CSSC on ProcGen games. We use 500 training levels for 30 million timesteps of training across 3 seeds and evaluate generalization on the full distribution testing levels. We test the bigram CSSC in comparison with the normal version of Rainbow across three different seeds and significant improvement learning curve of bigram CSSC is summarized in Figure 4 To evaluate the mean normalized score for bigram CSSC across 8 games in hard mode, also follow the normalization method used in
testing performance. The

---

Under review as a conference paper at ICLR 2021
Cobbe et al. (2020) and show the result in Figure 2b In the following summarize the main findings below:
As shown in F CSSC substantially improves both training and testing performance in Bigfish, Dodgeball, Starpilot, Fruitbot and Bosstight.
In particular, we see nearly 4x performance jump in Bigfish at 30M timestep in comparison with the base Rainbow model. The mean episodic return 30M timestep of bigram CSSC is even higher than that of PPO at 200M timestep as shown in Figure of Cobbe et al. (2020).
In the case of Heist (a puzzle-solvin task in maze-like layout), coinrun(a 2-D scroll plat- form game) and Plunder(a challenging shooting game), the gain from CSSC is not obvi- ous because all these games require careful manipulation and planning. Add constraint that modifies state representation directly would be risky or even detrimental for the learning process of the base RL. algorithm.
5.4 GENERALIZATION ON PROCGEN IN EASY MODE WITH PPO
We test the CSSC with PPO on all the 16 games of the Procgen benchmark in Easy mode. We conduct the training of 25 million timesteps across 3 seeds on 200 training levels and evaluate the testing levels across all 16 ProcGen games.
generalization improvement on the full distribution From the normalized learning curve shown in Figure [5 we can tell that CSSC helps reduce the gap between training and testing performance. For detailed final scores and learning curves, please refer to Table 7 and Figure 14 15 in the Appendix.
5.5 PERFORMANCE ON GYM ATARI WITH RAINBOW
We take Gym Atari benchmark [Brockman et al. 2016) as our second set of environments to measure the effectiveness of CSSC. Even though this classic benchmark does not explicitly split the training levels out of the training levels in all environments, we still see significant improvement on 23 out of 38 games tested as shown Figure 5b For training hyperparameters and scoreboard on Gym Atari benchmark, please refer to Table and in the Appendix.
5.6 VISUALIZATION OF REPRESENTATION EMBEDDING
To give a more tangible explanation about the effectiveness of CSSC, we plot the representation distribution in the embedding feature space across 4096 states from the replay buffer. We first perform dimension reduction on these state embeddings using principal decomposition analysis and label each embedding point with the index of corresponding action. Here we show the embedding space of both Rainbow and bigram- CSSC models trained to play the Bigfish game. In figure 6 we can see that states of the same action are more clustered bigram-CSS than that of the vanilla Rainbow model. We believe this is the possible reason behind the significant improvement generalization as shown in Figure For representation visualization other ProcGen games, please refer to A.4
6 CONCLUSION
In this paper we propose novel regularization on state representation learning based on the con- nection between agent behavior and the visual input. Our hypothesis is derived from the observation that the behavior of a rational agent would have certain relationship with general cross-state features or patterns. We see significant improvement on generalization brought by the proposed cross-state self-constraint(CSSCC
most of the games in OpenAl ProcGen benchmark. The connection be- tween behavior and conceived visual input can be considered as some kind of "motivation" that acts decisive factor behind the learned policy. It worth further study to better understand the derivation and transformation of agent's 's "motivation" during the Markov Decision Process, and we believe the concept proposed in this work would facilitate more research in this direction.
8

---

Under review as a conference paper at ICLR 2021
107
angiam urggam thart)
Mean Normalized Score
32
-10
-10-
1000
(b) Log- Scale Improvement on Gym Atari using Rain- bow with bigram CSSC
Timesteps (M)
(a) PPO CSSC easy mode
Figure 5: Left: Mcan normalized score curve OpenAl ProcGen environment. We normalized the episodic return of each game by the constants from Cobbe et al.] (2020) and report the mean score. Every curve is averaged across seeds and is smoothed with exponential moving average of 0.95 to improve readability. Right: the performance improvement of bigram CSSC with Rainbow on 38 games from the Gym Atari benchmark. All scores are recorded using the random seed.
(a) Bigfish: Rainbow + bigram CSSC
(b) Bigfish: Rainbow
Figure 6: Representation embedding projected to 2D space by PCA. We use the same 4096 state frames to extract representation using the bigram-CSSC and vanilla Rainbow model and display the distribution across all state representation.

---

Under review as a conference paper at ICLR 2021
REFERENCES
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling- The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, Jun 2013. ISSN 1076-9757. doi: 10. 10.1613/jair.912. URL URLhttp::// http://00...21...11.1.......
jair. 3912
Mare G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning, 2017.
Marc G. Bellemare, Will Dabney, Robert Dadashi, Adrien Ali Taiga, Pablo Samuel Castro, Nico- las Le Roux, Dale Schuurmans, Tor Lattimore, and Clare Lyle. A geometric perspective on optimal representations for reinforcement learning, 2019.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Karl Cobbe, Oleg Klimov, Chris Hesse, Tachoon Kim, and John Schulman. Quantifying generaliza- tion in reinforcement learning, 2018.
Karl Cobbe. Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural genera- tion benchmark reinforcement learning, 2020.
Will Dabney, André Barreto, Mark Rowland, Robert Dadashi, John Quan, Mare G. Bellemare, and path: Towards better representations for reinforcement
David Silver. The
learning, 2020.
Jesse Farebrother, Marlos C. Machado, and Michael Bowling. Generalization and regularization in dqn, 2018.
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg- Noisy networks for exploration, 2017.
Matthew Hausknecht and Peter Stone. The impact of determinism on learning atari 2600 games. In AAAI Workshop on Learning for General Competency Video Games, Austin, Texas, USA, January 2015. URL http: www. wuw.aaaaa.......
aaai15
Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning, 2017.
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks, 2016.
Michael Kearns, Yishay Mansour, and Andrew Y. Ng. Approximate planning in large pomdps via reusable trajectories. In Proceedings of the 12th International Conference on Neural Information Processing Systems, NIPS 99, pp. 1001-1007, Cambridge, MA, USA, 1999. MIT Press.
Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels, 2020.
Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Re- inforcement learning with augmented data, 2020.
Kimin Lee, Kibok Lee, Jinwoo Shin, and Honglak Lee. Network randomization: A simple technique for generalization deep reinforcement learning, 2019.
L. Li, T. Walsh, and M. Littman. Towards a unified theory of state abstraction for mdps. In ISAIM, 2006.
Marlos C. Machado, Mare G. Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents, 2017.
10

---

Under review as a conference paper at ICLR 2021
Andrew Kachites Mccallum and Dana Ballard. Reinforcement Learning with Selective Perception and Hidden State. PhD thesis, 1996. AA19618237.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Joannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-leve control through deep reinforcement learning. Nature, 518(7540):529 533, February 2015. ISSN 0028-0836. doi: 10. 10.1008/na 14255/ URL https://doii ltr.ooo..oooo..o.. /ttp://00..00/m/mrraelalal
Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme Bpr: Bayesian personalized ranking from implicit feedback, 2012.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui. Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of go without human knowledge. Nature, 550(7676)335 0836. doi: 10. 10.1038/mature222777
October 2017. ISSN 0028-
Salhttts::: /tcetps://oo... www.oio.rr//00 103. www.:////nnnune222.2
Xingyou Song, Yiding Jiang, Stephen Tu, Yilun Du, and Behnam Neyshabur, Observational over- fitting in reinforcement learning, 2019.
Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised representa- tions for reinforcement learning, 2020.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Ried- miller. Deepmind control suite, 2018.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning, 2015.
Ziyu Wang. Tom Schaul, Matteo Hessel, Hado van Hasselt, Mare Lanctot, and Nando de Freitas. Dueling network architectures for deep reinforcement learning, 2015.
Chiyuan Zhang, Oriol Vinyals. Remi Munos, and Samy Bengio. A study overfitting in deep reinforcement learning, 2018

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 NEURAL NETWORK ARCHITECTURE
Rainbow IMPALA network
Conv Sequence channel)
Conv. 3x3,
3x3,
Conv. 3x3, stride
Conv. Sequence [32 channel)
Residual Block
ReLU
IMIALA CNN
Residual Block
Conv. 3x3, stride 1
Conv. Sequence [32 channel)
Flatten
feature vector
Noisy Layer (5yer(512)
Noisy Laver(5122
ReLU
ReLU
Nolsy Layer(N_Atoon
Nolsy Layer(N _cctioN__Attoo
Advantages
Figure 7: Neural network architecture used for the Rainbow CSSC experiment. The output of the feature vector for each input state taken the state embedding for CSSC loss.
PPO IMPALA network
Conv. Sequence (16 channel)
Conv. 3x3, stride 1
ReLU
Max 3x3, stridl 2
Conv. 3x3, stride
Conv: Sequence (32 channel)
Residual Block
ReLU
IMPALA CNN
Residual Block
Conv. 3x3, stride
Sequence (32 channel)
Flatten
ReLU
FC (256)
ReLU
feature vector
FC (64)
FC (64)
Tanih
Tanh
FC (64)
(644
Tanh
Tanh
Pi
Value
Figure 8: Neural network architecture used for the PPO CSSC experiment. The output of the feature vector for each input state is taken as the state embedding for CSSC loss.
12

---

Under review as a conference paper at ICLR 2021
A.2 HYPERPARAMTERS AND MODEL SETTING FOR OPENAI PROCGEN AND GYM ATARI BENCHMARK
Table 3: Hyperparameters of Rainbow for OpenAI ProcGen
Env
ProcGen(Easy)
ProcGen(Hard)
Atari
Use stacked Frames
False
False
True
Image size(Channel, Height, Width)
(3, 64, 64)
(3, 64, 64)
(4, 84, 84)
CNN network
IMPALA CNN
IMPALA CNN
Nature CNN
state feature vector size
2048
2048
3136
Training levels
200
500
None
Training start Level
123
123
None
Testing levels
0
0
None
Testing start level
0
0
None
Batch size
128
128
32
CSSC type
unigram, bigram
bigram
bigram
CSSC constant Bossc
0.01
0.01
0.01
Multi step return
3
3
gamma
0.99
0.99
0.99
Learning rate
0.00025
0.00025
0.0000625
Replay buffer size
500000
500000
1000000
Replay alpha
0,5
0.5
0.5
Replay beta start
0.4
0.4
0.4
Minimum replay size for sampling
20000
20000
80000
Replay frequency
16
16
Target network update period
8000
8000
8000
Total timesteps
25M
30M
50M
Table 4: Hyperparameters of PPO for OpenAl ProcGen
Env
ProcGen(Easy)
Use stacked Frames
False
Image size(Channel, Height, Width)
(3, 64, 64)
state feature vector size
256
Training levels
200
Training start Level
0
Testing levels Testing start level
0
0
Batch size number of epoch coef entropy coef
2048
3
0.25
0.01
gamma
0.999
lambda
0.95
clip range
0.2
clip range vf
0.2
max grad norm
0.5
CSSC type
unigram, bigram, trigram
CSSC constant Bassc
0.01
Learning rate
0.0005
rollout buffer size number of envs
16384
64
steps per
256
Total timesteps
25M
13

---

Under review as a conference paper at ICLR 2021
A.3 SCOREBOARDS ON OPENAI PROCGEN
Table 5: Rainbow: Easy mode scores evaluated after 25M timesteps of training on 1 seed.
Train Unigram
Test Unigram
Game
Rainbow
Bigram
Rainbow
Bigram
Starpilot
27.79
43.75
46.13
26.37
39.72
43.40
Caveflyer
9.70
10.63
11.43
5.83
7.22
7.23
Dodgeball
5,46
9.08
9.48
2.74
4.25
5.04
Fruitbot
26.25
29.27
29.68
20.75
23.81
22.91
Chaser
1.71
2.33
2.74
1.34
2.07
2.43
Jumper
8.96
8.73
8.62
4.54
4.99
4.97
Leaper
8.81
9.58
9.39
5.42
5.83
5.88
Maze
0,41
0.71
0.45
0.21
0.50
0.38
Bigfish
14.80
26.60
26.04
13.16
14.87
16.91
Bossfight
8.26
8.16
8.57
7.52
6.90
7.32
Heist
0.77
0.77
1.06
0.26
0.34
0.42
Plunder
4.34
4.51
5.58
3.32
2.75
3,95
Table 6: Rainbow: Hard mode scores evaluated after 30M timesteps of training on seeds.
Train (mean + std)
Test (mean + std)
Game
Rainbow
Bigram
Rainbow
Bigram
Coinrun
5.99110.18
5.78=:028
5.13
5.46:10.37
Starpilot
5.63:10.23
7.30=1027
5.1710.15
6.16:10.26
Bossfight
4.42-10.43
5.64
3.89+0.24
5.38:10.28
Bigfish
1.88+0041
7.22+4.27
1.24+0.38
5.26+13.28
Dodgeball
3.74-10.15
5.23:
3.094-0.21
4.26-10.08
Fruitbot
1.33-10.14
1.65
0.12+-0.60
0.69
Heist
0.38-10.06
0.09
0.21
0.20+0.03
Plunder
2.43:10.08
2.3810.07
1.81
1.9140.04
Table 7: PPO: Easy mode scores evaluated after 25M timesteps of training 3 seeds.
Train (mean std)
Test (mean + std)
Game
PPO
Unigram
Bigram
PPO
Unigram
Bigram
Bigfish
319631.35
6.77 +1.86
6.30 =1.93
1.45
0.30
3.29
11.45
2.33 0066
Bossfight
8.36 +..24
7.21 +0.51
7.51 +0.38
7.64 0.14
6.79
+0.42
6.38 -0.61
caveflyer
+5.29
2.53 10.48
1.97 10.87
0.76 =0.4
1.78 20.33
10.27
Chaser
1.98 +0.23
1.93 10.14
1.77 10.14
1.73 -0.08
1.63 #0.11
1..07
Climber
2.63
3.22 10.82
1.42 10.17
1.53
Coinrun
+0.67
6.84
+0.52
-0.34.34
6..62
5.12
Dodgeball
1..30
1.33 +0.20
+0.03
0.89 -0.10
=0.02
Fruitbot
29.2740.51
29.8611043
30.05.10.38
24.99-0.71
27.00-10.45
25.63-0.13
Heist
1.38 10.21
1.77 10.03
2.42 10.45
0.27 20111
0.39 10.16
0.35 10.55
Jumper
10.09
40.20
40.27
4.16
40.51
=0.60
Leaper
4.38 #0.33
4..34
4.39 =0.18
4..122.12
0.35
Maze
6.10+0.7
4.75 +0.97
40.42
2.55 0.36
2.70 +0.37
2.58 0.15
Miner
7.37 +1.62
3.59 +0.45
1.80 0.45
1.24 +0.06
1.21 +0.08
Ninja
3.76 +0.54
+0.26
4.40 +0.37
2.86 0.07
2.53 +0.36
3.63 0.29
Plunder
3.37 +0.36
3.71 10.36
10.26
2.96 -0.34
3.17 10.16
2.94 10.24
Starpilot
27.1912.442
26.1911..7
28.41+12.23
19.82-0.85
22.12+1.87
14

---

Under review as a conference paper at ICLR 2021
(a) Dodgeball: Rainbow bigram CSSC
(b) Dodgeball: Rainbow
Figure 9: Dodgeball: representation embedding projected to 2D space by PCA
444
04
(a) Starpilot: Rainbow bigram CSSC
(b) Starpilot: Rainbow
Figure 10: Starpilot: representation embedding projected to 2D space by PCA
(a) Coinrun: Rainbow bigram CSSC
(b) Coinrun: Rainbow
Figure 11: Coinrun: representation embedding projected to 2D space by PCA
A.4
VISUALIZATION REPRESENTATION EMBEDDING ON OPENAI PROCGEN GAMES
A.5
GENERALIZATION ON PROCGEN IN EASY MODE FOR RAINBOW UNIGRAM-CSSC
A.6
GENERALIZATION ON PROCGEN IN EASY MODE FOR PPO WITH CSSC
A.7 PSEUDOCODE FOR CSSC EXPERIMENT
A.8
EVALUATION OF BIGRAM CSSC WITH RAINBOW ON GYM ATART BENCHMARK
15

---

Under review as a conference paper at ICLR 2021
Algorithm Rainbow + CSSC (unigram)
1: Input: minibatch size k. multi-step return n. replay period K and buffer size N, exponents and B. gamma y, CSSC coefficient Besses total-timesteps T.
2: Initialize replay memory H = Ø, feature extractor 0, return distribution predictor O =
3: Observe state 8q and choose action ag ~ (x))
4: for to 1 do
5: Observe state 8t, reward (()), done flag de, new state 8t+n
6: Store transition j: (st, Te (n)
de, Sttn) in H. with maximal priority pr maxi<t
7: if mod 0 then
8:
Sample batch of transition J Jo, julj P())
/ /Eppf] W [wo, Wi,..., Wik uyy
9:
Compute importance-sampling weight:
(N
P())) -B/ //axx wi]
Predict state feature vector co(s;), co(s)))) and return distribution dj, d (eea (s,)), ((en(s)+n)) for each transition in the batch
11:
Compute LRannbow tion function Ø
[/u 4..444444 DKL(#_d) [[d,)] using distribution projec-
12:
Update transition priority Py l for each transition in the batch Sample positive pair (sp, $q). p * q with Jp- Jy J and behavior 6). b) for each
13:
transition Jp in J
Sample negative pair (sp, Sr). F with Jp: and behavior b). * by. for each transition jp in
15:
Compute Lossc = |s0 St. Skmwaer 16 Apppps = -1.0 x (ee((8))) c es(s))
ee((s,)))
16:
Compute Lrotel = Raanboow + (Bessc Lcssc)) @ W where + and @ are element-
wise add and multiplication respectively
17:
Fit θ and with Adam optimizer to minimize Sxotat-maan)
18:
end if
19:
end for
16

---

Under review as a conference paper at ICLR 2021
0.1
03
Ranbon thaie
Rarbow (raa)
Mean Normalized Score
Normalized
81-
Timesteps (M)
Timesteps (M)
(a) unigram-CSt CSSC
(b) bigram- CSSC
Figure 12: Mean normalized score on OpenAl ProcGen environment in easy mode. Every curve is smoothed with an exponential moving average of 0.95 improve readability.
Saup ler
Cavelyer
Eodgebal
Fru Exl
Cheae
Jumper
Loaper
Map0
Bayis
Bossight
Mold
Plundor
Timestops (M)
Pabllne
Figure 13: Learning Curve of Rainbow with unigram CSSC in 12 ProcGen Games. We report the raw episodic return for both training and testing. Every curve is smoothed with exponential moving average of 0.95 to improve readability.
17

---

Under review as a conference paper at ICLR 2021
Bigfish
Boeelight
Cavellyer
Chaser
0.5
0.0
Climber
Cainrun
Dodgeball
Fruitbot
Heist
Jumper
Score
Loaper
Ninja
Plundar
Starpilot
20
PPO CSBC unitton Drain) PRO GSSC aigrom (hoin)
(rán)
Timesteps (M)
Figure 14: Learning Curves of PPO in 16 ProcGen Games on training levels. Every curve is averaged across 3 seeds and smoothed with an exponential moving average of 0.95 to improve readability.
Biglish
Bocolight
Cavellyer
Climber
Coinrun
Dodgeball
Fruitbot
Helst
Jumper
2.5
Score
1.00
0.75
350
20
0.25
Leaper
Maze
Miner
Ninja
Plunder
Starpilot
2.0
1.5
1.0
0.5
0.0
unigram Deee:
M90 Ser
Timesteps (M)
PPO CBSC algram nosh
in 16 ProcGen Games testing levels. Every curve Figure 15: Learning Curves of PPO with CSSC is averaged across 3 seeds and smoothed with an exponential moving average of 0.95 to improve readability.
18

---

Under review as a conference paper at ICLR 2021
Algorithm 2 PPO CSSC (unigram)
1: Input: minibatch size k. CSSC coefficient Besse, total-timesteps T. clip range
2: Initialize rollout buffer Ø feature extractor 0, policy parameters θ value function param- eters Ø
3: for = to T do
4: Collect set of trajectories D; {T:} by running policy Te (0x)) in the environment
5: Compute advantages estimates A, using current value function Vi
6: Store transition je = (8t, Oz, Tt, de, A., (og()((x))) H
7: if rollout buffer 7. is full then
8: for each batch in H. do Predict state feature vector eo(s;), log prob, values using Ty and V₂ for transitions
9:
in the batch
10:
Compute ratio wxw(leepppppppee batch.old_ (oooh.o)d_log-proh) Compute policy
11:
clamp(ratio, 1 range, cliperange) Compute Cpolicy = policylllloooooo los...ssssss...
13:
Sample positive pair (sp: 5q), p # q with jp: Jy J and behavior by b) for each transition j, in batch transitions
14:
Sample negative pair (sp, 8,), with Jr e J and behavior 63 * 6.. for each
transition jp in batch transitions
15:
Compute csse cssc_loss. loss
Skpeour ppppars = -1. (.0x0((o($p))) co((s)))
es(s) co(s,,) ))
16:
Compute Lesse essc loss.mean()
17:
Fit θ, 0 and é with Adam optimizer to minimize Ltotal Ditiiboow + Bease' Lcsse
18:
end for
end if
20: end for
19

---

Under review as a conference paper at ICLR 2021
Table 8: Gym Atari scores evaluated after 50M timesteps of training on I seed.
Train (mean)
Game alien
Rainbow
Bigram
3,738.3
4,659.5
amidar
1,951.2
1,579.2
assault
18,292.2
19,690.2
asterix
480,431.1
647,559.0
asteroids
2,041.0
296,158.1
atlantis
2,419,949.7
3,090,829.1
bankheist
1,292.9
1,595.3
battlezone
77,924.6
62,216.0
beamrider
35,120.4
27,937.9
berzerk
2,974.7
3,130.8
bowling
55.5
74.0
boxing 417.9
94.8
99.6
breakout
413.5
centipede
18,103.3
19,926.0
chopper command
28,468.0
30,509.5
crazy climber
243,610.0
221,021.5
demon attack
506,716.2
535,951.2
double dunk
-22.6
-22.8
enduro
7069.1
6391.6
fishing derby
36.18
50.4
freeway
32.0
33.8
frostbite
11688.2
10268.2
gopher
151,072.1
34,115.3
gravitar
2,084.5
1,426.3
hero
33,534.5
33,965.5
icehockey
4.9
-0.125
Jamesbond
36,417.5
47,388.5
kangaroo
14,525.0
14.740.5
krull
7,843.5
8,913.0
kungfumaster
51,319.0
42,107.0
mspacman
3,468.2
4,420.7
namethisgame
11,525.2
13,574.9
phoenix
479,379.5
488,284.2
pong
20.7
20.9
privateeye
99.5
100.0
qbert
26,394.9
27,159.0
timepilot
22930.0
27053.0
tutankham
170.4
150.7
20


Reviews:
[{"review_id": "nEWv37XHrQi", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors modify the Rainbow loss through an additional loss based on a cross-state similarity. The notion of similarity is called in the paper \"cross-state self-constraint\" - CSSC. The CSSC loss looks very simple and can be applied with other RL algorithms. The loss is defined in terms of an embedding e of the visual input into a 1-dim space. Given three states x_p, x_q, and x_r, their similarity is defined as the scalar product of e(x_p) and e(x_q) minus the scalar product of e(x_q) and e(x_r). It would be useful if the authors include pseudocode how the loss is exactly computed for a given batch of samples. My understanding of the description is that for a given batch the authors generate triples with the same action set and then take as an auxiliary loss the weighted sum of the logarithms of sigmoids of similarities of these triples.\n\nThe results on the ProcGen suite of environments are reported in Tables 1 and 2 and they look promising. Unfortunately, the results are reported without information about the variance. Also, for some environments, e.g. for Chaser, the final numbers are weak compared to the ProcGen paper.  The inclusion of a more systematic comparison with PPO (using numbers from the ProcGen paper) would be helpful. \n\nThe visualizations look interesting, though I am not sure how the authors back their claim that  \"in figure 5 we can see that states of the same action are more clustered in bigram-CSSC\". Can you include some numerical information about clustering and/or add some extra annotations to figure 5?\n\nMy understanding of the paper is that the standard IMPALA architecture was equipped with an additional \"embedding head\" (I am inferring it from Figure 1). Though I cannot find a precise description of this architecture change, in particular how large are the embedding vectors. \n\nThe method looks simple and interesting, though the paper in my opinion needs a substantial re-write before publication at ICLR.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZojNjKF1HtE", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper proposes to introduce an auxiliary contrastive loss to improve generalization on procedurally generated environments. In particular, it encourages latent representations of states to be more similar when those states lead the current agent to take the same sequence of future actions. \n\nPositive:\n(+)  Clear idea & description\n(+) Simple, but novel idea\n(+) Positive experimental results\n\nWeaknesses:\n(-) Insufficient comparison to baselines\n(-) Only one set of benchmark environments\n\nRecommendation: 5\nI'm currently recommending (weak) rejection but would change my score based on additional experimental results.\nIn particular: As noted by the authors, prior work has shown that \"standard\" regularization techniques like L2 (weight decay), Batchnorm or Dropout can improve performance on both training and testing. As the authors introduce a novel type of regularization technique, I believe it is important to not only compare against a \"vanilla\" (i.e. unregularized) agent, but agents utilizing various simpler regularization techniques. \nTheir technique would not even need to outperform agents which are, for example, regularized by weight decay, but I'd find it sufficient to show that CSSC can further improve the performance of such already regularized agents. \n\nMinor point: \nA second set of other procedurally generated environments, other than ProcGen, would help show that the method is not overfit to ProcGen. \n\nOther remarks (no need to answer, didn't have impact on evaluation):\n- I found section 4.2 hard to understand, maybe it could be reformulated?\n- Page 5, first line: I can't believe that e_\\theta maps to a 1d feature vector (i.e. a scalar)?\n- Equation (3): x_j should probably be x_r", "extra_scores": {}, "extra_text": {}}, {"review_id": "Mbzm3vjn8un", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles the problem of representation learning from visualized input. The paper presents \"cross-state self-constraint(CSSC)\", a technique for regularizing the representation feature space by favoring representation similarity (scalar product between representations) between representations when the agent behaves similarly. The approach is tested with deep RL on the OpenAI ProcGen benchmark.\n\nThe method developped in this paper seems to provide interesting empirical results. The motivation of bringing in all cases the representation closer when the agent takes the same actions is not necessarily obvious because very different reasons might lead to the same sequence of actions, but this empirical study seems to shows the interest of this idea. \n\nMy main concern is related to the following remarks/questions that I believe require some clarifications:\n\n- In equation 3, what does j stands for? \n- Can $\\hat x_{pqr}$ be negative? In that case, what happens with the loss? \n- n sometimes correspond to \"a set of action series of length $n$\" and sometimes to the batch size. Usually a batch of tuples is selected randomly in the replay memory (without notion of sequence). What does n corresponds to in practice in the experiments?\n- In table 1 and 2: How many seeds are used? Why is the standard deviation not given?\n- No information on the neural network architecture seems to be given. What is the structure of the abstract representation?\n\nSome typos (the 12 game tested, Rainbwo model)", "extra_scores": {}, "extra_text": {}}, {"review_id": "JFXisCVBLN", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary \n\nThis paper proposes a new method for learning representations of images with the goal of improving generalization in RL. The key idea is to regularize the learned feature space by forcing embeddings of states followed by the same action (or sequence of actions) to be more similar than embeddings of states followed by different actions. The method is evaluated on the Procgen and achieves superior performance relative to Rainbow, a standard RL algorithm.\n\n\nStrengths \n\nI do like the core idea of this paper and I think it makes sense intuitively. I also think the paper is trying to solve an important problem in RL which has been neglected until recently. In addition, the proposed method is simple and achieves much better generalization than Rainbow. However, I think the paper needs more work to be ready for publication for the reasons laid out below. \n\n\nWeaknesses\n\nThe main weakness of this paper is the lack of comparisons with previously proposed methods for improving generalization in RL. While the results are strong relative to Rainbow, no other baselines are included. I suggest adding comparisons to other methods which have recently shown good results on Procgen such as PPO (used as a baseline in the paper that introduced Procgen from Cobbe et al. 2020), DrAC (Raileanu et al. 2020), ITER (Igl et al. 2020), and DRIML (Mazoure et al 2020).  \n\nIn addition, various choices made by the authors make it difficult to compare with prior art.  For the hard mode, you only train the agents for 30m steps while the original paper trains them for 200m steps in order to achieve decent performance. You also only show results on 12 or 8 of the games instead of all 16. Following the same training practices and evaluation metrics as the ones proposed in Cobbe et al. 2020 would make comparisons with previous and future work easier and more transparent. \n\nYou mention using a 1-dimensional feature vector for the state representation. That seems very low and I don’t understand how it could capture all the relevant details in an observation. Does the method work with higher dimensional representations? Additional experiments and discussion to motivate this choice would be useful. \n\nAs I understand, the embeddings are constrained using the CSSC at the same time as the policy is being learned. But at the beginning of training, the actions are random so constraining representations based on the actions taken doesn’t seem ideal. Wouldn’t this slow down training because the CSSC loss forces the “wrong constraint” at the beginning and thus lead to poor representations which would then make it harder to learn a good policy? Can you include some experiments and discussion about how the performance varies with the coefficient for the CSSC loss. \n\nHave you tried using trigrams? It seems like using bigrams is better than unigrams so it would be useful to see whether adding more actions continues to improve performance or not. \n\nWhile I liked the visualization of the learned embeddings, I think it would be more insightful to compare the embeddings of observations from train and test (colored by their actions) in order to understand how well the structure in the embedding space generalizes to unseen observations. Do you see similar structure in the embeddings of test observations as in the train ones?\n\nI found the related work section to be rather disconnected. The authors mention relevant works but do not discuss them in relation to the proposed approach. I suggest the authors discuss in more detail the limitations of current methods and explain why (and when) the proposed approach might be preferable. In addition, the section is missing a number of closely related papers that address representation learning for improving generalization in RL such as Farebrother et al. 2018, Igl et al. 2019 and 2020, Lee et al. 2020, Srinivas et al. 2020, Laskin et al. 2020, Roy et al. 2020, Mazoure et al. 2020, Zhang et al. 2020, Raileanu et al. 2020.\n\nI think the clarity of the paper can be significantly improved. For example, at the end of the intro you mention that there are significant improvements for most Procgen games, but it would be better to be more precise and mention exactly on how many of them it is better and even by how much (i.e. outperforms on X out of Y games and achieves z% better test performance on the entire benchmark). The following sentence can also be more specific: “ Inspired by the pair-wise structure used in BPR-opt(Rendle et al., 2012), we design the self-constraint based on our hypothesis and utilize implicit feedback between positive and negative state pairs in the replay buffer“ by replacing the word “hypothesis” with something more descriptive. I would also more gently introduce the relation to BPR-opt for readers who don’t know what that is -- this is the first mention of BPR and has no explanation of what it is, including what the acronym stands for. There are lots of other places in the paper where clarity needs to be improved. \n\n\nMinor Points\n\nSome of the formatting seems off e.g. there is no space before starting a parenthesis.\n\nSometimes you use “representation feature space” which seems redundant. Using “representation space” or “feature space” would be more concise.\n\nThe use of “visualized input” instead of “visual input” sounds strange to me and it’s not a common phrasing in this field as far as I know.\n\nTypos:\nIn equation 3, I believe it should be x_r instead of x_j\n“Has conceived” instead of “has conceive” in the intro\n“Novel constraint that performs” instead of “...that perform” in the intro\n“Rational agent behaves” instead of “...behave” in intro\nSome letters are capitalized after a comma. \n“The most effective” instead of “...effect” in related work\n“Demonstrates” instead of “demonstrates” in 3.1\nMany others…\n\n\nRecommendation\n\nWhile the proposed method is simple, sensible, and appears effective, the paper needs significant improvements as I explained above. Thus, I cannot recommend it for publication at this stage but I encourage the authors to continue working on it. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I thank the authors for their submission and participation in the author response period. The updated experiments are appreciated. However, after discussion all reviewers unanimously agree that the paper is not ready for publication and encourage resubmission to another venue. In particular, R2 and R3 have raised concerns regarding additional widely available baselines that need to be evaluated against and that the rebuttal has not addressed. I agree with this assessment, and thus recommend rejection.
Decision: Reject