Learning Good Policies By Learning Good Perceptual Models
Authors: Yilun Du
Phillip Isola
Venue:  (2020)
Keywords: visual representation learning
reinforcement learning
curiosity

Abstract:     Reinforcement learning (RL) has led to increasingly complex looking behavior in recent years. However, such complexity can be misleading and hides over-fitting. We find that visual representations may be a useful metric of complexity, and both correlates well objective optimization and causally effects reward optimization. We then propose curious representation learning (CRL) which allows us to use better visual representation learning algorithms to correspondingly increase visual representation in policy through an intrinsic objective on both simulated environments and transfer to real images. Finally, we show better visual representations induced by CRL allows us to obtain better performance on Atari without any reward than other curiosity objectives.

Under review as a conference paper at ICLR 2020
LEARNING GOOD POLICIES BY LEARNING GOOD PER- CEPTUAL MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Reinforcement learning (RL) has led to increasingly complex looking behavior in recent years. However, such complexity can be misleading and hides over-fitting We find that visual representations may be useful metric of complexity, and both correlates well objective optimization and causally effects reward optimization. We then propose curious representation learning (CRL) which allows us to use better visual representation learning algorithms correspondingly increase visual repre- sentation in policy through an intrinsic objective on both simulated environments and transfer to real images. Finally, we show better visual representations induced by CRL allows us to obtain better performance Atari without any reward than other curiosity objectives.
INTRODUCTION
In recent years, reinforcement learning(RL) has lead to increasingly complex behavior from simulated environments (Silver et al., 2016; OpenAI, 2018; Mnih et al., 2013; Andrychowicz et al., 2018). Yet despite this, there lacks quantitative measure of intelligence in these agents. Qualitative measures can be deceptive. Consider agent Alice and Bob in Minecraft. Alice is capable house while Bob appears only be able to navigate around the world. While at face value it may then appear that Alice is more complex, upon closer inspection we may find that Alice has simply memorized set of actions to construct house in that particular environment!
constructing a
How be certain that agents are not simply not memorizing set of moves? One hypothesis that the more intelligent an agent is, the more likely the inner representations in its policy will exhibit disentangled properties of the world. Towards this end, we investigate the emergent visual representations that occur in RL policies.
We investigate on various objectives and environment conditions, and find that the quality visual representation learning correlates well with progress in reward optimization. Similarily, we find improved visual representations help agents perform better reward optimization. Thus, another natural question to ask is, how can enable agents to have better visual representations?
While there are ways to hardcode reward functions to enable agents perform well, can come up with a generic objective that our agents can optimize that will directly lead them to have good representations? One idea towards this is to recent work in curiosity. In curiosity, agents are typically given rewards corresponding to surprisal of state. But another view of curiosity is that of a minimax game where curious agent is seeking to maximize the surprisal of an uncertainty model, while the uncertainty model seeks become less surprised about new states.
Thus, to enable policy to learn good visual representations, we can treat the uncertainty model as a representation learning model. We then seek policy that wants to lower the loss of the representation learning objective, while the model itself tries to optimize this loss. Under this objective, a policy must learn good visual representations, so that it is able to find visually surprising inputs for the vision model. We call this overall objective, Curious Representation Learning (CRL).
By coupling policy learning with representation learning, we find that CRL allows us to get better policy visual representations simply by applying better visual representation learning algorithms to the model. As a result, we find that CRL obtains consistently good representations in policies across environment size and type, often beating many hard-coded domain specific objectives. As added bonus, we find that CRL is also able to achieve better visual representation learning than other data collection methods, as it actively sees diverse inputs that surprise it.

---

Under review as a conference paper at ICLR 2020
Policy
Representation Learning Model
Observations
Grey
Color
Reward
(Loss)
x(als)
Dooloz
Maximize
Minimize
Figure 1: Overview of CRL (curious representation learning). We use the loss of a representation learning algorithum intrinsic reward for agent. Correspondingly, we train the representation learning algorithm on the experience generated by the agent. This forces the agent and model compete in minimax game learn good visual representations
In addition, we investigate optimizing visual representation through CRL as curiosity bonus. We find that on Atari, CRL is able to obtain better overall performance compared to other approaches such Forward Dynamics and RND, when used as sole intrinsic bonus.
Our contributions are threefold. 1) We first show that visual representation learning corresponds well reward optimization RL policies. 2) We propose CRL, a method that allows us use better visual representation algorithms to obtain better visual representation in policies on simulated environments and real images. 3) We apply CRL to Atari and show that the better visual representation obtained in policies allows to out-compete forward dynamics (Burda et al., 2018a) and Random Network Distillation benchmarks (Burda et al., 2018b)
2 RELATED WORK
Visual representation learning has seen large amount of interest in recent years (Zhang et al., 2016; Gidaris et al., 2018; Hjelm et al., 2018; Bengio et al., 2013). Approaches towards unsupervised learning include predicting image rotations (Gidaris et al., 2018), colorizing images (Zhang et al., 2016), maximizing mutual information between observations and representations (Hjelm et al., 201 8), and many other approaches. Much of the previous visual representation work has focused passive datasets - here we focus on representation learning in active environments where agents must interact with the environment obtain data.
Curiosity has also been studied extensively in the past years (Houthooft et al., 2016; Pathak et al., 2017; Bellemare et al., 6; Burda et al., 201 8b). Such objectives typically encourage agents to minimize some uncertainty in the world, based off dynamics (Houthooft et al., 2016; Pathak et al., 2017) or random features (Burda et al., 2018b). Our proposed method for curiosity differs from past method algorithm and target policy. This formulation can help explain some of the strong results of curiosity as well pave direction towards better curiosity algorithms.
propose curiosity
minimax game between generic representation learning
Our work also relates past work in active learning. Active learning involves sampling datapoints that maximize an uncertainty/error measure, while the learner itself thereby aims to minimize un- certainty/error (Settles, 2009; Roy and McCallum). This entails adversarial game where learners get better and better, it becomes increasingly difficult to train on, an idea refered to as curriculum learning (Bengio al., 2009). In our work, explicitly formulate this arms race as an explicit minimax optimization problem with representation learning model.
FORMULATION
In this section we describe our approach towards achieving/meassringg visual representations. We describe visual representation learning and associated algorithms in Section 3.1 and discuss curious representation learning (CRL) Section 3.2. We further detail our architectures in Section 3.3 and describe our environment setup in Section 3.4.
3.1 VISUAL REPRESENTATION LEARNING
To evaluate visual representation learning, we test disentanglement of features at higher layers in neural network. To test this, train linear map frozen feature embedding from backbone architecture specified Section 3.3 on classification task, and report test set accuracy as the visual representation value. We note that Kolesnikov al. (2019) shows that linear maps are adequate
2

---

Under review as a conference paper at ICLR 2020
overall disentanglement of features representation learning. We consider different approaches towards visual representation learning which we detail below.
Colorization Zhang et al. (2016) propose to predict a color of image from its input luminance. Intu- itively, by learning accurate color images, a model must be able to have good visual representation
of objects.
Autoencoding tries to reconstruct an image through bottleneck Bengio et al. (2013) By being forced to compress an image into a compressed representation, model should learn good visual representation of objects.
Autoencoding
Burda et al. (2018b) propose to train a network to predict random features obtained from another network. Such task requires a model to learn behavior of another model, and we surprisingly find this leads reasonable representation learning on VizDoom environments.
RND
We also evaluated rotation prediction based off (Gidaris et al., 2018), but found that it performed poorly in ViZDoom, due to obvious rotation artifacts.
On the ViZDoom environment, our classification task is a curated dataset of different VizDoom objects (monsters, health packs, and guns). For the Habitat environment, our classification task is room scenes from the Places365 dataset (Zhou et al., 2017).
3.2 CURIOUS REPRESENTATION LEARNING
When training agents in reinforcement learning. an agent receives a reward r, at timestep and seeks to produce policy т (s;;; 0p), represented by a neural network that maximizes the expected overall reward
max E (cr:0p) [Eore] 8p
With curiosity, this reward can be decomposed into components of which is extrinsic and r/. which is an intrinsic reward that is dependent on separate model 6m. so that we seek parameters 9m. Bps so that we have
max E (s,,0,, (L) (s:)l: 0p
min E (Eggo.. Om
(2)
for some surprisal objective fe... and modeling objective 9um- This suggests that by setting surprisal objective So,, to be the same as the modeling objective go... we obtain a Minimax objective
max min Exx(,. Op
[E (st)
8m
between an agents policy Op and some world model Om
The above formulation of curiosity now gives us a natural method to improve visual representation learning in both policy and models. By replacing 90.,, (s:) with the loss of generic visual repre- sentation learning algorithm Leep. we can now simultaneously make both the models and policies have good visual representation in an environment. This is because the representation learning model 9m is able learn on more diverse data from policy optimized explore data that confuse the model, while the policy 9 is forced focused to learn good visual representations to confuse the chosen representation learning model. We provide an illustration of the above object in Figure 1.
3.3 ARCHITECTURE SETUP
To balance capacity and enable comparison between representation learning models and between policies, we use identical base architecture for both RI. policies and representation learning models. For both, we use 7x7 convolution with stride, followed by max-pooling layer, followed by 2 residual blocks (with down-pooling), with residual blocks having 64 and 128 filters. We then evaluate linear classification performance performance after flattening the 128 filter residual block. Models are trained and evaluate 168x168 images.
For reinforcement learning policies, when training the base architecture, we flatten observations from the 128 filter residual block and then apply an LSTM cell to which we apply MLPs to decode action and value heads. We use proximal policy optimization (Schulman et al., 2017) to train our policies, using the default models, for RND, we flatten then residual block and then apply an MLP to predict the random target. For colorization, we apply 3 additional convolutions to the 128 filter residual block to colorize. For autoencoding, use 128 hidden units bottleneck with 2 residual blocks of upsampling.
parametrts from Atari Environments. In representation learning
3

---

Under review as a conference paper at ICLR 2020
3.4 ENVIRONMENTS
We evaluate visual representations on two different visual environments:
VizDoom We first evaluate on the ViZDoom (Kempka et al., 2016) environment. We use the a ran- dom level generator Oblige to generate diverse different Doom environments and report configuration details in the appendix. Agents have the ability to move forward, backwards, left and right, turn left and right, attack, speed, and interact with different objects.
Agents are spawned in maps with a large number of rooms (greater than 10), with starting room and ending room through which agents must interact to leave/enter. There are a large number of monsters scattered across the rooms, and agents must either avoid shoot monsters, as well avoid hazardous traps.
We evaluate agent performance on varying numbers of randomly generated training maps. Environ- ments with different number of maps assess our visual representation learning algorithmss in different ways. On a single environment, to obtain the best visual representation, it is necessary to explore over the entire space. In contrast, in the setting with large number of environments, to obtain good visual representation, agents may need just directed way to explore the environments. We compare with set of designed objective listed below:
1. (Counts Based): Agents are given a counts based reward based off divided by the square root of the visitation count at the state. States are based off discretized grid and discretized rotation requirement absolute position and orientation an agent.
2. (Movement Distance): Agents are given reward based off the maximum distance from start location. Agents are given a positive reward every time it moves further from the origin than before and otherwise returns 0 reward.
3. (Shooter Optimization): Agents are given reward based off damage dealt to enemies, number of enemies killed, and any health gained.
Habitat We further investigate visual representation learning in the Habitat (Manolis Savva and Batra, 2019) environment, using the Matterport and Gibson house scans, The Habitat simulator consists of sets of rooms in houses. Under this setting, agents have the ability to move forward or turn left and right. We compare performance with the explicit objective called PointNav (Manolis Savva" and Batra, 2019), in which agents are given a relative distance to goal they must reach.
Atari As a benchmark for measuring the intrinsic curiosity, we also evaluate on the Atari benchmark, using CRL the only intrinsic reward. We evaluate on Atari environments of ReamRider, Breakout, Montezuma's 's Revenge, Pong, Abert, Riverraid, Seaquest and Spacelnvaders as done in (Burda et al., 2018a).
4 EXPERIMENTS
We provide empirical and qualitative evaluations of our hypothesis. First, we evaluate the correspon- dence between policy learning and the visual representation learning of the policy in Section 4. on different objectives. Next we evaluate visual representations that emerge in CRL on both synthetic environments and real images. Finally, evaluate CRL's ability enable good performance on Atari with no extrinsic reward in Section 4.3.
4.1
How DOES VISUAL REPRESENTATION LEARNING CORRESPOND WITH POLICY PERFORMANCE?
We measure the relationship of visual representations and policy reward on ViZDoom (Section 3.4). We systematically varied the number of different training levels generated to train policies with different complexities. We see from Table I that across different objectives and different environment numbers (as proxy of environment complexity), there is strong correlation that is consistent between both the criterion and environment number.
We provide a example visualization of the trend of representation learning with score optimization in Figure 2. In general, we find that visual representations correlate well with initial increases with objective (steps before le7). However, once the maximum score is achieved and training has plateaued, then the visual representation correspondingly decreases slightly (steps after Ie7), suggesting that it may be beneficial to stop RL policy training once reward performance stops.
We further investigated the opposite effect on whether better visual representations would lead to better objective optimization. To test this, we optimize the movement distance training objective
4

---

Under review as a conference paper at ICLR 2020
Score
Percestual Representation
0.7
Accuracy
20
0.5
Time Steps
167
Time Steps
167
Figure 2: An visualization of objective optimization with corresponding perceptual representation learning on the maximum distance from origin objective with 10 environments.
Environment Number
Counts Based
Movement Distance
Shooter Opt.
CRL
Training Reward
Classification Performance
Model
1
0.794
0.719
0.573
Autoencoding
47.63
0.799
10
0.911
0.733
0.549
RND
15.61
0.721
100
0.970
0.922
0.647
Colorization
29.69
0.766
1000
0.689
0.826
0.635
Random
19.61
0.468
Table Correlation coefficient of measured represen- tation accuracy and objectives (reward across different environment numbers in ViZDoom evaluate across 3 different seeds). values all the le-6 le-12 range.
Table 2: Relative performance on Movement Distance (10 levels) objective when using CRL ini- tialized with curious policies with different repre- sentation learning values indicated compared with random policy after frames of training.
with policies initialized with different visual representations from CRL. We report the the relative magnitude of overall reward compared with initializing from random policy after 1 million frames of training in Table 2. Overall, we find that better visual representations obtained from CRL are able to lead better relative performance after I million frames, with policy initialization with CRL with an autoencoding objective outperforming a policy from scratch by 242% in relative performance.
This result suggests that better visual representations both naturally emerge through RI. policy training, and further may be generic objective to optimize to lead to faster reinforcement learning on many different tasks.
4.1.1
WHAT VISUAL REPRESENTATIONS EMERGE FROM CURIOUS REPRESENTATION LEARNING?
Environment Number
Policy
Representation Model Colorization
Autoencode
Culorization
RND
Autoencode
RND
0.748 (0.016)
0.676 (0.022)
0.672 (0.014)
0.864 (0.014)
0.746 (0.018)
0.751 (0.024)
10
0.794 (0.005)
0.733 (0.010)
0.746 (0.013)
0.872 (0.002)
0.742 (0.017)
0.789 (0.020)
100
0.811 (0.006
0.762 (0.007)
0.810 (0.012)
0.863 (0.013)
0.767 (0.010)
0.792 (0.020)
1000
0.819 (0.011)
0.747 (0.005)
0.806 (0.011)
0.855 (0.002)
0.770 (0.002)
0.793 (0.019)
Table Correspondence between both better representations policies and representation learning models (RND = random network distillation) under CRL. Value evaluated across 3 difffernt seeds with standard emror in parentheses. These result show stronger visual representation learning algorithms give stronger visual representations in policies.
We next analyze the effect of CRL. inducing good visual representations. In Figure 4 we show scatter plot of policy representation and model representation under CRL. In Table 3, we provide quantitative numbers of visual representations from CRL on both policies and representation learning models. We find that better representation learning algorithms lead to significantly better policy representations, with autoencoding being the best representation learning algorithm in VizDoom. We see identical rankings between policy visual representations and representation learning models visual representations. These results help suggest that the search for generic representation learning models corresponding lead to policies with even better visual representations.
5

---

Under review as a conference paper at ICLR 2020
Policy Model in CRL
0.85
Model Representation
Linear Finetune Accuracy
0.85
0.75
CRL (Autoencode)
CRL (Colorization)
0.80
0.65
CRL (RND)
Movement Distance
Shooter Optimization
0.75
0.55
Counts Based
0.75
0.80
0.85
100
1000
Policy Representation
Environment Number
Figure 4: Plots of policy visual representations with representation model visual representation un- der CRL from 10, 100, 1000 environments. There is good correlation between policy representation and model representation
Figure 3: Plots of policy visual representa- tions when trained under different objectives with different number of environments
Environment Number
Counts Based
Movement
Shooter Optimization
CRL
Distance
Autoencode
Colorization
RND
0.749 (0.028)
0.717 (0.017)
0.772 (0.019)
0.748 (0.016)
0.676 (0.022)
0.672 (0.013)
10
0.783 (0.046)
0.780 (0.010)
0.679 (0.025)
0.794 (0.011)
0.733 (0.010)
0.746 (0.010)
100
0.818 (0.019)
0.778 (0.013)
0.723 (0.010)
0.811 (0.012)
0.762 (0.007)
0.810 (0.012)
1000
0.623 (0.079)
0.832 (0.002)
0.763 (0.006)
0.819 (0.021) 0,747 (0.005)
0.805 (0.012)
Table 4: Comparison of visual representations learned in policies from CRL and other objectives. We find that CRL with autoencoding objective consistently gives relatively good visual representations while other objective, such counts based may sporadically give better visual representation, but not consistently across environments.
Environment Number
Counts Based
Movement
Shooter Optimization
CRL
Random Policy
Distance
1
0.858 (0.008)
0.862 (0.003)
0.820 (0.005)
0.864 (0.014)
0.834 (0.012)
10
0.865 (0.006)
0,862 (0.001)
0,851 (0.004)
0.872 (0.003)
0.855 0.002
100
0.862 (0.002)
0.860 (0.001)
0.847 (0.002)
0.863 (0.002)
0.853 (0.003)
Table 5; Table of visual representations learned from autoencoding model from data collected from different policies trained in different number of ViZDoom environment. CRL incentives the policy generate diverse data that allows the best model representations across different environment numbers.
In Table 4, we compare visual representations learned in policies trained on fixed objectives to those that emerge CRL. We provide qualitative graph in Figure We find that CRL with an autoencoding objective obtains the best or close to the best visual representations in policies across different environment. While on different environments numbers, either shooter optimization, movement distance, or count based may obtain slightly better policy visual representations, CRL with autoencoding objective has significantly lower variance in observed visual representations, and performs consistently well across different environment numbers. Furthermore, note that CRL does not require any specification of source of task and works generically on any environment, while other objective may require significant manual specification.
We also compare visual representations of representation learning models under different objectives Table For non CRL objectives, we train an autoencoding model (best performing representation learning model on VizDoom) on data collected by policy. We compare CRL model representations with autencoding models trained on data from policies initialized from scratch or trained on counts based, movement distance, shooter optimization, and CRL objectives. We find, regardless of the environment number, that CRL allows the representation learning model to have best possible visual representation in the environment.

---

Under review as a conference paper at ICLR 2020
Map
Random Heatmap
CRL Hcatmap
Figure 5: Comparison of random policy with CRL (curious representation learning) on test level. CRI. (with autoencoding ohjective) is able to induce diverse paths that explore different sets rooms.
We further qualitatively evaluate the effectiveness CRL in inducing exploration of the surrounding environment. Figure 5 shows that CRL is able to lead to effective exploration schemes throughout large mapped environment compared to random policy and a test level.
4.2 Do VISUAL POLICY REPRESENTATIONS TRANSFER TO REAL IMAGES?
We next evaluate on Habitat (Section 3.4) to see if the visual representations learned through CRL transfer to real images from rooms in the Places dataset. We compare representations learned through CRL on RND, autoencoding. and colorization objectives well the PointNav objective.
Random
CRL
Query Image
Nearest Neighbors
Query Image
Nearest Neighbors
Figure 6: Illustration of nearest neighbors
in Places of a CRL model trained on Habitat compared a random network. CRL training simulation transfers to real scene as seen in the beds the row.
On real world images, similar to the VizDoom enviroment, we find that CRL is able to enable good visual features in Table 6. Trends between better model and visual representation hold, similar to in VizDoom with CRL, with the most effective representation learning algorithm being colorization. We find that CRL enables us to train representation learning models and policies to have significantly better visual features then agents optimizing the PointNav goal, with linear classification accuracy of 0.193 compared to 0.086 in PointNav for policies and 0.253 compared to 0.211 for the representation learning model. These linear classification accuracies significantly better than random (0.084) and somewhat close to the linear classification accuracy of a colorization model directly trained on Places room scenes (0.324).
7

---

Under review as a conference paper at ICLR 2020
PointNav
CRL (AE)
CRL (CI.)
CRL (RND)
CL. (oracle)
Random
Policy 0.086
0.175
0.193
0.164
Model
0.211
0.1066
0.253
0.085
0.324
0.084
Table 6: Comparison on linear finetuning classification accuracy on room scenes in Places using policies and representation learning models (using colorization or specified CRL objective) trained from data from Point Navigation and different CRL objectives. We also compare with colorization model trained data from Places room scenes and randomly initialized model.
To qualitatively study the visual representations in Habitat, we construct nearest neighbors in embed- ding space of a trained CRL colorization model and a random model in Figure We find that CRL trained models Habitat are able cluster certain images in Places room scenes together such as beds. 4.3
DOES INCENTIVIZING BETTER VISUAL REPRESENTATIONS LEAD TO MORE CURIOUS BEHAAIIOR?
BeamRider
Breakout
Pong
Obert
1500
CRL
RND
50
CRL
-10
CRL
RND
3000
CRL
1250
Dynamics
40
Dynamics
-12
Dynomics
w
Dynamics
-14
2000
1000
30
750
20
-16
500
-18
1000
10
250
-20
2000000
4000000
0
2000000 4000000
2000000
4000000
2000000 4000000
Riverraid
Seaquest
Spaceinvaders
MontezumaRevenge
3500
CRL
700
CRL
3000
RND
400
600
RND
Dynamics
600
Dynamics
2500
500
2000
RND
400
400
M
1500
Dynamics
300
CRL
1000
200
100
500
100
Dynamics
2000000
4000000
0
2000000 4000000
2000000
4200000
2000000 4000000
Figure Comparison of CRL (curious representation learning with autocncoding objective) vs RND vs Dynamics on Atari using only intrinsic reward across different seeds. CRL performs favorably and gets the highest score in 6 of the 8 evaluated environments.
Next we investigate whether using CRL intrinsic reward gives better performance than other curiosity models, We compare with Random Network Distillation and Forward Dynamics in Figure 7 across 3 different random seeds. We choose the autoencoding objective CRL for the representation learning objective, as we find it gets the best visual representations on VizDoom.
Overall, we find that CRL performs favorably compared to both RND and forward dynamics, and gets the best overall score 6 of the 8 evaluated environments. This suggests that obtaining best visual representations may also be a manner to improve curiosity.
CONCLUSION
In this paper, we have shown visual representations correspond and help reward optimization. Motivated by this insight, we propose a new method, CRL, that allows to get improved visual representations in policies through better visual representations model. We further illustrate that these better visual representation can provide incentives to explore more in reward scenarios. We hope that our results will inspire further exploration on both better visual representation learning models/policies and better reward optimization.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Marcin Andrychowicz, Bowen Baker, Macick Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Leamning dexterous in in-hand manipulation. arXiv preprint (xXiv:8808.0017, 2018.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, pages 1471-1479, 2016.
Yoshua Bengio, Jéröme Louradour, Ronan Collobert, and Jason Weston, Curriculum learning. In ICML, 2009. 2
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: review and new perspectives. IEEE TPAMI, 35(8):1798 (538)):798-18222) 1828, 2013. 2,3
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei Efros. Large-scale study of dediosity-driver learning. arXiv preprint arXiv:1808.04335. 2018a. 2,
Yuri Burda, Harrison Edwards. Amos Storkey, and Oleg Klimov. Exploration by random network distillation.
arXiv preprint 11..::88100118/// 2018b. 2,3
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations, arXiv preprint arXiv: 1803.0778,, 2018. 3.3
Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 1808.888867777 2018.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. In Advances in Neural Information Processing Systems, pages 1109- 1117, 2016. 2
Michat Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojeiech Jaškowski. Vizdoom: A doom- based ai research platform for visual reinforcement learning. In 2016 IEEE Conference on Computational Intelligence and Games (CIG), pages 1-8. IEEE, 2016. 4
Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting self-supervised visual representation Icarning. arXiv preprint arXiv: 190199991000,... 2019. 2
Oleksandr Maksymets Yili Zhao Frik Wijmans Bhavana Jain Julian Straub Jia Liu Vladlen Koltun Jitendra Malik Devi Parikh Manolis Savva*, Abhishek Kadian and Dhruv Batra. Habitat: platform for embodicd ai research. arXiv preprint arXiv: 1904,01201, 2019. 4
Volodymyr Mnih. Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. In NIPS Workshop, 2013.
OpenAl. Openai five. https: //wlgg.....o////ooo oponai. cooon/opnaa-fiveee 2018.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self- supervised prediction. In /CML, 2017. 2
Nicholas Roy and Andrew McCallum. Toward optimal active learning through monte carlo estimation of error reduction. 2
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv: 0el::/777.6600 707.06347, 2017.
Burr Settles. Active learning literature survey. Technical report, University of Wisconsin-Madie Department of Computer Sciences, 2009. 2
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panncershelvam, Mare Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap. Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nat., 529(7587):4844497 2016.
Richard Zhang, Phillip Isola, and Alexci A Efros. Colorful image colorization. In ECCV, 2016. 2,3
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEFF transactions on pattern analysis and machine intelligence, 40(6): 1452-1464, 2017.

---

Under review as a conference paper at ICLR 2020
A
APPENDIX
A. DOOM SETUP
We the following setup for setting the Oblige random map generator.
Configuration
Value
Length
Regular
Size
Regular
Health
Normal
Weapons
Very Soon
Theme
Jumble
Game
Doom
Mons
More
Ammo
More
Strength
Harder
Outdoors
Mixed
Doom Level
5
Table 7: List of configurations used for the Oblige game engine.
A.2 NEAREST NEIGHBORS DOOM
Random
CRL
Query Image
Nearest Neighbors
Query
Nearest Neighbors
Image
Figure 8: Illustration nearest neighbors Doom CRL(AE) policy trained on Habitat compared to random network. Nearest neighbor in CRL space is able to cluster more visually similar images
We further show nearest neighbor images on VizDoom in Figure 8. The leftmost column is the query image while the other 4 columns are the 4 nearest neighbors in embedding space. Training through CRL allows clustering of various doom objects.
10


Reviews:
[{"review_id": "SkepPxT0YB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a framework called curious representation learning (CRL) which uses a better visual representation in RL. They show that better visual representation helps reward maximization.\n\nI have to recommend rejection for this paper. It appears 1) the idea of using curiosity is not originated from this paper; 2) I do not see what is a \"better visual representation\"; 3) the comparison with baselines does not show that the new method is consistently better.\n\nThe paper is also very hard to read. I would think the name \"curious representation learning\" means \"representation learning is curious\". There are many inaccurate languages used in the paper. To list a few: \"complex behavior\", \"in curiosity\", \"disentanglement\"... I do not really understand what does it mean.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hyg8m3oTtr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper formulates curiosity based RL training as learning a visual representation model, where the policy tries to maximise the loss of a shared visual model minimising an auxiliary task (such as autoencoding the input).\n\nCuriosity is an important topic in the RL field and this paper is well motivated. I also like the approach taken as it looks into this problem through the lens of better representation learning (LR) and arguing that with focusing on better LR and maximising model loss for novel scenes, we are going to get also better overall performance.\n\nHowever, there are a few key question marks that are still open and I would suggest them to be answered explicitly in the paper:\n\n1) What is the relationship with methods that use auxiliary tasks for unsupervised training in RL (e.g. Jaderberg et al, ICLR 2017)? It's clear that this method doesn't use any extrinsic reward function but the underlying architecture is similar.\n\n2) Similar to above, the comparisons and contrasts to Burda et al, ICLR 2019 could be made more explicit as the objective functions such as autoencoding which seems to be working well in this paper has also been studied in that work.\n\n3) Continuing with comparisons, it's not clear if this method delivers better performance compared to other curiosity based methods. For examples, the top scores in Fig 7 are considerably lower than those achieved in Burda et al, ICLR 2019 (Fig 2). Similarly, we don't know how the method compares to state-of-the-art on other tasks considered in the paper. As a result, the paper lacks good benchmarking against state-of-the-art in this space and discussion on pros and cons.\n\n4) It seems that in Tab 1 the correlation collapses for the last row, any reason why this is happening?\n\n5) It would be good to add both a system diagram as well as a network architecture to clarify how everything is wired.\n\n6) The training details are missing, both in terms of hyperparameters as well as optimisation strategy for solving minmax.\n\n7) Minor: RND is used in the experimental section to refer to both random feature prediction and random network distillation, so would be better to use different references.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1lPAuj2tS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an empirical study of using error reduction as a curiosity measure. The authors consider an auto-encoder model, a colorization model and RND as intrinsic motivation signals. I find the write up very unclear and have trouble understanding what the claims are and how they are backed up. \n\nMajor points:\n* About the claims as stated on page 2: 1) The first claim I don't understand, I think what is meant is on navigation tasks they find \"their measure of representation learning\" (proposed in kolesnikov 2019) seems to correlate well with reward optimization. In 3.1 to back this claim they claim to test disentanglement but seem to test classification. I see no reason to not put that part in 4.1. 2) The authors claim to propose a new method: it can't be a separate representation network to derive rewards because that is what burda et al. and many many others do, it can't be the minimax formulation because that has been known for a while (e.g. predictability minimization schmidhuber) so I am not sure what the claim is about. What exactly is novel about the model. 3) CRL seems at best to outperform baselines on beamrider, qbert and riverraid but the results are impossible to assess. We don't know what the x axis is in figure 7 (is it frames, with or without repeats, is updates etc). Pong -12 is far from learnt for instance and its one of the easiest games. \n* suprisal objective and modeling objectives are very high level concepts that we can talk about in the introduction and conclusion but much more precise terms need to be employed in the model exposition. The readers need to know what sort of properties they should have ideally easily identify examples (without having to read 2 papers and a large survey). I would start with the minimax formula and then explain what is considered, how the different intrinsic rewards are added, if they are normalized, how they are weighted etc. etc.\n* the details about failed experiments, environments and architecture should IMO be relegated to experiments\n* It should be very clear early on that the model is separate from the representation. \n* auto-encoders are a large family of models and it is not clear from the paper which exact model is meant by the authors. Also, citing Bengio 2013 is NOT a valid citation for auto-encoders. The right citation depends on the model you apply please use that! \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates using "curiosity" to improve representation learning. This paper is not ready for publication. The main issues was the reviewers found the paper did not support the claim contributions in terms of (1) evaluating the new representations and improvement due to the representation, and (2) the novelty of the method compared to the long literature in this area. In general the reviewers found the empirical evidence unconvincing, and the too many missing details.

The results in this paper have many issues: claims of performance based on three runs; undefined error measures; bolding entries in tables which appear not significantly better without explanation; unclear/informal meta-parameter tuning. 

Finally, there are some terminology issues in this paper. I suggest an excellent paper on the topic: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3858647/  
Decision: Reject