Diagnosing and exploiting the computational demands of videos games for deep reinforcement learning
Authors: Lakshmi Narasimhan Govindarajan
Rex G Liu
Drew Linsley
Alekh Karkada Ashok
Max Reuter
Michael Frank
Thomas Serre
Venue: Submitted to ICLR 2023 (2023)
Keywords: Cognitive Science
Deep Reinforcement Learning
Perceptual Grouping
Neuroscience

Abstract: Humans learn by interacting with their environments and perceiving the outcomes of their actions. A landmark in artificial intelligence has been the development of deep reinforcement learning (dRL) algorithms capable of doing the same in video games, on par with or better than humans. However, it remains unclear whether the successes of dRL models reflect advances in visual representation learning, the effectiveness of reinforcement learning algorithms at discovering better policies, or both. To address this question, we introduce the Learning Challenge Diagnosticator (LCD), a tool that separately measures the perceptual and reinforcement learning demands of a task. We use LCD to discover a novel taxonomy of challenges in the Procgen benchmark, and demonstrate that these predictions are both highly reliable and can instruct algorithmic development. More broadly, the LCD reveals multiple failure cases that can occur when optimizing dRL algorithms over entire video game benchmarks like Procgen, and provides a pathway towards more efficient progress.

Under review as a conference paper at ICLR 2023
DIAGNOSING AND EXPLOITING THE COMPUTATIONAL DEMANDS OF VIDEOS GAMES FOR DEEP REINFORCE- MENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Humans learn by interacting with their environments and perceiving the outcomes their actions. A landmark in artificial intelligence has been the development deep reinforcement learning (dRL) algorithms capable of doing the same in video games, on par with or better than humans. However, it remains unclear whether the successes of dRL models reflect advances in visual representation learning, the effectiveness of reinforcement learning algorithms at discovering better policies, both. To address this question, we introduce the Learning Challenge Diagnos- ticator (LCDD, a tool that separately measures the perceptual and reinforcement learning demands of task. We use LCD discover novel taxonomy of chal- lenges in the Procgen benchmark, and demonstrate that these predictions are both highly reliable and can instruct algorithmic development. More broadly, the LCD reveals multiple failure cases that can occur when optimizing dRL algorithms over entire video game benchmarks like Procgen, and provides a pathway towards more efficient progress.
INTRODUCTION
Gibson famously argued that "The function of vision is solve the inverse problem and construct a veridical description of the physical world. [... It] is to keep perceivers in contact with behaviorally relevant properties of the world they inhabit" (reviewed in Warren 202I). The field of deep reinforcement learning (dRL) has followed Gibson's 's tenet since the seminal introduction of deep Q-networks (DQN) MMiih al. 2015). DQNs rely on reward feedback to train their policies and perceptual systems at the same time to learn to play games from a tabula rasa. This end-to-end approach of training on individual environments and tasks has supported steady progress in the field of dRL, and newer reinforcement learning algorithms have yielded agents that achieve human or super-human performance
variety challenges from Chess to Go and from Atari games to Starcraft (Mnih al. 2015 Silver et al., 2017 2018 Vinyals et al.) 2019). But Gibson also argued that the ecological niche of animals allows them to exploit task-agnostic mechanisms to simplify the perceptual behavioral demands of important tasks, like how humans rely optic flow for igation (Warren) 2021). In the decades since Gibson's 's writings, it has been found that humans can efficiently find learn bespoke perceptual features that aid performance on single task (Li al., 2004 (Scott et al.) 2007 Roelfsema et al.) 2010 Emberson (017)). or they can exploit previously learned generalist representations and task abstractions that are useful across multiple tasks and en- vironments (Wiesel & Hubel 1963 Watanabe et al. 2001 Emberson, 2017 Lehnert et al. 2020 O O'Reilly 2001). While there have been attempts at building similarly flexible dRL agents through meta-reinforcement learning (Frans et al. 22018 8 Xu et al.) 2018 2020 Houthooft et al. 2018 Gupta et al. 2018, Chelu et al., 2020) Pong et al. (022) I), these approaches ignore the complexities of per- ceptual learning and carry large computational burdens that limit them to simplistic scenarios. There a pressing need for approaches to training dRL agents that can meet the computational demands of wide variety of environments and tasks.
One way to build generalist agents is to first reliably diagnose where the computational challenges of given enviromment and task lie and adjust the agent to those demands. Is the perceptual challenge onerous? Is the reward signal for credit assignment especially sparse? Even partial answers to these questions are instructive for improving an agent, for instance, by pre-determining the extent which

---

Under review as a conference paper at ICLR 2023
Bigftsh
Easy Hard
Hard yr
Starpilot
Fruitbot
Bossfight
Reinforcement learning challenge
Maza
Plunder
Leaper
Dodgeball
Cavellyer
Gavelllyrr
Fruiboot
Dodgeball
Plunder
i-T
Ninja
Nova
Jumper
Coinrun
Chasee
Clibber
Easy Easy
Bosslight
Starpilot
Hard d
Easy
ip
(a)
Ninet
Climber
Bigfish
(d)
Perceptual challenge (g))
Perceptual challenge (/))
Reinforcement learning challenge (1/
Meaisure impact different visual representations on performance.
Meesure impact reward stochasticity performance.
Dodgeball:
Reward:
Reward: +1
Bigfish: Eat fihee smaller
Visual representations sinuctured
You (arean) 130 the
other
Reward: 40
Reward: +0
Reward:
difference total reware
Reward: +4.
-ffewerd: +1
Reward: +1
Compute difference in total roward
Most drrustueed
Fig/groun seg.
Samantic segmentation
Reward: +1
'Reward: +0
Reward: +1
Acquired: 22
Acquired:
Aequired: +2
(b)
(c)
Semantic segmentaiion
Semantto segmentation
Figure 1: The Learning Challenge Diagnosticator (LCD) separately measures the reinforce- ment learning or the perceptual challenge of an environment and task. (a) We use LCD to compute perceptual and reinforcement learning challenges in each game of the Procgen Benchmark. (b) Perceptual challenges are assessed by comparing the total reward accrued by agents that learn over visual representations with different amounts of structure: pixels, figure/ground segmentation masks, and semantic segmentation masks (see Appendix. (A.6). (c) Reinforcement lcarning chal- lenges are computed by feeding agents perceptually organized scenes (i.c., semantically segmented; second and third rows), and then comparing the total reward accrued by agent playing in environment with stochastic manipulations of reward (randomly masked and modulated; third row) to another agent playing in the normal environment (second row). (d) The taxonomy of Procgen reveals each game's 's relative perceptual and reinforcement learning challenge.
relies on feedback from the world tune its policy and perception versus drawing from previously learned representations and task abstractions. The introduction of diverse video game challenges for dRL, such as the Procgen Benchmark (Cobbe al., 2020), can serve as starting point for this investigation. For example, take the game "Plunder" from Procgen (Figure Th). Th) Plunder has simple gameplay rules but poses visual challenge: an agent is asked to shoot all objects that look like provided cue. The difficulty here lies in assessing whether each object in the environment is the same or different than the cue; a visual routine that is difficult to learn for neural networks (Vaishnav et al.) 2022 Kim al. 2018). For this reason, agent that draw from prior experience in learning relevant perceptual routines may perform better than one with a perceptual system tuned for this specific task from scratch. In contrast, the objects and environments of game like "Leaper"
2

---

Under review as a conference paper at ICLR 2023
(Figure [ can be identified by color. The game asks agents to use those objects to get from one side of the screen to the other, a task that can likely be learned from reward feedback.
Contributions. The ability to systematically, autonomously, and reliably identify appropriate strategy for learning a task in a given environment would enable the design of generalist dRL agents that can flexibly adapt challenges as humans do. To make progress towards this goal, we introduce the Learning Challenge Diagnosticator (LCD), a novel tool that identifies the specific computational challenges of an environment and task. When applied to video games, like those Procgen, the LCD measures the difficulty of encoding task-relevant perceptual properties versus the difficulty of optimal policy discovery (i.e., determining which actions lead to rewarding outcomes).
We develop modified, parameterizable version of Procgen, to systematically manipulate the perceptual and reinforcement learning challenges presented to agents in each individual game. We provide our version of Procgen and all experimental code at butttts: hnttps://noorymmuus /aanamonnoooouuso.m../ 4open.
science/r//cd-procpe
The L.D reveals a novel taxonomy of computational challenges in the games of our modified Procgen. Some are more visually complex, some more challenging from the point of view reinforcement learning, and others strain agents across both of these axes. This taxonomy is preserved across different dRL algorithms and the heterogeneity of the challenges associated with the Procgen games suggests that adopting a single "one size fits all" approach to learning, as is standard in dRL, suboptimal.
To address the visual challenges of Procgen games, we adopt a self-supervised visual front-end, which learns perceptual groups from motion cues games without reward feedback. To test reinforcement learning challenges can, in principle, be alleviated, we develop a proof-of-concept approach with reward shaping (Skinner 1938).
We exploit the computational taxonomy of Procgen revealed by the LCD to shape the design of agents for each game. These agents learn more efficiently and perform significantly better than one-size-fits-all dRL agents, suggesting the potential for "adaptive" agents.
RELATED WORK
Representation learning in dRL Perhaps the main contribution of DQNs was to demonstrate that is possible to jointly learn visual representations and policies through reward maximization from a tabula rasa to achieve high scores on Atari games (Mnih et al. 2015). Despite this extraordinary achievement, the extrinsic rewards available to algorithms like DQNs have been found to yield poor sample complexity, limit model scale, and cap maximum performance (Jaderberg et al., 2016 Laskin ct al., 2020). For this reason, there has been a growing number accounts showing that standard reward learning be augmented with auxiliary learning objectives and/or pre- pre-training via self- supervision to take steps towards ameliorating limitations of reward- based learning (Jaderberg et al.]
2016 Shelhamer et al., 2017: Stooke al. 2021 Laskin et al. 2020; Dittadi al.] 2021 [Zhang et al. 2020 Radosavovii et al. Xiao et al.). Relatedly, it has been shown that initializing agents with
structured visual representations can lead faster learning and better performance in reinforcement
learning (Tassa et al. 2018: Davidson & Lake 2020: Laskin et al. 20200 Stooke et al. 2021).
Measuring visual challenges in artificial intelligence An essential contribution that the cognitive sciences have made to artificial intelligence over the past decade has been the introduction of com- putational challenges that are easy for humans solve but strain the capabilities of neural networks. These challenges have identified deficiencies in the solutions deep neural networks (DNNs) tend to learn for contour tracing (Linsley et al.) 2018b Kim et al.. 2020 Tay et al., 2021). segmenta- tion (Kim et al.) 2020). object tracking (Linsley et al., 2021 object recognition (Geirhos et al.) 2021
2018), and visual reasoning problems (Fleuret et [0.] Kim et al. 2018 Vaishnav et al,
2022
The essential lesson from these studies is that measuring model performance data sampled from the same distribution that training data came from is not sufficient for establishing challenges and measuring progress them (Funke et al. 2021). Instead, it is critical to test models on out-of- distribution samples to find and evaluate challenges (eerrhos et al.) 2022 2018) Linsley et al. 2021 2020). We adopt this strategy measure dRI. agent performance this work.
3

---

Under review as a conference paper at ICLR 2023
Self-supervised training
Learning predict optical transformm to fef
difference
Flow encoder
166
Decoder
(b)
Frames ( and fef
Predicted
True
Testing
Notion encodings pinsedd dRi agenta for leamning
Object
16C
encoder
(a)
(c)
Frome
Figure 2: A DNN that is self-supervised to predict future frames also learns a concomitant representation of object-ness that is useful for reinforcement learning. (a) Our routine for self- supervising perceptual representations that can solve challenges identified by the LCD utilizes three networks: ResNer18 (He al. 2013) for predicting the optic flow between frame / frame + (Flow encoder), a ResNet18 feature pyramid network (FPN) with a 16-channel recurrent neural network (Linsley et al., (018a) to encode the content of frame (Object encoder), and an FPN- ResNet18 decoder that receives the two encoder's outputs to transform frame t to frame t + 1 via a differentiable warping module. After optimizing for next-frame prediction, the Object encoder learns object-like representations that can be used for reinforcement learning without additional training. This approach generates reasonable segmentations the important objects in Procgen games, including Bigfish, Starpilot (b), and Bossfight (c).
Biologically inspired mechanisms for improving neural networks There are multiple examples of the DNN challenges being resolved by the introduction of biologically inspired mechanisms. Mechanisms for attention have made significant contributions by enabling DNNs to limit noise and clutter, amplify task-relevant features, and subsequently achieve better performance and sample efficiency during learning (Linsley al. 2019). Others have found that horizontal connections, inspired by those in the early visual cortex of primates, can resolve DNN limitations solving contour tracing tasks (Linsley al. 2018a). There is also evidence that forcing dRL agents to acquire compositional abstractions, inspired by those described in the cognitive sciences, helps them learn policies that generalize to novel environments (Lehnert
al. 2020). The mechanisms propose to solve the computational challenges revealed by the LCD in Procgen are similarly inspired by solutions to those problems adopted by primates and humans.
Metrics for measuring progress in dRL Progress in dRL has been classically measured by the aggregate rewards and sample efficiency of models in games (Taylor & Stone 2009 Agarwal al. 2021 Kirk et al., 2021). The Procgen benchmark took a major step forward beyond what is stan- dard the field by measuring agent performance on game parameterizations that fell outside of the distribution used for training. While current metrics can adjudicate between different algo- game, they do not offer insights into the specific computational challenges that agents face. Our LCD therefore represents a major conceptual leap beyond current approaches to measuring progress in dRL, by quantifying the specific perceptual and reinforcement learning challenges found in given game, and enabling dRL researchers to more precisely resolve the deficiencies agents.
rithms based on performance
THE LEARNING CHALLENGE DIAGNOSTICATOR
The ability to identify and measure the computational challenges a dRL agent faces is an important step toward designing better algorithms for reinforcement learning. However, to the best of our knowledge, this problem has not yet been addressed the field. The LCD achieves this goal through three-step procedure, which reveals the computational challenges facing agent. By revealing the computational demands of every game, the LCD enables more efficient algorithm development and insights into how these demands may interact not during learning (see Section 4).

---

Under review as a conference paper at ICLR 2023
Here, focus on identifying the perceptual versus reinforcement learning challenges offered by any specific game, yielding the following steps: (i) We begin by modifying game to put its percep- tual representations and reinforcement learning problem under experimental control. (ii) Next, set of agents learn to play different versions of game. In each version, either the perceptual challenge is perturbed while the reinforcement learning challenge is held constant, or vice versa. These per- turbations are implemented by sampling either different perceptual representation of the game or reward scheme. (iii) Finally, we measure the integrated reward trajectories of agents in response to each type of perturbation. By separately averaging the scores of agents facing perceptual perturba- tions and reward perturbations, we compute scores describing the perceptual challenge (ω) and the reinforcement learning challenge (1)) of each game (Fig. D and c).
Procgen benchmark. We applied the LCD to Procgen (Cobbe et al. 2020). challenging dRL benchmark consisting of 16 games, each with distinct graphies and gameplay. Procgen is open- source and implemented in Python and C++, which made it possible to customize variety of game parameters for implementing the LCD. The parameters we manipulated were game assets (sprites and backgrounds) and reward functions.
Procgen generates game levels procedurally, enabling precise control over the levels used for train- ing versus testing and as a result, proper tests of generalization. We followed the training and testing protocol outlined in (Cobbe et al. (020)) in each of our experiments. In brief, all agents were trained for 200M steps across 500 training levels and evaluated on Procgen's 's held-out games measure generalization.
dRL algorithms. We apply the LCD to agents trained with either the standard and popular prox- imal policy optimization (Schulman et al. 2017) (PPO) or phasic policy gradient (PPG) (Cobbe et al. 2021), a more recent algorithm that attained saateoof-theaar performance in Procgen. PPO hyperparameters were selected to match those used in (Cobbe et al. 2020 for Procgen's 's hard mode. Similarly, for PPG we used the hyperparameters from (Cobbe et al. (021). The walltime for single training run of one of these agents averaged 24 hours per game on a standard 16-core Intel Xeon Gold 6242 CPU with a 24G Titan RTX GPU. We ~ 450 experiments across 32 GPUs, which took over 10K GPU hours of computing time. Agent performance at test time was computed using the interquartile mean (IQM) of the final returns (Agarwal al., 2021
LCD: Perceptual challenge. To perturb the perceptual complexity in game and measure its perceptual challenge, we varied the visual input provided to an agent when learning to play the game. Motivated by decades of cognitive science work on the visual representations that humans rely on for efficiently learning to interact with their environments (Ullman 1984, Roelfsema et al.) 2000: Roelfsema 2006b), we tested how well agents could learn to play games when given three different types of visual inputs with varying structure: the original frame, figure- figure-ground ground segmentations of each frame, or semantic segmentations of each frame (Fig. [p.c). sigure-ground, segmentations labeled background and object pixels differently, whereas semantic segmentations labeled each element of the frame differently, with consistent labels across frames (see Appendix. A.6).
We measured the perceptual challenge of each game by comparing the performance of three agents trained to solve it, where each agent learned a policy over one of the three different types of vi- sual inputs (Appendix Fig. 6). In this condition, rewards and the reinforcement learning challenge in games were kept as normal. We began by computing performance for each agent as the area under the cumulative reward curves (AUC) achieved by agents during generalization (Appendix Fig. 8 which implicitly captured both maximum performance and the sample efficiency needed to get there. Next, we computed the relative change AUC from the original frame figure-ground and semantic AUCs, and took the average of these ratios as our final score ó (Fig. [p). The percep- tual challenge score of each game was then normalized to the range [0, 1] according to the min and max & across all games (Appendix A.2). A game with a á approaching indicates that it presents greater perceptual challenge for reward-based learning, whereas small e means that reward-based discovery of task-relevant features is sufficient to perform well that task.
LCD: Reinforcement learning challenge. To perturb reinforcement learning in game, ma- nipulated the sparsity of rewards available to an agent. By manipulating reward sparsity, we were able to degrade "learning signal" available agent while also preserving the optimal policy,
5

---

Under review as a conference paper at ICLR 2023
Easy Ø
Leaper
Chaser
Bossfight
Bigfish
MM
Score
Self-supervised Pixels
Hard db Starpilot
10.7
Dodgeball
16.4
12.3
Climber
Mean normalized score
Score
0.3
0.0
All games Hard
Easy
(a)
0.5
0.7
0.4
(b)
100M
Figure 3: Self-supervised pre-training for learning object-ness addresses perceptual challenges in Procgen identified by the LCD. (a) We measured the impact of our self-supervised perceptual front-end for learning games classified having perceptual challenges (Hard Φ) or not (Easy Φ). The game which is helped the least by the front-end, "Leaper", requires agents to recognize colors feature that not preserved the front-end.
(b) When looking at aggregate performance across the entirety of Procgen, the benefit of pre-training is only seen on those games that the L.C deems as perceptually challenging (Hard (), indicating that the current state progress in dRL is being slowed by inability to find benchmarks (or subsets of benchmarks) which are aligned with the algorithmic solutions being tested. One-tailed t-tests assessing the effect learning from the perceptual front-end versus pixels are denoted by lines, * = 0.05.
across all perturbations (Appendix A.3). We did this by decreasing the probability p of receiving a reward every rewarding event, while also scaling reward magnitudes by 1/p preserve the ex- pected magnitude of rewards. We used reward perturbations of E {1, 0.75, 0.50, 0.25} (Appendix Fig. 7). All agents were given semantic segmentations as inputs, which minimized the perceptual challenge of games and controlled for potential perceptual confounds, such as how the perceptual complexity of the pixel inputs of given game may non-linearly interact with this reinforcement learning perturbation.
We measured the reinforcement learning challenge of each game by comparing the performance of agents trained to solve each of the reward perturbations (Appendix Fig. (9). First, we again computed AUCs the generalization reward curves for each agent, as above, but here for different values of Next, we computed the absolute AUC change from an agent trained averaged across all changes, yielding the score v (Appendix AA.4), which was normalized to [o, 1] in the same way Φ vo close to 1 means that it presents significant reinforcement learning challenge. Conversely, a game with a small và indicates that reward sparsity is not the primary challenge.
to the next and
Diagnosing the computational challenges the Procgen benchmark. We began by applying the LCD to PPO agents trained to solve each game in Procgen. Doing revealed a clear taxonomy of the computational challenges of each game (Fig. [d): Fruitbot, Jumper, Miner, Heist, Chaser, and Bossfight are, in relative terms, perceptually easy and have easy reinforcement learning. Cave- flyer, Dodgeball, Plunder, Coinrun, Ninja, Climber, and Starpilot are perceptually hard and have easy reinforcement learning. Bigfish and Leaper perceptually easy and have hard reinforcement learning, and Maze is challenging for perception and reinforcement learning. The challenges of Maze, in partieular, are likely driven by the need trace and plan paths through maze; routines

---

Under review as a conference paper at ICLR 2023
Dodgecall
1-
Miner
Dodgiball
Pluncer
Jumper
Covetlyer
Jumper (ighth
Nirja
Crvvetyer
Plunder
Heial
d/PPG
Booslight
Helst
Coinrun
Fruitoll
Fruibol
Colnrun
Loaper
Miner
Cliber
=
0.63, p 0.01
Starplot
Chaser
0,45, 0.05
Lesper
0
0-
Boestight
Begfish
POO
PPOO
Figure 4: LCD measurements of the perceptual and reinforcement learning challenges in Proc- gen translate across dRL algorithms. Measurements of visual challenges Φ (left) and reinforce- ment learning challenges 3/ (right) faced by PPO agents were significantly correlated with the chal- lenges of PPG agents, indicating that results derived from applying the LCD to simpler and lower- overhead algorithms can guide the development of agents pushing the state-of-the-art.
which are known to be challenging for neural networks and draw upon feedback mechanisms in humans (Linsley et al. 2018a Roelfsema et al. 2000; RReefssmaa 2006a Ullman 1984).
VALIDATING LCD TAXONOMIES BY BUILDING BETTER AGENTS
Our general approach for validating the computational taxonomy of the LCD is to demonstrate that the gains of targeted solutions to the challenges coincide with the LCD''s predictions. We develop separate solutions for perceptual challenges and reinforcement learning challenges and describe those methods and results here.
Developing solutions to perceptual challenges. We began by constructing new perceptual front-end that we believed could address the perceptual challenges of Procgen games. Inspired by the reliance of humans and other animals on optic flow to effectively and efficiently navigate through their environments (Warrnn et al. 2001 Warren 202I), we turned to motion perception as the base inductive bias for our agents. Specifically, we rely on the motion of objects in the world to extract perceptual groups that could instruct appropriate behavior.
It has been found that training DNNs to predict the optic flow between successive frames of video can induce the ability to segment object-like superpixels from complex scenes (Liu et al. 2021). Here, we build off this prior work to develop self-supervised pre-training strategy for solving per- ceptual challenges in Procgen. Our architecture consists of three separate DNNs: (i) a ResNet18 (He) al. 2015) feature pyramid network (FPN) (Lin et al., (016) for predicting the optic flow (Flow encoder) between two frames from successive timepoints of sequence, frame t and frame f + 1, (ii) a biologically inspired recurrent neural network (Linsley al. 2018a) (Object encoder) for en- coding the contents of frame t, and (iiz) fully convolutional (Long et al. 2013) decoder that learns how to use the two encoders' outputs to match frame to frame + via a differentiable warping module.
This model was pre-trained to learn visual representations in a purely self-supervised manner, by minimizing the mean squared error between two successive frames sampled from the seven games of Procgen which had consistent motion (Chaser, Leaper, Dodgeball, Climber, Bossfight, Starpilot, and Bigfish). We sampled frames for training from 14,000 videos (2,000 per game) in batches of 4 for 50k iterations using the Adam optimizer (Kingma & Ba 2014) and a learning rate le 4. As an alternative to approach for self-supervision. the-art features for one-shot recognition from a clip- clip-ViT-B-33 model pre-trained on 400M natural images and captions (Radford et al. 2021).
also evaluated the effectiveness of state-of-
7

---

Under review as a conference paper at ICLR 2023
Perceptual challenge evaluations. We the LCD predictions of the perceptual challenges o in each game by testing if preprocessing frames with our self-supervised front-end improved agent performance more for games with higher values of & (i.e., the front-end was fixed and not trained with reward). Because our self-supervised front-end relied on the game motion to extract perceptual groups, we focused our analysis on the same seven games with dynamic elements that it was pre- trained on. As predicted by the LCD, our front-end improved performance significantly more for pereeptually challenging games (hard σ) than perceptually simple games (easy o; Fig. 3b). Overall, the benefit of this front-end correlated significantly with the & predicted for each game (Spearman's s P 0.79 between & and the AUC of performance curves, p < .05). To understand the extent that our results merely demonstrate the value of pre-trained visual representations, we repeated our analysis while using a pre-trained CLIP as the perceptual front-end for agents. But while CLIP embeddings yield state-of-the -hat--rtt performance in one-shot classification (Radford et al. [202T), they did not help agents learn more effective policies, faster (Appendix Fig. [2). other words, LCD's perceptual challenges are more readily addressed by algorithmic solutions that induce object-like representations.
Guiding algorithmic development with the LCD, Given the success of our self- obvious
co-training pereepttiooo peaiieeeee
Bigfish
supervised perceptual front-end, approach towards building better agents is to simply use this front-end on every game. However, the impact of our front-end on performance was significantly blunted when measured on all games (the difference between
together,
Fruithot
Starpilot
perception action
Beneftt from
Jumper
Dodgoball
Caverlyer
Ninja
green bars in Hard & versus All Games,
Chaser Maze
о p
Plunder
Fig. BD). This finding points to a larger problem in algorithmic development for RL:
Cainrun
Bossfight
Leaper
Heist
Miner
without a reliable approach for diagnosing the specific computational challenges in a game, the effectiveness of reasonable solutions can be blunted or even appear to hurt agents, as our front end nominally does here.
co-training perception and doesn help
Beneflt from learning perception and action separatoly
Figure 5: A subset of games in Procgen see a synergistic benefit from co-training percep- tion and action/policies end-to-end. We com- pared the magnitude of improvement over base- line (PPO/pixels) agents experienced when they were given an ideal reinforcement learning algo- rithm (PPG) and perceptual representations (se- mantic segmentation) to the additive improve- ment of agent given PPG over pixels and an agent given PPO and semantic segmentation in- puts. Only 6/16 games benefit from co-training.
Reward shaping for resolving reinforce- ment learning challenges. The reinforce- ment learning challenges revealed by the LCD due the sparsity of rewards available to agents. As a partial solution to this prob- lem, we investigated the impact of "reward shaping" (Skinner 1938) on performance. Re- ward shaping technique from animal train- ing where supplemental rewards are provided make the learning problem easier Wiewiora (010)) We adopted this for three games amenable to reward shaping: Heist, Leaper, and Maze; the latter two LCD identified as having significant reinforcement learning challenges.
In Heist, agents must collect three keys and unlock their matching doors before they can reach the rewarding gem. We provided agents with additional rewards when they collected a key or opened door. In Leaper, agents must safely get across several lanes of roads followed by several rows of rivers. Lanes have moving cars agents must avoid, while rivers have moving logs agents must stay We gave agents additional rewards whenever they reached new lane or river.
In Maze, agents must wander through a complex maze to get to a reward of cheese; however, the topological complexity of the mazes deem rewards difficult to discover. To address this problem, we encourage exploration of novel locations through intermediate rewards [Bellemare et al. (2016); Ecoffet et al. 2021). In all of these cases, agents did not get these auxiliary rewards during test time.
Reinforcement learning challenge evaluations. Agents trained with reward shaping on Heist = 1.69, p < .05), Leaper (1 = 41.03. p 0001), and Maze (1 = 2.69, p < .01) performed signifi-
8

---

Under review as a conference paper at ICLR 2023
cantly better than those trained on the normal versions of each game (all tests one-tailed t-tests; Appendix Fig. [3 Moreover, the improvements in performance from reward shaping significantly correlated with the values of each game tested here (Spearman's 's p 0.92, p < .001).
The LCD-derived taxonomy translates across dRL. algorithms. Our findings thus far indicate that an LCD calibrated with PPO agents makes reliable predictions about the challenges individual games in Procgen. But despite the popularity of PPO, it has been surpassed by more recent dRL algorithms, like PPG, which achieved state-of-the-art performance on Procgen. To what extent does the LCD taxonomy derived from PPO translate to more effective dRL algorithms? We tested this question by applying the LCD to agents trained with PPG (Appendix Figs. [O [T] and found that its predictions were significantly correlated with those from an LCD calibrated on PPO agents (Fig. 4). This means that our PPO-derived taxonomy can be relied on prospectively for algorithmic development Procgen.
The whole is only sometimes greater than the sum of the parts in dRL. Training perception- for-action, end-to-end, is the standard approach to dRL ever since the introduction of DQNs. Here, we tested whether agents trained on Procgen experienced a superlinear benefit of co-training per- ception and action or not. We did this by computing two performance differentials. In both cases, the baseline was a PPO agent trained on pixels. (i) the performance of a PPG agent trained on semantic segmentation inputs minus baseline, and (ii) the performance of PPG agent trained on pixels minus baseline plus the performance of a PPO agent trained on semantic segmentation inputs minus baseline (Fig.F 5
Co-training perception for action helped performance in 6 of 16 games: Leaper, Maze, Chaser, Jumper, Starpilot, and Bigfish. Co-training did not help in 10 of games: Heist, Dodgeball, Fruitbot, Miner, Coinrun, Bossfight, Plunder, Ninja, Caveflyer, and Climber. This finding poses an intriguing prospect. It suggests the existence of different "minibenchmarks" within Procgen that researchers should tap into, depending on whether they wish to quantify "improvements" to an agent's 's policy learning algorithm, its representation learning system, or synergies between the two.
DISCUSSION
Ever since the introduction of the DQN, the field of dRL has consistently succeeded in reaching and exceeding human performance on defined tasks: beating grand masters at the game of Go, or exceeding the average performance of humans on the Atari benchmark. However, even in these successes, was clear that certain games presented different challenges than others. Take "Mon- tezuma's Revenge", which the original DQN could not solve. Subsequent analysis of Montezuma's Revenge led to the conclusion that the failures of the DQN were because of its long time horizon and sparse rewards; challenges which, these days, have positioned it as critical test for dRL algo- rithms (Ecoffet et al. 2019] Roderick et al. 2018 [Salimans & Chen 2018). We believe that the field of dRL can benefit from systematic approach to diagnosing such computational challenges in any benchmark. Our LCD tool is a major step towards this goal.
By applying our LCD the Procgen benchmark, we discover a novel taxonomy of those games, organized according to the relative perceptual and reinforcement learning challenges of each. This taxonomy delivers two prescriptions for algorithmic development in dRL. First, because the com- putational challenges of games in benchmarks like Procgen are not i.i.d.. there will be low signal for adjudicating between algorithms designed to improve problem that not over-represented in the benchmark. Second, although humans are capable of learning perception-for-actionn agents generally learn in this way, not all games in benchmarks like Procgen benefit from doing so. In the short-term, these problems can be addressed by using the LCD to select games in benchmark that are aligned with the algorithmic design goals of researchers. In the long-term, there is a need in the field for new benchmarks that can tap into wider range of computational challenges with less bias than is found in Procgen. We believe that our LCD is an important tool for advancing the pace of progress in dRL,, and release experimental code to support this goal.
and dRL.

---

Under review as a conference paper at ICLR 2023
6 ETHICS STATEMENT
Our work is a contribution to deep reinforcement learning and so inherits the concerns already inherent in dRL applications. Currently, our work has been applied to harmless toy videogame tasks, but improper application of dRL to real world applications can have negative externalities on society, especially when algorithms make poor decisions. Depending on the domain of application, these can include discriminatory decisions against marginalized groups serious injury and fatality control of vehicles or critical systems are involved. Advancing agents' ability to make better decisions would help mitigate this issue and eventually broaden the domains in which dRL. can help, and our contribution of identifying limitations in current dRL agents will help in that regard.
REPRODUCIBILITY STATEMENT
We have taken several steps to ensure the reproducibility of our work. We have made all code for modified Procgen environment and the self-supervised model for perceptual grouping avail- able at https: fteps::/aanoymouut wttp://ssspppep..e..ieeeeoc httpa//eeeeeeee//aaaarrrreeeeeee used the code available at https::/ the hyperparameters found in Cobbe et al. (2020). For PPG, we have used the code at hhtps: 7/github. com/ opena httpphasicccoo p- icy gcy-gradieett and the hyperparameters in Cobbe et al. (2021). Finally, we have reported the computing resources used and the number of experi- ments run under "dRL algorithms" of Section
4open
For PPO, we have along with
com http://////eeeeaaaaaaaaaoaooooo/
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Mare G Bellemare. Deep reinforcement learning at the edge of the statistical precipice. August 2021.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In D. Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Sys- tems, volume 29, Curran Associates, Inc., 2016. URL https: /ttp::/ee////ooeeeee/geeee http://roreednnn.....ii..sss.ss. neurips. cc/paper/2016// /ttp:/2//////eeeaaaa22222222 45222 4 3133bb66666 7tw.l/6622268bb5999 Paper. pdf
Veronica Chelu, Doina Precup, and Hado van Hasselt. Forethought and hindsight in credit assign- ment. In Proceedings of the 34th International Conference on Neural Information Processing Systems, number Article 191 in NIPS 20, pp. 2270-2281, Red Hook, NY, USA, December 2020. Curran Associates Inc.
Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In Hal Daumé li and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 2056. PMLR, 2020.
Karl W Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman. Phasic policy gradient. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 2020-2027. PMLR, 2021.
Guy Davidson and Brenden M Lake. Investigating simple object representations in Model-Free deep reinforcement learning. February 2020.
Andrea Dittadi, Frederik Träuble, Manuel Wüthrich, Felix Widmaier, Peter Gehler, Ole Winther, Francesco Locatello, Olivier Bachem, Bernhard Schölkopf, and Stefan Bauer. The role of pre- trained representations for the OOD generalization of reinforcement learning agents. July 2021.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-Fxplore: new approach for Exrd--xxporttion problems. January 2019.
Adrien Ecoffet, Joost Huizinga. Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore. Nature, 590(7847):580-555 February 2021.
10

---

Under review as a conference paper at ICLR 2023
L L Emberson. Chapter one how does experience shape early development? considering the role of Top-Down mechanisms. In Janette B Benson (ed.), Advances Child Development and Behavior, volume 52. pp. 1-41. JAI, January 2017.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: Scalable distributed Deep-RL with importance weighted Actor-Learner architectures. In Jennifer Dy and Andreas Krause (eds.), Proceedings the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1407-1416. PMLR, 2018.
François Fleuret, Ting Li, Charles Dubout, Emma Wampler, Steven Yantis, and Donald Geman. Comparing machines and humans visual categorization test. Proc. Natl. Acad. Sci. U. S. A., 108(4)): October 2011.
Kevin Frans, Jonathan Ho. Xi Chen, Pieter Abbeel, and John Schulman. Meta learning shared hierarchies. In International Conference on Learning Representations, 2018. URL https:
//openzeview. net. formt 1d=SyX0IeWAM ftte:////oo
Christina M Funke, Judy Borowski, Karolina Stosio, Wieland Brendel, Thomas S A Wallis, and Matthias Bethge. Five points to check when comparing visual perception in humans and ma- chines. J. Vis., 21,3):16, March 2021.
Robert Geirhos, Carlos R M Temme, Jonas Rauber, Heiko H Schütt, Matthias Bethge, and Fe- lix A Wichmann. Generalisation in humans and deep neural networks. In S Bengio, H Wallach, H Larochelle, K Grauman, N Cesa-Bianchi, and R Garnett (eds.), Advances in Neural Information Curran Associates, Inc., 2018.
Processing Systems 31, pp. 7549-7561.
Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thieringer, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Partial success in closing the gap between human and machine vision. June 2021.
Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta-reinforcemmet learning of structured exploration strategies. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL www.eeeeecccee1999e---eee 4de754248ca 96c859c44444
cc/ cc//papeer/211//E11ee/ .ddf]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. December 2015.
Rein Houthooft, Yuhua Chen, Phillip Isola, Bradly Stadie, Filip Wolski, OpenAl Jonathan Ho, and Pieter Abbeel. Evolved policy gradients. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL htw.hhtts////mm http:tts///toceeiiinoo .ewrips...... cc/ Paper. pdf
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. November 2016.
Junkyung Kim, Matthew Ricci, and Thomas Serre. Not-So-CLEVR: learning same-differen rela- tions strains feedforward neural networks. Interface Focus, ((4):2018001) August 2018.
Junkyung Kim, Drew Linsley, Kalpit Thakkar, and Thomas Serre. Disentangling neural mechanisms for perceptual grouping. International Conference on Representation Learning, 2020.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. December 2014.
Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. survey of generalisation in deep reinforcement learning. November 2021.
11

---

Under review as a conference paper at ICLR 2023
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. CURL: Contrastive unsupervised representa- tions for reinforcement learning. Hal Daumé li and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5639-5650. PMLR, 2020.
Lucas Lehnert, Michael L Littman, and Michael Frank. Reward-predictte representations gen- eralize across tasks in reinforcement learning. PLoS Comput. Biol., 16(10)=e1088317, October 2020.
Wu Li, Valentin Piëch, and Charles D Gilbert. Perceptual learning and top-down influences in primary visual cortex. Nat. Neurosci,, 7(6))651-657) June 2004.
Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. December 2016.
Drew Linsley, Junkyung Kim, Vijay Veerabadran, and Thomas Serre. Learning long-range spatial dependencies with horizontal gated-recurrent units. May 2018a.
Drew Linsley, Dan Shiebler, Sven Eberhardt, and Thomas Serre. Glohal-and-local attention net- works for visual recognition. May 2018b.
Drew Linsley, Dan Shiebler, Sven Eberhardt, and Thomas Serre. Learning what and where to attend with humans in the loop. In International Conference on Learning Representations, 2019.
Drew Linsley, Ashok, I. Govindarajan, Rex Liu, and Thomas Serre. Stable and expressive recur- rent vision models. NeurlPS, 2020.
Drew Linsley, Girik Malik, Junkyung Kim, Lakshmi N Govindarajan, Ennio Mingolla, and Thomas Serre. Tracking without re-recognition humans and machines. May 2021.
Runtao Liu, Zhirong Wu, Stella Yu, and Stephen Lin. The emergence of objectness: Learning zero-shot segmentation from videos. Advances in Neural Information Processing Systems, 34: 13137-13152, 2021.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431-3440, 2015.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver. Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533 February 2015.
C O'Reilly. Generalization in interactive networks: the benefits of inhibitory competition and hebbian learning. Neural Comput., 13(6): 1241, 2001.
Vitchyr H Pong, Ashvin Nair, Laura Smith, Catherine Huang, and Sergey Levine. Offline Meta- Reinforcement learning with online Self-Supervision. July 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. February 2021.
Ilija Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell. robot learning with masked visual pre- pre-training. CoRL.
Melrose Roderick, Christopher Grimm, and Stefanie Tellex. Deep abstract Q-Networks. In Pro- ceedings of the 17th International Conference Autonomous Agents and MultiAgent Systems, AAMAS 18, pp. 131-138, Richland, SC, July 2018. International Foundation for Autonomous Agents and Multiagent Systems.
12

---

Under review as a conference paper at ICLR 2023
P R Roelfsema, V A Lamme, and H Spekreijse. The implementation of visual routines. Viston Res., 40<10-122:1385-141,,
Pieter R Roelfsema. Cortical algorithms for perceptual grouping. Anmu. Rev. Neurosci, 29:203-227, January 2006a.
Pieter R Roelfsema. Cortical algorithms for perceptual grouping. Annu. Rev. Neurosci, 29:203-227, January 2006b.
Pieter R Roelfsema, Arjen Ooyen, and Takeo Watanabe. Perceptual learning rules based reinforcers and attention. Trends Cogn. Sci., 14(2):64-71, February 2010.
Tim Salimans and Richard Chen. Learning montezuma's revenge from a single demonstration. December 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. July 2017.
Lisa S Scott, Olivier Pascalis, and Charles A Nelson. A Domain-Genena theory of the development of perceptual discrimination. Curr. Dir. Psychol. Sci., 16(4):197-201, 2007.
Evan Shelhamer, Parsa Mahmoudich, Max Argus, and Trevor Darrell. Loss is its own reward: Self- Supervision for reinforcement learning. February 2017.
David Silver, Julian Schrittwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap. Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of go without human knowledge. Nature, 350-7676):354-55, October 2017.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Si- monyan, and Demis Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419): 1440-1144,, December 2018.
B F Skinner. The Behavior of Organisms: An Experimental Analysis. D. Appleton & Company, December 1938.
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 9870-9879. PMLR, 2021.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Ried- miller. DeepMind control suite. January 2018.
Yi Tay, M Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, J Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. ICLR, 2021.
Matthew E. Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey Journal of Machine Learning Research, (0(56):1633-1685 2009. URL http: /mlr.. org/
S Ullman. Visual routines. Cognition, 18(1-3):97-159, December 1984.
Mohit Vaishnav, Remi Cadene, Andrea Alamia, Drew Linsley, Rufin VanRullen, and Thomas Serre. Understanding the computational demands underlying visual reasoning. Neural Comput., 34(5): 1075-1099, April 2022.
13

---

Under review as a conference paper at ICLR 2023
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czarnekki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung. David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P Agapiou, Max Jaderberg, Alexander S Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L. Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünnshh, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7782): 350- 354, November 2019.
W H Warren, Jr, A Kay, W Zosh, A P Duchon, and S Sahuc. Optic flow used to control human walking. Nat. Neurosci,, 4(2):213-21 216, February 2001.
William H Warren. Information is where you find it: Perception as an ecologically Well-Posed problem. Iperception, 1212):2041665521 1000366, March 2021.
T Watanabe, J E Náñez, and Y Sasaki. Perceptual learning without perception. Nature, 413(6888): 844-848, October 2001.
T N Wiesel and D H Hubel. Single-cell responses in striate cortex of kittens deprived of vision in one eye. Neurophysiol., 26: 1003-1017, November 1963.
Eric Wiewiora. Reward shaping. In Claude Sammut and Geoffrey Webb (eds.), Encyclopedia of Machine Learning, pp. 863-865. Springer US, Boston, MA, 2010.
Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for motor control. arXiv:2203. 06173.
Zhongwen Xu, Hado P van Hasselt, and David Silver. Meta-Gradient reinforcement learning. In Bengio, H Wallach, H Larochelle, Grauman, N Besa-Bianchi, and R Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Zhongwen Xu, Hado Hasselt, Matteo Hessel, Junhyuk Oh, Satinder Singh, and David Silver. Meta-gradient reinforcement learning with an objective discovered online. In Proceedings of the 34th International Conference Neural Information Processing Systems, number Article 1279 in NIPS 20, pp. 15254-15264, Red Hook, NY, USA, December 2020. Curran Associates Inc
Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invari- ant representations for reinforcement learning without reconstruction. June 2020.
A APPENDIX
A.1 NETWORK ARCHITECTURE
Following Cobbe et al. 22220), we used the convolutional actor-critic IMPALA architecture de- scribed in Espeholt et al. (2018). This is deeper network than what has been used previously, consisting of 15 convolutional layers, 16 residual blocks, and 1.6M parameters. We refer the reader to (Espeholt et al. 2018) for details. For raw pixel, semantic segmentation, and figure- grouee seg- mentation, the number of input channels was fixed to three. This controls the model architecture to input. For semantic or figure-ground masks, we replicated the masked in-
be identical regardless put three times and fed that into each of the three convolutional input channels. Our self-supervised visual model outputs 16 channel perceptual masks (at the same spatial resolution the inputs) which concatenated with the RGB values and fed into the actor-critic network for policy learning.
A.2 REWARD NORMALIZATION
Given that each task involves varying reward magnitudes, episode rewards r were normalized to facilitate comparison across tasks. Normalized rewards r' were computed as Rmaz= Rmas where Ryaa is the theoretical maximum score attainable in an episode while Ryan is the average score a random agent would attain.
r-Riiin
14

---

Under review as a conference paper at ICLR 2023
A.3 STOCHASTIC REWARD FEEDBACK
Here, we prove that the expected return under policy does not change under our manipulation of reward probability. We proceed in two parts. First, we show that the expected reward received from rewarding event remains unchanged. Then we show that the action-value function Q" (s, for policy т remains unchanged. By implication, this means the optimal policy does not change either.
Let R(s, denote the original reward function at state and action In our manipulation, rewards are generated stochastically with probability p(s, a). but reward magnitudes are rescaled by p(s.a)"
Definition A.l (Modified reward function). The modified reward function Rt(s, a) is given by
R(s, a) - R(s, a) a(s,a) ((s,
where 7185,) Bernlp(ssa)) a)].
Lemma A.1. The expected value of R(s, a) is R(s,a).
Proof.
R(s,
E, R(s, a) & En
n(s.a)
pps.
R(s.
E, [n(s,a)]
p(s,a)
R(s,a),
since E, [m(s a).
Using this lemma, we now prove our main result. Let (" denote the action-value function induced by the original reward function R(s, a) while following policy n, and let (s,(s, a) denote the action-value function induced by R(s, a) while following the same policy.
Theorem A.2. The action-value function Q (8, a) is given by (s,
Proof. As is common practice in RL, we shall discount future rewards by [0, 1] on each future time step. Before beginning, shall introduce some notation. We shall denote the current time step by f. We shall denote the state and action at any time by 8, and notation, we shall use |- denote expectations over state-action trajectories generated by both the policy π and the state-transition probability p(s" [s, a), where s' is the successor state.
Finally, by an abuse of
We now proceed to the proof.
0"((sal
A E
R(s444+1100444)) 8t 8,0,
E
E.
a) g= -84tt11 0++k++1
8,4;
(law of total expectation)
k=
E
*((544411 =,a,
(Lemma A.I
k=
Q" (s)
Note that in our experimental manipulations, we used a single reward probability p that was inde- pendent of and a.
15

---

Under review as a conference paper at ICLR 2023
A.4 PERFORMANCE CHANGE WITH INCREASING CREDIT ASSIGNMENT DIFFICULTY
To quantify the average change with increasing credit assignment difficulty, we first computed the reward curve AUC AUC(p) for 0.25, 0.5, 0.75, 1.0. We next computed the absolute change
in AUC between successive values of p: that is, (p = 1.0) - AUC(pp = 0.75)l, AUC((" 0.75) ApC 0.5)], and (p 0.5) AUC 0.25) Finally, we averaged over these measures to get our final measure of average change.
A.5
SUPPLEMENTARY FIGURES
coinrun
starpilot
caveflyer
dodgeball
1.0
0.8
0.7
0.3
0.1
0.0
fruitbot
chaser
miner
jumper
0.9
o.s
1.0
0.8
0.1
leaper
maze
bigfish
heist
0.9
0.8
0.4
o.6
0.1
0.3
0.0
0.2
climber
plunder
ninja
bossfight
10
semantic
pixets
0.1
0.1
0.1
0.0
figure-ground
200M
Figure 6: Normalized reward curves while systematically varying perceptual complexity. X- axis denotes the number of "interactions" an agent performs with its environment during training and Y-axis denotes normalized rewards. We trained all these agents for a total of 200M steps.
16

---

Under review as a conference paper at ICLR 2023
coinrun
starpilot
caveflyer
dodgeball
1.2
0.8
0.7
fruitbot
chaser
miner
jumper
o.s
1.0
0.8
0.1
leaper
maze
bigfish
heist
0.5
0.1
climber
plunder
ninja
bossfight
0.5
p=1
p=0.5
p=0,25
0.0
200M
Figure 7: Normalized reward curves while systematically varying reward stochasticity. X-axis denotes the number of "interactions" an agent performs with its environment during training and Y-axis denotes normalized rewards. We trained all these agents for total of 200M steps. The perceptual input to these agents were semantic representations. While performing evaluations, the reward mechanisms were set the original configuration thus effectively only testing the impact of these perturbations on policy learning.
17

---

Under review as a conference paper at ICLR 2023
coinrun
starpilot
caveflyer
dodgeball
0.8
0.5
Ww
a.s
0.6
www.mmma
-0.4
...
0.0
0..
fruitbot
chaser
miner
jumper
WW
-0.0
0.0
0.0
-..0
leaper
maze
bigfish
heist
0.8
mm
0.2
m Mn
0.1
-..
mar
0.1
0.11
0.0
.0.2
climber
plunder
ninja
bossfight
0.8
0.3
mary
0.9
0.8
mmmmg
semantic
pixels
Figure 8: Normalized reward curves, on i.i.d. generalization levels, while systematically varying perceptual complexity. X-axis denotes the number of "interactions" an agent performs with its environment during training and Y-axis denotes normalized rewards. We trained all these agents for total of 200M steps. Agents were evaluated approximately every 60K steps.
18

---

Under review as a conference paper at ICLR 2023
coinrun
starpilot
caveflyer
dodgeball
0.7
0.3
0.4
0.6
.0.2
0.0
0.0
.0.0
fruitbot
chaser
miner
jumper
0.4
0.5
0.9
0.4
leaper
maze
bigfish
heist
0.5
0.2
0.5
0..
climber
plunder
ninja
bossfight
0.8
0.4
0.9
o.s
p=1
p=0,5
p=0.25
0.1
0.0
0.0
-0.0
200M
Figure 9: Normalized reward curves, on i.i.d. generalization levels, while systematically vary- ing reward stochasticity. X-axis denotes the number of "interactions" an agent performs with its environment during training and Y-axis denotes normalized rewards. We trained all these agents for a total of 200M steps. The perceptual input to these agents were semantic representations. Agents were evaluated approximately every 60K steps. While performing evaluations, the reward mechanisms were set to the original configuration thus effectively only testing the impact of these perturbations on policy learning.
19

---

Under review as a conference paper at ICLR 2023
coinrun
starpilot
caveflyer
dodgeball
9.9
23.0
10.3
13.5
1.8
0.7
fruitbot
chaser
miner
jumper
-0.7
0.7
0.9
0.9
leaper
maze
bigfish
heist
6.0
climber
plunder
ninja
bossfight
12.3
12.8
10.0
12.1
pixels
semantic
2.6
1.3
0.0
100M
Figure 10: Training dRL agents with varying levels of perceptual complexity using the prox- imal policy gradient algorithm. X-axis denotes the number of "interactions" an agent performs with its environment during training and Y-axis denotes rewards training levels. We trained these agents for total of 100M steps.
20

---

Under review as a conference paper at ICLR 2023
coinrun
starpilot
caveflyer
dodgeball
23.2
10.3
13.5
fruitbot
chaser
miner
jumper
26.4
8.3
12.1
9.0
0.5
leaper
maze
bigfish
heist
30.9
Lea I1
7.5
0.6
2.3
0.2
0.0
climber
plunder
ninja
bossfight
12.4
18.5
10.3
p=1
p= 0.75
p=0.5
pm0.25
2.5
0.0
100M
Figure 11: Training dRL agents with varying levels of stochastic feedback using the proximal policy gradient algorithm. X-axis denotes the number of "interactions" agent performs with its environment during training and Y-axis denotes rewards on training levels. We trained these agents for total of 100M steps. The perceptual inputs these agents were semantic representations. While performing evaluations, the reward mechanisms were set to the original configuration thus effectively only testing the impact these perturbations on policy learning.
21

---

Under review as a conference paper at ICLR 2023
coinrun
starpilot
caveflyer
dodgeball
2.1
0.8
fruitbot
chaser
miner
jumper
7.3
leaper
maze
bigfish
heist
climber
plunder
ninja
bossfight
picels mode] CUP [easy mode]
CLIP [hard mode]
Policy learning experiments in dRL agents operating visual representations learned from natural language supervision. We test the extent to which generalist visual represen- tations support policy learning. We find that while CLIP Radford al. (2021), waasformer-basee architecture, is state-of-the-art on zero-shot image classification, its representations do not support policy learning adeptly. We train agents both the "easy" and "hard" versions of Procgen. While our agent was able to learn certain tasks in the "easy" mode, they predominantly struggled in the "hard" mode. X-axis denotes the number of "interactions" an agent performs with its environment during training and Y-axis denotes rewards. We trained these agents for a total of 200M steps.
Figure 12:
22

---

Under review as a conference paper at ICLR 2023
0.86
Generalization performance
No reward shaping
Reward shaping
200M
The effect of "reward shaping" on Leaper high z/ task as predicted by our taxonomy. In the traditionally challenging reinforcement learning problem we find that encouraging agents to discover sub- goals significantly improved both the sample efficiency and generalization capacity (1 41.03, p < .001). X-axis denotes the number of "interactions" an agent performs with environment during training and Y-axis denotes normalized rewards. We trained both these agents semantic representations for total of 200M steps. While performing evaluations, revert to the original reward scheme to remain compatible for comparing to the "naive" model.
Figure 13:
A.6 SPECIFICS OUR PERCEPTUAL PARAMETERIZATIONS THE Procgen BENCIMARK
23

---

Under review as a conference paper at ICLR 2023
Image
Game
Mask Details
bigfish
There are two entity IDs: (1) the player, and (2) all other fish.
There are seven entity IDs: (1) the player, (2) player projectiles, (3) the enemy, (4) enemy projectiles, (5) the enemy's shield, (6) meteors, and (7) explosions.
bossfight
There are eight entity IDs: (1) the player, (2) enemy ships, (3) player pro- jectiles, (4) targets, (5) the goal landing pad, (6) meteors, (7) explosions, and (8) the surrounding terrain.
caveflyer
There entity IDs: (1) the player, (2) enemies, (3) enemies in their weakened state, (4) enemy eggs, (5) small collectible squares, (6) large collectible stars, and (7) the walls.
chaser
There are four entity IDs: (1) the player, (2) enemies, (3) the star rewards, and (4) the surrounding walls, ground, and platforms.
climber
There are eight entity IDs: (1) the player, (2) enemies, (3) enemy barriers, (4) saw obstacles, (5) lava obstacles, (6) the goal star, (7) crates, and (8) the surrounding walls and ground.
coinrun
There entity IDs: (1) the player, (2) player projectiles, (3) the enemies, (4) enemy projectiles, (5) the locked exit, (6) the unlocked exit, and (7) the walls.
dodgeball
There are eight entity IDs: (1) the player, (2) player projectiles, (3) reward- ing objects, (4) penalizing objects, (5) locks, (6) locked doors, (7) the goal, and (8) barriers and the terrain below and above the beginning and end of each episode, respectively.
fruitbot
Table 1: Outlining the "Semantic" organization of each environment in the Procgen benchmark.
24

---

Under review as a conference paper at ICLR 2023
Image
Game
Mask Implementation
There are six entity IDs: (1) the player, (2-4) to three unique keys and their corresponding locks, (5) the goal crystal, and (6) the walls.
heist
There six entity IDs: (1) the player, (2) spikes, (3) the goal, (4) the needle of the compass and the bar indicating the distance to the goal, (5) the circle of the compass, and (6) the surrounding terrain.
jumper
There are six entity IDs: (1) the player, (2) cars, (3) the road, (4) log plat- forms, (5) the water, and (6) the finish line.
leaper
There are three entity IDs: (1) the player, (2) the goal cheese, and (3) the walls.
maze
There are five entity IDs: (1) the player, (2) boulders, (3) crystals, (4) the exit, and (5) dirt.
miner
There are seven entity IDs: (I) the player, (2) player projectiles, (3) the goal mushroom, (4) bombs, (5) explosions, (6) the bar indicating jump charge, and (7) the surrounding walls, ground, and platforms.
ninja
There are eight entity IDs: (1) the player and ally ships, (2) player projec- tiles, (3) enemy ships, including the ship, (4) the circle behind the cue ship, (5) barriers, (6) explosions, (7) the time-countdown status bar, and (8) the points-accrued status bar.
plunder
There are eleven entity IDs: (1) the player, (2) player projectiles, (3) slow enemies flyers, (4) fast enemies flyers, (5) enemy flyer projectiles, (6) en- emy turrets, (7) enemy turret projectiles, (8) meteors, (9) clouds, (10) the finish line, and (11) explosions.
starpilot
Table 2: Outlining the "Semantic" organization each environment in the Procgen benchmark.
25

---

Under review as a conference paper at ICLR 2023
Image
Game
Mask Details
bigfish
All entities share the same ID.
bossfight
All entities share the same ID.
caveflyer
All entities share the same ID.
chaser
All entities share the same ID).
climber
All entities share the same ID.
coinrun
All entities share the same ID.
dodgeball
All entities share the same ID.
fruitbot
All entities share the same ID.
Table 3: Outlining the "Figure/Ground" organization of each environment in the Procgen benchmark.
26

---

Under review as a conference paper at ICLR 2023
Image
Game
Mask Implementation
There four entity IDs: (1-3) up to three unique keys and their corre- sponding locks, and (4) all other entities.
heist
There three entity IDs: (1) the needle of the compass and the bar in- dicating the distance to the goal, (2) the circle of the compass, and (3) all other entities.
jumper
leaper
There are two entity IDs: (1) the road and water, and (2) all other entities.
maze
All entities share the same ID),
miner
All entities share the same ID,
There are two entity IDs: (1) the bar indicating jump charge, and (2) all other entities.
ninja
There five entity IDs: (1) the player and friendly ships, (2) enemy ships, including the cue ship, (3) the time-countdown status bar, (4) the points- accrued status bar, and (5) all other entities.
plunder
starpilot
All entities share the same ID.
Table 4: Outlining the "Figure/Ground" organization of each environment in the Procgen benchmark.
27

---

Under review as a conference paper at ICLR 2023
A.7 IMPROVEMENTS IN GENERALIZATION PERFORMANCE ACROSS EASY AND HARD Ø TASKS
Easy (ø) % improvement
Chaser 7.5%
Leaper - -117.%
Bigfish 37.9%
Bossfight -39.8%
Hard (Ø) % improvement Starpilot 7%
Dodgeball 71.4%
Climber 3364.6%
28


Reviews:
[{"review_id": "nLmyyGYYxOh", "reviewer": "Reviewer_Ni8m", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper tackled with disentangling the challenges from representaion learning and policy learning in deep RL for vision-based input. The main contribution is the introduction of \"Learning Challenge Diagnosticator\" (LCD), which is metric to measure the aforementioned problem. LCD is computed by examing the performance change when perturbating the vision input and the reward signal, respectively. The authors did experiments in 16 Procegen game environments to support their claims. Furthermore, for different challenges in deep RL, the paper suggested to use pre-trained representation learning and reward-shaping to overcome the two challenges respectively. ", "summary_of_the_review": "This study aims to address an increasingly significant question in deep RL, that is disentangling the difficulty of representation learning and policy learning in deep RL with image observations. The paper put a step forward in this direction, while the proposed method has some room to improve. I encourage the authors to update their method by overcoming the limitations I mentioned above, in which case the work will be a fantastic contribution to the deep RL society. However, currently I vote for a weak rejection.", "strength_and_weaknesses": "**[Strength]**\n- The paper tackles an important yet unclear question in deep RL (disentangling the challenges from representation learning and reward-maximization).\n- The proposed LCD is a practical and effective way to numerically address the question in Procgen environments.\n- The authors give useful suggestion to improve deep RL performance when LCD is known.\n\n**[Limitations]**\n- The way of obtaining LCD is hindsight, it is still unclear where the challenge locates before conducting RL.\n- The way of computing LCD is not general, which requires segmentation of image input already known.\n- The computational cost of LCD is high (require repeating the whole training process for several times).\n\n**[Suggestions]**\n- The way of measuring the reinforcement learning challenge could be more compresensive: the difficulty in RL does not solely come from sparse reward (e.g., in MuJoCo tasks, the reward is non sparse, but it is still sometimes challenging).\n- I would like to see more experimental results besides the ProcGen benchmark.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UvHe3Os7D7", "reviewer": "Reviewer_8UAX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a taxonomy of the tasks, namely the Learning Challenge Diagnosticator (LCD), based on the perceptual and reinforcement learning challenges in the Procgen benchmark. The games are parameterized to perturb the perceptual representation (original pixels, figure-ground or semantic segmentations) or to vary the sparsity of the reward scheme. The integrated reward trajectories of a set of agents of sampled perturbed environments are used to compute scores that describe the perceptual and reinforcement challenge of that game. Specifically, the score is the normalized average absolute AUC difference of a policy across all changes; where a score of 0 indicates sparse reward is not much of a challenge while 1 represents a significant RL challenge. These agents are trained with either PPO or PPG for 200M steps (~24hrs per game). The main advantage of LCD is that it reveals failure cases and instructs algorithmic development. For the perceptual challenge, the work proposes self-supervised pretraining on in-domain videos to compute the optic flow between two frames giving a better visual representation for policy learning than the alignment (clip-ViT-B-32) model trained on images and captions. For the reinforcement challenge, the work demonstrates reward shaping on 3 games: Heist, Leaper, and Maze, demonstrating how the improvements correlated with the LCD taxonomy scores for RL complexity.", "summary_of_the_review": "Overall, the paper presents an interesting study of how abstractness in perceptual representations and reward sparsity affects performance in video games and motivates the readers to broadly think about the issues in solving perceptual challenges with pretraining and reinforcement challenges with reward shaping. However, certain assumptions for pretraining and evaluation are not fully justified. A discussion in highlighting the potential limitations of these assumptions will clarify the takeaways of this work. \n\n----\nReferences: \n\nA few relevant citations for visual pre-training: Xiao et al. “Masked Visual Pre-training for Motor Control.” *ArXix* abs/2203.06173 (2022)", "strength_and_weaknesses": "### Strengths\n\nThe paper presents a task taxonomy separated by perceptual and reinforcement learning. This is a novel reward performance-based taxonomy, for which the authors train policies in different perceptual and reward representations in each environment. \n\nThe work also has a detailed appendix with performance plots for 16 games under different perturbations. \n\nThe paper discusses interesting effects of the assumption that Leaper does not benefit from the self-supervised pretraining to predict optic flow as the color attributes of the object are lost which are important for it to solve the task.\n\n### Weaknesses\n\nThe paper motivates the introduction with DQN but does not consider any off-policy algorithm while discussing the reinforcement learning challenge or reward scheme. Why are only on-policy algorithms (like PPO, and PPG) considered?\n\nWhile the motivation of ranking tasks based on perceptual and reinforcement challenges is clear, the choice of input representation in pixels, figure-background vs semantic segmentation, and reward schemes are limited and not fully motivated. Estimating the mean reward achieved by the agents could be possible due to some spurious correlations at times, which the current process does not take into account. The perceptual and reinforcement learning challenge scores are computed separately while holding the other part appears on the easiest possible level. An underlying assumption is that the two challenges can be completely disentangled, which may not always be true if other types of perturbations are considered. \n\nThe proposed taxonomy depends on the DRL algorithm (PPO) and semantic segmentation representation used and does not use any task-intrinsic elements directly. Is there any theoretical or empirical evidence on how much the taxonomy would shift when other DRL algorithms or perceptual representations are used to compute the respective scores?\n\nWhile the authors present a paragraph in section 4 on how co-training perception for action helps in 6 out of 16 games, with 3 being on the separating line, the readers will benefit from discussing or cross-referencing this in section 3. The paper discusses a separate policy trained for each of the 16 games and does not discuss possible perceptual and task-solving benefits of learning one policy for all [Reed et al., 2022 [A Generalist Agent](https://arxiv.org/pdf/2205.06175.pdf)]. \n\nFinally, an assumption used here is “training DNNs to predict the optic flow between successive frames of video can induce the ability to segment object-like superpixels from complex scenes” from Liu et al. 2021. Does this only hold true when the camera is fixed, such that only objects move in computing the optic flow with a fixed background? Can this assumption be applied to an agent in First-Person-Shooter (FPS) videogames with egocentric perceptual inputs?   \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MnMwp4-_Kth", "reviewer": "Reviewer_tmph", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces the Learning Challenge Diagnosticator (LCD) that is a tool to measure the contribution of advances I visual representation learning and the effectiveness of reinforcement learning algorithms at discovering policies to the successes of deep reinforcement learning models. The proposed approach is applied to challenges in Procgen benchmark and shows the effectiveness using  the self-supervised visual representations and  the reward shaping.", "summary_of_the_review": "Incorporating the recent advances of visual representations learning is good, and \nproposing the environment to measure the performance of perceptual part and reinforcement learning part is good. But the  definitions of levels of perception and  \ndifficulties of reinforcement learning is not well designed and considered.", "strength_and_weaknesses": "The strength of this paper is to incorporate recent great successes of visual representation learning into deep reinforcement algorithms and to propose new taxonomy of challenges in the Procgen bechmark. The weakness of this paper is the definition of perception  challenge as the structure of the scenes and the definition of reinforcement learning as the total rewards. Those definitions are very  limited and evaluations on more variations of the definitions would support the claims. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "4m-yIcb-gbP", "reviewer": "Reviewer_SeAm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces the Learning Challenge Diagnosticator (LCD), a tool for assessing the difficulties of perceptual learning and reinforcement learning for a video game environment. The authors apply LCD on the Procgen benchmark, and reveals that how much perceptual learning and reinforcement learning affects the performance for each game. Instead of having a single “one size fits all” approach, the knowledge gained from LCD helps us tailor the algorithmic improvements (perceptual-oriented or RL-oriented) for a specific game.", "summary_of_the_review": "In summary, the paper conducts solid experiments to discover a novel taxonomy of challenges in the Procgen benchmark, though I have concerns on how such knowledges can benefit future research. I would like to hear the authors' opinions on it.", "strength_and_weaknesses": "Strength:\n* The paper is overall clear and easy to follow.\n* The experiments on procgen are comprehensive and the results are statistically significant.\n\nComments:\n* In Section 4, the authors choose 7 games for pretraining visual representations with self-supervision. I am wondering why these 7 games are chosen. In other words, I don't quite get what \"consistent motion\" means.\n* The results in Figure 3 seem to be dominated by two games (Leaper and Climber). For other games, the improvements look similar across easy and hard $\\phi$. Thus, I do not think the conclusion drawn from the results is convincing enough.\n* In Page 8 (Reinforcement learning challenge evaluations), the authors state that \"the improvements in performance from reward shaping significantly correlated with the values of each game tested here\". However, it depends on the reward shaping methods used (different envs use different reward shaping). So it looks to me that such correlation does not imply anything meaningful.\n* In \"The whole is only sometimes greater than the sum of the parts in dRL\" on Page 8, I do not fully understand the connection between the experiments setup (i) /(ii) and the implication of learning perception and action together/separately. Can the authors elaborate on it?\n* In Figure 9, for some games, using $p < 1$ yields much better performance. Do the authors have any interpretation of such results?\n\nWeaknesses:\n* My major concern is about the insights offered by this paper. It is not surprising that some games require more perceptual learning while others require less. Even if we know (from LCD) that a particular game falls into the former category, can we totally disregards the RL part for this game? I guess the answer is probably no. So eventually, we still need to consider both perceptual learning and RL. In short, I think the benefits brought by LCD are not clear.\n\nMinor Issues:\n* The citation of Procgen benchmark is incorrect. It should be \"Leveraging Procedural Generation to Benchmark Reinforcement Learning\".\n* For Figure 4 and Figure 5, it is better to keep the equal axis aspect ratio, i.e., making it a square instead of a rectangle.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject