Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a taxonomy of the tasks, namely the Learning Challenge Diagnosticator (LCD), based on the perceptual and reinforcement learning challenges in the Procgen benchmark. The games are parameterized to perturb the perceptual representation (original pixels, figure-ground or semantic segmentations) or to vary the sparsity of the reward scheme. The integrated reward trajectories of a set of agents of sampled perturbed environments are used to compute scores that describe the perceptual and reinforcement challenge of that game. Specifically, the score is the normalized average absolute AUC difference of a policy across all changes; where a score of 0 indicates sparse reward is not much of a challenge while 1 represents a significant RL challenge. These agents are trained with either PPO or PPG for 200M steps (~24hrs per game). The main advantage of LCD is that it reveals failure cases and instructs algorithmic development. For the perceptual challenge, the work proposes self-supervised pretraining on in-domain videos to compute the optic flow between two frames giving a better visual representation for policy learning than the alignment (clip-ViT-B-32) model trained on images and captions. For the reinforcement challenge, the work demonstrates reward shaping on 3 games: Heist, Leaper, and Maze, demonstrating how the improvements correlated with the LCD taxonomy scores for RL complexity.
strength_and_weaknesses: ### Strengths

The paper presents a task taxonomy separated by perceptual and reinforcement learning. This is a novel reward performance-based taxonomy, for which the authors train policies in different perceptual and reward representations in each environment. 

The work also has a detailed appendix with performance plots for 16 games under different perturbations. 

The paper discusses interesting effects of the assumption that Leaper does not benefit from the self-supervised pretraining to predict optic flow as the color attributes of the object are lost which are important for it to solve the task.

### Weaknesses

The paper motivates the introduction with DQN but does not consider any off-policy algorithm while discussing the reinforcement learning challenge or reward scheme. Why are only on-policy algorithms (like PPO, and PPG) considered?

While the motivation of ranking tasks based on perceptual and reinforcement challenges is clear, the choice of input representation in pixels, figure-background vs semantic segmentation, and reward schemes are limited and not fully motivated. Estimating the mean reward achieved by the agents could be possible due to some spurious correlations at times, which the current process does not take into account. The perceptual and reinforcement learning challenge scores are computed separately while holding the other part appears on the easiest possible level. An underlying assumption is that the two challenges can be completely disentangled, which may not always be true if other types of perturbations are considered. 

The proposed taxonomy depends on the DRL algorithm (PPO) and semantic segmentation representation used and does not use any task-intrinsic elements directly. Is there any theoretical or empirical evidence on how much the taxonomy would shift when other DRL algorithms or perceptual representations are used to compute the respective scores?

While the authors present a paragraph in section 4 on how co-training perception for action helps in 6 out of 16 games, with 3 being on the separating line, the readers will benefit from discussing or cross-referencing this in section 3. The paper discusses a separate policy trained for each of the 16 games and does not discuss possible perceptual and task-solving benefits of learning one policy for all [Reed et al., 2022 [A Generalist Agent](https://arxiv.org/pdf/2205.06175.pdf)]. 

Finally, an assumption used here is “training DNNs to predict the optic flow between successive frames of video can induce the ability to segment object-like superpixels from complex scenes” from Liu et al. 2021. Does this only hold true when the camera is fixed, such that only objects move in computing the optic flow with a fixed background? Can this assumption be applied to an agent in First-Person-Shooter (FPS) videogames with egocentric perceptual inputs?   

clarity,_quality,_novelty_and_reproducibility: The paper is well-written and novel. The appendix provides a detailed set of reward plots for all 16 games with different perceptual representations and reward sparsity. The work discusses how the taxonomy correlates with techniques needed to improve the training. The authors present the code for parameterized ProcGen environments, training, and computational details for reproducibility. 

### Minor clarifications/suggestions:

Figure 3b is unclear in terms of what is the takeaway message.
summary_of_the_review: Overall, the paper presents an interesting study of how abstractness in perceptual representations and reward sparsity affects performance in video games and motivates the readers to broadly think about the issues in solving perceptual challenges with pretraining and reinforcement challenges with reward shaping. However, certain assumptions for pretraining and evaluation are not fully justified. A discussion in highlighting the potential limitations of these assumptions will clarify the takeaways of this work. 

----
References: 

A few relevant citations for visual pre-training: Xiao et al. “Masked Visual Pre-training for Motor Control.” *ArXix* abs/2203.06173 (2022)
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces the Learning Challenge Diagnosticator (LCD) that is a tool to measure the contribution of advances I visual representation learning and the effectiveness of reinforcement learning algorithms at discovering policies to the successes of deep reinforcement learning models. The proposed approach is applied to challenges in Procgen benchmark and shows the effectiveness using  the self-supervised visual representations and  the reward shaping.
strength_and_weaknesses: The strength of this paper is to incorporate recent great successes of visual representation learning into deep reinforcement algorithms and to propose new taxonomy of challenges in the Procgen bechmark. The weakness of this paper is the definition of perception  challenge as the structure of the scenes and the definition of reinforcement learning as the total rewards. Those definitions are very  limited and evaluations on more variations of the definitions would support the claims. 

clarity,_quality,_novelty_and_reproducibility: This paper is well structured, but doesn't have enough  clarity and quality to support claims, especially the effectiveness of the proposed taxonomy.
The novelty is low.  This paper applies basic self-supervised representation learning  using some features often used in computer vision for the effectiveness of the proposed taxonomy.
summary_of_the_review: Incorporating the recent advances of visual representations learning is good, and 
proposing the environment to measure the performance of perceptual part and reinforcement learning part is good. But the  definitions of levels of perception and  
difficulties of reinforcement learning is not well designed and considered.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces the Learning Challenge Diagnosticator (LCD), a tool for assessing the difficulties of perceptual learning and reinforcement learning for a video game environment. The authors apply LCD on the Procgen benchmark, and reveals that how much perceptual learning and reinforcement learning affects the performance for each game. Instead of having a single “one size fits all” approach, the knowledge gained from LCD helps us tailor the algorithmic improvements (perceptual-oriented or RL-oriented) for a specific game.
strength_and_weaknesses: Strength:
* The paper is overall clear and easy to follow.
* The experiments on procgen are comprehensive and the results are statistically significant.

Comments:
* In Section 4, the authors choose 7 games for pretraining visual representations with self-supervision. I am wondering why these 7 games are chosen. In other words, I don't quite get what "consistent motion" means.
* The results in Figure 3 seem to be dominated by two games (Leaper and Climber). For other games, the improvements look similar across easy and hard $\phi$. Thus, I do not think the conclusion drawn from the results is convincing enough.
* In Page 8 (Reinforcement learning challenge evaluations), the authors state that "the improvements in performance from reward shaping significantly correlated with the values of each game tested here". However, it depends on the reward shaping methods used (different envs use different reward shaping). So it looks to me that such correlation does not imply anything meaningful.
* In "The whole is only sometimes greater than the sum of the parts in dRL" on Page 8, I do not fully understand the connection between the experiments setup (i) /(ii) and the implication of learning perception and action together/separately. Can the authors elaborate on it?
* In Figure 9, for some games, using $p < 1$ yields much better performance. Do the authors have any interpretation of such results?

Weaknesses:
* My major concern is about the insights offered by this paper. It is not surprising that some games require more perceptual learning while others require less. Even if we know (from LCD) that a particular game falls into the former category, can we totally disregards the RL part for this game? I guess the answer is probably no. So eventually, we still need to consider both perceptual learning and RL. In short, I think the benefits brought by LCD are not clear.

Minor Issues:
* The citation of Procgen benchmark is incorrect. It should be "Leveraging Procedural Generation to Benchmark Reinforcement Learning".
* For Figure 4 and Figure 5, it is better to keep the equal axis aspect ratio, i.e., making it a square instead of a rectangle.
clarity,_quality,_novelty_and_reproducibility: * Clarity: Good.
* Quality and Novelty: Average. The results are unsurprising and the benefits are not clear.
* Reproducibility: Good
summary_of_the_review: In summary, the paper conducts solid experiments to discover a novel taxonomy of challenges in the Procgen benchmark, though I have concerns on how such knowledges can benefit future research. I would like to hear the authors' opinions on it.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper tackled with disentangling the challenges from representaion learning and policy learning in deep RL for vision-based input. The main contribution is the introduction of "Learning Challenge Diagnosticator" (LCD), which is metric to measure the aforementioned problem. LCD is computed by examing the performance change when perturbating the vision input and the reward signal, respectively. The authors did experiments in 16 Procegen game environments to support their claims. Furthermore, for different challenges in deep RL, the paper suggested to use pre-trained representation learning and reward-shaping to overcome the two challenges respectively. 
strength_and_weaknesses: **[Strength]**
- The paper tackles an important yet unclear question in deep RL (disentangling the challenges from representation learning and reward-maximization).
- The proposed LCD is a practical and effective way to numerically address the question in Procgen environments.
- The authors give useful suggestion to improve deep RL performance when LCD is known.

**[Limitations]**
- The way of obtaining LCD is hindsight, it is still unclear where the challenge locates before conducting RL.
- The way of computing LCD is not general, which requires segmentation of image input already known.
- The computational cost of LCD is high (require repeating the whole training process for several times).

**[Suggestions]**
- The way of measuring the reinforcement learning challenge could be more compresensive: the difficulty in RL does not solely come from sparse reward (e.g., in MuJoCo tasks, the reward is non sparse, but it is still sometimes challenging).
- I would like to see more experimental results besides the ProcGen benchmark.



clarity,_quality,_novelty_and_reproducibility: **[Clarity]**

The paper is basically clear and well-written. The plots are vivid.

**[Quality]**

As an empirical study, the quality is mostly OK, while there is some space to improve (see above).

**[Novelty]**

The proposed approach is novel to my knowledge and it touches a vital problem in deep RL.

**[Reproducibility]**

The source code is available, while I have not checked it.
summary_of_the_review: This study aims to address an increasingly significant question in deep RL, that is disentangling the difficulty of representation learning and policy learning in deep RL with image observations. The paper put a step forward in this direction, while the proposed method has some room to improve. I encourage the authors to update their method by overcoming the limitations I mentioned above, in which case the work will be a fantastic contribution to the deep RL society. However, currently I vote for a weak rejection.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: I would really want this paper to be in an acceptable state, as I find the topic fascinating and the idea novel. The problem addressed is how much of the challenge in an RL problem is in perceptual processing, and how much is in sparse rewards. In essence, the method is to vary both perceptual (graphical) elements and reward sparsity and see how it affects agents. I'm enthusiastic about the overall approach and would love to see it developed further. However, the paper as it stands arguably doesn't take it far enough. In particular, it only tests the method on seven games from ProcGen and the way it varies perceptual difficulty is rather limited. It could also be argued that we don't actually learn all that much from the experiments in the paper.

justification_for_why_not_higher_score: Arguably needs more work. Trying on different games, broadening the concept of perceptual difficulty.
Decision: Reject