Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: #### Summary

This paper works on unsupervised discovering keypoints in an Atari game frame to help improving Atari game performance. The keypoint discovery is based on predicting "predictable" local structure. I.e., the authors consider points that can not be predicted from its neighbor as good. Experiments show the learned keypoints performs better on 3 Atari games (Table. 1) than a counterpart keypoint discovery method, Transporter. 

#### Strength

- The key idea of finding non-locally-predictable points as a representation of the game state is interesting, and specifically suitable for Games, where the backgrounds are mostly static and predictable. 

- The technical implementation of the framework (Fig. 2) is clear and makes sense to me.

- Experimental results in Table. 1 is healthy. It shows the proposed method decently outperforms the Transporter counterparts.

#### Weaknesses

- The ablation studies are not exciting. These (number of keypoints/ spatial resolution of the embedding) are mostly design choice experiments and are better to be in the supplement materials. A more interesting ablation would be to quantitatively evaluate the quality of the points. Currently the paper only qualitatively shows the keypoint discovering results in Fig. 2 and claims an advantage to Transporter. This is not clear to the reviewer. The reviewer understands that there is no existing metric to evaluate keypoint discovery quality. However some proxy evaluation would also be helpful. For example, are the learned points temporally stable?

- As the key contribution is the keypoint discovery, it would be more convincing to compare with other unsupervised keypoint discovery methods besides Transporter if applicable. E.g., PointNet (Jakab et al 2018) that considers keypoints as a pixel-level reconstruction bottle-neck. 


#### Summary

The paper proposed an interesting idea with reasonable results (better than a recent counterpart, Transporter). However, the reviewer does not have backgrounds in the specific experimental settings (Atari games), and can not assess the significance of the improvements. Comparisons with more keypoint discovery methods would make the results more convincing. My current rating is 6, but might change based on other reviews. 

#### Post rebuttal

Thank you for providing the rebuttal. The rebuttal addressed my concern on comparing to other baselines. And it's fine to keep the design choice experiments in the main paper. However a proxy evaluation of key point evaluation is still missing and it will further strengthen the paper (I don't have a clear idea for the evaluation either). I keep my original rating of 6. 
review_title: Interesting idea

final_score: rating: 7: Good paper, accept
review_content: The authors propose a novel approach to unsupervised key point detection based on predictability. The demonstrate their model on Atari tasks comparing to other key point detectors.

 Quality:

The authors compare their work both qualitatively and quantitatively to Transporter. 

The authors show that their model picks out important key points that Transporter does not. Figure 5 is a great! It would also be good to show the distribution of predicted key points over multiple runs for other levels.

The authors train agents on Atari and compare their model to suitable baselines. It’s interesting that the GNN does not always outperform the CNN. This paper could be improved by also comparing to Transporter + GNN. 

When constructing the error map, is this approach very sensitive to the receptive field and the number/ location of the neighbours? How would this approach handle larger / more complex objects?

Section 5.2 is interesting. What about if you have new objects that are not predictable, but are distractors? Would the model not create key points here? For example, adding in some smaller distractor shapes? Some randomly coloured dots? Or some missing pixels?

The ablation study is great!

This results section of this paper is very thorough and addressed a lot of the questions that came to mind when reading the introduction and methods section of this paper. 

Clarity:

The authors argue that local predictability is an intrinsic property of an object with out giving more evidence for this. Perhaps the authors are hypothesising that this interpretation of objects will be more useful for downstream tasks? It’s not clear to me otherwise why this is an intrinsic property of an object?

The authors could improve their paper by being very clear about distinguishing focus on image regions that are unpredictable” from “local regions in input space that have high internal predictive structure” when describing objects and key points.

With the exception to the above, the introduction is well written and the methods section is easy to follow.

The model is designed to pick out points that are harder to predict, which is useful for ignoring background and finding the agent as nicely demonstrated in Figure 2 of Montezuma Revenge, but it’s not clear that this is a good definition of an object? For example the platforms in Frostbite may be very easy to predict, but you would need to know where they are in order to successfully navigate the environment. Also, it seems that predictability of a feature may depend heavily on the environment and any new object in an environment would be immediately picked out even if they are irrelevant to the task (i.e. new colours etc).

Could you explain better why you thing that the platforms in frostbite are assigned key points as they are?

Figure 3 is a really clear and nice example.

Originality and Significance:

This approach is novel and interesting and offers a new perspective on what an object can be and what definitions of an object may be useful for training agents.

Pros:
- Well written (with minor exception)
- Thorough results section.
- A novel approach to thinking about what an object is.
- Improvement over Transporter on Atari tasks.

Cons:
- Some confusion in the introduction about their definition of an object.
- There may be some limitation of the types (size and shape) of the objects that this model can assign key points to.
- There are some examples of objects being detected where it is not clear why those points are being detected according to the definition given in the paper. Explaining this more clearly would improve the paper.

review_title: A very good paper! Very thorough experimentation.

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: The authors tackle the problem of self-supervised representation learning, and validate their approach on downstream Reinforcement Learning tasks. Building on the insight that predictability in local patches is a good inductive bias for salient regions that characterize objects, the authors propose a well-reasoned, well-engineered and thoroughly validated pipeline for learning object keypoints without supervision. The authors present a wide range of ablative studies to validate their design choices, and demonstrate the superiority of their method both illustratively as well as quantitatively on a number of standard Atari benchmarks. 

The paper is very well written and clearly explained, with the assumptions clearly stated, and design choices thoroughly validated through ablative studies. Overall the authors make a very compelling argument for using local predictability as an intrinsic property of objects, leading me to recommend accepting this paper for publication.

Pros:
+ The intro motivates the problem well, contrasting the proposed method with a number of key recent methods. The implementation details are well recorded in the supplementary, with the added mention of releasing the source code 
+ The keypoint detection pipeline is well reasoned and well explained: using the error map obtained through the spatial prediction task to recover keypoints via a bottleneck with limited capacity is a neat idea. The authors ablate a number of design choices (number of keypoints, which encoder layers to use); Figure 1 and 2 are great at showing the high-level components of the method as well as (intermediate) outputs
+ The comparison against Transporter is thorough and well analyzed. Fig2.b. provides a very clear insight into the limitations of Transporter, showing that the method proposed by the authors is able to achieve some robustness to visual distractors. PKey-CNN uses a similar method as Transporter for encoding keypoint features for downstream tasks, and thus serves to show that the keypoints identified are indeed superior. PKey-GNN further increases performance on a number of Atari games. 
+ Very good ablative analysis and qualitative examples.

Some questions:
+ Do the authors have any further insights regarding why PKey-GNN would perform worse than PKey-CNN? While the authors’ reasoning makes sense, in my understanding a GNN based approach should be able to model any kind of interaction. 
+ The authors demonstrate impressive results on a number of Atari games. I am wondering how this method would perform on a slightly more complex environment, i.e. CarRacing in OpenAI’s gym environment, or maybe even going as far as CARLA?
+ As I understand, PermaKey is first trained on Atari game rollouts, with the policy trained afterwards. Would it be possible to optimize both the keypoints and the policy together, end-to-end?

Post Rebuttal:
I thank the authors for their detailed and thorough response. All my questions and concerns were addressed and I appreciate the discussion on end-to-end learning as well as the “Transporter + GNN experiment”. I am happy to maintain my original rating and recommend acceptance.


review_title: Official Blind Review #4

Meta-review: Reviewers all agreed that this submission has an interesting new idea for learning object/keypoint representations: parts of a visual scene that are not easily predictable from their neighborhoods are good object candidates. Experimental gains on various Atari games are convincing. The main drawback at this point is that the evaluation is limited to visually rather simple settings, and it is unclear how the approach will scale to more realistic scenes. 
Decision: Accept (Spotlight)