Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: # Review ICLR20, Visual Hide and Seek

This review is for the originally uploaded version of this article. Comments from other reviewers and revisions have deliberately not been taken into account. After publishing this review, this reviewer will participate in the forum discussion and help the authors improve the paper.



## Overall

**Summary**

The authors introduce a new RL environment and task, "Visual Hide and Seek", in which they analyze how the agent's learned visual representations are impacted by its speed, auxiliary rewards, and opponent behavior.


**Overall Opinion**

This paper presents a thorough analysis and great visualizations of agent behaviors and representations under different conditions. I wish more papers would put this much effort into analyzing their agents. I'd highly recommend this paper get accepted since I believe the analysis carried out here and the conclusions reached are quite novel and the paper is overall well-written.
However, at the same time, the work of [Baker et al., 2019][1] was published with significantly more fanfare. I hope their work does not overshadow this one since they are only related in the general task concept.

[1]: https://arxiv.org/abs/1909.07528

Some major issues I had with this work:

- In general, please run more random seeds. Just reporting on a single random seed is not enough, as per [Henderson et al., 2018][2].
- There are some sections of the paper where the order of paragraphs is confusing. You start the introduction by stating what you've done and letting the reader wonder "why?". The explanation is only given in the second paragraph. So I'd suggest rotating the second paragraph upwards before the first. Similarly, at the beginning of section 4, you just mention the results - this should either be shorter (1 sentence, as an overview of the work in this section) or moved to the end of that section.
- You're missing a section about future work and flaws/problems of your work at the very end (the latter if which should be in "Discussion"), which is common to include in ICLR publications.

[2]: https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/viewFile/16669/16677

Here are some minor...

## Specific comments and questions

### Abstract

all good

### Intro

- Fig.1: What is that white thing that the seeker/hider have on the capsule?

### Rel. Work

all good

### Method

- What's the speed (FPS) of that Unity engine? Why didn't you use Mujuco/(Py)Bullet/Gym-Miniworld?
- You need to add some measurements and units: the arena size doesn't have a unit, the size (diameter) of the hiders/seekers is unclear, turning left/right is unclear (how far left/right, after action-repeat)
- You mention any real-valued position to be valid - so the agents can step into obstacles? And how about through obstacles?
- "Affordance Learning" is usually not used for static environment geometry like obstacles (e.g. [Georgia-Tech course on "Human-Robot Interaction"][3])
- Why 4-layer CNNs? Why not 6 or 8 or a ResNet? Would you think the features would be stronger/weaker in an 8-layer CNN?

[3]: https://www.cc.gatech.edu/~athomaz/classes/CS8803-HRI-Spr08/MayaChandan/Site/Affordance_Learning.html

### Experiments

- 4.1 "... learned this play game" -> "... learned this game".
- 4.2 "mid-level features" -> what's that? The activation of the convolutional kernels after the second layer CNN? What's the dimension? And why did you pick the 2nd layer, not any of the other 3?
- Tab.3: This is averaged over how many frames of rollout?
- "... case can moves a lot faster." -> "... case can move a lot faster".
- Fig.2 is very interesting. Well done.
- Fig.3: the font is not consistent with other figures
- Fig.4: remove the blueish background to increase contrast. Increase the font size of the ticks on the left. Make the legend color boxes slightly bigger. Add more space or a visual divider between the different states - especially on the right side it's hard to make out where one stops and the next begins.
- Fig.4/5: This analysis is lovely and we need more of this in DRL.
- 4.4 in the text, you sometimes write "not S" and sometimes "¬S". Please change the "not s"
- Fig.5: (suggestion) Merge/sum the 2 columns (in both A/B merge the left and right plot into one by summing); subtract the random policy values as you did with Fig.4.
- "We summarize representative cases, and put the full results for all combinations in the Appendix" - no you didn't.
- Fig.6: What is going on in the left third of this diagram? What is this colorful mush? If this is by any chance indicating a change over time, do you maybe want to spread a single, very colorful plot of distance over time into multiple less colorful plots? At least add a legend, please. Also, I'd recommend smoothing (moving average or smoothing spline). 
- Also maybe add reward over time plots, as is common in DRL, to show that your policies converged after 8 mil. steps.

### Conclusion

All good, save for the missing future work and critical analysis of your work.

### Appendix

all good
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: Visual Hide and Seek

In this paper, the authors propose a two-agent hide and seek environment, and uses reinforcement learning to train the hider. The authors interpret the learned representation by using the learnt features to do classification.

I tend to vote rejection for this paper, mostly due to my feeling that the empirical contribution is interesting, but not novel enough for this conference.
There is still a lot of potential improvement available for this project, which is summarized below.

Pros:
- The environment itself, once open-sourced, can be quite valuable to the community.
I think it is fair to say that the proposed environment in this project is better than the hide-and-seek environment from OpenAI in that it provides the visual input. 
By the way, I believe OpenAI environment is also partially observable, where unseen agents’ information is masked out.
- It is an interesting finding on how meaningful features and performance correlate with each other.

Cons:
- Findings in the project are very practical and interesting, but they do not seem to provide valuable information for future research.
For example, to improve the quality of the paper, is it possible to utilize the features and performance correlation to improve unsupervised visual feature learning?
Is it possible to draw a connection to Psychology? Does this “Representation vs. Survival Time” also happens in real-life, or is it just some empirical things that happen in optimization?

- There are a lot more potential to improve the environments. It will greatly increase the impact of the paper by, for example, considering multi-agent training, self-play, unsupervised training.

In general, I think this project still have a lot of room for fine-tuning.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: 
Paper Summary: The paper studies what an embodied agent that is trained using RL learns in the context of the hide and seek game. There are two agents in a simple environment (one hider and one seeker). The strategy for the seeker is fixed, but the hider learns a policy based on ego-centric visual input. The paper attempts to analyze how the learned representation differs with different capabilities of the agents or environment structure.

Originality:

The study of the learned representations in the context of hide and seek is new. There is concurrent work on hide and seek (e.g., Baker et al., 2019), but unlike this paper, they have access to groundtruth location information of the agents and it is not based on visual input.

Quality:

- The conclusions of the paper are either counter-intuitive or are based on some hypotheses and guesses.

For example, it is strange that "visibilityreward" performs worst for "Awareness of Self-Visibility". It is exactly trained for that task.

There are conclusions about "temporal events" (e.g., the hider learns to first turn away from the seeker then run away) just based on a single frequency number (Figure 4). There are many other possibilities that can result in the same frequency. If the authors draw conclusions about temporal events, they should show how values change over time. Single frequency numbers cannot be used for such conclusions.

- To check the learned representation, the common practice is to use features on a very different task. However, this paper addresses only two tasks "Seeker Recognition" and "Awareness of Self-Visibility" which are very close to the original task. One of the models even receives explicit rewards for the latter case.

- No standard deviation is reported for the results. Given the random nature of RL algorithms, standard deviations should be reported.

Significance:

I expected a more rigorous analysis since this is an analysis paper and there is no new methodology in the paper. Conclusions that are just based on a guess about some qualitative result or based on two state changes as in Figure 5 are groundless. More thorough analysis based on quantitative results are required to justify the claims and provide generalizable conclusions.

Clarity:

The paper is mostly clear, but there is some missing information:

- In Figure 3, which dot corresponds to which model? It is claimed that "When the model has a weakness, the model learned to overcome it by instead learning better features". To justify that, we need to know what the dots are.

- What do "randominits" in Figure 5 correspond to?

- What is red, yellow, blue and green plots in Figure 6? What is the standard deviation? 
review_title: Official Blind Review #1

Meta-review: This paper proposes a technique for training embodied agents to play Visual Hide and Seek where a prey must navigate in a simulated environment in order to avoid capture from a predator. The model is trained to play this game from scratch without any prior knowledge of its visual world, and experiments and visualizations show that a representation of other agents automatically emerges in the learned representation. Results suggest that, although agent weaknesses make the learning problem more challenging, they also cause useful features to emerge in the representation.

While reviewers found the paper explores an interesting direction, concerns were raised that many claims are unjustified. For example, in the discussion phase a reviewer asked how can one infer "hider learns to first turn away from the seeker then run away" from a single transition frequency? Or, the rebuttal mentions "The agent with visibility reward does not get the chance to learn features of self-visibility because of the limited speed hence the model received samples with significantly less variation of its self-visibility, which makes learning to discriminate self-visibility difficult". What is the justification for this? There could be more details in the paper and I'd also like to know if these findings were reached purely by looking at the histograms or by combining visual analysis with the histograms.

I suggest authors address these concerns and provide quantitative results for all of the claims in an improved iteration of this paper. 
Decision: Reject