Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Paper summary:

This is an empirical study of the representations learned by a reinforcement learning agent. An agent is trained, using a standard RL algorithm, to solve puzzles by navigating through a 3D visual environment (Unity obstacle tower challenge). The analyses in the paper show that the visual representations of the trained agents are sparse and cluster according to the actions performed by the agent. The goal of the paper is to show that these features are due to the embodied nature of the agent. Specifically, the paper states that “the quality of the representations learned shows the strength of embodied learning and its advantages over fully supervised approaches with regards to robustness and generalizability”.

Decision:

I suggest to reject this paper. While the topic is interesting and the paper is clearly written, there is a lack of control/comparison experiments, such that the paper’s conclusions are not backed up by the analyses. However, with more experiments, I think this line of research has large potential.

Further justification for the decision:

My main criticism is that the paper claims to show that embodiment is important for representation learning, but never actually compares representations learned by an embodied agent to representations learned in some other way.

Comparing to a random network is a sanity check, but not sufficient to support the paper’s claims.

One way to experimentally dissociate embodiment/agency from supervised learning would be to train two separate models, one that is “embodied”/active, and another that gets the same sensory input, but without embodiment (“passive”). The passive network could be trained using the sensory inputs recorded while training the active network. The passive network could be trained to predict the value and/or the actions of the active network. Thus, the passive network would be trained in a supervised way, whereas the active network would be trained by RL (i.e. in an embodied way).

The representations of the two networks could then be compared using the analyses used in the paper. This setup would experimentally isolate the effect of embodiment. Without such comparisons, it is unclear whether representations learned in a supervised way would be any different from those learned by RL.

In addition to the descriptive analyses presented in the paper, a transfer learning approach could test whether there is actually a functional difference between the “embodied” and the supervised representations: Take the representations of the “active agent” and the “passive agent” and freeze the weights. Then re-initialize and re-train only the dense layer before the action probabilities on the RL task, leaving everything else frozen. Does the model from the “active agent” do better on the RL task? This would suggest that the embodied agent learned better representations.

Minor comment: The website contains a link to a YouTube profile that is not completely anonymous (it contains the first name and a profile photo). While I did not identify the authors when visiting the paper website, I recommend removing links to personal YouTube profiles and create submission-specific anonymous accounts.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This work builds on the embodied cognition literature, hypothesizing that representations learned in embodied agents will be of improved “quality” compared to non-embodied models, such as neural networks trained on static supervised datasets. 

The authors provide an excellent motivation to the work, with the introduction nicely laying the groundwork for their hypothesis. In general the motivation is quite strong, and research along these directions will no doubt be of value to the field.

To assess their hypothesis, the authors compare the representations learned in trained vs. random agents on the Unity Obstacle Tower Challenge, and demonstrate that trained agents develop semantically meaningful, sparse representations, without explicit regularization. 

Unfortunately, the work doesn’t provide adequate baselines to properly assess the hypothesis. With the data presented, we can only make claims about *trained* vs. *untrained* agents (both of which are embodied!). In other words, an embodied agent with a random policy is not equivalent to a non-embodied agent. Thus, the data only support the conclusion that performance and task-relevant policies drive good representation learning in embodied agents, which is altogether not surprising, as one wouldn’t expect representations to be good in a randomly initialized network. 

To assess wither *embodiment* is a critical factor for driving good representations, the authors should compare to a model that learns from a static supervised dataset. Curiously, this idea is alluded to in the introduction, but not followed up on.

Overall, there are some nicely presented ideas but the work is unfortunately incomplete, and the results cannot support the hypothesis laid out.

As a final note, the authors are encouraged to remove any assignments of gender to the agent (“he/him” is unnecessarily used throughout).

review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: 
Paper Summary: The goal of the paper is to analyze what information is encoded in the representation learned using RL for a specific game. The paper shows that the activations are sparse and the activation patterns are distinct and shows that the conceptually similar images are clustered together in t-sne visualization.

Paper Strengths:

The paper starts with a nice introduction that embodiment is useful for perception. However, the main content of the paper is very different from the introduction.

Paper Weaknesses:

The conclusions of the paper are either already known or very trivial. So there is nothing new for the community to benefit from. Please refer to my comments below for more information.

The conclusion of section 3.3 is that "the agent learns to use sparse activation patterns and even leaves some of the available neurons completely unused". This has nothing to do with embodiment. The same patterns are observed in non-dynamic tasks such as image classification. The CNNs are usually over-parameterized.

The conclusion of section 3.4 is that "When comparing this with the encodings of a random untrained agent one can see that there is a clear association between the learned image encoding and the actions". That is the whole point of training. We train the models to find correlations between actions and observations. It is obvious that there is more correlation compared to a random agent.

It is shown in section 3.5 that similar images will be next to each other in the t-sne visualization. It is obvious that this happens.

Due to the issues mentioned above, I do not think there is anything new in the paper and I vote for rejection.


review_title: Official Blind Review #1

Meta-review: What is investigated is what kind of representations are formed by embodied agents; it is argued that these are different than from non-embodied arguments. This is an interesting question related to foundational AI and Alife questions, such as the symbol grounding problem. Unfortunately, the empirical investigations are insufficient. In particular, there is no comparison with a non-embodied control condition. The reviewers point this out, and the authors propose a different control condition, which unfortunately is not sufficient to test the hypothesis.

This paper should be rejected in its current form, but the question is interesting and hopefully the authors will do the missing experiments and submit a new version of the paper.
Decision: Reject