Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: 
########## Post-Rebuttal Summary ###########
The authors engaged actively in the rebuttal discussion and in the process we were able to concretize the motivation of the submission (as a result in increased my score). However, I think that the submission in its current form lacks experimental analysis of the proposed conditional probes, especially the trade-offs on the reliability of the representation analysis when performed with a conditional probe as well as a clear motivation for the need of a language interface. I can therefore not recommend the submission in its current form for acceptance.
(for detailed discussions + suggested experiments please see rebuttal discussions)
#######################################


## Paper Summary
The paper tackles the problem of analyzing the information captured in the learned representation of a deep neural network. It proposes to replace the commonly used "probing networks" that try to directly infer the information from the learned representation, with a language interface in the form of a QA model which is trained post-hoc without propagating gradients into the learned representation. The authors argue that such a language interface provides a more natural interface for probing the information in a learned representation. The paper shows representation analysis results for the internal representations of agents trained on an exploratory task in a simple, simulated household environment similar to DM Lab. The authors conduct additional analysis on representations learned with auxiliary generative and contrastive objectives.

## Strengths
- the language of the paper is clear and easy to follow
- the paper covers the related work well
- the provided explanations help understand the content of the paper
- the analysis of how the information captured in the agent's representation changes over the course of a trajectory is interesting (more such visualizations in the appendix would be nice!)

## Weaknesses
(1) the motivation for the proposed problem does not convince: why do I need to train a Q/A system to infer which components of the true state are captured by the learned representation? For each of the properties of the environment I could train a probing network (as is done e.g. in [1,2]) and would get much more precise answers; ground truth labels need to be available whether I train QA or probing networks. The authors provide two motivations for the proposed approach which both do not convince me:
	(a) QA "provides an intuitive investigation tool for humans": I cannot imagine a workflow in which researchers would prefer to ask questions to their model over a plot showing explicit regression accuracies aggregated across many data samples (which the conventional probing networks provide).
	(b) "the space of questions is open-ended, [...] enabling comprehensive analysis of [an agent's] internal state": for each new question type we need to provide a sufficiently large number of question-answer-pairs to train the QA system for this question type. we could therefore also train a probing network using the same labels and would get a better overview of whether the state information is captured in the representation. I fail to see how substituting probing networks with a language interface helps the investigation of the representation's properties.
	
(2) the paper only provides minor novelty: the main proposal is to replace the probing networks, which were extensively used in prior work, with a natural language interface; a proposal that does not seem to provide a clear advantage (see above). The paper does not provide any further technical novelty.
	
(3) it is possible that the analysis of the contrastive model could improve substantially with a different sampling scheme of the negative examples: maybe sampling from different sequences makes the task of discriminating too easy so that the model is not encouraged to learn a rich representation. It would be interesting to see whether the representation captures more information if a more standard contrastive objective is chosen that discriminates between future frames with different offset in the same sequence (see for example the objective in [3]).

(4) it seems that both environment and chosen task will have high influence on the representation the agent can learn from the collected data. Therefore the fact that the authors evaluate their approach only on a single environment / task combination, both of which they introduce themselves, weakens any conclusions the authors draw from their experiments. It would help strengthen the message of the paper to apply their methodology to previously proposed environment / task combinations, for example in the AI2-THOR environment [4]


[Novelty]: minor
[technical novelty]: minor
[Experimental Design]: not comprehensive
[potential impact]: low


#########################
[overall recommendation]: Reject - In its current form the paper does not provide a convincing argument for why learning a language interface for probing a representation is better than learning the usual probing networks. Further there are some doubts on the setup of the contrastive objective and the paper lacks comprehensive evaluation on standard environments. Therefore I cannot recommend acceptance in its current form.
[Confidence]: High


[1] Neural Predictive Belief Representations, Guo et al., 2018
[2] Shaping Belief States with Generative Environment Models for Reinforcement Learning, Gregor et al., 2019
[3] Representation Learning with Contrastive Predictive Coding, Van den Oord et al., 2018
[4] AI2-THOR: An Interactive 3D Environment for Visual AI, Kolve et al., 2017



#### Final Rebuttal comment (to make it visible to the authors) ####
I understand the authors point that every method for probing representations can potentially be flawed in that the probing mechanism might not be expressive enough to extract the information that is indeed present in the representation. The point I was trying to make in my previous response is, that in the case of conditional probes that try to solve multiple such "probing tasks" in parallel, such uncertainties accumulate because the probing mechanism might trade off performance on one task for performance on another (if solving all of them at once is too challenging). If the submission's key contribution is to make probes conditional this seems like a trade-off that should be experimentally investigated, as it is vital to practitioners how much they can trust their probing method to extract the relevant information if it is actually in the representation.

One possible experiment would be to train an unconditional probe per attribute (maybe on a subset of all possible attribute-object combinations) and then a conditional probe across all of them to show that the conditional probe's assessments of the representation agree with the ensemble of unconditional probes. In my previous response I additionally raised the point that even if it can be shown that conditional probes are reliable, the authors still need to provide arguments why that conditioning should work via a language interface, not for example a symbolic/one-hot interface. If the claim is that this allows for generalization, it should again be shown that this generalization does not increase the error of the probe substantially, such that meaningful conclusions about the representation are still possible.

Regarding testing on more diverse environments: the questions these experiments are supposed to answer (i.e. how reliable are conditional probes) are inherently empirical, so verifying across diverse environments will make the analysis more conclusive.

I agree with the author's point that training 7k unconditional probes would not be practical, probably the current approach would be to train an image decoder that then reconstructs a top-down view of the whole scene. This, however, would have the same problems as a conditional probe, i.e. the probing decoder could trade-off performance between reconstructing different parts of the image. Therefore, I agree with the authors that investigating conditional probes is an interesting direction, but the submission does not provide a comprehensive analysis of this question.

On a final note, I appreciate the continued, factual discussion with the authors and think that the refinement of the focus towards conditional vs unconditional probes is a step in the right direction. To acknowledge that I am raising my score from "reject" to "weak reject". Yet, I think that the work in its current form lacks experimental analysis of the proposed conditional probes. During the rebuttal discussion I highlighted concerns and proposed possible experiments. I cannot recommend acceptance of the submission in its current form, but I encourage the authors to incorporate the refined motivation and add more comprehensive experimental evaluation for a possible resubmission.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The authors propose a framework to assess to which extent representations built by predictive models such as action-conditional CPC or SimCore contain sufficient information to answer questions about the environment they are trained/test on. The main idea is to train an independent LSTM (a Question-answer decoder) so that given the hidden state of the predictive model and a question about the environment, it is able to answer the question. 

The authors give empirical evidence that the representations created by SimCore contain sufficient information for the LSTM to answer questions quite accurately while the representations created by CPC (or a vanilla LSTM) do not contain sufficient information. Based on the experimental results, the authors argue that the information encoded by SimCore contains detailed and compositional information about objects, properties and spatial relations from the physical environment.

The idea is clearly explained and seems sensible, the paper is well written, the execution is competent and the authors provide a sufficient amount of details so that reproducibility should be possible. As a result, I am positive, however, I think it would be best accepted as a workshop paper given that:

- The experiment are only carried out on a single environment, however, their claims are rather general. To support such general claims, experiments on additional environments seem necessary.

- While the idea is sensible, the study is quite narrow because it only compares three models.

- While sensible, the methodological contribution is rather straightforward.

- The take home is quite brief.
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: The authors propose question answering (QA) as a tool to investigate what agents learn about the world, i.e., how much about the world is encoded in their internal states. The authors argue that this is an intuitive method for humans and allows for arbitrary complexity. 
Concretely, they train agents on exploration of a 3D environment using reinforcement learning and then ask them a set of non-trivial questions. This includes unseen combinations of seen attributes ("zero-shot"), showing that, what the agents learn, is to some degree compositional. Importantly, agents are not trained to answer questions explicitly.

The authors investigate multiple agents and find that LSTM and CPC|A representations are no better than chance, SimCore's representations seem to be the best for the QA task, and there is still a big performance difference between SimCore and the upper bound "No SG".

I think this paper is interesting and well done. I agree with the authors that QA is an intuitive probing tool, which can be used for similar agent analyses in the future.


review_title: Official Blind Review #2

Meta-review: This paper proposes question-answering as a general paradigm to decode and understand the representations that agents develop, with application to two recent approaches to predictive modeling. During rebuttal, some critical issues still exist, e.g., as Reviewer#3 pointed out, the submission in its current form lacks experimental analysis of the proposed conditional probes, especially the trade-offs on the reliability of the representation analysis when performed with a conditional probe as well as a clear motivation for the need of a language interface. The authors are encouraged to incorporate the refined motivation and add more comprehensive experimental evaluation for a possible resubmission.
Decision: Reject