Probing Emergent Semantics in Predictive Agents via Question Answering
Authors: Abhishek Das
Federico Carnevale
Hamza Merzic
Laura Rimell
Rosalia Schneider
Alden Hung
Josh Abramson
Arun Ahuja
Stephen Clark
Greg Wayne
Felix Hill
Venue:  (2020)
Keywords: question-answering
predictive models

Abstract: Recent work has demonstrated how predictive modeling can endow agents with rich knowledge of their surroundings, improving their ability to act in complex environments. We propose question-answering as a general paradigm to decode and understand the representations that such agents develop, applying our method to two recent approaches to predictive modeling – action-conditional CPC (Guo et al., 2018) and SimCore (Gregor et al., 2019). After training agents with these predictive objectives in a visually-rich, 3D environment with an assortment of objects, colors, shapes, and spatial configurations, we probe their internal state representations with a host of synthetic (English) questions, without backpropagating gradients from the question-answering decoder into the agent. The performance of different agents when probed in this way reveals that they learn to encode detailed, and seemingly compositional, information about objects, properties and spatial relations from their physical environment. Our approach is intuitive, i.e. humans can easily interpret the responses of the model as opposed to inspecting continuous vectors, and model-agnostic, i.e. applicable to any modeling approach. By revealing the implicit knowledge of objects, quantities, properties and relations acquired by agents as they learn, question-conditional agent probing can stimulate the design and development of stronger predictive learning objectives.

Under review as a conference paper at ICLR 2020
PROBING EMERGENT SEMANTICS IN PREDICTIVE AGENTS VIA QUESTION ANSWERING
Anonymous authors Paper under double-blind review
ABSTRACT
Recent work has demonstrated how predictive modeling can endow agents with rich knowledge of their surroundings, improving their ability to act in complex environments. We propose question-answering as a general paradigm and understand the representations that such agents develop, applying our method to two recent approaches to predictive modeling - action-conditions CPC (Guo et al., 2018) and SimCore (Gregor et al., 2019). After training agents with these predictive objectives a visually-rich, 3D environment with an assortment ob- jects, colors, shapes, and spatial configurations, resentations with a host of synthetic (English) questions, without backpropagating gradients from the -anssionnnnswering decoder into the agent. The performance of different agents when probed in this way reveals that they learn to encode de- tailed, and scemingly compositional, information about objects, properties and spatial relations from their physical environment. Our approach is intuitive, i.e. humans can easily interpret the responses of the model as opposed to inspecting continuous vectors, and model-agnostic, i.e. applicable to any modeling approach. By revealing the implicit knowledge of objects, quantities, properties and relations acquired by agents as they learn, cevesiin-condttional agent probing can stimulate the design and development of stronger predictive learning objectives.
decode
probe their internal state rep-
INTRODUCTION
Some of the biggest successes artificial intelligence have relied on learning representations from large labeled datasets (Krizhevsky et al., 2012; Sutskever et al., 2014) or dense reward signals (Mnih et al., 2015; Silver et al., 2016). However, an intelligent agent that is capable of functioning in a complex, open-ended environment should be capable of learning general representations that are task-agnostic, and should not require exhaustive labeled data collection or careful reward design.
One of the main challenges in developing such agents is the need for general approaches to evaluate and analyze agents' internal states. In this work, we propose question-anse answering as an evaluation paradigm for analyzing how much objective knowledge about the external environment is encoded an agent's 's internal representation. Our motivation do so twofold. First, question-answering provides intuitive investigative tool for humans one can simply ask an agent what it knows about its environment and get an answer back, without having to inspect internal activations. Second, the space of questions is fairly open-ended we can pose arbitrarily complex questions to an agent, has previously been
enabling a comprehensive analysis of its internal states. studied in textual (Rajpurkar al., 2016; 2018), visual (Malinowski & Fritz, 2014; Antol et al., 2015; Das et al., 2017) and embodied (Gordon et al., 2018: 8; Das et al., 201 2018a) settings. Crucially, however, these systems are trained end-to-end for the goal of answering questions. Here, we utilize question- answering simply to probe an agent's 's internal representation, without backpropagating gradients from the question-answering decoder into the agent. That is, we view question-answeet general purpose decoder of environment knowledge designed to assist the development of agents.
as
We are particularly interested in agents that can learn general task-agnostic representations of the external world. One promising way to achieve this is via self-supervised predictive modeling. In- spired by learning in humans (Elman, 1990; Quiroga et al., 2005; Nortmann et al., 2013; Rao & Ballard, 1999; Clark, 2016; Hohwy, 2013; Seth, 2015), predictive modeling, i.e. predicting future sensory observations, has emerged as powerful method to learn general-purposs neural network

---

Under review as a conference paper at ICLR 2020
Agent's view
Prediction Loss
lime I+k
Top-down
Simulation Network
stop gradient
-uestion-Answering Decoder
Three
How many wardrobes are there?
Figure I: We train predictive agents to explore visually-rich 3D environment with an assortment of objects of different shapes, colors and sizes. As the agent navigates (trajectory shown in white the top-down map), it contains auxiliary network that learns simulate representations of future observations (labeled 'Simulation Network') say steps into future self-supervised by ground-truth egocentric observation at è k. Simultaneouslyy another decoder network is trained to extract answers to variety of questions about the environment, conditioned on the agent's 's internal memory but without affecting it (notice 'stop gradient' gradients from the QA decoder not backpropagated into the agent), We use this question-answering paradigm to decode and understand internal representations that such agents develop. Note that top-down map is only shown for illustration purposes not available to agent.
on agent's future prediction against
representations (Elias, 1955; Atal & Schroeder, 1970; Schmidhuber, 1991; Schaul & Ring, 2013; Schaul et al., 2015; Silver et al., 2017; Wayne et al., 2018; Guo et al., 2018; Gregor et al., 2019; Recanatesi al., 2019). These representations with an environment in a task-agnostic manner, and later exploited for goal-directed behavior.
be learned while exploring in and interacting
We evaluate predictive answering testbed to investigate how much objective knowledge about environment semanties can be captured solety by egocentric prediction. By semantics, here specifically refer to information about objects quantity, colors, shapes, spatial relations. The set questions is intended to be holistic, i.e. they require representation of relevant aspects of the whole environment and in general cannot answered from a single observation, nor few consecutive observations of the episode and test variety of local and global scene understanding, visual reasoning, and recall skills.
non-predictive agents (both trained for exploration) on our question-
Concretely, we make the following contributions:
In a visually rich 3D room environment developed in the Unity game engine', we define and develop set of questions designed probe range of semantic, relational and spatial knowledge - from identifying shapes and colors (' ('What shape is the red object?') to counting ('How many blue objects are there?') to spatial relations ("What is the color of the chair near the table?'), exhaustive search ('Is there cushion?'), and comparisons ( ('Are there the same number of tables as chairs?'),
We train RI. agents augmented with predictive loss functions 1) action-conditional CPC (Guo al., 2018) and 2) SimCore (Gregor et al., 2019) for exploration task and analyze the internal representations they develop by decoding answers to our suite of questions. Crucially, the QA de- coder is trained independent of the predictive agent and we find that QA performance is indicative of the agent's ability to capture global environment structure and semantics solely through egocen- tric prediction. We compare these predictive agents strong non- con-predietive LSTM baselines as well as to an agent that is explicitly optimized for the
task.
We establish generality of the semantic knowledge by testing zero-shot generalization of a trained QA decoder compositionally novel questions (unseen combinations ing degree of compositionality in the internal representations captured by predictive agents.
attributes), suggest-
uni tymd.ccmmmmm
2

---

Under review as a conference paper at ICLR 2020
2
RELATED WORK
Our work relates to, and builds on, prior work on predictive modeling and auxiliary loss functions in reinforcement learning as well as grounded language learning and embodied question answering.
Predictive modeling and auxiliary loss functions in RL. The power of predictive modeling for representation learning has been known since at least the seminal work of Elman (1990) on emer- gent language structures. More recent examples include Word2Vec (Mikolov et al., 2013), Skip- Thought vectors (Kiros et al., 2015), and BERT (Devlin et al., 2019) in language. while in vision similar principles have been applied to context prediction (Doersch et al., 2015; Noroozi & Favaro, 2016), unsupervised tracking (Wang& & Gupta, 2015), inpainting (Pathak et al., 2016) and coloriza- tion (Zhang et al., 2016). More related to us is the use of such techniques in designing auxiliary loss functions for training model-free RL agents, such as successor representations (Dayan, 1993; Zhu et al., 2017a), value and reward prediction (Jaderberg et al., 2016; Wayne et al., 2018), contrastive predictive coding (CPC) (Oord et al., 2018; Hénaff et al., 2019), and SimCore (Gregor et al., 2019).
Grounded language learning. Inspired by the work of Winograd (1972) on SHRDLU, several recent works have explored linguistic representation learning by grounding language into actions and pixels in physical environments - in 2D gridworlds (Andreas et al., 2017; Yu et al., 2018; Misra et al., 2017) 1). 3D (Chaplot al., 2018; Hermann et al., 2017; Das et al., 2018a; Gordon et al., 2018; Cangea et al., 2019; Puig et al., 2018; Zhu et al., 2017a; Anderson et al., 2018; Gupta et al., 2017;
Zhu et al., 2017b; Oh et al., 2017; Shu et al., 2018; Vogel & Jurafsky, 2010) and textual (Matuszek
et al., 2013; Narasimhan et al., 2015) environments. Closest to our work is the task of Embodied Question Answering (Gordon et al., 2018; Das et al., 201 ba;b; Yu et al., 2019; Wijmans et al., 2019) where an embodied agent in environment (e.g. a house) is asked to answer a question (e.g. "What color is the piano?"). Typical approaches to EmbodiedQA involve training agents to move for the goal of answering questions. In contrast, our focus is on learning a good predictive model in a goal-agnostic exploration phase and using question-answerin as post-hoc testbed for evaluating the semantic knowledge that can emerge in the agent's representations from predicting the future.
Neural population decoding. Probing an agent using QA decoder can be viewed as a variant of neural population decoding, successfully used as an analysis tool in neuroscience (Georgopoulos et al., 1986; Bialek et al., 1991: Salinas & Abbott, 1994) and more recently in deep learning (Guo et al., 2018; Gregor et al.. 2019; Azar et al., 2019; Alain & Bengio, 2016). The idea is test if desired information is encoded in the learned representation by feeding it as input to a network tasked to extract the desired information. In deep learning, this is done by training network to predict the desired parts of the ground-truth state of the environment, such as an agent's 's position or orientation, without backpropagating through the agent's internal state. We extend this idea by using question-answerrng as a general purpose decoder conditioned by an arbitrary question, through which we can attempt extract complex high-level information from the agent's internal state.
Neurocognition. Predictive modeling is thought to be fundamental component of human neu- rocognition (Elman, 1990; Clark, 2016; Hohwy, 2013; Seth, 2015). In particular, it has been pro- posed that human learning and decision- rely on the minimization of prediction error (Clark, 2016; Friston, 2010; Friston et al., 2017: Hohwy, 2013; Seth, 2015). A well-established strand of work has focused on decoding predictive representations in brain states (Quiroga et al., 2005; Nort- mann et al., 2013; Huth et al., 2016). The question of how prediction of sensory experience relates to higher-order conceptual knowledge is complex and subject to debate (Williams, 2018; Roskies & Wood, 2017), though some have proposed that conceptual knowledge, planning, reasoning, and other higher-order functions emerge in deeper layers of predictive network. We focus on the emer- gence of semantics in an artificial predictive agent's 's internal representations.
3 ENVIRONMENT & TASKS
Environment. We use Unity-based visually-rich 3D environment (see Figure 1). It is a single L-shaped room that can be programmatically populated with an assortment of objects of different colors at different spatial locations and orientations. In total, we use library of 50 different objects, referred to as 'shapes' henceforth (e.g. chair, teddy, glass, etc.), in 10 different colors (e.g. red, blue, green, etc.). For complete list of objects, attributes, and other environment details, see section A.4.
3

---

Under review as a conference paper at ICLR 2020
At every step, the agent gets 96 72 first-person RGB image its observation, and the action space consists of movements object pick-up and manipulation (4 DoF: yaw, pitch, roll, and movement along the axis between the agent and object). See Table in the Appendix for the full set of actions.
back, 1eft, right)), turns (turn- (up, (up,down, down, 1eft, right)). and
Question type
Template
Level codename
QA pairs
Attribute
What is the color of the sishape:>?
color
500
What shape is the <collrr object?
shape
500
Count
How many <sshape> there?
count_shaap
200
How many <color> objects are there?
count_color
40
Exist
Is there <shape>>?
existence_shar
100
Compare + Count
Are there the same number <colort> objects as <color?> objects? number <shapell <shape2>?
compare_count_com
180
Are there the
compare_count_sie
1900
Relation
Attribute
What is the color of the <s <shape1>> > near the <tshape:>??
near_color
24500
What <color>> object
<cshape>??
near_shape
25000
Table 1: cuestion-anssering task templates. In every episode, objects and their configurations are randomly generated, and these task templates get translated to question-answer pairs for all unambiguous <shape, color> combinations. There are 50 shapes and 10 colors in total. For more details, see section A.4,
Question-Answering Tasks. We develop a range of question-answeriig tasks of varying com- plexity that test the agent's local and global scene understanding, visual reasoning, and memory skills. Inspired by Johnson et al. (2017); Das et al. (2018a); Gordon et al. (2018), we programmat- ically generate dataset of questions (see Table These questions ask about the presence or ab- sence of objects (existence_shapp shape), their attributes (color, shape), counts (count_.oloo, count_shape), compare_count_siee chanrtsppentt_ppp_ and elementary spatial rela- tions (near_color, near_shape). Unlike the fully-observable setting in CLEVR (Johnson et al., 2017), the agent does not get a global view of the environment, and must answer these questions from sequence of partial egocentric observations. Moreover, unlike prior work on Embodied Question Answering (Gordon et al., 2018; Das al., 2018a), the agent is not being trained end-to-end to move to answer questions. It is being trained to explore, and answers are being decoded (without backpropagating gradients) from its internal representation. Thus order to answer these questions, the agent must learn to encode relevant aspects of the environment in representation amenable to easy decoding into symbols (e.g. what does the word "chair" mean? or what representations does computing "how many" require?).
quantitative comparisons (compare_comm) ctwprre__cotttooorrroroo
4 APPROACH
Learning an exploration policy. Predictive modeling has proven to be an effective approach for an agent develop general knowledge of its environment it explores and behaves in the service of its principal goal, typically maximising environment returns (Gregor et al.. 2019; Guo et al., 2018). Since wish to evaluate the effectiveness of predictive modeling independent of the agent's specific goal, we define a simple task that stimulates the agent to visit all of the 'important' places in the environment (i.e. acquire an exploratory but otherwise tesk-neutral policy). This is achieved by giving the agent a reward of +1.0 every time it visits an object in the room for the first time. After visiting all objects, rewards are refreshed and available to consumed by the agent again (i.e. re-visiting an object the agent has already been to will now again lead to + 1.0 reward), and this process continues for the duration of each episode (30 seconds or 900 steps).
During training on this exploration task, the agent receives first-person RGB observation o, at every timestep f, and processes using a convolutional neural network to produce εt. This is input to an LSTM policy whose hidden state is ht and output a discrete action The agent optimizes the discounted sum of future rewards using an weportance-weiighed actor-critic algorithm (Espeholt et al., 2018).
Training the QA-decoder. The question-answerte decoder is operationalized initialized with the agent's internal representation h and receives the question as input at every timestep (see Figure 2). The question is string that tokenise into words and then map to learned embeddings. The question decoder LSTM is then unrolled for a fixed number of computation steps after which it predicts softmax distribution the vocabulary of one-word answers questions
LSTM that
4

---

Under review as a conference paper at ICLR 2020
Action conditinaniiioonalll GPC (Bue et 2013)
SimCore
(Gregoret 2018)
at-1
a4
Paaiciss
24+1 F++2
R++
OA Desocer
hy
h4-1
ht
oct1
ac+T
Oc+1
d++T
Zt-1
Zt.
Snowe
rtop
gradkant
question
queston
question
1
Zt
Figure 2: Overview our approach: at every timestep f, the agent receives an RGB observation Iy as input, processes it using a convolutional neural network to produce 21, which is then processed by an LSTM to select action ar. The agent is learning to explore it receives a reward of 1.0 for navigating to each object in the environment. As explores the environment, it builds up internal representation from an auxiliary predictive modulc to capture environment semantics its actions multiple steps into the future. We experiment with vanilla LSTM agent and two recent predictive approaches action-conditional CPC (Guo et al., 2018) and SimCore (Gregor et al., 2019). The learnt internal representations are then analyzed via guestion-answering decoder whose gradients are not backpropagated to the agent core. The QA decoder is LSTM initialized with and receiving the question every timestep.
which receives pressure accurately predict consequences
in Table I, and is trained via a cross-entropy loss. Crucially, this QA decoder is trained independent of the agent policy; i.e. gradients from this decoder not allowed to flow back into the agent. We evaluate question-answering performance by measuring top-1 accuracy at the end of the episode- we consider the agent's 's top predicted answer at the last time step of the episode and compare that with the ground-trutt answer.
The QA decoder can be seen as a general purpose decoder trained to extract object-specific knowl- edge from the agent's ' internal state without affecting the agent itself. If this knowledge not re- tained in the agent's 's internal state, then this decoder will not be able to extract it. This is important difference with respect to prior work (Gordon et al., 2018; Das et al., 2018a) wherein agents were trained move to answer questions, i.e. all parameters had access to linguistic information. Recall that the agent's 's navigation policy has been trained for exploration, and so the visual information required to answer question need not be present in the observation the end of the episode. Thus, through
we are evaluating the degree to which agents encode relevant aspects of the environment (object colors, shapes, counts, spatial relations) in their internal representations and maintain this information in memory beyond the point at which it was initially received
4.1 AUXILIARY PREDICTIVE LOSSES
We augment the baseline architecture described above with an auxiliary predictive head consisting of a simulation network (operationalized an LSTM) that is initialized with the agent's 's internal state h and deterministically simulates future latent states si.... .t. s..... in an open-loop manner, re- ceiving the agent's action sequence as input. We evaluate two predictive losses cction-conditional CPC (Guo et al., 2018) and SimCore (Gregor et al., 2019). See Fig. 2 for overview, A. 1.1.2 for details.
oction-conditional CPC (CPCIA, Guo al. (2018)) makes use of noise contrastive estimation model to discriminate between true observations processed by the convolutional neural network 2++ (A steps into the future) and negatives randomly sampled from the dataset 24+k. in from other episodes the minibatch. Specifically, at each timestep t + (up to a maximum), the output of the simulation core sy and 2 are fed to MLP to predict 1, and of and z are used to predict O.
"See Scction A.1.3 for details about the QA decoder.
5

---

Under review as a conference paper at ICLR 2020
Navigation palicy reward
What shape is the <color> object?
LSTM
30
SimCore
Q-only
0.8
CPC/A
QA Accuracy
Reward
0.8
15
0.4
0.2
5
0
8.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
1.6
steps
1e9
steps
1e9
Figure 3: L Reward in episode. R Top-1 1 QA accuracy. Averaged over seeds. Shaded region is SD.
Sim Core (Gregor al., 2019) uses the simulated state s to condition generative model based on ConvDRAW (Gregor et al., 2016) and GECO (Rezende & Viola, 2018) that predicts the distribution of true observations p(*++*</y, (t.....(--k)) pixel space.
Baselines. We evaluate and compare the above approaches with 1) a vanilla RL agent without any auxiliary predictive losses (referred to 'LSTM'), and 2) question-only agent that receives zero- masked observations as input and is useful to measure biases in our question-answering testhed. Such baseline is critical, particularly when working with simulated environments, cover biases in the environment's generation of tasks that can result in strong but uninteresting performance from agents capable of powerful function approximation (Thomason et al., 2019).
No stop gradient. We also compare against an agent without blocking the QA decoder gradients (labeled 'No SG'). This model differs from the above in that it is trained end-to-end with supervi- sion to answer the set of questions in addition to the exploration task. Hence, represents an agent receiving privileged information about how to answer the questions and its performance provides an upper bound indicative of how challenging these question-ansee answering tasks are in this context.
5 EXPERIMENTS & RESULTS
5.1 QUESTION-ANSWERRN PERFORMANCE
We begin by analyzing performance on single question-answering task shape which consists of questions of the form "what shape is the <color> object?". Figure 3 shows the average reward accumulated by the agent one episode (left) and the QA accuracy the last timestep of the episode (right) for all approaches over the course of training. We make the following observations:
All agents learn to explore. With the exception 'question-only', all agents achieve high reward on the exploration task. This means that they visited all objects in the room more than once each and therefore, in principle, have been exposed to sufficient information to answer all questions.
Predictive models aid navigation. Agents equipped with auxiliary predictive losses CPCIA and SimCore collect the most rewards, suggesting that predictive modeling helps navigate the environment efficiently. This is consistent with findings in Gregor al. (2019).
QA decoding from LSTM and CPCIA representations no better than chance.
SimCore's internal representations lead to best QA accuracy. SimCore gets a QA accuracy of ~72% indicating that its representations best capture environment knowledge and are the most suitable decoding answers to questions. Figure 4a shows example agent predictions.
Wide gap between SomCore and No SG. There still a ~ 24% gap between SimCore and the No SG oracle, suggesting scope for developing better auxiliary predictive losses.
worth emphasizing that answering this shape question from observations is not a challenging task in and of itself. The No SG agent, which is trained end-to-end optimize both for exploration and QA, achieves almost perffett accuracy (~96%). The challenge arises from the fact that we are not training the agent end-to-end from pixels navigation QA - but decoding the answer from

---

Under review as a conference paper at ICLR 2020
cownt_.color color
compar caint, ahape
comport R. count
coant shape
count, count_coolo color
shape
near .color
Overall
shape
color
eriat
Baseline: Question-only
0.29
0.01
0.1
0.63
0.21
0.24
0.19
0.70
0.01
0.09
LSTM
0.31
0.04
0.1
0.54
0.34
0.38
0.53
0.70
0.04
0.09
CPC
0.06
0.39
0.50
0.10
SimCore
0.60
0.72
0.81
0.72
0.39
0.57
0.56
0.73
0.30
0.59
Oracle: No SG
0.63
0.96
0.81
0.60
0.45
0.37
0.51
0.76
0.41
0.72
Table 2: Top-1 accuracy question-answering tasks.
the agent's internal state, which learned agnostic the question. The answer can only be decoded if the agent's ss internal state contains relevant information represented in an easily-decodable way.
Decoder complexity. To explore the possibility that answer-relevant information is present in the agent's internal state but requires a more powerful decoder, we experiment with QA decoders of range of depths. As detailed in Figure 6 in the appendix, we find that using deeper QA decoder with SimCore does lead to higher QA accuracy (from 1 - 12 layers), although greater decoder depths become detrimental after 12 layers. Crucially, however, in the non-predictive LSTT agent, the correct answer cannot be decoded irrespective the capacity of the QA decoder. This high- lights an important aspect of question-answeriing evaluation paradigm that while the absolute accuracy at answering questions may also depend on decoder capacity, relative differences provide an informative comparison between the internal representations developed by different agents.
Table shows QA accuracy for all QA tasks (see Figure 7 in appendix for training curves). The results reveal large variability in difficulty across question types. Questions about attributes (color and shape), which can be answered from a single well-chosen frame of visual experience, are the easiest, followed by spatial relationship questions (near_ molor_ and near phapes) and the hardest are counting questions (count_color and count_shape). We further note that:
All agents perform question-only, which captures any biases in the environment question distributions (enabling strategies such as constant prediction of the most-common answer).
CPCIA representations are not better than LSTM most question types.
SimCore representations achieve higher QA accuracy than all other approaches, sub- stantially above question-only on count_color (57% vs. 24%), near_shape (30% vs. 4%) and near_color (59% vs. 9%), demonstrating strong tendency for encoding and retaining informa- tion about the identity of objects, their properties, and both spatial as well temporal relations.
Finally, as before, the No SG agent trained to answer questions without stopped gradients achieves highest accuracy for most questions, although not - perhaps due to trade-offs between simulta- neously optimizing performance for different QA losses and the exploration task.
5.2 COMPOSITIONAL GENERALIZATION
While there is high degree of procedural randomization in our environment and QA tasks, over- parameterized neural-network-baal rote memorization. We therefore constructed a test of the generality of the information encoded in the internal state of an agent. The test involves a variant of the shape question type (i.e. ques- tions like "what shape is the <color>> object?"), but in which the possible question-answer pairs are partitioned into mutually exclusive training and test splits. Specifically, the test questions are constrained such that they are compositionally novel the <color, shape>> combination involved in the question-answer pair is never observed during training, but both attributes are observed in other contexts. For instance, test question-an -eeste pair "Q: what shape is the blue object?, A: table" is excluded from the training set of the QA decoder, but "Q: what shape is the blue object?, A: car" and "Q: What shape is the green object?, A: table" are part of the training set (but not the test set).
models limited environments are always prone to overfitting
We evaluate only the SimCore agent on this test of generalization (since other agents perform poorly on the original task itself). Figure 4b shows that the QA decoder applied to SimCore's 's internal states performs at substantially above-chance (and all baselines) on the held-out test questions (although somewhat lower than training performance). This indicates that the QA decoder extracts and applies

---

Under review as a conference paper at ICLR 2020
object?
Train qjedtoo set
0.8
Test queetion net
Question-on
Accurary
0.6
aispeia graardeed?
(a) Sample trajectory (1 1) and QA decoding prodictions (for top 5 most probable
answers) for the question 'What shape is the green object?' from the SimCore agent.
68.00
0.75
0.50
0.75
steps
1.00
1.25
1.50
1.75
100
2.00
Note that the top-down map is only shown for illustration and not available to the agent.
(b) QA accuracy on disjoint train and test splits.
Figure
information in comparatively factorized (or compositional) manner, and suggests (circumstan- tially) that the knowledge acquired by the simcore may also be represented in this way.
DISCUSSION
We introduced question-answering paradigm to evaluate and analyze representations learned by artificial agents. In particular, we tested how much knowledge about the external environment can be decoded from predictive vs. non-predictive RL agents. We started by developing a range of question-answering tasks in visually-rich 3D environment serving as diagnostic test of an agent's scene understanding, visual reasoning, and memory skills. Next, we trained agents to optimize an exploration objective with and without auxiliary self-supervised predictive losses, and evaluated the representations they form as they explore an environment via this question -anseerinnn testbed. We found that predictive agents (in particular SimCore (Gregor et al., 2019)) are able to reliably capture detailed environment semanties their internal states, which can be easily decoded as answers to questions, while non-predictive agents do not, even if they optimize the exploration objective well.
Interestingly, not all predictive agents equally good at forming these representations. We com- pared model explicitly learning the probability distribution of future frames in pixel space via generative model (SimCore (Gregor al., 2019)) with model based discriminating frames through contrastive estimation (CPCIA (Guo et al., 2018)). We found that while both learned to navigate well, only the former developed representations that could be used for answering ques- tions about the environment. Gregor et al. (2019) previously showed that the choice of predictive model has a significant impact on the ability to decode an agent's position, orientation and top-down map reconstructions of the environment, Here we extend this idea to more high-level and complex aspects of the environment and show the value of our qmestion-answeriin approach in comparing existing agents and its potential utility tool for developing better ones.
Finally, the fact that agent's internal representations learned solely from egocentric future predictions without exposing the agent to any questions is encouraging. It indicates that the agent is learning form and maintain invariant object identities and properties (modulo limitations in decoder capacity) in its internal state without explicit supervision. structures and semantic organization can emerge in the units of a neural network as consequence of the simple objective of predicting the next word in sequence. This work corroborates Elman's belief in the power of prediction by demonstrating the diversity of knowledge that can emerge when a situated neural-network agent endowed with powerful predictive objectives applied to raw pixel observations. We think we have just scratched the surface of this problem, and hope our work inspires future research evaluating predictive agents using natural linguistic interactions.
decode answers to questions (i.e. symbolic information) from an
almost 30 years since Elman (1990) showed how syntactic
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Guillaume Alain and Yoshua Bengio. Understanding intermediatc layers using lincar classifier probes. arXiv preprint urXiv:1610.01644, 2016.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision-and-languaae navigation: Interpreting risually-grounde navigation instructions in real environments. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 3
Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In Proceedings of the International Conference on Machine Learning (ICML), 2017.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, Lawrence Zitnick, and Devi Parikh. VQA: Visual Question Answering. In Pruceedingy of IEEE International Conference on Computer Vision (ICCV), 2015.
Bishnu S Atal and Manfred R Schroeder. Adaptive predictive coding of speech signals. Bell System Technical Journal, 1970. 2
Mohammad Gheshlaghi Azar, Bilal Piot, Beraardo Avila Pires, Bean-Bastian Grill, Florent Altché, and Rémi Munos. World discovery models. arXiv preprint arXiv: 1902:0099020783 2019.
Charles Beattie, Joel Z. Leibo, Denis Teplyashin, Tom Ward, Mareus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Victor Valdés, Amir Sadik, Julian Schrittwisser, Keith Anderson, Sarah York, Max Cant, Adam Cain, Adrian Bolton, Stephen Gaffney, Helen King, Demis Hassabis, Shane Legg, and Stig Petersen. Deepmind lab. CoRR, abs/1612.038011 2016. URL
03801. 16, 18
William Bialek, Fred Ricke, RR De Ruyter Van Steveninck, and David Warland. Reading a neural code. Science, 252(5014):18544
1991.
Cätälina Cangea, Fugene Belilovsky, Pietro Liò, and Aaron Courville. Videonavqa: Bridging the gap hetween visual and embodied question answering. arXiv preprint arXiv: arXiv: 190,00450,, 2019. 3
Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pasumarthi, Dheeraj Rajagopal, and Ruslan Salakhutdinov. ence on Artificial Intelligence, 2018."
architectures for task orientree language grounding. In AAAI Confer-
A. Clark. Surfing umcertainty. Oxford University Press, Oxford, 2016.
1,
Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José M.F. Moura, Devi Parikh, and Dhruv Batra. Visual Dialog. In CVPR, 2017. 1
Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, Dhruv Batra. Embodied Ques- tion Answering. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
2018a. 1,4,
Abhishck Das, Georgia Gkioxari, Stefan Lec, Devi Parikh, and Dhruv Batra. Neural Modular Control for Embodied Question Answering. In Proceedings of International Conference on Robot Learning (CoRL), 2018b. 3
Peter Dayan. Improving generalization for temporal difference learning: The successor representation. Neural Computation. 1993.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre- Pre-training of deep bidirectional transformers for language understanding, In NAACL, 2019.
Carl Doersch, Abhinav Gupta, and Alexei Efros. Unsupervised visual representation learning by context prediction. In Proceedings of IEEE International Conference on Computer Vision (ICCV), 2015.
Peter Elias. Predictive coding - IRE Transactions on Information Theory, 1955.
Jeffrey L. Elman. Finding structure in time. Cognitive science, 1990. 1, 3,
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-leamer architcctures. arXiv preprint 1802.0000..0166 2018. 4,
K. Friston. The free-energy principle: A unified brain theory? Nature Reviews Neuroscience, 11:127-138, 2010. 3

---

Under review as a conference paper at ICLR 2020
K. Friston, T. FitzGerald, F. Rigoli, P. Schwartenbeck, and G. Pezzulo. Active inference: A process theory. Neural Computation, 29:1-49, 2017.
Apostolos P Georgopoulos, Andrew B Schwartz, and Ronald E Kettner. Neuronal population coding of move- ment direction. Science, 233(4771):1416 14444, 1986. 3
Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox. and Ali Farhadi. IQA: Visual Question Answering in Interactive Environments. In Proceedings of IEFE Conference on Com- puter Vision and Pattern Recognition (CVPR), 2018. 1, 1,3, 4,5
Karol Gregor, Frederic Besse, Danilo Jimenez Rezende, Ivo Danihelka, and Daan Wierstra. Towards conceptual compression. In Advances in Neural Information Processing Systems (NeurlPS), 2016. 6, 13
Karol Gregor, Danilo Jimenez Rezende, Frederic Besse, Yan Wu, Hamza Merzic, and Aaron van den Oord. Shaping Belief States with Generative Environment Models RI.. In Advances in Neural Information Processing Systems (NeurIPS), 2019. 1, 2, 3, 5, 6, 8, 13, 15
Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Bernardo A Pires, Toby Pohlen, and Rémi Munos. Neural predictive bclief representations. arXiv preprint 1rXiv:1811.06407 2018. 2, 4, 13
Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, and Jitendra Malik. Cognitive mapping and planning for visual navigation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 13
Olivier J Hénaff, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recog- nition with contrastive predictive coding. arXiv preprini 1rXvv:1905.09273 2019. 3
Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojtek Czarnecki, Max Jaderberg, Denis Teplyashin, et al. Grounded language learning simulated 3D world. arXiv preprint arXiv:/706.06::: 705.06651,,200000 2017.
J. Hohwy. The predictive mind. Oxford University Press, Oxford, 2013. 1,3
Alexander G Huth. Tyler Lee, Shinji Nishimoto, Natalia Y Bilenko, An T Vu, and Jack L Gallant. Decoding semantic content natural movies from human brain activity. Frontiers in systems neuroscience, 2016.
3
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo. David Silver, and Koray Kavukcuoglu. Reinforeement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.05533 2016.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitiick and Ross Gir- shick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning, In CVPR, 2017. 4
Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. In Advances in Neural Information Processing Systems (NIPS), 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoff Hinton. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems (NIPS), 2012. 1
Mateusz Malinowski and Mario Fritz. Multi World Approach to Question Answering about Real- World Scenes based on Uncertain Input. Advances Neural Information Processing Systems (NIPS), 2014.
Cynthia Matuszek, Evan Herbst, Luke Zettlemoyer, and Dieter Fox. Learning to parse natural language com- mands to robot control system. In Experimental Robotics, 2013. 3
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint 1rXiv:130.37888 2013.
Dipendra Misra, John Langford, and Yoav Artzi Mapping instructions and visual observations to actions with reinforcement learning. In Proceddings of the Annual Meeting Association for Computational Linguistics (ACI.), 2017.
10

---

Under review as a conference paper at ICLR 2020
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattic, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hass- abis. Human-leve control through deep reinforcement learning. Nature, 518(7540):529 533, 02 2015. URL http://dx. doi .org//. 1838/ 228/naauure1236.
Karthik Narasimhan, Tejas Kulkarni, and Regina Barzilay. Language understanding for text-based games using deep reinforcement lcarning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2015.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In Proceedings of European Conference on Computer Vision (ECCV), 2016,
Nora Nortmann, Sascha Rekauzke, Selim Onat, Peter König, and Dirk Jancke. Primary visual cortex represents the difference between past and present. Cerebral Cortex, 2013. 3,3
Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. Zero-shot task generalization with multi-task deep reinforcement learning. In Proceedings of the International Conference Machine Learning (ICML), 2017. 3
Aaron van den Oord, Yazhe Li. and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint ( 2018.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexci A Efros. Context encoders: Fea- ture learning by inpainting. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang. Sanja Fidler, and Antonio Torralba. Virtu- alhome: Simulating household activities via programs. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 3
R Quian Quiroga, Leila Reddy, Gabriel Kreiman, Christof Koch, and Itzhak Fried. Invariant visual representa- tion by single neurons in the human brain. Nature, 2005. 1,3
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2016.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't 't know: Unanswerable questions for squad. In Proceedings of the Annual Meeting on Association for Computational Linguistics (ACL., 2018. 1
Rajesh PN Rao and Dana H Ballard. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-fiele field effects. Nature neuroscience, 1999.
Stefano Recanatesi, Matthew Farrell, Guillaume Lajoie, Sophie Deneve, Mattia Rigotti, and Eric Shea- Brown. Predictive learning extracts latent space representations from sensory observations. bioRxiv, 2019.
Danilo Jimenez Rezende and Fabio Viola. Taming VAEs. arXiv preprint arXiv:1810000059, 2018.
A. Roskies and C. Wood. Catching the prediction wave in brain science. Analysis, 77:848-857, 2017. 3
Emilio Salinas and LF Abhott. Vector reconstruction from firing rates. Journal of computational neuroscience, 1(1-2)899-10) 1994. 3
Tom Schaul and Mark Ring. Better generalization with forecasts. In Pruceedings of the International Joint Conference on Artificial Intelligence, 2013. 2
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In Proceedings of the International Conference Machine Learning (ICML), 2015. 2
Jürgen Schmidhuher. Curious model-building control systems. In Proc. international joint conference on neural networks, 1991. 2
A. K. Seth. The cybernetic bayesian brain: From interoceptive inference to sensorimotor contingencies. In
T. Metzinger & J. M. Windt (cd.), Open MIND: 35(T). MIND Group, Frankfurt am Main, 2015. 1,
Tianmin Shu, Caiming Xiong, and Richard Socher. Hierarchical and interpretable skill acquisition in multi- task reinforcement learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
11

---

Under review as a conference paper at ICLR 2020
David Silver, Aja Huang, Christopher J. Maddison, Arthur Guez, Laurent Sifre, George den Driessche, Julian Schrittwieser, Joannis Antonoglou, Veda Panneershelvam, Mare Lanctot, Sander Dieleman, Do- minik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madelcine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 2016.
David Silver, Hado van Hasselt, Mattco Hessel, Tom Schaul, Arthur Guez, Tim Harley, Gabriel Dulac-Arnold, David Reichert, Neil Rabinowitz, Andre Barreto, et al. The predictron: End-to-end learning and planning. In Proceedings of the International Conference Machine Learning (ICML), 2017.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to Sequence Learning with Neural Networks. In Advances in Neural Information Processing Systems (NIPS), 2014.
Jesse Thomason, Daniel Gordan, and Yonatan Bisk. Shifting the baseline: Single modality performance on visual navigation & QA. In NAACL, 2019. 6
Adam Vogel and Dan Jurafsky. Learning to follow navigational directions. In Proceedings of the Annual Meeting on Association for Computational Linguistics (ACL), 2010. 3
Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using vidcos. In Proceed- ings IEEE International Conference Computer Vision (ICCV), 2015.
Greg Wayne, Chia-Chun Hung, David Amos, Mchdi Mirza, Arun Ahuja, Agnieszka Grabska- Barsska,a Jack Rac, Piotr Mirowski, Joel Z Leibo, Adam Santoro, al. Unsupervised predictive memory goal- goal-directed agent, arXiv preprint ar Xiv: 10760, 2018. 3,
Erik Wijmans, Samyak Datta, Oleksandr Maksymets, Abhishek Das, Georgia Gkioxari, Stefan Lee, Irfan Essa, Devi Parikh, and Dhruv Batra. Embodied Question Answering in Photorealistic Environments with Point Cloud Perception. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 3
Daniel Williams. Predictive coding and thought. Synthese, pp- 1-27, 2018. 3
Terry Winograd. Understanding natural language. Cognitive Psychology, 1972.
Haonan Yu, Haichao Zhang, and Wei Interactive Grounded Language Acquisition Generalization in World. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
Licheng Yu, Xinlei Chen, Georgia Gkioxari, Mohit Bansal, Tamara Berg, and Dhruv Batra. Multi-target embodied question answering. In Proceedings of IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), 2019. 3
Richard Zhang, Phillip Isola, and Alexci A Efros. Colorful image colorization. Proceedings of European Conference on Computer Vision (ECCV), 2016.
Yuke Zhu, Daniel Gordon, Eric Kolve, Dieter Fox, Li Fei-Fei, Abhinav Gupta, Roozbeh Mottaghi, and Ali Farhadi. Visual Semantic Planning using Deep Successor Representations. In Proceedings of IEEE Interna- tional Conference Computer Vision (ICCV), 2017a. 3
Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinay Gupta, Li Fei-Fei, and Ali Farhadi. Target- driven visual navigation in indoor seenes using deep reinforeement learning. In Proceedings of IEEE Inter- national Conference Robotics and Automation (ICRA), 2017b.
12

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 NETWORK ARCHITECTURES AND TRAINING SETUP
A.1.1 IMPORTANCE WEIGHTED ACTOR-LEARNEE ARCHITECTURE
Agents were trained using the IMPALA framework (Espeholt et al., 2018). Briefly, there are N parallel 'actors' collecting experience from the environment in a replay buffer and one learner taking batches of trajectories and performing the learning updates. During one learning update the agent network is unrolled, all the losses (RI, and auxiliary ones) are evaluated and the gradients computed.
A.1.2 AGENTS
Input encoder To process the frame input, all models in this work use residual network (He al., 2016) of 6 64-channel ResNet blocks with rectified linear activation functions and bottleneck channel of size 32. We use strides of (2, 1, 2. 1, 2, 1) and don't t use batch-norm. Following the convnet we flatten the ouput and use linear layer to reduce the size to 500 dimensions. Finally, We concatenate this encoding of the frame together with a one hot encoding of the previous action and the previous reward.
Core architecture The recurrent core of all agents is a 2-layer LSTM with 256 hidden units per layer. At each time step this core consumes the input embedding described above and update its state. We then use 200 units single layer MLP to compute a value baseline and an equivalent network to compute action logits, from where one discrete action is sampled.
Simulation Network Both predictive agents have a simulation network with the same architecture as the agent's core. This network is initialized with the agent state at some random time t from the trajectory and unrolled forward for a random number of steps up to 16, receiving only the actions of the agent inputs. We then use the resulting LSTM hidden state as conditional input for the prediction loss (SimCore or CPCIA).
Core We use the same architecture and hyperparameters described in Gregor et al. (2019). The output of the simulation network used condition a Convolutional DRAW (Gregor et al., 2016). This is conditional deep variational auto-encoder with recurrent encoder and decoder using convo- lutional operations and canvas that accumulates the results each step to compute the distribution over inputs. It features a recurrent prior network that receives the conditioning vector and computes a prior over the latent variables. See more details Gregor et al. (2019).
Action-conditional CPC We replicate the architecture used in Guo et al. (2018). CPCIA uses the output of the simulation network as input to MLP that is trained to discriminate true versus false future frame embedding. Specifically, the simulation network outputs a conditioning vector after k simulation steps which concatenated with the frame embedding ž produced by the image encoder on the frame Je+A and sent through the MLP discriminator. The discriminator has one hidden layer of 512 units, ReLU activations and linear output of size which is trained to binary classify true embeddings into one class and false embeddings into another. We take the negative examples from random time points the same batch of trajectories.
A.1.3 QA NETWORK ARCHITECTURE
Question encoding The question string is first tokenized to words and then mapped to integers corresponding to vocabulary indices. These are then used to lookup 32-dimensional embeddings for each word. We then unroll a 64-units single-layer LSTM for fixed number of 15 steps. The language representation is then computed by summing the hidden states for all time steps.
QA decoder. To decode answers from the internal state of the agents we use second LSTM initial- ized with the internal state of the agent's LSTM and unroll it for fix number of steps, consuming the question embedding at each step. The results reported in the main section were computed using 12 decoding steps. The terminal state is sent through two-layer MLP (sizes 256, 256) to compute a vector of answer logits with the size of the vocabulary and output the top-1 answer.
1.1.4
The hyper-parameter values used in all the experiments are in Table 3.
13

---

Under review as a conference paper at ICLR 2020
Agent
Learning rate
le-4
Unroll length
50
Adam 31
0.90
Adam S2
0.95
Policy entropy regularization
0,0003
Discount factor
0.99
No. of ResNet blocks
No. of channel in ResNet block
64
Frame embedding size
500
No. of LSTM layers
No. of units per LSTM layer
256
No. units in value MLP
200
No. of units in policy MLP
200
Simulation Network
Overshoot length
16
No. of LSTM layers
2
No. units per LSTM layer
256
No. of simulations per trajectory
No. of evaluations per overshoot
SimCore
No. of ConvDRAW Steps
GECO kappa
0.0015
CPCIA
MLP discriminator size
64
QA network
Vocabulary size
1000
Maximum question length
15
No. of units in Text LSTM encoder
64
Question embedding size
32
No. LSTM layers in question decoder
No. of units per LSTM layer
256
No. of units in question decoder MLP
200
No. of decoding steps
12
Table 3: Hyperparameters.
A.5. NEGATIVE SAMPLING STRATEGIES FOR CPCIA
We experimented with multiple sampling strategies for the CPCIA agent (whether or not negative examples are sampled from the same trajectory. the number of contrastive prediction steps, the number of negative examples). We report the best results in the main text. The CPCIA agent did provide better representations of the environment than the LSTM--based agent, as shown by the top-down view reconstruction loss (Figure 5a). However, none of the CPCIA agent variations that tried led better-than-chanee question-answerinn accuracy. As example, in Figure 5b compare sampling negatives from the same trajectory or from any trajectory the training batch.
A.2 EFFECT OF QA NETWORK DEPTH
To study the effect of the QA network capacity the answer accuracy, we tested decoders of differ- ent depths applied to both the SimCore and the LSTM agent's internal representations (6). The QA network is an LSTM initialized with the agent's internal state that unroll for fix number of steps feeding the question as input at each step. We found that, indeed, the answering accuracy increased with the number of unroll steps from 1 to 12, while greater number of steps became detrimental. We performed the same analysis on the LSTM agent and found that regardless the capacity of the QA network, could not decode the correct answer from its internal state, suggesting that the limiting
14

---

Under review as a conference paper at ICLR 2020
What shape the <color> object?
23.50
What shape is the <color> object?
CPC same trajectory
Top-down decoding loss
23.25
trajectories
0.14
CPC trajectory
23.00
LSTM
0.12
CPC trajectories
LSTM
22.75
Accuracy
0.10
22.50
0.08
21.75
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
step. counter
lee
0.3
0.5
1.5
2.0
2.5
3.5
4.0
(a) To whether the CPCIA provided improved represen-
1e8
tations we reconstructed environment top-down view, similar (b) QA accuracy the CPCIA agent is not better to Gregor al. (2019). Indeed reconstruction loss is lower LSTM agent, for both sampling stratogies of negatives. for CPCI than for the LSTM agent.
Figure
factor is not the capacity of the decoder but the lack of useful representations in the LSTM agent state.
SimCore Agent
LSTM Agent
1.0
1.0
decoder depth=1
decoder_ depth=6
0.8
decoder depth=12 12
0.8
QA Accuracy
decoder_ aecoder_depth-22 depth-24
QA Accuracy
0.6
0.6
0.4
0.4
0.2
0.2
0.0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
1.6
08.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
leg
1e9
Figure Answer accuracy over training increasing QA decoder's depths. subplot shows results the SimCore agent and right subplot for the LSTM baseline. For SimCore, the QA accuracy increases with the decoder depth, up to 12 layers. For the LSTM agent, QA accuracy is not better than chance regardless of the capacity of the QA network.
A.3 ANSWERING ACCURACY DURING TRAINING FOR ALL QUESTIONS
The QA accuracy over training for all questions is shown in Figure
A.4 ENVIRONMENT
Our environment single L-shaped 3D room, procedurally populated with an assortment of ob- jects.
Actions and Observations. The environment is episodic, and runs at 30 frames per second. Each episode takes 30 seconds (or 900 steps). At each step, the environment provides the agent with two observations: a 96x72 RGB image with the first-person view of the agent and the text containing the question.
The agent interact with the environment by providing multiple simultaneous actions to control movement (forward/back, left/right,, looking (up/down, left/right), picking up and manipulating objects degrees of freedom: yaw, pitch, roll movement along the axis between agent and object).
Rewards. To allow training using cross-entropy, as described Section 4, the environment provides the ground-truth answer instead of the reward to the agent.
Object creation and placement. We generate between 2 and 20 objects, depending the task, with the type of the object, its color and size being uniformly sampled from the set described in Table 4.
15

---

Under review as a conference paper at ICLR 2020
Are there sa same number of <colorl> than <color2> oojects?
What is the color of the <shape>?
1.0
1.0
No
LSTM
SimCorc
only
0.9
0.0
CPCJA
A.Acuurrcy
0.0
02
0.4
0.6
0.8
1.0
1.2
1.4
1.6 Lef
0..0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
1.6
number
1e9
<shapel> than <shapez> objects?
How many <color> objects are there7
1.0
1.0
0.8
0.8
0.5
0.6
0.4
0.4
0.2
0.00
0.90
How many eshapee there?
there <shape- in the room?
Accurecy
0.6
0.4
0.4
0.2
0.2
0.000
0.2
0.4
0.6
0.8
1.0
12
1.4
1.6
0.0
0.2
0.4
0.6
0.8
1.0
1.2
144
1.6
1c5
100
What the color of the <shape> near the <shape>?
What is the <color> object near the <shape>?
1.0
1.0
0.8
0.6
00
02
0.4
0.6
0.8
1.0
1.2
1.4
1.6
0.9
0.2
0.0
0.8
1.0
1.2
1.4
1.8
steps
Leb
staps
109
Figure 7: QA accuracy over training for all questions and all models.
Objects will be placed in a random location and random orientation. For some tasks, we required some additional constraints for example, if the question is "What is the color of the cushion near the bed?", we need ensure only one cushion close to the bed. This was done by checking the constraints and regenerating the placement in case they were not satisfied.
A.5 RESULTS IN THE DEEPMIND LAB ENVIRONMENT
In order to check if our results are robust the choice of environment, we developed similar setup using the DeepMind Lab (Beattie et al., 2016) environment. consists a rectangular room that is populated with random selection of objects of different shapes and colors in each episode. There are 6 distinct objects in each room, selected from a pool of 20 objects and 9 different colors. We use similar exploration reward structure in earlier environment train the agents navigate
16

---

Under review as a conference paper at ICLR 2020
Attribute
Options
Object
basketball, cushion, carriage, train, grinder, candle, teddy, chair, scissors, stool, book, football, rubber duck, glass, toothpaste, arm chair, robot, hairdryer, cube block, bathtub, TV, plane, cuboid block, car, tv cabinet, plate, soap, rocket, dining table, pillar block, potted plant, boat, tennisball, tape dispenser, pencil, wash basin, vase, picture frame, bottle, bed, helicopter, napkin, table lamp, wardrobe, racket, keyboard, chest, bus, roof block, toilet
Color
aquamarine, blue, green, magenta, orange, purple, pink, red, white, yellow
Size
small, medium, large
Table 4: Randomization of objects in the Unity room. 50 different types, 10 different colors and 3 different
scales.
Body movement actions
Movement and grip actions
Object manipulation
NOOP
GRAB
GRAB SPIN OBJECT RIGHT
FORWARD
GRAB + MOVE_ MOVE_
FORWARD
GRAB + SPIN TITN_OBJECTLLER
MOVE BACKWARD
GRAB MOVE BACKWARD
GRAB SPIN OBJECT UP
MOVE RIGHT
GRAB MOVE RIGHT
GRAB SPIN OBJECT DOWN
MOVE_LEFT
GRAB MOVE_ BOWNE_BA
GRAB + SPIN_OBJECT OOBJECTFFORWRRRD FORWARD
LOOK RIGHT
GRAB LOOK RIGHT
GRAB SPIN OBJECT BACKWARD
LOOK LEFT
GRAB + LOOK LEFT
GRAB PUSH OBJECT AWAY
LOOK_UP
GRAB + +LOKKUP
GRAB + PULL-OBIECT_CLOST
LOOK DOWN
GRAB LOOK DOWN
Table 5: Environment action set.
and observe all objects. Finally, in each episode, we introduce a question of the form Whht the color of the <shape>>? where <'shape>> is replaced by the name of an object present in the room.
Figure 8b shows question-anse answering accuracies the DeepMind Lab environment. Consistent with the results presented in the main text, internal representations of the SimCore agent lead to the high- est answering accuracy while CPCIA and the vanilla L.TTM agent perform worse and similar each other. Crucially, for running experiments in DeepMind Lab, we did not change any hyperparameters from the experimental setup described the main paper. This demonstrates that our approach not specific to single environment and that it can be readily applied in variety of settings.
17

---

Under review as a conference paper at ICLR 2020
is the color of <shapes?
0.8
QA Accurary
0.2
0.0
1455
(b) QA accuracy for questions (What the color the <shape>?) in Deep- Mind Lab. Consistent with results in the main paper, internal representations of the SimCore agent lead to the highest accuracy while CPCIA and LSTM perform worse and similar to each other.
(a) DeepMind Lab environment (Beattie al., 2016): Rectangular-shaped room with 6 randomly selected objects out of pool of 20 different objects of different colors.
Figure
18


Reviews:
[{"review_id": "rklyq2NAFS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a framework to assess to which extent representations built by predictive models such as action-conditional CPC or SimCore contain sufficient information to answer questions about the environment they are trained/test on. The main idea is to train an independent LSTM (a Question-answer decoder) so that given the hidden state of the predictive model and a question about the environment, it is able to answer the question. \n\nThe authors give empirical evidence that the representations created by SimCore contain sufficient information for the LSTM to answer questions quite accurately while the representations created by CPC (or a vanilla LSTM) do not contain sufficient information. Based on the experimental results, the authors argue that the information encoded by SimCore contains detailed and compositional information about objects, properties and spatial relations from the physical environment.\n\nThe idea is clearly explained and seems sensible, the paper is well written, the execution is competent and the authors provide a sufficient amount of details so that reproducibility should be possible. As a result, I am positive, however, I think it would be best accepted as a workshop paper given that:\n\n- The experiment are only carried out on a single environment, however, their claims are rather general. To support such general claims, experiments on additional environments seem necessary.\n\n- While the idea is sensible, the study is quite narrow because it only compares three models.\n\n- While sensible, the methodological contribution is rather straightforward.\n\n- The take home is quite brief.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rJx8VFf0Kr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose question answering (QA) as a tool to investigate what agents learn about the world, i.e., how much about the world is encoded in their internal states. The authors argue that this is an intuitive method for humans and allows for arbitrary complexity. \nConcretely, they train agents on exploration of a 3D environment using reinforcement learning and then ask them a set of non-trivial questions. This includes unseen combinations of seen attributes (\"zero-shot\"), showing that, what the agents learn, is to some degree compositional. Importantly, agents are not trained to answer questions explicitly.\n\nThe authors investigate multiple agents and find that LSTM and CPC|A representations are no better than chance, SimCore's representations seem to be the best for the QA task, and there is still a big performance difference between SimCore and the upper bound \"No SG\".\n\nI think this paper is interesting and well done. I agree with the authors that QA is an intuitive probing tool, which can be used for similar agent analyses in the future.\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "r1ep1oO0YS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n########## Post-Rebuttal Summary ###########\nThe authors engaged actively in the rebuttal discussion and in the process we were able to concretize the motivation of the submission (as a result in increased my score). However, I think that the submission in its current form lacks experimental analysis of the proposed conditional probes, especially the trade-offs on the reliability of the representation analysis when performed with a conditional probe as well as a clear motivation for the need of a language interface. I can therefore not recommend the submission in its current form for acceptance.\n(for detailed discussions + suggested experiments please see rebuttal discussions)\n#######################################\n\n\n## Paper Summary\nThe paper tackles the problem of analyzing the information captured in the learned representation of a deep neural network. It proposes to replace the commonly used \"probing networks\" that try to directly infer the information from the learned representation, with a language interface in the form of a QA model which is trained post-hoc without propagating gradients into the learned representation. The authors argue that such a language interface provides a more natural interface for probing the information in a learned representation. The paper shows representation analysis results for the internal representations of agents trained on an exploratory task in a simple, simulated household environment similar to DM Lab. The authors conduct additional analysis on representations learned with auxiliary generative and contrastive objectives.\n\n## Strengths\n- the language of the paper is clear and easy to follow\n- the paper covers the related work well\n- the provided explanations help understand the content of the paper\n- the analysis of how the information captured in the agent's representation changes over the course of a trajectory is interesting (more such visualizations in the appendix would be nice!)\n\n## Weaknesses\n(1) the motivation for the proposed problem does not convince: why do I need to train a Q/A system to infer which components of the true state are captured by the learned representation? For each of the properties of the environment I could train a probing network (as is done e.g. in [1,2]) and would get much more precise answers; ground truth labels need to be available whether I train QA or probing networks. The authors provide two motivations for the proposed approach which both do not convince me:\n\t(a) QA \"provides an intuitive investigation tool for humans\": I cannot imagine a workflow in which researchers would prefer to ask questions to their model over a plot showing explicit regression accuracies aggregated across many data samples (which the conventional probing networks provide).\n\t(b) \"the space of questions is open-ended, [...] enabling comprehensive analysis of [an agent's] internal state\": for each new question type we need to provide a sufficiently large number of question-answer-pairs to train the QA system for this question type. we could therefore also train a probing network using the same labels and would get a better overview of whether the state information is captured in the representation. I fail to see how substituting probing networks with a language interface helps the investigation of the representation's properties.\n\t\n(2) the paper only provides minor novelty: the main proposal is to replace the probing networks, which were extensively used in prior work, with a natural language interface; a proposal that does not seem to provide a clear advantage (see above). The paper does not provide any further technical novelty.\n\t\n(3) it is possible that the analysis of the contrastive model could improve substantially with a different sampling scheme of the negative examples: maybe sampling from different sequences makes the task of discriminating too easy so that the model is not encouraged to learn a rich representation. It would be interesting to see whether the representation captures more information if a more standard contrastive objective is chosen that discriminates between future frames with different offset in the same sequence (see for example the objective in [3]).\n\n(4) it seems that both environment and chosen task will have high influence on the representation the agent can learn from the collected data. Therefore the fact that the authors evaluate their approach only on a single environment / task combination, both of which they introduce themselves, weakens any conclusions the authors draw from their experiments. It would help strengthen the message of the paper to apply their methodology to previously proposed environment / task combinations, for example in the AI2-THOR environment [4]\n\n\n[Novelty]: minor\n[technical novelty]: minor\n[Experimental Design]: not comprehensive\n[potential impact]: low\n\n\n#########################\n[overall recommendation]: Reject - In its current form the paper does not provide a convincing argument for why learning a language interface for probing a representation is better than learning the usual probing networks. Further there are some doubts on the setup of the contrastive objective and the paper lacks comprehensive evaluation on standard environments. Therefore I cannot recommend acceptance in its current form.\n[Confidence]: High\n\n\n[1] Neural Predictive Belief Representations, Guo et al., 2018\n[2] Shaping Belief States with Generative Environment Models for Reinforcement Learning, Gregor et al., 2019\n[3] Representation Learning with Contrastive Predictive Coding, Van den Oord et al., 2018\n[4] AI2-THOR: An Interactive 3D Environment for Visual AI, Kolve et al., 2017\n\n\n\n#### Final Rebuttal comment (to make it visible to the authors) ####\nI understand the authors point that every method for probing representations can potentially be flawed in that the probing mechanism might not be expressive enough to extract the information that is indeed present in the representation. The point I was trying to make in my previous response is, that in the case of conditional probes that try to solve multiple such \"probing tasks\" in parallel, such uncertainties accumulate because the probing mechanism might trade off performance on one task for performance on another (if solving all of them at once is too challenging). If the submission's key contribution is to make probes conditional this seems like a trade-off that should be experimentally investigated, as it is vital to practitioners how much they can trust their probing method to extract the relevant information if it is actually in the representation.\n\nOne possible experiment would be to train an unconditional probe per attribute (maybe on a subset of all possible attribute-object combinations) and then a conditional probe across all of them to show that the conditional probe's assessments of the representation agree with the ensemble of unconditional probes. In my previous response I additionally raised the point that even if it can be shown that conditional probes are reliable, the authors still need to provide arguments why that conditioning should work via a language interface, not for example a symbolic/one-hot interface. If the claim is that this allows for generalization, it should again be shown that this generalization does not increase the error of the probe substantially, such that meaningful conclusions about the representation are still possible.\n\nRegarding testing on more diverse environments: the questions these experiments are supposed to answer (i.e. how reliable are conditional probes) are inherently empirical, so verifying across diverse environments will make the analysis more conclusive.\n\nI agree with the author's point that training 7k unconditional probes would not be practical, probably the current approach would be to train an image decoder that then reconstructs a top-down view of the whole scene. This, however, would have the same problems as a conditional probe, i.e. the probing decoder could trade-off performance between reconstructing different parts of the image. Therefore, I agree with the authors that investigating conditional probes is an interesting direction, but the submission does not provide a comprehensive analysis of this question.\n\nOn a final note, I appreciate the continued, factual discussion with the authors and think that the refinement of the focus towards conditional vs unconditional probes is a step in the right direction. To acknowledge that I am raising my score from \"reject\" to \"weak reject\". Yet, I think that the work in its current form lacks experimental analysis of the proposed conditional probes. During the rebuttal discussion I highlighted concerns and proposed possible experiments. I cannot recommend acceptance of the submission in its current form, but I encourage the authors to incorporate the refined motivation and add more comprehensive experimental evaluation for a possible resubmission.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes question-answering as a general paradigm to decode and understand the representations that agents develop, with application to two recent approaches to predictive modeling. During rebuttal, some critical issues still exist, e.g., as Reviewer#3 pointed out, the submission in its current form lacks experimental analysis of the proposed conditional probes, especially the trade-offs on the reliability of the representation analysis when performed with a conditional probe as well as a clear motivation for the need of a language interface. The authors are encouraged to incorporate the refined motivation and add more comprehensive experimental evaluation for a possible resubmission.
Decision: Reject