Embodied Multimodal Multitask Learning
Authors: Devendra Singh Chaplot
Lisa Lee
Ruslan Salakhutdinov
Devi Parikh
Dhruv Batra
Venue:  (2020)
Keywords: Visual Grounding
Semantic Goal Navigation
Embodied Question Answering

Abstract: Visually-grounded embodied language learning models have recently shown to be effective at learning multiple multimodal tasks such as following navigational instructions and answering questions. In this paper, we address two key limitations of these models, (a) the inability to transfer the grounded knowledge across different tasks and (b) the inability to transfer to new words and concepts not seen during training using only a few examples. We propose a multitask model which facilitates knowledge transfer across tasks by disentangling the knowledge of words and visual attributes in the intermediate representations. We create scenarios and datasets to quantify cross-task knowledge transfer and show that the proposed model outperforms a range of baselines in simulated 3D environments. We also show that this disentanglement of representations makes our model modular and interpretable which allows for transfer to instructions containing new concepts.

Under review as a conference paper at ICLR 2020
EMBODIED MULTIMODAL MULTITASK LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Visually- embodied language learning models have recently shown be effective at learning multiple multimodal tasks such as following navigational instructions and answering questions. In this paper, address two key limitations of these models, (a) the inability to transfer the grounded knowledge across different tasks and (b) the inability transfer new words and concepts not seen during training using only a few examples. We propose a multitask model which facilitates knowledge transfer across tasks by disentangling the knowledge of words and visual attributes in the intermediate representations. We create scenarios and datasets to quantify cross-task knowledge transfer and show that the proposed model outperforms a range of baselines in simulated 3D environments. We also show that this disentanglement of representations makes our model modular and interpretable which allows for transfer to instructions containing new concepts
INTRODUCTION
Humans learn language by interacting with dynamic perceptual environment, grounding words into visual entities and motor actions (Smith and Gasser 20055 Barsalou 2008). In recent years, there has been an increased focus on training embodied agents capable of grrounddd language learning. These include multimodal tasks involving one-way communication, such as mapping navigational instructions to actions (MacMahon et al.) 2006) Chen and Mooney| 201T Artzi and Zettlemoyer 2013, Mei et al. 2016 Misra et al., 2018 and tasks involving two- way communication such as embodied question answering (Gordon et al. 2018 Das et al.] 2018)) and embodied dialogue (de Vries) et al. 2018). Other studies have shown that grounded semantic goal navigation agents can be effective at exploiting the compositionality of language to generalize to unseen instructions with an unseen composition of semantic attributes (Hermann et al. 2017 Chaplot et al. 2018). unseen
composition of steps in a multi-step instruction (Oh et al. 2017).
However, current grounded language learning models have certain limitations. Firstly, these models are typically trained only for a single multimodal task and lack the ability to transfer grounded knowledge of "concepts 7 across tasks. For example, if an agent learns follow the instruction 'Go to the red torch' and answer the question 'What color is the pillar?', then ideally it should also understand 'Go the red pillar' and 'What color is the torch?' without additional training. Training multitask grounded-langusee
models can also improve training sample efficiency, as these multimodal tasks share many common learning challenges including perception, grounding, and navigation.
The second limitation is the inability of trained models to quickly transfer to tasks involving unseen concepts. For example, consider a houschold cestruction-oolloinng robot trained on an existing set of objects. We would like the robot to follow instructions involving new object 'lamp' that has been added to the house. Existing models would need be trained with the new object, which typically requires thousands of samples and can also lead to catastrophic forgetting of known objects. Even if the models were given some labeled samples to detect the new objects, they would require additional training to learn to combine existing grounded knowledge with the new concept (e.g., 'blue lamp' if "blue' is already known). In this paper, we tram à multimodal multitask learning model for two tasks: Semantic Goal Navigation, where the agent is given a language instruction to navigate to a goal location, and Embodied Question Answering. where the agent is asked a question and it can navigate in the environment to gather information to answer the question (see Figure (5). We make the following contributions this paper:
SSee https::///// /se.es.tssssee//ee google. com/ / / ew //w//emmm eemmmmmmmmmm/ iillll--lll icl r2020 for demo videos.
"In this paper, we refer to the knowledge of a word and its grounding in the visual world as the knowledge of concept (for example, concept 'torch' involves word 'torch' and how torch looks visually).

---

Under review as a conference paper at ICLR 2020
Table Table showing training and test sets both the tasks, Semantic Goal Navigation (SGN) and Embodied Question Answering (FQA). The test set consists unseen instructions and questions. The dataset evaluates model for cross-task knowledge transfer hetween the embodied multimodal tasks, SGN and EQA.
Task
Train Set
Test Sct
SGN
Instructions containing 'red' & 'pillar': 'Go to the blue object'
Instructions containing 'red" 'pillar":
'Go to the red pillar'
'Go to the torch'
'Go to the tall red object'
EQA
Questions not containing 'blue' & 'torch":
Questions containing "blue' or 'torch':
'Which object is red in color?'
'Which object is blue in color?'
'What color the tall pillar?
'What color the torch?'
First, we define a cross-task knowledge transfer evaluation cri- terion to test the ability of multimodal multi-task models to transfer knowledge of concepts across tasks. We show that several prior single-task models, when trained on both tasks, fail to achieve cross-task knowledge transfer. This is because the visual grounding of words is often implicitly learned as a by-product of end-to-end training of the underlying task, which leads to the entanglement of knowledge of concepts in the learnt representations. We propose novel Dual-Altentic model which learns task-invariant disentangled visual and tex- tual representations and explicitly aligns them with each other. We create datasets and simulation scenarios for testing cross- task knowledge transfer and show an absolute improvement of 43-61% % on instructions and 5-26% for questions over baselines.
Instruction: Go to the red pillar Question:
What color is the torch?
Figure I: Examples of embodicd multi- modal tasks, following instructions and answering questions.
Second, the disentanglement and explicit alignment of representations makes our model modular and interpretable. We show that this allows us transfer the model to handle instructions involving unseen concepts by incorporating the output of object detectors. We also show that model is able to combine the knowledge of existing concepts with a new concept without any additional policy training.
Finally, we show that the modularity and interpretability of our model also allow us to use trainable neural modules (Andreas al., 2016) to handle relational tasks involving negation and spatial relationships and also tackle relational instructions involving new concepts.
2 RELATED WORK
A lot of early work on visual instruction-folloiingg in the embodied space such as in robotics
applications Tellex et al. 2011 Matuszek et al. 2012 Hemachandra al., 2015 Misra al. 2016) and on mapping natural language instructions to actions (MacMahon et 2006 Chen and Mooney 2011 Artzi and Zettlemoyer 2013 Mei et al.
[he6 symbolic representations. Recently, there have been efforts on learning to follow navigational instructions from raw visual observations (Anderson et al.) 2018 Misra et al.) 2018 Chen et al.) 2019 Blukis al.)
2016) required hand-designn
2018). Oh et al. (2017); Chaplot et al. (2188): Hermann et al. (201)) 7) study the language learning aspect of instruction-followe agents are able to learn spatial and logical reasoning and exploit the compositionality of language to generalize to new instructions.
in more controlled setting, and show that grounded language learning
Question Answering in the embodied space has been comparatively less-studied with recent work studying QA which requires exploration. navigation, and interaction with objects in the environ- ment (Gordon et al.] 2018 Das et al. 2018). In contrast to the prior work which tackles a single grounding task, we tackle both instruction followiinn and question answering in the embodied space and study the ability to transfer the knowledge of concepts across the tasks and tackle instructions with new concepts.
In addition to the above, there is a large body of work on multimodal learning in static settings which do not involve navigation or reinforcement learning. Some relevant works which use attention mechanisms similar to the ones used in our proposed model include Perez et al. (2018): Fukui et al.) (2016); Xu and Saenko (2016); Hudson and Manning ( Answering and Zhao al. (2018) for grounding audio vision.
Gupta et al. (211)) for Visual Question
2

---

Under review as a conference paper at ICLR 2020
Text y
Ba Words
chatcoloris is the torch? ?
Time step [1]
Fitow
Policy prediction
X5
State rep.
[WW]
Policy Module
GRU Linear
Asent Sentence (Vp:
Dual-Attention
Aóny
Answer
Prediction [K]
v(G. T)
Conv
Task Indicator(1)
Value prediction
Imagelz
[VxH*W)
Figure 3: Overview of our proposed architecture, described in detail in Section
3 PROBLEM FORMULATION
Consider an autonomous agent interacting with an episodic environment as shown in Figure I At the beginning of each episode, the agent receives textual input T' specifying task. T could be an instruction to navigate to a target object or a question querying some visual detail of objects in the environment. At each time step the agent observes state 8, (I, T) where I is the first-person (egocentric) view of the environment, and takes an action ag, which could be a navigational action answer action. The agent's 's objective is to learn policy x (() which leads to successful completion of the task specified by T.
Environments. We adapt the ViZDoom-base (Kempka et al. 2016) language grounding environ- ment proposed by [Chaplot et al. (2018) for embodied multitask learning. It consists of a single room with 5 objects. The objects are randomized in each episode based on the textual input. We use two difficulty settings for the Doom domain as shown in Figure Easy: The agent is spawned at fixed location. The candidate objects are spawned at five fixed locations along a single horizontal line in the field of view of the agent. Hard: The candidate objects and the agent are spawned at random locations and the objects may or may not be in the agent's 's field of view in the initial configuration. The agent must explore the map to view all objects. The agent can take 4 actions: 3 navigational actions (forward, left, right) and answer action. When the agent takes the answer action, the answer with the maximum probability in the output answer distribution is used.
Datasets. We use the set 70 instructions from Chaplot et al. (2018) and create dataset of 29 questions using the same set of objects and attributes. These datasets include instructions and questions about object types, colors, relative sizes (tall/short) and superlative sizes (smallest/largest). We create train-test splits for both instructions and questions datasets explicitly test a multitask model's ability to transfer the knowledge of concepts across different tasks. Each instruction in the test set contains a word that is never seen in any instruction in the training set but is seen in some questions in the training set. Similarly, each question in the test set contains word never seen in any training set question. Table T illlusrrate the train- test split of instructions and questions used in our experiments] unseen words can be present in the answer.
Figure 2: Sample starting states in Easy (top) and Hard (bottom) settings. Note that for the EQA trainset,
4 PROPOSED METHOD
In this section, we describe our proposed architecture (illustrated in Figure 3). At the start of each episode, the agent receives a textual input T (an instruction or question) specifying the task. At each time step I, the agent observes an egocentric image L, which is passed through convolutional neural network (LeCun et al. 1995) with ReLU activations (Glorot et al.] 2011) to produce the image representation x = f [I,; Boom) E xV where dconv denotes the parameters of the convolutional network, V is the number of feature maps in the convolutional network output which is by design set equal to the vocabulary size (of the union of the instructions and questions training sets), and H and W are the height and width of each feature map. We use two representations for the textual input T: (1) the bag-of-words representation denoted by TBoW E 0, V and (2) a sentence representation Z'gent = [TT; 0sent) E RVV which is computed by passing the words in T through a Gated Recurrent Unit (GRU) (Cho et al., 2014) network followed by a linear layer. Here, 0 denotes the parameters of the GRU network and the linear layer with ReLU activations. Next, the
More details about the datasets and the environments are deferred to the supplementary material.
3

---

Under review as a conference paper at ICLR 2020
Qual-Attention unit SDA combines the image representation with the text representations get the complete state representation 2y and answer prediction ZAnsi
z5, x Ans [2), 2BoW, Tsent)
(1)
Finally, 2s and Ans- along with a time step embedding and a task indicator variable (for whether the task SGN or EQA), are passed to the policy module to produce an action.
4.1 TUAL-ATTENTION UNIT
The Dual-Attention unit uses two types of attention mechanisms, Attent-Atentionn fGa and Spatial- Attention SSA. to align representations in different modalities and tasks.
Gated-Attention (GA). The GA unit (Chaplot et al.) 2018) attends to the different channels in the image representation based on the text representation. For example, if the textual input is the instruction 'Go the red pillar', then the GA unit can learn attend channels which detect red things and pillars. Specifically, the GA unit takes as input 3-dimensional tensor image representation yI E x and a text representation yT E Rd, and outputs a 3-dimensional tensor E xxxx Note that the dimension of yr is equal to the number of feature maps and the size of the first dimension of yr. In the GA unit, each element of yr is expanded a H x W matrix, resulting a 3-dimensional tensor M yr E RR xx W whose (1. j. 1-)th element is given by May [ k] VT [i]. This matrix is
multiplied element-wise with the image representation: z = SGA (3I,, yT) = Myr C yr, where @
denotes the Hadamard product (Horn 1990p.
Spatial- Aatall-Atteniioon (SA). We propose an SA unit which is analogous to the Gated-Attention unit except that it attends to different pixels in the image representation rather than the channels. For example, if the textual input is the question 'Which object is blue in color?', then we would like to spatially attend to the parts of the image which contain blue object in order recognize the type of the blue object. The Spatial-Attention unit takes as input a 3-dimensional tensor image representation y7 dx and 2-dimensional spatial attention map ys IR H W and outputs a tensor Z E RdxH> X Note that the height and width of the spatial attention map are equal to the height and width of the image representation. In the spatial-attention unit, each element of the spatial attention map is expanded to a d dimensional vector. This again results in a 3 3-dimensional tensor Mys E Rdx WxW whose (i. j, k)th element is given by: Mye J.. j. = ys [j, k). Just like in the Gated-Atter Attention unit, this matrix is multiplied element-wiss with the image representation: (y), us) Mys ② yr-
Dual-Attention. We now describe the operations in the Dual-Attention unit shown in Figure 4 as well as motivate the intuitions behind each operation. Given Ty, JBo. and Isents the Dual-Attention unit first computes a Gated-Attention over v using TBow:
fcx (11, (Bow) V
FGAI
Intuitively, this GA unit grounds each word in the vocabulary with feature map in the image representation. A particular feature map is activated if and only if the corresponding word occurs in the textual input. Thus, the feature maps in the convolutional output learn to detect different objects and attributes, and words in the textual input specify which objects and attributes are relevant to using BoW representation attends feature maps detecting
the current task. The corresponding objects and attributes, and masks all other feature maps. We use the BoW representation for the first GA unit as it explicitly aligns the words in textual input irrespective of whether it is a question or an instruction.
Next, the output of the GA unit AGAI is converted to a spatial attention map by summing over all channels followed by softmax over H x W elements:
Zspat Σ =GAI E R "xw
where the softmax oz); (/p(2)//>> Ex exp(2=) ensures that the attention map is spatially normal- ized. Summation of ÆGAI along the depth dimension gives a spatial attention map which has high activations at spatial locations where relevant objects attributes are detected. ReLU activations in the convolutional feature maps makes all elements positive, ensuring that the summation aggregates the activations of relevant feature maps.
4

---

Under review as a conference paper at ICLR 2020
Reld
Answer predictions
Senttocc meberidnn
GRU 5
Bap-df-Wlerd
Witch tall eot i aode?
Culest-Adentionn
Glated/ Atttec-ttrrdonn
Text T
Atoiion
KAna
feA
XD42
ÍGA
Conr
TGA1
Mles Ocam))
Xyent
Image
Xy
Xspu2
Imsgge rep.
Spulii-Alunlion
fsA
Figure 4: Architecture of the Dual-Attention unit with example intermediate representations and operations.
sppt and σ are then passed through SA unit:
ZSA fSA (21, Zspat) E R V
(4)
The SA unit outputs all attributes present at the locations where relevant objects and attributes are detected. This is especially helpful for question answering, where single Aateed--ttttiion may not be sufficient. For example, if the textual input is 'Which color is the pillar?', then the model needs to attend not only to feature maps detecting pillars (done by the attributes at the spatial locations where pillars are seen in order to predict their color.
but also to other
JSA is then passed through another GA unit with the sentence-level text representation:
TGA2 SGA (ISA, Tsent) E R V xHxW
(5)
This second GA unit enables the model to attend to different types of attributes based on the question. For instance, if the question is asking about the color ("Which color is the pillar?"), then the model needs to attend to the feature maps corresponding to colors; if the question is asking about the object type ('Which object is green in color?') ), then the model needs to attend to the feature maps corresponding object types. The sentence embedding Zeent can learn to attend to multiple channels based on the textual input and mask the rest.
Next, the output transformed to answer prediction by again doing summation and softmax but this time summing over the height and width instead of the channels:
H.W Σ j.k
JGA2 IGA2
k]
E RR
(6)
Summation of IGA2 along each feature map aggregates the activations for relevant attributes spatially, Again, ReLU activations for sentence embedding ensure aggregation of activations for each attribute or word. The answer space is identical to the textual input space IR
Finally, the Dutl-Attention unit SDA outputs the answer prediction JAa and the flattened spatial attention map æs vec(zs)at). where vec(-) denotes the flattening operation.
Policy Module. The policy module takes as input the state representation zy from the -nal-Attention unit, time step embedding i, and a task indicator variable (for whether the task is SGN EQA). The inputs are concatenated then passed through a linear layer, then a recurrent GRU layer. then linear layers estimate the policy function π (a; 4, T) and the value function V(I,T),
All above operations are differentiable, making the entire architecture trainable end-to-end. Note that all attention mechanisms in the Dual-Attention unit only modulate the input image representation, ie., mask or amplify specific feature maps or pixels. This ensures that there is an explicit alignment between the words in the textual input, the feature maps in the image representation, and the words in the answer space. This forces the convolutional network to encode all the information required with respect certain word in the corresponding output channel. For example, predict 'red' as the answer, the model must detect red objects in the corresponding feature map. This explicit iiss-invariant& alignment between convolutional feature maps and words the input and answer space facilitates grounding and allows for cross-task knowledge transfer. As shown in the results later, this also makes our model modular and allows easy addition of objects and attributes to a trained model.
5

---

Under review as a conference paper at ICLR 2020
Table 2: Accuracy of all models for both Easy & Hard difficulties. 'MT' stands for multi-task.
Easy
Hard
No Aux
Aux
No Aux
Aux
Train MT
Test
Train MT
Test
Train MT
Test
Train MT
Test
Model
SGN
EQA
SGN
EQA
SGN
EQA
SGN
EQA
Text only
0.33
0.20
0.33
0.31
0.20
0.33
0.36
0.20
0.33
0.36
0.20
0.33
Image only
0.41
0.20
0.09
0,40
0.21
0.08
0.36
0.16
0.08
0.36
0.15
0.08
Concat
0.97
0.33
0.21
0.99
0.31
0.19
0.57
0.20
0.26
0.71
0.39
0.22
GA
0.97
0.27
0.18
0.99
0.35
0.24
0.44
0.18
0.11
0.71
0.22
0.24
FILM
0.97
0.24
0.11
0.99
0.34
0.12
0.52
0.12
0.03
0.55
0.25
0.15
PACMAN
0.66
0.26
0.12
0.79
0.33
0.10
0.56
0.29
0.33
0.54
0.11
0.27
Dual-Atiention
0.99
0.86
0.53
0.99
0.96
0.58
0.85
0.86
0.38
0.90
0.82
0.59
4.2
OPTIMIZATION
The entire model is trained to predict both navigational actions and answers jointly. The policy is trained using Proximal Policy Optimization (PPO) (Schulman et al. 2017). For training the answer predictions, we use a supervised cross-entropy loss. Both types of losses have common parameters as the answer prediction is essentially an intermediate representation for the policy.
Auxiliary Task. As mentioned earlier, the feature maps in the convolutional output are expected to detect different objects and attributes. Consequently, cross-entropy loss) to detect the object or attribute in the convolutional output channels corresponding to the word in the bag-of-wordh representation. Rather than doing fine-grained object detection, keep the size of the auxiliary predictions the same as the convolutional output to avoid an increase in the number of parameters, and maintain the explicit alignment with the words. Consequently, auxiliary labels are (V x H x --)dimesiional tensors, where each of the V channels corresponds word the vocabulary, and each element in a channel is 1 if the corresponding object or attribute is present in the corresponding frame.
add spatial auxiliary task (trained with
the convolutional feature maps
EXPERIMENTS & RESULTS
Jointly learning semantic goal navigation and embodied question answering essentially involves fusion of textual and visual modalities. While prior methods are designed for a single task, we adapt several baselines for our environment and tasks by using their multimodal fusion techniques. We use two naive baselines, Image only and Text only; two baselines based prior semantic goal navigation models, Concat (used by Hermann et al.] (2017); Misra et al. (2017)) and Gated- Attention (GA) (Chaplot et al.) 2018); and two baselines based Question Answering models, FiLM (Perez et al. (018)) and PACMAN (Das et al.] (2018). For fair comparison, we replace the proposed Dual-Attention unit with multimodal fusion techniques in the baselines and keep everything else identical to the proposed mode !F
5.1 RESULTS
We train all models for 10 million frames in the Easy setting and 50 million frames in the Hard setting. We use a +1 reward for reaching the correct object in SGN episodes and predicting the correct answer in EQA episodes. We use small negative reward of -0.001 per time step to encourage shorter paths to the target and answering questions as soon as possible. We also use bistance-based reward shaping for SGN episodes, where the agent receives a small reward proportional to the decrease in distance to the target. In the next subsection, we evaluate the performance of the proposed model without the reward shaping. SGN episodes end when the agent reaches any object, and EQA episodes end when the agent predicts any answer. All episodes have a maximum length of 210 time steps. We train all models with and without the auxiliary tasks using identical reward functions.
All models are trained jointly for both the tasks and tested on each task separately In Table 2 we report the performance of all models for both Easy and Hard settings The Dual-Attention (DA) model and many baselines achieve 99% accuracy during training in the Easy-Aux setting; however,
See the supplementary material for implementation details of all baselines.
See https::// /ssttsstss....e../.eeee.eeeeesss gwwgle..oooooooooooooooooooo.o com/ view/ emmn- 1elll-icr1222 c1x2020/ for visualization videos. See the supplementary material for training performance for all models.

---

Under review as a conference paper at ICLR 2020
Table Aceuracy all the ablation models on SGN and EQA test sets for the Easy setting.
Table 4: The performance of the trained policy appended with object detectors on instructions containing unseen words ('red' and 'pillar").
Aux SGN
Aux
Instruction
Acc
Model
EQA
SGN
EQA
Go to the <color name> pillar.
1.00
w/o S.A
0.20
0.16
0.20
0.15
Go to the red <object name>
1.00
w/o GA1
0.14
0.25
0.16
0.38
Go to the tall/short red pillar
0.99
w/o GA2
0.80
0.33
0.97
0.15
Go the red pillar
0.99
w/o Task Indicator
0.79
0.47
0.96
0.56
w/o Reward Shaping
0.82
0.49
0.93
0.51
Go to the <color name> object that is not pillar
0.91
DA Single-Task
0.63
0.31
0.91
0.34
Go to the <object_ name> that left of the object
0.96
DA Multi-Task
0.86
0.53
0.96
0.58
Go to the red object that is right of the pillar
0.95
the test performance of all the baselines considerably lower than that of the DA model (see Table 2 (left)). Performance of all the baselines is worse than the "Text only' model on the EQA test set, although the training accuracy higher. This indicates that baselines tend to overfit on the training set and fail to generalize to questions which contain words never seen in training questions. As expected, using spatial auxiliary tasks improves performance all models. Even without auxiliary tasks, the DA model achieves test accuracy 86% (SGN) and 53% (EQA), compared to the best baseline performance of 33% (SGN& & EQA).
For the Hard setting, the DA model achieves higher training (90% vs 71% with Aux) as well as test performance (82% vs. 39% for SGN, 59% vs. 33% for EQA with Aux) than the baselines (see Table [2] (right)). These results confirm the hypothesis that prior models, which are designed for single task, lack the ability to align the words in both the tasks and transfer knowledge across tasks.
Lower test accuracy EQA (vs. SGN) for most models (Table 2 indicates that EQA is more challenging as it involves alignment between not just input textual and visual representations but also with the answer space.
5.2 ABLATION TESTS
We perform series of ablation tests in order to analyze the contribution of each component in the Dual-Attention unit: without Spatial-Attention (w/o SA), without the first Gated-Attention with B.o (w/o GA1), and without the second Gated-Attention with Tvent (w/o GA2). We also try removing the task indicator variable (w/o Indicator Variable), removing reward shaping (w/o Reward Shaping), and training the proposed model on single task, SGN or EQA (DA Single-Task).
In Table 3 we report the test performance of all ablation models. The results indicate that SA and GA1 contribute the most to the performance of the full Dual- Attenittention model. GA2 is critical for performance on EQA but SGN (see Table 3). This is expected as GA2 is designed to attend to different objects and attributes based on the question and is used mainly for answer prediction. It is not critical for SGN as the spatial attention map consists of locations of relevant objects, which is sufficient for navigating the correct object
We observe that reward shaping and indicator variable help with learning speed, but have little effect on the final performance (see Table 3. DA models trained only on single tasks work well on SGN, especially with auxiliary tasks, because the auxiliary task for single task models includes object detection labels corresponding the words in the test set. This highlights key advantage of our model's 's modular and interpretable design: the model can be used for transferring the policy to new objects and attributes without fine-tuning as later discussed the Section 5.4
5.3 HANDLING RELATIONAL TASKS
The instructions and questions considered so far contained single target object. We propose simple extension to our model to handle relational tasks, such as 'Which object is to the left of the torch?', where the agent required to attend to the region left of the torch, not the torch itself.
See the supplementary material for visualization of convolutional feature outputs, spatial attention map. sentence representation of the textual input and answer predictions
7

---

Under review as a conference paper at ICLR 2020
We consider three relational operations: 'left of", 'right of" and 'not'. We add questions and instruc- tions with all objects and attributes using these rela- tional operations to the existing dataset and perform experiments in the Easy-Aux setting. We assume that the knowledge of relational words, and the words they modify, are given. We train separate module corresponding to each relational operation, and apply it the convolutional output of the words that are modified. For example, for the above question, we apply the module for relation 'left of" to the convo- lutional output channel corresponding to the word 'torch'. Fach relational module is trainable convolutional network which preserves the size of the input. The rest the operations are identical to the Dual-Attention Unit. The relational modules are learned end-to-end without any additional supervision.
green
blue
Seycard
not green
of blue
oight_ keycard
Figure 5: Outputs for relations 'not', 'left of", and "right of" learned by the relational modules.
In Figure show convolutional outputs of the relational modules learned by model. While the original DA model achieves test performance of 0.48 (SGN) and 0.44 (EQA), this simple extension achieves 0.97 (SGN) and 0.64 (EQA).
5.4 TRANSFER TO NEW CONCEPTS
Suppose that the user wants the agent to follow instructions about new object such 'pillar' or a new attribute such as 'red' which the agent has never seen during training. Prior SGN models (Chaplot et al. 2018 Hermann al. 2017: Yu et al., 2018) cannot handle instructions containing new concept. In contrast, our model can be used for handling such instructions by training an object detector for each we train the DA model in the Easy setting on the training set for only instructions. We use auxiliary tasks but only for words in the vocabulary of the instructions training set. After training the policy, we test the agents on instructions containing test concept words 'red' and 'pillar', which the agent has never seen in textual input during training and never received any supervision about how this attribute or object looks visually.
concept and appending it to the image representation æ7. In order to test this,
For transferring the policy, we assume access to two object detectors for 'red' and 'pillar' separately, We resize the object detections to the size of a feature map in the image representation (H W) and append them as channels to the image representation. We also append the words 'red' and 'pillar" to the bag-of-words representations in the same order such that they are aligned with the appended feature maps. We randomly initialize the embeddings of the new words for computing the sentence embedding.
The results in Table 4show that this policy generalizes well to different types of instructions with unseen concepts, including: combining knowledge of existing attributes with knowledge of existing objects with new attribute; and composing a new attribute with a new object. The results shown in the lower part of Table indicate that the model also generalizes well to relational instructions containing new concepts. This means that given an object detector for a new object 'pillar', the model can (without any additional training) detect and differentiate between green and blue pillars, or between tall and short pillars; and understand left of/right of pillar, or the negation of pillar. The model can also combine 'pillar' with another new attribute 'red' to detect red pillars and understand relational instructions involving both red objects and pillars. This suggests that a trained policy can be scaled more objects provided the complexity navigation remains consistent.
object, or
CONCLUSION
We proposed a Dual-Attention model for Assuall-groundee multitask learning which uses Gated- and Spatial-Attention to disentangle attributes in feature representations and align them with the answer space. We show that the proposed model able to transfer the knowledge of concepts across tasks and outperforms the baselines both Semantic Goal Navigation and Embodied Question Answering by a considerable margin. We showed that disentangled and interpretablew representations make our model modular and allow for easy addition of new objects attributes to trained model. For future work, the model can potentially be extended to transferring knowledge across different domains by using modular interpretable representations of objects which are Nomain-invariant.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. 2018. Vision-and-language navigation: Interpreting disually-grunnde navigation instructions in real environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3674-3683
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein, 2016. Neural module networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 39-48.
Yoav Artzi and Luke Zettlemoyer. 2013. Weakly supervised learning of semantic parsers for mapping instructions to actions. Transactions of the Association for Computational Linguistics, 1:49-62.
Lawrence W Barsalou. 2008, Grounded cognition. Annu. Rev. Psychol., 59:617-645.
Valts Blukis, Dipendra Misra, Ross A Knepper, and Yoav Artzi. 2018. Mapping navigation instruc- tions to continuous control actions with position-visitation prediction. Proceedings of The 2nd Conference on Robot Learning.
Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pasumarthi, Dheeraj Rajagopal, and Ruslan Salakhutdinov. 2018. Gated-attention architectures for task-oriented language grounding. AAAI.
David L. Chen and Raymond J Mooney. 2011. Learning to interpret natural language navigation instructions from observations. In Twenty-Fifth AAAI Conference on Artificial Intelligence.
Howard Chen, Alane Shur, Dipendra Misra, Noah Snavely, and Yoav Artzi 2019. Touchdown: Natural language navigation and spatial reasoning in visual street environments. In CVPR.
Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the proper- ties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:14099 1259.
Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. 2018 8. Embodied question answering. In CVPR.
Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. 2016. Multimodal compact bilinear pooling for visual question answering and visual grounding. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 315-323.
Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi. 2018 Iqa: Visual question answering in interactive environments. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 4089-4098.
Tanmay Gupta, Kevin J Shih, Saurabh Singh, Derek Hoiem, Kevin Shih, Arun Mallya, Wei Di, Vignesh Jagadeesh, Robinson Piramuthu, K Shih, et al. 2017. Aligned image-word representations improve inductive transfer across lasion-language tasks. In ICCV, pages 4223-4232.
Sachithra Hemachandra, Felix Duvallet, Thomas M Howard, Nicholas Roy, Anthony Stentz, and Matthew R Walter. 2015. Learning models for following natural language directions in unknown environments. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 1608-5615. IEEE.
Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojtek Czarnecki, Max Jaderberg, Denis Teplyashin, et al. 2017. Grounded language learning in a simulated 3d world. arXiv preprint arXiv:1706.066 770.00651.
Roger A Horn. 1990. The hadamard product. In Proc. Symp. Appl. Math, volume 40, pages 87-169.
Drew Hudson and Christopher D Manning, 2018. Compositional attention networks for machine reasoning. In ICL.R

---

Under review as a conference paper at ICLR 2020
Michat Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojeiech Jaškowski. 2016. Vizdoom: A doom-based ai research platform for visual reinforcement learning. In 2016 IEEE Conference Computational Intelligence and Games (CIG), pages 1-8. IEEE
Yann LeCun, Yoshua Bengio, al. 1995. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 3361((0):1995.
Matt MacMaaon, Brian Stankiewicz, and Benjamin Kuipers. 2006. Walk the talk: Connecting language, knowledge, and action in route instructions. Def. 2(6):4.
Cynthia Matuszek, Nicholas FitzGerald, Luke Zettlemoyer, Liefeng Bo, and Dieter Fox. 2012. A joint model of language and perception for grounded attribute learning. arXiv preprint arXiv: 1206.6423.
Hongyuan Mei, Mohit Bansal, and Matthew R Walter. 2016. Listen, attend, and walk: Neural mapping of navigational instructions to action sequences. In Thirtieth AAAI Conference on Artificial Intelligence.
Dipendra Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, and Yoav Artzi. 2018. Mapping instructions to actions in 3d environments with visual goal prediction. In Pro- ceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2667-2678.
Dipendra Misra, John Langford, and Yoav Artzi 2017. Mapping instructions and visual observations to actions with reinforcement learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1004-1015.
Dipendra K Misra, Jaeyong Sung, Kevin Lee, and Ashutosh Saxena. 2016. Tell me dave: Context- sensitive grounding of natural language to manipulation instructions. The International Journal of Robotics Research, 35(1-3):281-30
Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. 2017. Zero-shot task generalization with multi-task deep reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 2661-2670. JMLR. org.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. 2018. Film:
Visual reasoning with a general conditioning layer. In AAAI.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347.
Linda Smith and Michael Gasser. 2005, The development of embodied cognition: Six lessons from babies. Artificial life. 11(1-2):13-29,
Stefanie A Tellex, Thomas Fleming Kollar, Steven R Dickerson, Matthew R Walter, Ashis Banerjee, Seth Teller, and Nicholas Roy. 2011. Understanding natural language commands for robotic navigation and mobile manipulation.
Harm de Vries, Kurt Shuster, Dhruv Batra, Devi Parikh, Jason Weston, and Douwe Kiela. 2018. Talk the walk: Navigating new york city through grounded dialogue. arXiv preprint arXiv: 1007103070000
Huijuan Xu and Kate Saenko. 2016. Ask, attend and answer: Exploring question-guides spatial attention for visual question answering. In European Conference on Computer Vision, pages 451-466. Springer.
Haonan Yu, Xiaochen Lian, Haichao Zhang, and Wei Xu. 2018. Guided feature transformation (gft): A neural language grounding module for embodied agents. In Conference on Robot Learning, pages 81-98.
Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. 2018 The sound of pixels. In Proceedings of the European Conference on Computer Vision (ECCV), pages 570-586.
10

---

Under review as a conference paper at ICLR 2020
A VISUALIZATIONS OF MODEL COMPONENTS
Figure 6 shows an example indicating that the sentence-level embedding for the question attends to relevant words. also shows visualization of spatial-attention maps and answer predictions for each frame in an example EQA episode. The spatial attention map shows that the model attends to relevant objects and attributes based on the question. The answer predictions change as the agent views more objects. These visualizations show that the textual and visual representations are aligned with each other, as well as with the answer space, as expected. In Figure 7 we visualize the convolutional network outputs corresponding to 7 words for the same frame for both Aux and No Aux models. As expected, the Aux model predictions are very close to the auxiliary task labels. More interestingly, the convolutional outputs of the No Aux model show that words and Dbjects/properiec in the images have been properly aligned even when the model is not trained with any auxiliary task labels.
borth
sho it
fornest
Which is the smallest
T =
blue object?
1
XSA
Ans
Action
Clie
Tum Right
the
torch
vyllooo
objjct
tall
short
skullley
color
which
ceed
Tum Right
Cojjett
gent
chulllon
seccora
Biue
Eorch
Tum Right
lect
Bia
turdis
yellow
de jent
largent
Answer: skullke
Figure 6: Spatial Attention and Answer Prediction Visualizations. An example EQA episode with the question "Which is the smallest blue object?". The sentence embedding of the question is shown on the top (Teent). As expected, the embedding attends object type words ('torch', 'pillar', 'skullkey'. etc.) the question asking about object type ('Which object'). The rows show increasing time steps and columns show the input frame, the input frame overlaid with the spatial attention map, the predicted answer distribution, and the action at each time step. As the agent is turning. the spatial attention attends to small and blue objects. Time steps 1. 2: The model is attending to the yellow skullkey but the probability of the answer is not sufficiently high, likely because the skullkey is not blue. Time step 3: The model cannot the skullkey anymore attends the armor which is next smallest object. Consequently, the answer prediction also predicts armor, but the policy decides not to answer due to low probability. Time step As the agent turns more, it observes and attends to the blue skullkey. The answer prediction for 'skullkey' has high probability because it is small and blue, so the policy decides to answer the question.

---

Under review as a conference paper at ICLR 2020
Convolutional output channel corresponding to each word
Word
Aux label
DA No Aux model
DA Aux model
Frame
Pillar
Red
Skullkey
Smallest
Tall
Torch
Figure 7: Visualizations of convolutional output channels. We visualize the convolutional channels corre- sponding to words (one in row) for the same frame (shown the rightmost column). The first column shows the auxiliary task labels for reference. The second column and third column show the output of the corre- sponding channel for the proposed Dual-Attentio model trained without and with auxiliary tasks, respectively. As expected, the Aux model outputs very close the auxiliary task labels. The convolutional outputs of the when the
No Aux model show that words and objects/properties in the images have heen properly aligned not provide any auxiliary label for words 'smallest"
model not trained with any auxiliary task labels. and 'largest' as they are not properties of an object and require relative comparison of objects. The visualizations in row (corresponding to 'smallest') ) indicate that both models are able to compare the sizes of objects and detect the smallest object in the corresponding output channel even without any aux labels for the smallest object.
12

---

Under review as a conference paper at ICLR 2020
B ADDITIONAL RESULTS FOR THE DOOM ENVIRONMENT
EASY
HARD
0.8
0.8
Accuracy
Accuracy
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
2
1e7
1e7
lime steps
# Time steps
Text only
Image only
Concal
FILM
PACMAN
Dual-Allention
Figure 8: Training accuracy of all models trained with auxiliary tasks for Easy (left) and Hard (right).
EASY
HARD
1.0
1.0
0.8
0.8
Accuracy
Accuracy
0.6
0.6
0.4
0.4
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
2
le7
le7
lime steps
# Time steps
only
Image only
PACMAN
Figure 9: Training accuracy of all models trained without auxiliary tasks for Easy (left) and Hard (right).
No Aux
Aux
1.0
1.0
0.8
Accuracy
Accuracy
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0.0
0.2
0.4
0.6
0.0
0.2
0.4
0.6
0.8
1.0
#Time steps
1e7
# 4Time steps
1e7
Dual-Attention
w/o SA
wja GA1
w/o GA7
w/n Tack Indicator
w/o Reward Shaping
Figure 10: Training accuracy of proposed Dual-Attention model with all ablation models trained without (left) and with (right) auxiliary tasks for the Easy environment.
13

---

Under review as a conference paper at ICLR 2020
C DOOM ENVIRONMENT DETAILS
The Doom objects used our experiments are illustrated Figure Instructions and questions used for training and evaluation are listed in Tables
Torches
Key cards
Skull keys
Armors
Figure 11: Objects of various colors and sizes used in the ViZDoom environment.
Table Instructions and questions for ViZDoom experiments. We used 5 object classes (torch, pillar, keyeard, skullkey, armor), colors (red, green, blue, yellow). sizes (tall, short), and superlative sizes (smallest, largest).
SGN
Instruction Type
Train Instructions:
Test Instructions:
Not containing 'red' & 'pillar'
Coutaining 'red' or 'pillar' pillar red
(object). Go to the (color) object.
torch, keycard, skullkey, ammor yellow, green, blue
Go to the (size) object.
tall, short
Go to the (color) (ubject).
blue torch, green torch, green
red torch, red skullkey, red pillar, green pillar, red keycard, red armor
blue skullkey, blue keyeard.
yellow keycard, yellow skullkey short torch, tall terch green
Go to the (size) (object).
tall pillar, shoet pillar short, red
Go to (color) (size) object.
tall, blue shor,
green short
Go to the (size) (solor) object.
tall green, tall blue, short blue. short
short red, tall red
Go to the (color) (size) (object).
green tall torch, green short torch, blue short torch,
red short pillar, red short torch, red tall pillar, green tall pillar. red tall torch, green short pillar short red pillar, short red torch, tall red pillar, tall green pillar. tall red torch, shori green pillar
torch
Go to the (size) (color) (object).
green torch, short green torch, short blue torch, tall blue torch
Go to the (superlative) object.
largest, smallest
Go to the (superlative) (color) objoct.
smallest yellow, smallest blue, smallest green, largest blue.
largest red,, smallest red
largest green, largesi yellow
EQA
Question Type
21 Train Questions:
8 Test Questions: Containing 'blue' "torch"
Not containing 'blue' & "torch'
(ohject)?
pillar, keycard, skullkey, armor short pillar, tall pillar
What color is the (sizc) (object)?
short torch, tall torch blue
Which object is (color) color?
red, yellow, green
Which (size) abject (color) color?
shori red, tall red, short green, tall green
short blue, tall blue
Which is the (superlative) object?
largest, smallest
Which is the (superlative) (color) object?
largest red, largest yellow.
largest blue, smallest blue
largest green, smallest
smallest yellow, smallest green
14

---

Under review as a conference paper at ICLR 2020
D ADDITIONAL EXPERIMENTAL DETAILS
Time [1]
Policy Module
Policy prediction
State rep.
X5
Linear
T)
[[W]
XP1
Linear
Xpy (126)
GRU
Xps [126]
Answer Pradiction
Xans
Concet
Linear
V(4. T)
Value prediction
Task Indicator [1]
Figure 12: Architecture of the policy module.
Image representision y'
z- (hx3y) My,
Image representation y
= fox foabruss) Oy
Expand
Tixst represontatiin yr
M,
Spalisl mep Ys
Mys
Figure 13: Gated-Attention unit fGA
Figure 14: Spatial- unit Ssa
D.1 HYPERPARAMETERR AND NETWORK DETAILS
The input image is rescaled to size 3 x 168 x 300. The convolutional network for processing the image consisted of convolutional layers: convl containing 32 8x8 filters with stride 4, conv2 containing 64 4x4 filters with stride 2, and conv3 containing V 3 x 3 filters with stride 2. We use ReLU activations for convl and conv2 and sigmoid for conv3, as its output is used as auxiliary task predictions directly. We use word embeddings and GRU of size 32 followed by a linear layer of size V to get the sentence-level representation. The policy module uses hidden dimension 128 for the linear and GRU layers (see Figure] 12)
For reinforcement learning, we use Proximal Policy Optimization (PPO) with 8 actors and a time horizon of 128 steps. We use single batch with 4 PPO epochs. The clipping parameter for PPO is set to 0.2. The discount factor (y) 0.99. We used Adam optimizer with learning rate 2.5e-4 for all experiments.
Figures 13 and 14 illustrate the Gated-Attention unit and Spatial-Attention unit discussed in Sec- tion
D.2 RELATIONAL TASKS
Here, discuss additional experimental details for the relational tasks described in Section 5.3 We consider three relations, R = =f ['not', 'left of", 'right of'). J. As mentioned previously, we assume that the knowledge of relational words, and which words modified by relational words, are given as input to the model.
D.2.1 EXTENDED DUAL- ARCHITECTURE
Given textual input T and relation E R, define yr (T) RV to be the indicator vector for words that are modified by relation r. For example, if T is the instruction 'Go to the torch that is not red and left_ of pillar', then Yna(T) has 1 the index for 'red" and O everywhere else, and ar() has I at the index for 'pillar' and 0 everywhere else.
Given an indicator vector y E R let My E V be the 3-dimensional tensor obtained by expanding each element of y to a H x W matrix. Thus, the (i, k)th element given by M, jū. j. v[i]. This matrix is multiplied element-wise with convolutional output tensor TR. VxH to select the channels corresponding words indicated by y.
15

---

Under review as a conference paper at ICLR 2020
In the extended Dual-Attention architecture, train separate module fr for each relation R, where each fr is a convolutional network (5 x 5 kernel size, stride I, and padding 2) that preserves the size of the input. We apply the module f, the channels in 2GA1 R VxVllxW which correspond to the words that are modified by relation r:
[IXHHW
Σ M MT)(TT
Σ (T) O fRA(FCAI) E RVXHW
TRA
EGAI
TE
PER
Then TRA create the spatial attention map in equation
Jssptt Σ =
IR
(8)
We also zero out the convolutional channels of 2, corresponding to relational words, in order to improve generalization across different modified words. The rest of the operations are identical the Dual-Attention architecture.
D.2.2 DATASET GENERATION
The relational tasks are generated in the Easy-Aux setting (Section 5.3) which has five candidate objects spawned along a horizontal line in the field of view of the agent. At the start of each episode, we sample word from R U U ('none').
If "none' is sampled, then an instruction (SGN) or a question (EQA) sampled identically in the original experiments.
If 'not" is sampled, then we sample an instruction (SGN) or question (EQA) from the original dataset, then sample a word in this instruction or question and negate it (e.g. 'Go to the red torch' - 'Go to the torch which is not red').
If 'left of" is sampled, then we sample a correct object among the candidate objects in positions {0, 1, 3). generate short description for the object immediately the right of the correct object (c.g., 'red object', ,red torch', 'short torch', or "torch'), ), and append 'left of x' to the instruction or question.
If 'right of" is sampled, then sample correct object among the candidate objects in positions {1, 2, 3. 4), generate a short description x for the object immediately to the left of the correct object, and append 'right of x' to the instruction question.
D.3 BASELINE DETAILS
Image only: Naive baseline of just using the image representation: = vec(zz) where vec(.)
denotes the flattening operation.
Text only: Naive baseline of just using the textual representations: [TBow Zsemt]
Concat:
The image and textual representations are concatenated: .zs [vec(z]) (. TBo Tsent Note
that concatenation is the most common method of combining representations. |Hermann et al. (011))
concatenate convolutional image and bag-of-words textual representations for SGN, whereas Misra et al. (2017) use concatenation with sentence-level textual representations.
Gated Attentio:: Adapted from Chaplot et al. (2018), who used Gated-Attentin with sentence-level textual representations for SGN:
(
FiLM: Perez al. 2018 introduced a general-purpose conditioning method called Feature-wise Linear Modulation (FiLM) for Visual Question Answering. Using FiLM, æs )x Ozy +3 are learnable projections of the sentence representation.
where ( and 3 ((
Das al. 2018) presented hierarchical RL model for EQA. We adapt their method by using the attention mechanism in their QA module, which takes the last frames and the text as input, and computes the similarity of the text with each frame using dot products between image and sentence-lewe text representations. These similarities are converted into attention weights using softmax, and the Aitention-weightee image features concatenated with question embedding and
PACMAN:
16

---

Under review as a conference paper at ICLR 2020
passed through softmax classifier to predict the answer distribution. For this particular baseline, use the last frames as input at each time step, unlike the proposed model and all other baselines which use single frame input. The iitention-weighter image features are used as the state representation. The PACMAN model used pretrained QA module, but we train this module jointly with the Navigation model for fair comparison with the proposed model.
For each the above method except PACMAN, we use linear layer f with ReLU activations followed by softmax a get V-dimensional answer prediction from the state representations: Ans o (as; (rin)). )) Iş and Ans are concatenated and passed to the policy module along with the time step and task indicator variable just as the proposed model.
17

---

Under review as a conference paper at ICLR 2020
E SCALABILITY EXPERIMENTS
We perform additional experiments a sym- bolic 2D environment test the scalability of our model with respect to vocabulary size. We use 5 different abstract attribute types (e.g., object type, size, color, texture, etc.), where each attribute can take on one of K values (e.g., the 'color' attribute can take on values 'blue', 'red', etc.). We construct
Table 6: Performance of the DA model trained in 2D envi- ronment with different vocabulary sizes.
Vocabulary size
Number of Number of Questions
Train MT
Test
Instructions
SGN
EQA
25
> 5
5*
0.79
0.77
0.69
100
> 10
10a
0.77
0.76
0.64
square maze of size 7x7, and spawn the agent with a 3x5 field of view and candi- date objects random locations identical to the Hard setting in the 3D environment. The questions and instructions are created identically to the 3D datasets except superlative questions and instructions. We assume perfect perception, meaning that the dual-attention unit receives convolutional output which detects each attribute perfectly. We perform experiments with different values of K (5, 20, 100} leading vocabulary size of up to 500 words and more than 10 instructions and 10th questions.
500
> 10
> 10a
0.73
0.70
0.60
The results in Tinietoindicate that the cross-task knowledge transfer performance scales well with vocabulary size. Furthermore, results in the previous subsection indicate that attributes can be swapped in and out per requirement due the modularity and interpretability of the model.
18

---

Under review as a conference paper at ICLR 2020
F HOUSE3D EXPERIMENTS
In the House3D domain, we train on one house environment and randomize the colors of each object the start of each episode. The agent's spawn location is fixed. We create instructions and questions dataset for this house similar to the Doom domain. The House3D objects used in our experiments are illustrated in Figure I5 Instructions and questions used for training and evaluation are listed in Table g
Each model is trained for 50 million frames jointly on both SGN and EQA, without the auxiliary tasks and using identical reward functions. Similar to Doom, we use +1 reward for reaching the correct object in SGN episodes and predicting the correct answer in the EQA episodes. We use small negative reward of -0.001 per time step to encourage shorter paths to tar- get and answering the questions as We also use distance-based reward shaping for both SGN and EQA episodes, where the agent receives a small reward proportional to the decrease in dis- tance to the target. SGN episodes end when the agent reaches any object and EQA episodes when agent predicts any answer. All episodes have a maximum length of 420 time steps.
SGN
EQA
Model
Train
Test
Train
Test
Text only
0.63
0.33
0.22
0.23
Image only
0.28
0.01
0.12
0.22
Concat
0.65
0.13
0.31
0.13
GA
0.98
0.20
0.92
0.03
possible.
FiLM
0.99
0.37
0.92
0.24
PACMAN
0.73
0.20
0.40
0.21
0.99
0.47
0.89
0.29
Table 7: Accuracy of all the models the SGN and EQA train and test sets the House3D main.
In Table 7 we report the train and test performance of all the models on both SGN and EQA. The results are similar as in Doom: the Dual-Attentie model outperforms the baselines by a considerable margin.
(a) blue fish tank, green fireplace
(b) blue sofa, red sofa
(c) yellow bed, blue affice_ chair
Figure 15: Examplc first-person views of the House3D environment with sample objects of various colors.
SCN
Instruction Type
22 Train Instructions:
11 Test Instructions:
Not containing 'red" & 'bed'
Containing 'red' or 'bed'
Go to the (object). Go (color) (object).
refrigerator, chhir, fish, tank, fireplace green refrigerator, green office chaiz.
bed
red bed, green bed, bed.
green fish, lank, green fireplace, green sofa, blue refrigerator.
yellow bed, red refrigerator,
office chair. blue lish, lank, blue fireplave, blue sola. yellow refrigerator, yellow office, chair, yellow fish, bank, yellow fireplace, yellow sofa groen, bluc, yellow
red office chair, red fish, tank,
red lireplace, red sofa
Go to the (color) object.
red
EQA
Question Type
Train Questions:
Questions:
Not containing 'blue' & 'sofa'
Containing 'blue' or "sofa"
What color the (object)?
refrigerator, office, chair, fish tank, fireplace, bed red, green, yellow
What object is (color) in color?
blue
Table 8: Instructions and questions for House3D experiments. We used 6 object classes (refrigerator, effiie_chair, fish tank, fireplace, bed, sofa) and 4 colors (red, green, blue, yellow).
19


Reviews:
[{"review_id": "HyxBKMA6KB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Summary \n\nThe paper describes a Dual-Attention model using Gated- and Spatial-Attention for disentanglement of attributes in feature representations for visually-grounded multitask learning. It has been shown that these models are capable of learning navigational instructions and answering questions. However, they addressed two limitations of previous works about visually-grounded embodied language learning models.  The first is the inability to transfer grounded knowledge across different\ntasks, and the other is the inability to transfer to new words and concepts not seen during the training phase. To overcome the problem, a multitask model is introduced. The model can transfer knowledge across tasks via learning disentanglement of the knowledge of words and visual attributes. The paper shows that the proposed model outperforms a range of baselines in simulated 3D environments. \n\n\n*Decision and supporting arguments\n\nI think the paper is on the borderline. The reason is as follows. \nThe motivation of the study is described appropriately, and the performance is quantitatively evaluated, as shown while Table 2. \nHowever, the generality of the proposed method, i.e., dual attention, is still ambiguous. Though the devised module performs effectively in this specific simulation environment and specific two tasks, an explanation of the theoretical basis and generality of dual attention seem to be missing.\nEven though the title has the phrase \"multitask learning,\" what the system copes with is just two specific tasks.  If the system is designed to solve the two specific tasks simultaneously, it's better to change the title. The title seems to be misleading.\nSome of the main contributions, e.g., \"modularity and interpretability\" and \"transfer to new concepts,\" are not evaluated quantitatively.\n\n\n*Additional feedback\nIn conclusion, \"interpretablew\" -> \"interpretable\"", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJxnPOdRYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I thank the authors for their detailed response and appreciate their hard work in bringing us this paper. \n\nI think that my main point is that this work relies too much on the extra information/constraints in the synthetic env. E.g., 1. since the vocab size is small, thus the feature map could be designed 'equal to the vocabulary size' 2. The bag-of-words representation is effective but it is not the case for natural language. Although the authors kindly point me to some recent works on sim2real, I am still not convinced whether this proposed method could be transferred to real setups based on the referenced papers.\n\nHowever, it is a personal research taste that I always take real setup into considerations, because I have worked on both synthetic and real setup (on both lang and visn sides) for years and observed a large gap. My opinion is that methods of synthetic setups are not naturally convertible to the real ones. If AC/meta-reviewer considers the ability of vision-and-language interactions could be effectively studied through this setup with synthetic language and simulated-unrealistic images, I am OK with acceptance. I have downgraded my confidence scores (but kept my overall score) for this purpose.\n\n\n-----------------------------------------------------------------------------------\n\nPros: \n(1) The proposed model makes sense to me, which tries to have two attention layers to extract the information related to the questions. It seems to have the ability to deal with \"and\"/\"or\" logical relationships as well. \n\n(2) Fig. 4 is impressive. It is clear and well-designed. \n\n(3) The results in Table 2 are convincing. They show that both the proposed dual-attention method and multi-task learning would contribute to the performance. \n\nCons:\n(1) It seems that the two main contributions are related to the language. Thus the synthetic language might not be proper to study. For example, in Eqn. 2, the first GA multiplies the BOW vector with the vision feature map, which could filter out unrelated instruction. This method could not be directly transferred to a real setup where natural language and natural images are involved.\n\n(2) The designed attention modules is lack of generalizability. It implements a two-step attention module, while the first step selects the related visual regions w.r.t the words and the second step gathers the information regarding these attended regions. However, it might not be aware of the spatial relationships and thus be limited to simple questions. For example, if the question is \"What is the object on top of the apple?\". To my understanding, the current module would not explicitly handle this one-hop spatial relationship. \n\nComments:\n(1) According to Sec. 3, 70 instructions and 29 questions are involved in this task. Using GRU to encoder these questions seems to be redundant. A simple one-hot embedding for these instructions might already be enough to encode the information.\n\n(2) I am not sure why the visual attention map x_S could be used as the state of the module.\n\n(3) After Eqn. 3, the paper says that \"ReLU activations ... make all elements positive, ensuring ...\". I am confused about the intuition behind this argument because of the softmax activation. Softmax will projects 0 to 1. So the sum of the all-zero vector would still be non-zero after softmax. \n\nTypo:\n- In Sec. 4, X_{BoW} \\in \\{0, 1\\}^V.\n- In Sec. 4.1, \"this matrix is multiplied ...\" --> this tensor.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJgz_vSntH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe paper explores multi-task learning in embodied environments and proposes a Dual-Attention Model that disentangles the knowledge of words and visual attributes in the intermediate representations. It addresses two tasks, namely Semantic Goal Navigation (SGN) and Embodied Question Answering (EQA), using a simple synthetic environment. The paper compares against a few simple baselines and baselines adapted from models in each task.\n\nI would recommend for acceptance, as the experimental results show that the proposed approach successfully transfers knowledge across tasks.\n\nHowever, I would also like to note that the paper has a few drawbacks.\n\nFirst, the paper uses a new environment to evaluate the SGN and EQA task instead of the benchmark environments for these two tasks, making it difficult to compare performance to previous work. The environment in the paper is small (compared to e.g., House3D for EQA) and has a limited variety. Also, the paper only compares to relatively out-of-date approaches on EQA and SGN, instead of the state-of-the-art approaches on them.\n\nIn addition, the paper should also discuss its connections to other multi-task learning approaches in the related work section.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper offers a new approach to cross-modal embodied learning that aims to overcome limited vocabulary and other issues.  Reviews are mixed.  I concur with the two reviewers who say the work is interesting but the contribution is not sufficiently clear for acceptance at this time.
Decision: Reject