Year: 2019

Abstract:
Visually-grounded embodied language learning models have recently shown to be effective at learning multiple multimodal tasks such as following navigational instructions and answering questions. In this paper, we address two key limitations of these models, (a) the inability to transfer the grounded knowledge across different tasks and (b) the inability to transfer to new words and concepts not seen during training using only a few examples. We propose a multitask model which facilitates knowledge transfer across tasks by disentangling the knowledge of words and visual attributes in the intermediate representations. We create scenarios and datasets to quantify cross-task knowledge transfer and show that the proposed model outperforms a range of baselines in simulated 3D environments. We also show that this disentanglement of representations makes our model modular and interpretable which allows for transfer to instructions containing new concepts.†

heading: 1 INTRODUCTION
text: Humans learn language by interacting with a dynamic perceptual environment, grounding words into visual entities and motor actions (Smith and Gasser, 2005; Barsalou, 2008). In recent years, there has been an increased focus on training embodied agents capable of visually-grounded language learning. These include multimodal tasks involving one-way communication, such as mapping navigational instructions to actions (MacMahon et al., 2006; Chen and Mooney, 2011; Artzi and Zettlemoyer, 2013; Mei et al., 2016; Misra et al., 2018); and tasks involving two-way communication such as embodied question answering (Gordon et al., 2018; Das et al., 2018) and embodied dialogue (de Vries et al., 2018). Other studies have shown that grounded semantic goal navigation agents can be effective at exploiting the compositionality of language to generalize to unseen instructions with an unseen composition of semantic attributes (Hermann et al., 2017; Chaplot et al., 2018), or an unseen composition of steps in a multi-step instruction (Oh et al., 2017).
However, current grounded language learning models have certain limitations. Firstly, these models are typically trained only for a single multimodal task and lack the ability to transfer grounded knowledge of ‘concepts’∗ across tasks. For example, if an agent learns to follow the instruction ‘Go to the red torch’ and answer the question ‘What color is the pillar?’, then ideally it should also understand ‘Go to the red pillar’ and ‘What color is the torch?’ without additional training. Training multitask grounded-language models can also improve training sample efficiency, as these multimodal tasks share many common learning challenges including perception, grounding, and navigation.
The second limitation is the inability of trained models to quickly transfer to tasks involving unseen concepts. For example, consider a household instruction-following robot trained on an existing set of objects. We would like the robot to follow instructions involving a new object ‘lamp’ that has been added to the house. Existing models would need to be trained with the new object, which typically requires thousands of samples and can also lead to catastrophic forgetting of known objects. Even if the models were given some labeled samples to detect the new objects, they would require additional training to learn to combine existing grounded knowledge with the new concept (e.g., ‘blue lamp’ if ‘blue’ is already known). In this paper, we train a multimodal multitask learning model for two tasks: Semantic Goal Navigation, where the agent is given a language instruction to navigate to a goal location, and Embodied Question Answering, where the agent is asked a question and it can navigate in the environment to gather information to answer the question (see Figure 5). We make the following contributions in this paper:
†See https://sites.google.com/view/emml-iclr2020 for demo videos. ∗In this paper, we refer to the knowledge of a word and its grounding in the visual world as the knowledge
of a concept (for example, concept ‘torch’ involves word ‘torch’ and how torch looks visually).
First, we define a cross-task knowledge transfer evaluation criterion to test the ability of multimodal multi-task models to transfer knowledge of concepts across tasks. We show that several prior single-task models, when trained on both tasks, fail to achieve cross-task knowledge transfer. This is because the visual grounding of words is often implicitly learned as a by-product of end-to-end training of the underlying task, which leads to the entanglement of knowledge of concepts in the learnt representations. We propose a novel Dual-Attention model which learns task-invariant disentangled visual and textual representations and explicitly aligns them with each other. We create datasets and simulation scenarios for testing crosstask knowledge transfer and show an absolute improvement of 43-61% on instructions and 5-26% for questions over baselines.
Second, the disentanglement and explicit alignment of representations makes our model modular and interpretable. We show that this allows us to transfer the model to handle instructions involving unseen concepts by incorporating the output of object detectors. We also show that our model is able to combine the knowledge of existing concepts with a new concept without any additional policy training.
Finally, we show that the modularity and interpretability of our model also allow us to use trainable neural modules (Andreas et al., 2016) to handle relational tasks involving negation and spatial relationships and also tackle relational instructions involving new concepts.

heading: 2 RELATED WORK
text: A lot of early work on visual instruction-following in the embodied space such as in robotics applications (Tellex et al., 2011; Matuszek et al., 2012; Hemachandra et al., 2015; Misra et al., 2016) and on mapping natural language instructions to actions (MacMahon et al., 2006; Chen and Mooney, 2011; Artzi and Zettlemoyer, 2013; Mei et al., 2016) required hand-designed symbolic representations. Recently, there have been efforts on learning to follow navigational instructions from raw visual observations (Anderson et al., 2018; Misra et al., 2018; Chen et al., 2019; Blukis et al., 2018). Oh et al. (2017); Chaplot et al. (2018); Hermann et al. (2017) study the language learning aspect of instruction-following in a more controlled setting, and show that grounded language learning agents are able to learn spatial and logical reasoning and exploit the compositionality of language to generalize to new instructions.
Question Answering in the embodied space has been comparatively less-studied with recent work studying QA which requires exploration, navigation, and interaction with objects in the environment (Gordon et al., 2018; Das et al., 2018). In contrast to the prior work which tackles a single grounding task, we tackle both instruction-following and question answering in the embodied space and study the ability to transfer the knowledge of concepts across the tasks and tackle instructions with new concepts.
In addition to the above, there is a large body of work on multimodal learning in static settings which do not involve navigation or reinforcement learning. Some relevant works which use attention mechanisms similar to the ones used in our proposed model include Perez et al. (2018); Fukui et al. (2016); Xu and Saenko (2016); Hudson and Manning (2018); Gupta et al. (2017) for Visual Question Answering and Zhao et al. (2018) for grounding audio to vision.

heading: 3 PROBLEM FORMULATION
text: Consider an autonomous agent interacting with an episodic environment as shown in Figure 1. At the beginning of each episode, the agent receives a textual input T specifying a task. T could be an instruction to navigate to a target object or a question querying some visual detail of objects in the environment. At each time step t, the agent observes a state st = (It, T ) where It is the first-person (egocentric) view of the environment, and takes an action at, which could be a navigational action or an answer action. The agent’s objective is to learn a policy π(at|st) which leads to successful completion of the task specified by T .
Environments. We adapt the ViZDoom-based (Kempka et al., 2016) language grounding environment proposed by Chaplot et al. (2018) for embodied multitask learning. It consists of a single room with 5 objects. The objects are randomized in each episode based on the textual input. We use two difficulty settings for the Doom domain as shown in Figure 2: Easy: The agent is spawned at a fixed location. The candidate objects are spawned at five fixed locations along a single horizontal line in the field of view of the agent. Hard: The candidate objects and the agent are spawned at random locations and the objects may or may not be in the agent’s field of view in the initial configuration. The agent must explore the map to view all objects. The agent can take 4 actions: 3 navigational actions (forward, left, right) and 1 answer action. When the agent takes the answer action, the answer with the maximum probability in the output answer distribution is used.
Datasets. We use the set of 70 instructions from Chaplot et al. (2018) and create a dataset of 29 questions using the same set of objects and attributes. These datasets include instructions and questions about object types, colors, relative sizes (tall/short) and superlative sizes (smallest/largest). We create train-test splits for both instructions and questions datasets to explicitly test a multitask model’s ability to transfer the knowledge of concepts across different tasks. Each instruction in the test set contains a word that is never seen in any instruction in the training set but is seen in some questions in the training set. Similarly, each question in the test set contains a word never seen in any training set question. Table 1 illustrates the train-
test split of instructions and questions used in our experiments†. Note that for the EQA trainset, unseen words can be present in the answer.

heading: 4 PROPOSED METHOD
text: In this section, we describe our proposed architecture (illustrated in Figure 3). At the start of each episode, the agent receives a textual input T (an instruction or a question) specifying the task. At each time step t, the agent observes an egocentric image It which is passed through a convolutional neural network (LeCun et al., 1995) with ReLU activations (Glorot et al., 2011) to produce the image representation xI = f(It; θconv) ∈ RV×H×W , where θconv denotes the parameters of the convolutional network, V is the number of feature maps in the convolutional network output which is by design set equal to the vocabulary size (of the union of the instructions and questions training sets), and H and W are the height and width of each feature map. We use two representations for the textual input T : (1) the bag-of-words representation denoted by xBoW ∈ 0, 1V and (2) a sentence representation xsent = f(T ; θsent) ∈ RV , which is computed by passing the words in T through a Gated Recurrent Unit (GRU) (Cho et al., 2014) network followed by a linear layer. Here, θsent denotes the parameters of the GRU network and the linear layer with ReLU activations. Next, the
†More details about the datasets and the environments are deferred to the supplementary material.
Dual-Attention unit fDA combines the image representation with the text representations to get the complete state representation xS and answer prediction xAns:
xS, xAns = fDA(xI , xBoW, xsent) (1)
Finally, xS and xAns, along with a time step embedding and a task indicator variable (for whether the task is SGN or EQA), are passed to the policy module to produce an action.

heading: 4.1 DUAL-ATTENTION UNIT
text: The Dual-Attention unit uses two types of attention mechanisms, Gated-Attention fGA and SpatialAttention fSA, to align representations in different modalities and tasks.
Gated-Attention (GA). The GA unit (Chaplot et al., 2018) attends to the different channels in the image representation based on the text representation. For example, if the textual input is the instruction ‘Go to the red pillar’, then the GA unit can learn to attend to channels which detect red things and pillars. Specifically, the GA unit takes as input a 3-dimensional tensor image representation yI ∈ Rd×H×W and a text representation yT ∈ Rd, and outputs a 3-dimensional tensor z ∈ Rd×H×W . Note that the dimension of yT is equal to the number of feature maps and the size of the first dimension of yI . In the GA unit, each element of yT is expanded to aH×W matrix, resulting in a 3-dimensional tensor MyT ∈ Rd×H×W , whose (i, j, k)th element is given by MyT [i, j, k] = yT [i]. This matrix is multiplied element-wise with the image representation: z = fGA(yI , yT ) = MyT yI , where denotes the Hadamard product (Horn, 1990).
Spatial-Attention (SA). We propose an SA unit which is analogous to the Gated-Attention unit except that it attends to different pixels in the image representation rather than the channels. For example, if the textual input is the question ‘Which object is blue in color?’, then we would like to spatially attend to the parts of the image which contain a blue object in order to recognize the type of the blue object. The Spatial-Attention unit takes as input a 3-dimensional tensor image representation yI ∈ Rd×H×W and a 2-dimensional spatial attention map yS ∈ RH×W , and outputs a tensor z ∈ Rd×H×W . Note that the height and width of the spatial attention map are equal to the height and width of the image representation. In the spatial-attention unit, each element of the spatial attention map is expanded to a d dimensional vector. This again results in a 3-dimensional tensor MyS ∈ Rd×H×W , whose (i, j, k)th element is given by: MyS [i, j, k] = yS [j, k]. Just like in the Gated-Attention unit, this matrix is multiplied element-wise with the image representation: z = fSA(yI , yS) =MyS yI . Dual-Attention. We now describe the operations in the Dual-Attention unit shown in Figure 4, as well as motivate the intuitions behind each operation. Given xI , xBoW, and xsent, the Dual-Attention unit first computes a Gated-Attention over xI using xBoW:
xGA1 = fGA(xI , xBoW) ∈ RV×H×W (2)
Intuitively, this GA unit grounds each word in the vocabulary with a feature map in the image representation. A particular feature map is activated if and only if the corresponding word occurs in the textual input. Thus, the feature maps in the convolutional output learn to detect different objects and attributes, and words in the textual input specify which objects and attributes are relevant to the current task. The Gated-Attention using BoW representation attends to feature maps detecting corresponding objects and attributes, and masks all other feature maps. We use the BoW representation for the first GA unit as it explicitly aligns the words in textual input irrespective of whether it is a question or an instruction.
Next, the output of the GA unit xGA1 is converted to a spatial attention map by summing over all channels followed by a softmax over H ×W elements:
xspat = σ ( V∑ i xGA1[i, :, :] ) ∈ RH×W (3)
where the softmax σ(z)j = exp(zj)/ ∑
k exp(zk) ensures that the attention map is spatially normalized. Summation of xGA1 along the depth dimension gives a spatial attention map which has high activations at spatial locations where relevant objects or attributes are detected. ReLU activations in the convolutional feature maps makes all elements positive, ensuring that the summation aggregates the activations of relevant feature maps.
xspat and xI are then passed through a SA unit:
xSA = fSA(xI , xspat) ∈ RV×H×W (4)
The SA unit outputs all attributes present at the locations where relevant objects and attributes are detected. This is especially helpful for question answering, where a single Gated-Attention may not be sufficient. For example, if the textual input is ‘Which color is the pillar?’, then the model needs to attend not only to feature maps detecting pillars (done by the Gated-Attention), but also to other attributes at the spatial locations where pillars are seen in order to predict their color.
xSA is then passed through another GA unit with the sentence-level text representation:
xGA2 = fGA(xSA, xsent) ∈ RV×H×W (5)
This second GA unit enables the model to attend to different types of attributes based on the question. For instance, if the question is asking about the color (‘Which color is the pillar?’), then the model needs to attend to the feature maps corresponding to colors; or if the question is asking about the object type (‘Which object is green in color?’), then the model needs to attend to the feature maps corresponding to object types. The sentence embedding xsent can learn to attend to multiple channels based on the textual input and mask the rest.
Next, the output is transformed to answer prediction by again doing a summation and softmax but this time summing over the height and width instead of the channels:
xAns = σ H,W∑ j,k xGA2[:, j, k]  ∈ RV (6) Summation of xGA2 along each feature map aggregates the activations for relevant attributes spatially. Again, ReLU activations for sentence embedding ensure aggregation of activations for each attribute or word. The answer space is identical to the textual input space RV .
Finally, the Dual-Attention unit fDA outputs the answer prediction xAns and the flattened spatial attention map xS = vec(xspat), where vec(·) denotes the flattening operation. Policy Module. The policy module takes as input the state representation xS from the Dual-Attention unit, a time step embedding t, and a task indicator variable I (for whether the task is SGN or EQA). The inputs are concatenated then passed through a linear layer, then a recurrent GRU layer, then linear layers to estimate the policy function π(at | It, T ) and the value function V (It, T ). All above operations are differentiable, making the entire architecture trainable end-to-end. Note that all attention mechanisms in the Dual-Attention unit only modulate the input image representation, i.e., mask or amplify specific feature maps or pixels. This ensures that there is an explicit alignment between the words in the textual input, the feature maps in the image representation, and the words in the answer space. This forces the convolutional network to encode all the information required with respect to a certain word in the corresponding output channel. For example, to predict ‘red’ as the answer, the model must detect red objects in the corresponding feature map. This explicit task-invariant alignment between convolutional feature maps and words in the input and answer space facilitates grounding and allows for cross-task knowledge transfer. As shown in the results later, this also makes our model modular and allows easy addition of objects and attributes to a trained model.

heading: 4.2 OPTIMIZATION
text: The entire model is trained to predict both navigational actions and answers jointly. The policy is trained using Proximal Policy Optimization (PPO) (Schulman et al., 2017). For training the answer predictions, we use a supervised cross-entropy loss. Both types of losses have common parameters as the answer prediction is essentially an intermediate representation for the policy.
Auxiliary Task. As mentioned earlier, the feature maps in the convolutional output are expected to detect different objects and attributes. Consequently, we add a spatial auxiliary task (trained with cross-entropy loss) to detect the object or attribute in the convolutional output channels corresponding to the word in the bag-of-words representation. Rather than doing fine-grained object detection, we keep the size of the auxiliary predictions the same as the convolutional output to avoid an increase in the number of parameters, and maintain the explicit alignment on the convolutional feature maps with the words. Consequently, auxiliary labels are (V ×H ×W )-dimensional tensors, where each of the V channels corresponds to a word in the vocabulary, and each element in a channel is 1 if the corresponding object or attribute is present in the corresponding frame.

heading: 5 EXPERIMENTS & RESULTS
text: Jointly learning semantic goal navigation and embodied question answering essentially involves a fusion of textual and visual modalities. While prior methods are designed for a single task, we adapt several baselines for our environment and tasks by using their multimodal fusion techniques. We use two naive baselines, Image only and Text only; two baselines based on prior semantic goal navigation models, Concat (used by Hermann et al. (2017); Misra et al. (2017)) and GatedAttention (GA) (Chaplot et al., 2018); and two baselines based on Question Answering models, FiLM (Perez et al., 2018) and PACMAN (Das et al., 2018). For fair comparison, we replace the proposed Dual-Attention unit with multimodal fusion techniques in the baselines and keep everything else identical to the proposed model‡.

heading: 5.1 RESULTS
text: We train all models for 10 million frames in the Easy setting and 50 million frames in the Hard setting. We use a +1 reward for reaching the correct object in SGN episodes and predicting the correct answer in EQA episodes. We use a small negative reward of -0.001 per time step to encourage shorter paths to the target and answering questions as soon as possible. We also use distance-based reward shaping for SGN episodes, where the agent receives a small reward proportional to the decrease in distance to the target. In the next subsection, we evaluate the performance of the proposed model without the reward shaping. SGN episodes end when the agent reaches any object, and EQA episodes end when the agent predicts any answer. All episodes have a maximum length of 210 time steps. We train all models with and without the auxiliary tasks using identical reward functions.
All models are trained jointly for both the tasks and tested on each task separately§. In Table 2, we report the performance of all models for both Easy and Hard settings¶. The Dual-Attention (DA) model and many baselines achieve 99% accuracy during training in the Easy-Aux setting; however,
‡See the supplementary material for more implementation details of all baselines. § See https://sites.google.com/view/emml-iclr2020/ for visualization videos. ¶See the supplementary material for training performance curves for all models.
the test performance of all the baselines is considerably lower than that of the DA model (see Table 2 (left)). Performance of all the baselines is worse than the ‘Text only’ model on the EQA test set, although the training accuracy is higher. This indicates that baselines tend to overfit on the training set and fail to generalize to questions which contain words never seen in training questions. As expected, using spatial auxiliary tasks improves performance of all models. Even without auxiliary tasks, the DA model achieves a test accuracy 86% (SGN) and 53% (EQA), compared to the best baseline performance of 33% (SGN & EQA).
For the Hard setting, the DA model achieves a higher training (90% vs 71% with Aux) as well as test performance (82% vs. 39% for SGN, 59% vs. 33% for EQA with Aux) than the baselines (see Table 2 (right)). These results confirm the hypothesis that prior models, which are designed for a single task, lack the ability to align the words in both the tasks and transfer knowledge across tasks.
Lower test accuracy on EQA (vs. SGN) for most models (Table 2) indicates that EQA is more challenging as it involves alignment between not just input textual and visual representations but also with the answer space.

heading: 5.2 ABLATION TESTS
text: We perform a series of ablation tests in order to analyze the contribution of each component in the Dual-Attention unit: without Spatial-Attention (w/o SA), without the first Gated-Attention with xBoW (w/o GA1), and without the second Gated-Attention with xsent (w/o GA2). We also try removing the task indicator variable (w/o Indicator Variable), removing reward shaping (w/o Reward Shaping), and training the proposed model on a single task, SGN or EQA (DA Single-Task).
In Table 3, we report the test performance of all ablation models. The results indicate that SA and GA1 contribute the most to the performance of the full Dual-Attention model. GA2 is critical for performance on EQA but not SGN (see Table 3). This is expected as GA2 is designed to attend to different objects and attributes based on the question and is used mainly for answer prediction. It is not critical for SGN as the spatial attention map consists of locations of relevant objects, which is sufficient for navigating to the correct object ‖.
We observe that reward shaping and indicator variable help with learning speed, but have little effect on the final performance (see Table 3). DA models trained only on single tasks work well on SGN, especially with auxiliary tasks, because the auxiliary task for single task models includes object detection labels corresponding to the words in the test set. This highlights a key advantage of our model’s modular and interpretable design: the model can be used for transferring the policy to new objects and attributes without fine-tuning as later discussed in the Section 5.4.

heading: 5.3 HANDLING RELATIONAL TASKS
text: The instructions and questions considered so far contained a single target object. We propose a simple extension to our model to handle relational tasks, such as ‘Which object is to the left of the torch?’, where the agent is required to attend to the region left of the torch, not the torch itself.
‖See the supplementary material for visualization of convolutional feature outputs, spatial attention map, sentence representation of the textual input and answer predictions
not green
green
obs obs
blue
left_of blue
obs
keycard
right_of keycard
Figure 5: Outputs for relations ‘not’, ‘left of’, and ‘right of’ learned by the relational modules.
We consider three relational operations: ‘left of’, ‘right of’ and ‘not’. We add questions and instructions with all objects and attributes using these relational operations to the existing dataset and perform experiments in the Easy-Aux setting. We assume that the knowledge of relational words, and the words they modify, are given. We train a separate module corresponding to each relational operation, and apply it to the convolutional output of the words that are modified. For example, for the above question, we apply the module for relation ‘left of’ to the convo-
lutional output channel corresponding to the word ‘torch’. Each relational module is a trainable convolutional network which preserves the size of the input. The rest of the operations are identical to the Dual-Attention Unit. The relational modules are learned end-to-end without any additional supervision.
In Figure 5, we show convolutional outputs of the relational modules learned by our model. While the original DA model achieves test performance of 0.48 (SGN) and 0.44 (EQA), this simple extension achieves 0.97 (SGN) and 0.64 (EQA).

heading: 5.4 TRANSFER TO NEW CONCEPTS
text: Suppose that the user wants the agent to follow instructions about a new object such as ‘pillar’ or a new attribute such as ‘red’ which the agent has never seen during training. Prior SGN models (Chaplot et al., 2018; Hermann et al., 2017; Yu et al., 2018) cannot handle instructions containing a new concept. In contrast, our model can be used for handling such instructions by training an object detector for each new concept and appending it to the image representation xI . In order to test this, we train the DA model in the Easy setting on the training set for only instructions. We use auxiliary tasks but only for words in the vocabulary of the instructions training set. After training the policy, we test the agents on instructions containing test concept words ‘red’ and ‘pillar’, which the agent has never seen in textual input during training and never received any supervision about how this attribute or object looks visually.
For transferring the policy, we assume access to two object detectors for ‘red’ and ‘pillar’ separately. We resize the object detections to the size of a feature map in the image representation (H ×W ) and append them as channels to the image representation. We also append the words ‘red’ and ‘pillar’ to the bag-of-words representations in the same order such that they are aligned with the appended feature maps. We randomly initialize the embeddings of the new words for computing the sentence embedding.
The results in Table 4 show that this policy generalizes well to different types of instructions with unseen concepts, including: combining knowledge of existing attributes with a new object, or knowledge of existing objects with a new attribute; and composing a new attribute with a new object. The results shown in the lower part of Table 4 indicate that the model also generalizes well to relational instructions containing new concepts. This means that given an object detector for a new object ‘pillar’, the model can (without any additional training) detect and differentiate between green and blue pillars, or between tall and short pillars; and understand left of/right of pillar, or the negation of pillar. The model can also combine ‘pillar’ with another new attribute ‘red’ to detect red pillars and understand relational instructions involving both red objects and pillars. This suggests that a trained policy can be scaled to more objects provided the complexity of navigation remains consistent.

heading: 6 CONCLUSION
text: We proposed a Dual-Attention model for visually-grounded multitask learning which uses Gated- and Spatial-Attention to disentangle attributes in feature representations and align them with the answer space. We show that the proposed model is able to transfer the knowledge of concepts across tasks and outperforms the baselines on both Semantic Goal Navigation and Embodied Question Answering by a considerable margin. We showed that disentangled and interpretablew representations make our model modular and allow for easy addition of new objects or attributes to a trained model. For future work, the model can potentially be extended to transferring knowledge across different domains by using modular interpretable representations of objects which are domain-invariant.

heading: B ADDITIONAL RESULTS FOR THE DOOM ENVIRONMENT

heading: C DOOM ENVIRONMENT DETAILS
text: The Doom objects used in our experiments are illustrated in Figure 11. Instructions and questions used for training and evaluation are listed in Tables 5.
D ADDITIONAL EXPERIMENTAL DETAILS
D.1 HYPERPARAMETERS AND NETWORK DETAILS
The input image is rescaled to size 3 × 168 × 300. The convolutional network for processing the image consisted of 3 convolutional layers: conv1 containing 32 8x8 filters with stride 4, conv2 containing 64 4x4 filters with stride 2, and conv3 containing V 3× 3 filters with stride 2. We use ReLU activations for conv1 and conv2 and sigmoid for conv3, as its output is used as auxiliary task predictions directly. We use word embeddings and GRU of size 32 followed by a linear layer of size V to get the sentence-level representation. The policy module uses hidden dimension 128 for the linear and GRU layers (see Figure 12).
For reinforcement learning, we use Proximal Policy Optimization (PPO) with 8 actors and a time horizon of 128 steps. We use a single batch with 4 PPO epochs. The clipping parameter for PPO is set to 0.2. The discount factor (γ) is 0.99. We used Adam optimizer with learning rate 2.5e-4 for all experiments.
Figures 13 and 14 illustrate the Gated-Attention unit and Spatial-Attention unit discussed in Section 4.1.
D.2 RELATIONAL TASKS
Here, we discuss additional experimental details for the relational tasks described in Section 5.3. We consider three relations,R = {‘not′, ‘left of′, ‘right of′}. As mentioned previously, we assume that the knowledge of relational words, and which words are modified by relational words, are given as input to the model.
D.2.1 EXTENDED DUAL-ATTENTION ARCHITECTURE
Given textual input T and a relation r ∈ R, define yr(T ) ∈ RV to be the indicator vector for words that are modified by relation r. For example, if T is the instruction ‘Go to the torch that is not red and left_of pillar’, then ynot(T ) has 1 at the index for ‘red’ and 0 everywhere else, and yleft_of(T ) has 1 at the index for ‘pillar’ and 0 everywhere else.
Given an indicator vector y ∈ RV , let My ∈ RV×H×W be the 3-dimensional tensor obtained by expanding each element of y to a H × W matrix. Thus, the (i, j, k)th element is given by My[i, j, k] = y[i]. This matrix is multiplied element-wise with a convolutional output tensor x ∈ RV×H×W to select the channels corresponding to words indicated by y.
In the extended Dual-Attention architecture, we train a separate module fr for each relation r ∈ R, where each fr is a convolutional network (5× 5 kernel size, stride 1, and padding 2) that preserves the size of the input. We apply the module fr to the channels in xGA1 ∈ RV×H×W which correspond to the words that are modified by relation r:
xRA = ( 1V×H×W −
∑ r∈R Myr(T )
) xGA1 +
∑ r∈R Myr(T ) fRA(xGA1) ∈ R V×H×W (7)
Then we use xRA to create the spatial attention map in equation 3:
xspat = σ ( V∑ i xRA[i, :, :] ) ∈ RH×W (8)
We also zero out the convolutional channels of xI corresponding to relational words, in order to improve generalization across different modified words. The rest of the operations are identical to the Dual-Attention architecture.
D.2.2 DATASET GENERATION
The relational tasks are generated in the Easy-Aux setting (Section 5.3), which has five candidate objects spawned along a horizontal line in the field of view of the agent. At the start of each episode, we sample a word fromR∪ {‘none′}.
• If ‘none’ is sampled, then an instruction (SGN) or a question (EQA) is sampled identically as in the original experiments. • If ‘not’ is sampled, then we sample an instruction (SGN) or a question (EQA) from the
original dataset, then sample a word in this instruction or question and negate it (e.g. ‘Go to the red torch’→ ‘Go to the torch which is not red’). • If ‘left of’ is sampled, then we sample a correct object among the candidate objects in
positions {0, 1, 2, 3}, generate a short description x for the object immediately to the right of the correct object (e.g., ‘red object’, ‘,red torch’, ‘short torch’, or ‘torch’), and append ‘left of x’ to the instruction or question. • If ‘right of’ is sampled, then we sample a correct object among the candidate objects in
positions {1, 2, 3, 4}, generate a short description x for the object immediately to the left of the correct object, and append ‘right of x’ to the instruction or question.
D.3 BASELINE DETAILS
Image only: Naive baseline of just using the image representation: xS = vec(xI) where vec(.) denotes the flattening operation.
Text only: Naive baseline of just using the textual representations: xS = [xBoW, xsent].
Concat: The image and textual representations are concatenated: xS = [vec(xI), xBoW, xsent]. Note that concatenation is the most common method of combining representations. Hermann et al. (2017) concatenate convolutional image and bag-of-words textual representations for SGN, whereas Misra et al. (2017) use concatenation with sentence-level textual representations.
Gated-Attention: Adapted from Chaplot et al. (2018), who used Gated-Attention with sentence-level textual representations for SGN: xS = fGA(xI , xsent).
FiLM: Perez et al. (2018) introduced a general-purpose conditioning method called Feature-wise Linear Modulation (FiLM) for Visual Question Answering. Using FiLM, xS = γ(xsent) xI+β(xsent) where γ(xsent) and β(xsent) are learnable projections of the sentence representation.
PACMAN: Das et al. (2018) presented a hierarchical RL model for EQA. We adapt their method by using the attention mechanism in their QA module, which takes the last 5 frames and the text as input, and computes the similarity of the text with each frame using dot products between image and sentence-level text representations. These similarities are converted into attention weights using softmax, and the attention-weighted image features are concatenated with question embedding and
passed through a softmax classifier to predict the answer distribution. For this particular baseline, we use the last 5 frames as input at each time step, unlike the proposed model and all other baselines which use a single frame as input. The attention-weighted image features are used as the state representation. The PACMAN model used a pretrained QA module, but we train this module jointly with the Navigation model for fair comparison with the proposed model.
For each of the above method except PACMAN, we use a linear layer f with ReLU activations followed by softmax σ to get a V -dimensional answer prediction from the state representations: xAns = σ(f(xS; θLin)). xS and xAns are concatenated and passed to the policy module along with the time step and task indicator variable just as in the proposed model.

heading: E SCALABILITY EXPERIMENTS
text: We perform additional experiments in a symbolic 2D environment to test the scalability of our model with respect to vocabulary size. We use 5 different abstract attribute types (e.g., object type, size, color, texture, etc.), where each attribute can take on one of K values (e.g., the ‘color’ attribute can take on values ‘blue’, ‘red’, etc.). We construct a square maze of size 7x7, and spawn the agent with a 3x5 field of view and 5 candidate objects at random locations identical to
the Hard setting in the 3D environment. The questions and instructions are created identically to the 3D datasets except superlative questions and instructions. We assume perfect perception, meaning that the dual-attention unit receives convolutional output which detects each attribute perfectly. We perform experiments with different values of K ∈ {5, 20, 100} leading to a vocabulary size of up to 500 words and more than 1010 instructions and 108 questions.
The results in Table 6 indicate that the cross-task knowledge transfer performance scales well with vocabulary size. Furthermore, results in the previous subsection indicate that attributes can be swapped in and out as per requirement due to the modularity and interpretability of the model.

heading: F HOUSE3D EXPERIMENTS
text: In the House3D domain, we train on one house environment and randomize the colors of each object at the start of each episode. The agent’s spawn location is fixed. We create instructions and questions dataset for this house similar to the Doom domain. The House3D objects used in our experiments are illustrated in Figure 15. Instructions and questions used for training and evaluation are listed in Table 8.
SGN EQA Model Train Test Train Test
Text only 0.63 0.33 0.22 0.23 Image only 0.28 0.01 0.12 0.22 Concat 0.65 0.13 0.31 0.13 GA 0.98 0.20 0.92 0.03 FiLM 0.99 0.37 0.92 0.24 PACMAN 0.73 0.20 0.40 0.21 Dual-Attention 0.99 0.47 0.89 0.29
In Table 7, we report the train and test performance of all the models on both SGN and EQA. The results are similar as in Doom: the Dual-Attention model outperforms the baselines by a considerable margin.

Review summaries:
The paper describes a Dual-Attention model using Gated- and Spatial-Attention for disentanglement of attributes in feature representations for visually-grounded multitask learning. It has been shown that these models are capable of learning navigational instructions and answering questions. However, they addressed two limitations of previous works about visually-grounded embodied language learning models.  The first is the inability to transfer grounded knowledge across different

The paper explores multi-task learning in embodied environments and proposes a Dual-Attention Model that disentangles the knowledge of words and visual attributes in the intermediate representations. It addresses two tasks, namely Semantic Goal Navigation (SGN) and Embodied Question Answering (EQA), using a simple synthetic environment. The paper compares against a few simple baselines and baselines adapted from models in each task.