SlowGuess/NAACL_2021_OCR huggingface dataset:
Rodolfo Corona Daniel Fried Coline Devin Dan Klein Trevor Darrell UC Berkeley
{rcorona,dfried,coline,klein,trevordarrell}@berkeley.edu
Standard architectures used in instruction following often struggle on novel compositions of subgoals (e.g. navigating to landmarks or picking up objects) observed during training. We propose a modular architecture for following natural language instructions that describe sequences of diverse subgoals. In our approach, subgoal modules each carry out natural language instructions for a specific subgoal type. A sequence of modules to execute is chosen by learning to segment the instructions and predicting a subgoal type for each segment. When compared to standard, non-modular sequence-to-sequence approaches on ALFRED (Shridhar et al., 2020), a challenging instruction following benchmark, we find that modularization improves generalization to novel subgoal compositions, as well as to environments unseen in training.
Work on grounded instruction following (MacMahon et al., 2006; Vogel and Jurafsky, 2010; TELLEX et al., 2011; Chen and Mooney, 2011; Artzi and Zettlemoyer, 2013) has recently been driven by sequence-to-sequence models (Mei et al., 2016; Hermann et al., 2017), which allow end-to-end grounding of linguistically-rich instructions into equally-rich visual contexts (Misra et al., 2018; Anderson et al., 2018; Chen et al., 2019). These sequence-to-sequence models are monolithic: they consist of a single network structure which is applied identically to every example in the dataset.
Monolithic instruction following models typically perform well when evaluated on test data from the same distribution seen during training. However, they often struggle in compositional generalization: composing atomic parts, such as actions or goals, where the parts are seen in training but their compositions are not (Lake and Baroni, 2018; Ruis et al., 2020; Hill et al., 2020).
Turn left to move to the counter... Pick up the pan to the right of the toaster. Turn around to move in front of the sink. Place the pan in the sink ... Carry the pan ... Place the pan on the back of the counter by the wall.
Turn around and walk past the bed... Grab the cellphone off of the cabinet. Turn right and walk ... Place the cellphone on the dresser. Turn around ... Grab the cellphone off of the chair there. Turn left and walk ... Place the cellphone down ...
Figure 1: At evaluation time, an instruction following agent may need to generalize both to novel chains of subgoals encountered during training as well as to completely new environments. In the generalization condition above, the agent must generalize to multiple pickup actions (in green) at test time, whereas only single ones were seen at training, as well as to a new house. We propose a modular architecture to handle these cases.
In this work, we improve compositional generalization in instruction following with modular networks, which have been successful in non-embodied language grounding tasks (Andreas et al., 2016; Hu et al., 2017; Cirik et al., 2018; Yu et al., 2018; Mao et al., 2019; Han et al., 2019) and in following synthetic instructions or symbolic policy descriptions (Andreas et al., 2017; Oh et al., 2017; Das et al., 2018). Modular networks split the decision making process into a set of neural modules. Modules are each specialized for some function, composed into a structure specific to each example, and trained jointly to complete the task.
Prior work has found that modular networks often perform well in compositional generalization
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1033-1040 June 6-11, 2021. ©2021 Association for Computational Linguistics
Figure 2: Our modular approach first uses a controller (left) trained with supervised learning to segment a given instruction and label segments with subgoal types (e.g. GOTo, PICKUP) to execute. These subgoal types are used to chain together modules (right) to carry out instructions in the environment. Each module is a separately parameterized sequence-to-sequence model that conditions on an attended representation of the instruction sequence, the visual observations, and the action taken at the previous timestep. Modules pass recurrent hidden states to each other.
because of their composable structure (Devin et al., 2017; Andreas et al., 2017; Bahdanau et al., 2019; Purushwalkam et al., 2019), and that they can generalize to new environments or domains through module specialization (Hu et al., 2019; Blukis et al., 2020). However, all this work has either focused on grounding tasks without a temporal component or used a network structure which is not predicted from language.
We propose a modular architecture for embodied vision-and-language instruction following<sup>1</sup>, and find that this architecture improves generalization on unseen compositions of subgoals (such as navigation, picking up objects, cleaning them, etc.). We define separate sequence-to-sequence modules per type of subgoal. These modules are strung together to execute complex high-level tasks. We train a controller to predict a sequence of subgoal types from language instructions, which determines the order in which to execute the modules.
We evaluate models on the ALFRED dataset (Shridhar et al., 2020), an instruction-following benchmark containing a diverse set of household tasks. We focus on compositional generalization: carrying out instructions describing novel high-level tasks, containing novel compositions of actions (see Figure 1 for an example). We find that our modular model improves performance on average across subgoal types when compared to a standard, monolithic sequence-to-sequence architecture. Additionally, we find improved generalization to environments not seen in training.
2 Modular Instruction Following Networks
We focus on following instructions in embodied tasks involving navigation and complex object interactions, as shown in Figure 2.
In training, each set of full instructions (e.g. "Turn right and cross the room ... Place the vase on the coffee table to the left of the computer.") is paired with a demonstration of image observations and actions. In training, we further assume that the full instruction is segmented into subgoal instructions, and each subgoal instruction is labeled with one of a small number (in our work, 8) of subgoal types, e.g. ["Walk to the coffee maker": GoTo ], ["Pick up the dirty mug...": PICKUP], ..., and paired with the corresponding segment of the demonstration.
During evaluation, the agent is given only full instructions (which are unsegmented and unlabeled), and must predict a sequence of actions to carry out the instructions, conditioning on the image observations it receives.
Our modular architecture for compositional instruction following consists of a high-level controller (Figure 2, left), and modules for each subgoal type (Figure 2, right). The high-level controller chooses modules to execute in sequence based on the natural language instructions, and each chosen module executes until it outputs a STOP action. The modules all share the same sequence-to-sequence architecture, which is the same as the monolithic architecture. We initialize each module's parameters with parameters from the monolithic model, and then fine-tune the parameters of each module to specialize for its subgoal.
$^{1}$ Code and dataset splits may be found at github.com/rcorona/modular_compositional_alfred.
2.1 Instruction-Based Controller
Our instruction-based controller is trained to segment a full instruction into sub-instructions and predict the subgoal type for each sub-instruction. We use a linear chain CRF (Lafferty et al., 2001) that conditions on a bidirectional-LSTM encoding of the full instruction and predicts tags for each word, which determine the segmentation and sequence of subgoal types. This model is based on standard neural segmentation and labelling models (Huang et al., 2015; Lample et al., 2016).
We train the controller on the ground-truth instruction segmentations and subgoal sequence labels, and in evaluation use the model to predict segmentations and their associated subgoal sequences (Figure 2, top left). This predicted sequence of sub-goals determines the order to execute the modules (Figure 2, right). We use a BIO chunking scheme to jointly segment the instruction and predict a subgoal label for each segment.
Formally, for a full instruction of length  $N$ , the controller defines a distribution over subgoal tags  $s_{1:N}$  for each word given the instruction  $x_{1:N}$  as
$$
p \left(s _ {1: N} \mid x _ {1: N}\right) \propto \exp \sum_ {n = 1} ^ {N} \left(U _ {s _ {n}} + B _ {s _ {n - 1}, s _ {n}}\right)
$$
The subgoal tag scores  $U_{s_n}$  for word  $n$  are given by a linear projection of bidirectional LSTM features for the word at position  $n$ . The tag transition scores  $B_{s_{n-1}, s_n}$  are learned scalar parameters.
In training, we supervise  $s_{1:N}$  using the segmentation of the instruction  $x_{1:N}$  into  $K$  subgoal instructions and the subgoal label for each instruction. To predict subgoals for a full instruction in evaluation, we obtain  $\arg \max_{s_{1:N}} p(s_{1:N} \mid x_{1:N})$  using Viterbi decoding, which provides a segmentation into sub-instructions and a subgoal label for each sub-instruction.
The controller obtains  $96\%$  exact match accuracy on subgoal sequences on validation data.
2.2 Module Architecture
Our modularized architecture may be seen in Figure 2, right. The architecture consists of 8 independent modules, one for each of the 8 subgoals in the domain (e.g. GOTo, PICKUP). For each module, we use the same architecture as Shridhar et al. (2020)'s monolithic model. This is a sequence-to-sequence model composed of an LSTM decoder taking as input an attended embedding of the natural language instruction, pretrained ResNet-18 (He
et al., 2016) features of the image observations, and the previous action's embedding. Hidden states are passed between the modules' LSTM decoders at subgoal transitions (Figure 2, right).
At each time step, each module  $M^i$  computes its hidden state based on the last time step's action  $a_{t-1}$ , the current time step's observed image features  $o_t$ , an attended language embedding  $\hat{x}_t^i$ , and the previous hidden state  $h_{t-1}^i$ :
$$
e _ {t} ^ {i} = [ a _ {t - 1}; o _ {t}; \hat {x} _ {t} ^ {i} ]
$$
$$
h _ {t} ^ {i} = \mathrm {L S T M} _ {i} (e _ {t} ^ {i}, h _ {t - 1} ^ {i})
$$
Each module's attended language embedding  $\hat{x}_t^i$  is produced using its own attention mechanism over embeddings  $X = x_{1:N}$  of the language instruction, which are produced by a bidirectional LSTM encoder:
$$
z _ {t} ^ {i} = \left(W _ {x} ^ {i} h _ {t - 1} ^ {i}\right) ^ {\top} X
$$
$$
\alpha_ {t} ^ {i} = \operatorname {S o f t m a x} \left(z _ {t} ^ {i}\right)
$$
$$
\hat {x} _ {t} ^ {i} = (\alpha_ {t} ^ {i}) ^ {\top} X
$$
Finally, the action  $a_{t}$  and object interaction mask  $m_{t}$  are predicted from  $h_{t}^{i}$  and  $e_{t}^{i}$  with a linear layer and a deconvolution network respectively. More details about this architecture can be found in Shridhar et al. (2020). Both the action and mask decoders, well as the language encoder, are shared across modules.
Our use of subgoal modules is similar to the hierarchical policy approaches of Andreas et al. (2017), Oh et al. (2017), and Das et al. (2018). However, in those approaches, the input to each module is symbolic (e.g. FIND[KITCHEN]). In contrast, all modules in our work condition directly on natural language.
2.3 Training
We first pre-train the monolithic model by maximizing the likelihood of the ground-truth trajectories in the training data (Shridhar et al., 2020). We train for up to 20 epochs using the Adam optimizer (Kingma and Ba, 2014) with early stopping on validation data (see Appendix A.1 for hyperparameters). We use this monolithic model to initialize the parameters of each of the modules, which have identical architecture to the monolithic model, and
2The modules' instruction encoder is separate from the controller's encoder (Sec. 2.1), as we found it possible to achieve high performance on the subgoal prediction task using a smaller encoder than the one used by the modules.
fine-tune them using the same training and early stopping procedure on the same validation data, $^{3}$  allowing the monolithic model's parameters to specialize for each module. Each module predicts only the actions for its segment of each trajectory; however, modules are jointly fine-tuned, passing hidden states (and gradients) from module to module.
3 Generalization Evaluation
We evaluate models on out-of-domain generalization in two conditions (see below) using the ALFRED benchmark (Shridhar et al., 2020), comparing our modular approach to their non-modular sequence-to-sequence model. ALFRED is implemented in AI2-THOR 2.0 (Kolve et al., 2017), which contains a set of simulated environments with realistic indoor scene renderings and object interactions.
The dataset contains approximately 25K expert instruction-trajectory pairs, comprised of 3 instructions for each of 8K unique trajectories. The instructions include both a high level instruction and a sequence of low level instructions. In our experiments, we do not use the high level instructions, which Shridhar et al. (2020) found to produce comparable results when evaluated on generalization to unseen environments with these architectures.
Figure 1 shows two example trajectories and their associated instructions. Trajectories are composed (see Sec. 2) of sequences of eight different types of subgoals: navigation (GOTO) and a variety of object interactions (e.g. PICKUP, CLEAN, HEAT). Each subgoal's subtrajectory is composed of a sequence of low-level discrete actions which specify commands for navigation or object interactions (which are accompanied by image segmentations to choose the object to interact with).
3.1 Generalization Conditions
The ALFRED dataset was constructed to test generalization to novel instructions and unseen environments. However, all evaluation trajectories in the dataset correspond to sequences of subgoals that are seen during training. For example, some training and evaluation instances might both correspond to the underlying subgoal sequence GoTo, PICKUP, GoTo, PUT, but differ in their low-level actions, their language descriptions, and possibly also the environments they are carried out in.
Novel Tasks. We evaluate models' ability to generalize to different high-level tasks (compositions of subgoals) than seen in training. The dataset contains seven different task types, such as Pick & Place, as described in Appendix B.1. We hold out two task types and evaluate models on their ability to generalize to them: Pick Two & Place and Stack & Place. These tasks are chosen because they contain subgoal types that are all individually seen in training, but typically in different sequences.
We create generalization splits pick-2-seen and pick-2-unseen by filtering the seen and unseen splits below to contain only Pick Two & Place tasks, and remove all Pick Two & Place tasks from the training data. We create splits stack-seen and stack-unseen for Stack & Place similarly.
Novel Instructions and Environments This is the standard condition defined in the original ALFRED dataset. There are two held-out validation sets: seen, which tests generalization to novel instructions and trajectories but through environments seen during training, and unseen, which tests generalization to novel environments: rooms with new layouts, object appearances, and furnishings.
We compare our modular architecture with the monolithic baseline, averaging performance over models trained from 3 random seeds. For each generalization condition, we measure success rates over full trajectories as well as over each subgoal type independently. Due to the challenging nature of the domain, subgoal evaluation provides finer-grained comparisons than full trajectories.
We use the same evaluation methods and metrics as in Shridhar et al. (2020). Success rates are weighted by path lengths to penalize successful trajectories which are longer than the ground-truth demonstration trajectory. To evaluate full trajectories, we measure path completion: the portion of subgoals completed within the full trajectories. To evaluate the subgoals independently, we advance the model along the expert trajectory up until the point where a given subgoal begins (to maintain a history of actions and observations), then require the model to carry out the subgoal from that point.
We also report results from Shridhar et al. (2020) and Singh et al. (2020). We note that the approach of Singh et al. (2020) obtains higher performance on full trajectories than the system of Shridhar et al. (2020) (which we base our approach on)
3Additionally, we append a special STOP action to the end of each module's action sequence so that it can predict when to give control back to the high-level controller.
Model	Clean	Cool	Goto	Heat	Pickup	Put	Slice	Toggle	Avg.
seen	S+	82	87	49	85	32	80	23	97	67
MOCA	79	87	54	84	53	62	51	93	70
Mono.	82	88*	52**	82**	37	81	34	98	69*
Mod.	82	86	43	80	41**	80	37	97	68
unseen	S+	21	94	21	88	20	51	14	54	45
MOCA	71	38	32	86	44	39	55	11	47
Mono.	28	90	23**	89**	25	48	25	42	46
Mod.	67**	94*	14	85	28**	55	39*	73*	57**
(a) Standard (novel environments) validation splits
Model	Coyo	Pickup	Par	Avg.
seen	Mono.	18*	21	48	29
Mod.	14	35*	63*	37*
unseen	Mono.	14	12	14	13
Mod.	14	25**	33**	24**
(b) Pick-2 task splits
Model	Goto	Pickup	Put	Avg.
seen	Mono.	28*	15	54	32
Mod.	21	27*	58*	35*
unseen	Mono.	19	7	25	17
Mod.	14	16*	28	19*
primarily by introducing a modular object interaction architecture (shared across all subgoals) and a pre-trained object segmentation model. These techniques could also be incorporated into our approach, which uses modular components for individual subgoal types.
Novel Tasks. Table 1 shows for each split the success rates on subgoals appearing in at least 50 validation examples. The modular outperforms the monolithic model on both seen and unseen splits (Tables 1b and 1c). Full trajectory results for novel task generalization are shown in Table 2. In the double generalization condition (unseen environments for the held-out pick-2 and stack tasks) on full trajectories, neither model completes subgoals successfully. Overall, we find that modularity helps across most generalization conditions.
Generalization to novel environments. We also compare models on generalization to unseen environments. In the independent subgoal evaluation, the monolithic and modular models perform equally on average in the standard-seen split (Ta
(c) Stack task splits
Table 1: Path weighted subgoal success percentages, by subgoal type, on the various generalization splits, and averaged across subgoal types (Avg.). We compare the performance of the monolithic (Mono.) model to our modular model (Mod.). The modular model generalizes better on average to unseen environments (standard-unseen) and to both seen and unseen environments for two held-out task types: Pick-2 and Stack. Bolded numbers show the best model between Mono and Modular, with * and ** denoting differences that are statistically significant at the  $p < 0.15$  and  $p < 0.05$  levels, respectively, by a one-tailed t-test. S+ gives results from Shridhar et al. (2020) and MOCA from Singh et al. (2020).
Model	Standard seen	Standard unseen	Pick-2 seen	Stack seen
S+	9.4 (5.7)	7.4 (4.7)	—	—
MOCA	28.5 (22.3)	13.4 (8.3)	—	—
Mono.	10.9 (7.0)	7.1 (4.9)	1.3 (1.6)	1.3 (0.3)
Mod.	9.1 (6.6)	7.0 (5.5)	1.5 (1.6)	1.5 (0.4)
Table 2: We compare performance of the monolithic and modular models on full trajectories, reporting the percentages of subgoals correctly completed. Numbers in parentheses weight these percentages by path length. S+ gives results from Shridhar et al. (2020), and MOCA from Singh et al. (2020).
ble 1a, top). However, in the standard-unseen split (Table 1a, bottom), our modular model outperforms the baseline substantially, with an average success rate of  $57\%$  compared to the monolithic model's  $46\%$ . (On subgoal types not shown, the modular model still outperforms the monolithic, by margins up to  $16\%$ .) In the full trajectory results (Table 2) we see comparable performance between the monolithic and modular models on unseen environments.
We introduced a novel modular architecture for grounded instruction following where each module is a sequence-to-sequence model conditioned on natural language instructions. With the ALFRED dataset as a testbed, we showed that our modular model achieves better out-of-domain generalization, generalizing better at the subgoal level to novel task compositions and unseen environments than the monolithic model used in prior work. All of the module types in our model currently use separate parameterizations but identical architectures; future work might leverage the modularity of our approach by using specialized architectures, training procedures, or loss functions for each subgoal type. Furthermore, unsupervised methods for jointly segmenting instructions and trajectories without requiring labeled subgoal labels and alignments would be a valuable addition to our framework.
This material is based upon work supported by the National Science Foundation Graduate Research Fellowship Program under Grant No. DGE 1752814, a Ford Foundation fellowship to the first author, a Google PhD fellowship to the second author, and by DARPA through the XAI program and the LwLL program.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sunderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. 2018. Vision- and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Jacob Andreas, Dan Klein, and Sergey Levine. 2017. Modular multitask reinforcement learning with policy sketches. In Proceedings of the International Conference on Machine Learning (ICML), pages 166-175.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Neural module networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 39-48.
Yoav Artzi and Luke Zettlemoyer. 2013. Weakly supervised learning of semantic parsers for mapping instructions to actions. Transactions of the Association for Computational Linguistics (TACL), 1(1):49-62.
Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron Courville. 2019. Systematic generalization: what is required and can it be learned? In Proceedings of the International Conference on Learning Representations (ICLR).
Valts Blukis, Yannick Terme, Eyvind Niklasson, Ross A. Knepper, and Yoav Artzi. 2020. Learning to map natural language instructions to physical quadcopter control using simulated flight. volume 100 of Proceedings of Machine Learning Research, pages 1415-1438. PMLR.
David L. Chen and Raymond J. Mooney. 2011. Learning to interpret natural language navigation instructions from observations. In Proceedings of the Conference on Artificial Intelligence (AAAI).
Howard Chen, Alane Shur, Dipendra Misra, Noah Snavely, and Yoav Artzi. 2019. Touchdown: Natural language navigation and spatial reasoning in visual street environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Volkan Cirik, Taylor Berg-Kirkpatrick, and Louis-Philippe Morency. 2018. Using syntax to ground referring expressions in natural images. In Proceedings of the Conference on Artificial Intelligence (AAAI).
Abhishek Das, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. 2018. Neural modular control for embodied question answering. In Proceedings of the Conference on Robot Learning (CoRL).