SlowGuess/ACL_2020_OCR huggingface dataset:
Wang Zhu\*1 Hexiang Hu\*2 Jiacheng Chen2 Zhiwei Deng3
Vihan Jain<sup>4</sup> Eugene Ie<sup>4</sup> Fei Sha<sup>†</sup>2,4
$^{1}$ Simon Fraser University  $^{2}$ University of Southern California  $^{3}$ Princeton University
Learning to follow instructions is of fundamental importance to autonomous agents for vision-and-language navigation (VLN). In this paper, we study how an agent can navigate long paths when learning from a corpus that consists of shorter ones. We show that existing state-of-the-art agents do not generalize well. To this end, we propose BabyWalk, a new VLN agent that is learned to navigate by decomposing long instructions into shorter ones (BabySteps) and completing them sequentially. A special design memory buffer is used by the agent to turn its past experiences into contexts for future steps. The learning process is composed of two phases. In the first phase, the agent uses imitation learning from demonstration to accomplish BabySteps. In the second phase, the agent uses curriculum-based reinforcement learning to maximize rewards on navigation tasks with increasingly longer instructions. We create two new benchmark datasets (of long navigation tasks) and use them in conjunction with existing ones to examine BabyWalk's generalization ability. Empirical results show that BabyWalk achieves state-of-the-art results on several metrics, in particular, is able to follow long instructions better. The codes and the datasets are released on our project page https://github.com/Sha-Lab/babywalk.
Autonomous agents such as household robots need to interact with the physical world in multiple modalities. As an example, in vision-and-language navigation (VLN) (Anderson et al., 2018), the agent moves around in a photo-realistic simulated environment (Chang et al., 2017) by following a sequence of natural language instructions. To infer its whereabouts so as to decide its moves, the
agent infuses its visual perception, its trajectory and the instructions (Fried et al., 2018; Anderson et al., 2018; Wang et al., 2019; Ma et al., 2019a,b).
Arguably, the ability to understand and follow the instructions is one of the most crucial skills to acquire by VLN agents. Jain et al. (2019) shows that the VLN agents trained on the originally proposed dataset ROOM2ROOM (i.e. R2R thereafter) do not follow the instructions, despite having achieved high success rates of reaching the navigation goals. They proposed two remedies: a new dataset ROOM4ROOM (or R4R) that doubles the path lengths in the R2R, and a new evaluation metric Coverage weighted by Length Score (CLS) that measures more closely whether the ground-truth paths are followed. They showed optimizing the fidelity of following instructions leads to agents with desirable behavior. Moreover, the long lengths in R4R are informative in identifying agents who score higher in such fidelity measure.
In this paper, we investigate another crucial aspect of following the instructions: can a VLN agent generalize to following longer instructions by learning from shorter ones? This aspect has important implication to real-world applications as collecting annotated long sequences of instructions and training on them can be costly. Thus, it is highly desirable to have this generalization ability. After all, it seems that humans can achieve this effortlessly<sup>1</sup>.
To this end, we have created several datasets of longer navigation tasks, inspired by R4R (Jain et al., 2019). We trained VLN agents on R4R and use the agents to navigate in ROOM6ROOM (i.e., R6R) and ROOM8ROOM (i.e., R8R). We contrast to the performance of the agents which are trained on those datasets directly ("in-domain"). The results
<sup>1</sup>Anecdotally, we do not have to learn from long navigation experiences. Instead, we extrapolate from our experiences of learning to navigate in shorter distances or smaller spaces (perhaps a skill we learn when we were babies or kids).
\*Author contributed equally.  $\dagger$  On leave from University of Southern California
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2539-2556 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 1: Performance of various VLN agents on generalizing from shorter navigation tasks to longer ones. The vertical axis is the newly proposed path-following metric SDTW (Magalhaes et al., 2019), the higher the better. BABYWALK generalizes better than other approaches across different lengths of navigation tasks. Meanwhile, it get very close to the performances of the in-domain agents (the dashed line). Please refer to the texts for details.
are shown in Fig. 1.
Our findings are that the agents trained on R4R (denoted by the purple and the pink solid lines) perform significantly worse than the in-domain agents (denoted the light blue dashed line). Also interestingly, when such out-of-domain agents are applied to the dataset R2R with shorter navigation tasks, they also perform significantly worse than the corresponding in-domain agent despite R4R containing many navigation paths from R2R. Note that the agent trained to optimize the aforementioned fidelity measure (RCM(fidelity)) performs better than the agent trained to reach the goal only (RCM(goal)), supporting the claim by Jain et al. (2019) that following instructions is a more meaningful objective than merely goal-reaching. Yet, the fidelity measure itself is not enough to enable the agent to transfer well to longer navigation tasks.
To address these deficiencies, we propose a new approach for VLN. The agent follows a long navigation instruction by decomposing the instruction into shorter ones ("micro-instructions", i.e., BABY-STEPs), each of which corresponds to an intermediate goal/task to be executed sequentially. To this end, the agent has three components: (a) a memory buffer that summarizes the agent's experiences so that the agent can use them to provide the context for executing the next BABY-STEP. (b) the agent first learns from human experts in "bitsize". Instead of trying to imitate to achieve the ground-truth paths as a whole, the agent is given
the pairs of a BABY-STEP and the corresponding human expert path so that it can learn policies of actions from shorter instructions. (c) In the second stage of learning, the agent refines the policies by curriculum-based reinforcement learning, where the agent is given increasingly longer navigation tasks to achieve. In particular, this curriculum design reflects our desiderata that the agent optimized on shorter tasks should generalize well to slightly longer tasks and then much longer ones.
While we do not claim that our approach faithfully simulates human learning of navigation, the design is loosely inspired by it. We name our approach BABYWALK and refer to the intermediate navigation goals in (b) as BABY-STEPS. Fig. 1 shows that BABYWALK (the red solid line) significantly outperforms other approaches and despite being out-of-domain, it even reach the performance of in-domain agents on R6R and R8R.
The effectiveness of BABYWALK also leads to an interesting twist. As mentioned before, one of the most important observations by Jain et al. (2019) is that the original VLN dataset R2R fails to reveal the difference between optimizing goal-reaching (thus ignoring the instructions) and optimizing the fidelity (thus adhering to the instructions). Yet, leaving details to section 5, we have also shown that applying BABYWALK to R2R can lead to equally strong performance on generalizing from shorter instructions (i.e., R2R) to longer ones.
In summary, in this paper, we have demonstrated empirically that the current VLN agents are ineffective in generalizing from learning on shorter navigation tasks to longer ones. We propose a new approach in addressing this important problem. We validate the approach with extensive benchmarks, including ablation studies to identify the effectiveness of various components in our approach.
Vision-and-Language Navigation (VLN) Recent works (Anderson et al., 2018; Thomason et al., 2019; Jain et al., 2019; Chen et al., 2019; Nguyen and Daumé III, 2019) extend the early works of instruction based navigation (Chen and Mooney, 2011; Kim and Mooney, 2013; Mei et al., 2016) to photo-realistic simulated environments. For instance, Anderson et al. (2018) proposed to learn a multi-modal Sequence-to-Sequence agent (Seq2Seq) by imitating expert demonstration. Fried et al. (2018) developed a method that augments the
paired instruction and demonstration data using a learned speaker model, to teach the navigation agent to better understand instructions. Wang et al. (2019) further applies reinforcement learning (RL) and self-imitation learning to improve navigation agents. Ma et al. (2019a,b) designed models that track the execution progress for a sequence of instructions using soft-attention.
Different from them, we focus on transferring an agent's performances on shorter tasks to longer ones. This leads to designs and learning schemes that improve generalization across datasets. We use a memory buffer to prevent mistakes in the distant past from exerting strong influence on the present. In imitation learning stage, we solve fine-grained subtasks (BABY-STEPS) instead of asking the agent to learn the navigation trajectory as a whole. We then use curriculum-based reinforcement learning by asking the agent to follow increasingly longer instructions.
Transfer and Cross-domain Adaptation There have been a large body of works in transfer learning and generalization across tasks and environments in both computer vision and reinforcement learning (Andreas et al., 2017; Oh et al., 2017; Zhu et al., 2017a,b; Sohn et al., 2018; Hu et al., 2018). Of particular relevance is the recent work on adapting VLN agents to changes in visual environments (Huang et al., 2019; Tan et al., 2019). To our best knowledge, this work is the first to focus on adapting to a simple aspect of language variability — the length of the instructions.
Curriculum Learning Since proposed in (Bengio et al., 2009), curriculum learning was successfully used in a range of tasks: training robots for goal reaching (Florensa et al., 2017), visual question answering (Mao et al., 2019), image generation (Karras et al., 2018). To our best knowledge, this work is the first to apply the idea to learning in VLN.
3 Notation and the Setup of VLN
In the VLN task, the agent receives a natural language instruction  $\mathrm{X}$  composed of a sequence of sentences. We model the agent with an Markov Decision Process (MDP) which is defined as a tuple of a state space  $S$ , an action space  $\mathcal{A}$ , an initial state  $s_1$ , a stationary transition dynamics  $\rho : S \times \mathcal{A} \to S$ , a reward function  $r: S \times \mathcal{A} \to \mathbb{R}$ , and the discount factor  $\gamma$  for weighting future rewards. The agent
acts according to a policy  $\pi : S \times \mathcal{A} \to 0 \cup R^{+}$ . The state and action spaces are defined the same as in (Fried et al., 2018) (cf. §4.4 for details).
For each X, the sequence of the pairs  $(s,a)$  is called a trajectory  $\mathrm{Y} = \{s_1,a_1,\dots ,s_{|\mathrm{Y}|},a_{|\mathrm{Y}|}\}$  where  $|\cdot |$  denotes the length of the sequence or the size of a set. We use  $\hat{\pmb{a}}$  to denote an action taken by the agent according to its policy. Hence,  $\hat{\mathrm{Y}}$  denotes the agent's trajectory, while Y (or  $\pmb{a}$ ) denotes the human expert's trajectory (or action). The agent is given training examples of (X, Y) to optimize its policy to maximize its expected rewards.
In our work, we introduce additional notations in the following. We will segment a (long) instruction X into multiple shorter sequences of sentences  $\{\pmb{x}_m,m = 1,2,\dots ,\mathrm{M}\}$ , to which we refer as BABY-STEPS. Each  $\pmb{x}_{m}$  is interpreted as a microinstruction that corresponds to a trajectory by the agent  $\hat{\pmb{y}}_m$  and is aligned with a part of the human expert's trajectory, denoted as  $\pmb{y}_{m}$ . While the alignment is not available in existing datasets for VLN, we will describe how to obtain them in a later section ( $\S 4.3$ ). Throughout the paper, we also freely interexchange the term "following the  $m$ th microinstruction", "executing the BABY-STEP  $\pmb{x}_{m}$ ", or "complete the  $m$ th subtask".
We use  $t \in [1, |Y|]$  to denote the (discrete) time steps the agent takes actions. Additionally, when the agent follows  $\pmb{x}_m$ , for convenience, we sometimes use  $t_m \in [1, |\hat{y}_m|]$  to index the time steps, instead of the "global time"  $t = t_m + \sum_{i=1}^{m-1} |\hat{y}_i|$ .
We describe in detail the 3 key elements in the design of our navigation agent: (i) a memory buffer for storing and recalling past experiences to provide contexts for the current navigation instruction ( $\S 4.1$ ); (ii) an imitation-learning stage of navigating with short instructions to accomplish a single BABY-STEP ( $\S 4.2.1$ ); (iii) a curriculum-based reinforcement learning phase where the agent learns with increasingly longer instructions (i.e. multiple BABY-STEPs) ( $\S 4.2.2$ ). We describe new benchmarks created for learning and evaluation and key implementation details in  $\S 4.3$  and  $\S 4.4$  (with more details in the Appendix).
4.1 The BABYWALK Agent
The basic operating model of our navigation agent BABYWALK is to follow a "micro instruction"  $\pmb{x}_m$  (i.e., a short sequence of instructions, to which we
Figure 2: The BABYWALK agent has a memory buffer storing its past experiences of instructions  $\boldsymbol{x}_m$ , and its trajectory  $\hat{\boldsymbol{y}}_m$ . When a new BABY-STEP  $\boldsymbol{x}_m$  is presented, the agent retrieves from the memory a summary of its experiences as the history context. It takes actions conditioning on the context (as well as its state  $\boldsymbol{s}_t$  and the previous action  $\hat{\boldsymbol{a}}_t$ ). Upon finishing following the instruction, the trajectory  $\hat{\boldsymbol{y}}_m$  is then sent to the memory to be remembered.
also refer as BABY-STEP), conditioning on the context  $\hat{z}_m$  and to output a trajectory  $\hat{y}_m$ . A schematic diagram is shown in Fig. 2. Of particularly different from previous approaches is the introduction of a novel memory module. We assume the BABY-STEPs are given in the training and inference time - § 4.3 explains how to obtain them if not given  $a$  prior (Readers can directly move to that section and return to this part afterwards). The left of the Fig. 3 gives an example of those micro-instructions.
Context The context is a summary of the past experiences of the agent, namely the previous  $(m - 1)$  mini-instructions and trajectories:
$$
\hat {\boldsymbol {z}} _ {m} = g \left(f _ {\text {S U M M A R Y}} \left(\boldsymbol {x} _ {1}, \dots , \boldsymbol {x} _ {m - 1}\right), \right.
$$
$$
f _ {\text {S U M M A R Y}} \left(\hat {\mathbf {y}} _ {1}, \dots , \hat {\mathbf {y}} _ {m - 1}\right) \tag {1}
$$
where the function  $g$  is implemented with a multi-layer perceptron. The summary function  $f_{SUMMARY}$  is explained in below.
Summary To map variable-length sequences (such as the trajectory and the instructions) to a single vector, we can use various mechanisms such as LSTM. We reported an ablation study on this in § 5.3. In the following, we describe the "forgetting" one that weighs more heavily towards the most recent experiences and performs the best empirically.
$$
f _ {\text {S U M M A R Y}} \left(\boldsymbol {x} _ {1}, \dots , \boldsymbol {x} _ {m - 1}\right) = \sum_ {i = 1} ^ {m - 1} \alpha_ {i} \cdot u \left(\boldsymbol {x} _ {i}\right) \tag {2}
$$
$$
f _ {\text {S U M M A R Y}} \left(\hat {\mathbf {y}} _ {1}, \dots , \hat {\mathbf {y}} _ {m - 1}\right) = \sum_ {i = 1} ^ {m - 1} \alpha_ {i} \cdot v \left(\hat {\mathbf {y}} _ {i}\right) \tag {3}
$$
where the weights are normalized to 1 and inverse proportional to how far  $i$  is from  $m$ ,
$$
\alpha_ {i} \propto \exp (- \gamma \cdot \omega (m - 1 - i)) \tag {4}
$$
$\gamma$  is a hyper-parameter (we set to  $1 / 2$ ) and  $\omega(\cdot)$  is a monotonically nondecreasing function and we simply choose the identity function.
Note that, we summarize over representations of "micro-instructions"  $(\pmb{x}_m)$  and experiences of executing those micro-instructions  $\hat{\pmb{y}}_m$ . The two encoders  $u(\cdot)$  and  $v(\cdot)$  are described in § 4.4. They are essentially the summaries of "low-level" details, i.e., representations of a sequence of words, or a sequence of states and actions. While existing work often directly summarizes all the low-level details, we have found that the current form of "hierarchical" summarizing (i.e., first summarizing each BABY-STEP, then summarizing all previous BABY-STEPs) performs better.
Policy The agent takes actions, conditioning on the context  $\hat{z}_m$ , and the current instruction  $x_m$ :
$$
\hat {\boldsymbol {a}} _ {t} \sim \pi (\cdot | \boldsymbol {s} _ {t}, \hat {\boldsymbol {a}} _ {t - 1}; u (\boldsymbol {x} _ {m}), \hat {\boldsymbol {z}} _ {m}) \tag {5}
$$
where the policy is implemented with a LSTM with the same cross-modal attention between visual states and languages as in (Fried et al., 2018).
4.2 Learning of the BABYWALK Agent
The agent learns in two phases. In the first one, imitation learning is used where the agent learns to execute BABY-STEPS accurately. In the second one, the agent learns to execute successively longer tasks from a designed curriculum.
4.2.1 Imitation Learning
BABY-STEPs are shorter navigation tasks. With the  $m$ th instruction  $\boldsymbol{x}_m$ , the agent is asked to follow the instruction so that its trajectory matches the human expert's  $\boldsymbol{y}_m$ . To assist the learning, the context is computed from the human expert trajectory up to the  $m$ th BABY-STEP (i.e., in eq. (1),  $\hat{\boldsymbol{y}}\mathbf{s}$  are replaced with  $\boldsymbol{y}\mathbf{s}$ ). We maximize the objective
$$
\ell = \sum_ {m = 1} ^ {\mathrm {M}} \sum_ {t _ {m} = 1} ^ {| \boldsymbol {y} _ {m} |} \log \pi (\boldsymbol {a} _ {t _ {m}} | \boldsymbol {s} _ {t _ {m}}, \boldsymbol {a} _ {t _ {m} - 1}; u (\boldsymbol {x} _ {m}), \boldsymbol {z} _ {m})
$$
We emphasize here each BABY-STEP is treated independently of the others in this learning regime. Each time a BABY-STEP is to be executed, we "preset" the agent in the human expert's context
Figure 3: Two-phase learning by BABYWALK. (Left) An example instruction-trajectory pair from the R4R dataset is shown. The long instruction is segmented into four BABY-STEP instructions. We use those BABY-STEPs for imitation learning (§ 4.2.1) (Right) Curriculum-based RL. The BABYWALK agent warm-starts from the imitation learning policy, and incrementally learns to handle longer tasks by executing consecutive BABY-STEPs and getting feedback from external rewards (c.f. § 4.2.2). We illustrate two initial RL lectures using the left example.
and the last visited state. We follow existing literature (Anderson et al., 2018; Fried et al., 2018) and use student-forcing based imitation learning, which uses agent's predicted action instead of the expert action for the trajectory rollout.
4.2.2 Curriculum Reinforcement Learning
We want the agent to be able to execute multiple consecutive BABY-STEPs and optimize its performance on following longer navigation instructions (instead of the cross-entropy losses from the imitation learning). However, there is a discrepancy between our goal of training the agent to cope with the uncertainty in a long instruction and the imitation learning agent's ability in accomplishing shorter tasks given the human annotated history. Thus it is challenging to directly optimize the agent with a typical RL learning procedure, even the imitation learning might have provided a good initialization for the policy, see our ablation study in § 5.3.
Inspired by the curriculum learning strategy (Bengio et al., 2009), we design an incremental learning process that the agent is presented with a curriculum of increasingly longer navigation tasks. Fig. 3 illustrates this idea with two "lectures". Given a long navigation instruction X with M BABY-STEPs, for the  $k$ th lecture, the agent is given all the human expert's trajectory up to but not including the  $(\mathrm{M} - k + 1)$ th BABY-STEP, as well as the history context  $z_{\mathrm{M} - k + 1}$ . The agent is then asked to execute the  $k$ th micro-instructions from  $x_{\mathrm{M} - k + 1}$  to  $x_{\mathrm{M}}$  using reinforcement learning to produce its trajectory that optimizes a task related
R2R	R4R	R6R	R8R
Train seen instr.	14,039	233,532	89,632	94,731
Val unseen instr.	2,349	45,234	35,777	43,273
Avg instr. length	29.4	58.4	91.2	121.6
Avg # BABY-STEPS	1.8	3.6	5.6	7.4
Table 1: Datasets used for VLN learning and evaluation
Figure 4: The distribution of lengths of instructions and ground-truth trajectories in our datasets.
metric, for instance the fidelity metric measuring how faithful the agent follows the instructions.
As we increase  $k$  from 1 to  $\mathbb{M}$ , the agent faces the challenge of navigating longer and longer tasks with reinforcement learning. However, the agent only needs to improve its skills from its prior exposure to shorter ones. Our ablation studies show this is indeed a highly effective strategy.
4.3 New Datasets for Evaluation & Learning
To our best knowledge, this is the first work studying how well VLN agents generalize to long navigation tasks. To this end, we create the following datasets in the same style as in (Jain et al., 2019).
ROOM6ROOM and ROOM8ROOM We concatenate the trajectories in the training as well as the validation unseen split of the ROOM2ROOM dataset for 3 times and 4 times respectively, thus extending the lengths of navigation tasks to 6 rooms and 8 rooms. To join, the end of the former trajectory must be within 0.5 meter with the beginning of the later trajectory. Table 1 and Fig. 4 contrast the different datasets in the # of instructions, the average length (in words) of instructions and how the distributions vary.
Table 1 summarizes the descriptive statistics of BABY-STEPs across all datasets used in this paper. The datasets and the segmentation/alignments are made publicly available<sup>2</sup>.
4.4 Key Implementation Details
In the following, we describe key information for research reproducibility, while the complete details are in the Appendix.
States and Actions We follow (Fried et al., 2018) to set up the states as the visual features (i.e. ResNet-152 features (He et al., 2016)) from the agent-centric panoramic views in 12 headings  $\times 3$  elevations with 30 degree intervals. Likewise, we use the same panoramic action space.
Identifying BABY-STEPS Our learning approach requires an agent to follow microinstructions (i.e., the BABY-STEPS). Existing datasets (Anderson et al., 2018; Jain et al., 2019; Chen et al., 2019) do not provide fine-grained segmentations of long instructions. Therefore, we use a template matching approach to aggregate consecutive sentences into BABY-STEPS. First, we extract the noun phrase using POS tagging. Then, we employs heuristic rules to chunk a long instruction into shorter segments according to punctuation and landmark phrase (i.e., words for concrete objects). We document the details in the Appendix.
Aligning BABY-STEPs with Expert Trajectory Without extra annotation, we propose a method to approximately chunk original expert trajectories into sub-trajectories that align with the BABY-STEPs. This is important for imitation learning at the micro-instruction level (§ 4.2.1). Specifically, we learn a multi-label visual landmark classifier to identify concrete objects from the states along expert trajectories by using the landmark phrases
extracted from the their instructions as weak supervision. For each trajectory-instruction pair, we then extract the visual landmarks of every state as well as the landmark phrases in BABY-STEP instructions. Next, we perform a dynamic programming procedure to segment the expert trajectories by aligning the visual landmarks and landmark phrases, using the confidence scores of the multi-label visual landmark classifier to form the function.