Curriculum Learning for Vision-and-Language Navigation
Year: 2022

Abstract:
Vision-and-Language Navigation (VLN) is a task where an agent navigates in an embodied indoor environment under human instructions. Previous works ignore the distribution of sample difficulty and we argue that this potentially degrade their agent performance. To tackle this issue, we propose a novel curriculumbased training paradigm for VLN tasks that can balance human prior knowledge and agent learning progress about training samples. We develop the principle of curriculum design and re-arrange the benchmark Room-to-Room (R2R) dataset to make it suitable for curriculum training. Experiments show that our method is model-agnostic and can significantly improve the performance, the generalizability, and the training efficiency of current state-of-the-art navigation agents without increasing model complexity.

heading: 1 Introduction
text: Vision-and-Language (VLN) navigation task proposed recently by (Anderson et al., 2018) is a step towards building smart robots. It requires the agent to perceive the environment, understand human language instructions and finally unify the multi-modal information to make actions. Many stateof-the-art methods have been proposed. Some of them focus on the alignment between visual and textual inputs by improving model structure (Ma et al., 2019) or proposing novel auxiliary losses (Zhu et al., 2020) whereas others put their attention on the data augmentation (Fried et al., 2018; Tan et al., 2019; Hong et al., 2020; Ku et al., 2020). Large scale pre-training has also been employed to better generalize the downstream VLN tasks (Li et al., 2019; Majumdar et al., 2020; Hao et al., 2020; Huo et al., 2021).
Despite great progress previous works have made, very few of them care about how much the agent learns from the dataset, i.e. is the agent a good student? In computer vision, (Hlynsson et al., 2019) tries to answer this question by measuring the data efficiency — performance as a function of training set size — of deep learning methods. In vision-and-language navigation, (Huang et al., 2019) develops a discriminator that can filter the low quality instruction-path pairs to boost the learning efficiency. In this work, we focus on another aspect: can a VLN agent be further educated without model structure change and data modification?
We observe that many works neglect the internal distribution of the sample difficulty. For example, a navigation task within a single room should be considered easier than one that needs to travel two or more rooms. Current training methods simply flush in the data and do not distinguish difficulty levels among the training samples. As shown in Figure 1(a), navigation agents trained by such learning process do not perform well on those "easy" tasks even on the previously seen environments. We
∗Corresponding author.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
monitor the first error an agent makes during the navigation and demonstrate the ratio of different types of these errors in Figure 1(b). We find that when the navigation agent fails, about 50% errors are caused by the agent wrongly predicts the next in-room direction. The ratio of this kind of error decreases as the navigation task spans more rooms, but still remains a relatively high level. These phenomenon indicates that the navigation agent is limited by its ability to navigate inside one room and cross two rooms.
The poor performance of agents on those easy cases inspires us to borrow the idea from curriculum learning (Bengio et al., 2009) and propose a curriculum-based training paradigm. The basic idea of curriculum learning is to start small, learn easier aspects of the task and then gradually increase the difficulty level. The definition of "difficulty" is therefore worthwhile. We hypothesise that the difficulty of a navigation task is closely related to the rooms the agent needs to pass by towards the destination. On top of this, we introduce the curriculum design for VLN and re-arrange the benchmark Room-to-Room (R2R) dataset (Anderson et al., 2018) to make it suitable for curriculum learning (Section 3). We incorporate human prior knowledge about training samples into the training process of a navigation agent via self-paced curriculum learning (SPCL) (Jiang et al., 2015). We adapt the traditional SPCL algorithm to efficiently train deep learning models (Section 4). Experiments show that our method can consistently improve both the navigation performance and the training efficiency of navigation agents (Section 5).
In summary, our main contributions are:
• We propose to explicitly incorporate human prior knowledge about training samples into navigation agent training process.
• We design the curriculum for VLN and put forward the training paradigm of a navigation agent by curriculum learning without increasing the complexity of the model.
• We empirically validate that the role of curriculum learning is to smooth loss landscape and hence find a better local optima.

heading: 2 Related Work
text: Vision-and-Language Navigation. Vision-and-language Navigation (VLN) is a task where an agent navigates to a goal location in a photo-realistic 3D environment under human instructions. (Anderson et al., 2018) formalized this task, proposed the benchmark Room-to-Room (R2R) dataset and set up an attention-based sequence-to-sequence baseline model. Other related VLN datasets include Touchdown dataset (Chen et al., 2019), which is the first large-scale outdoor VLN dataset,
and CVDN dataset (Thomason et al., 2019) that emphasizes on the robot-human dialogues during navigation.
Embodied navigation tasks suffer from a limited size of training data, which results in several different research focuses. For data augmentation, (Fried et al., 2018) developed a speaker-follower model where the speaker model is usually used as a tool for instruction generation, and (Tan et al., 2019) proposed to apply a effective environmental dropout layer to mimic unseen environments during training. For better generalization, (Wang et al., 2018) introduced a hybrid model that integrated the model-based and model-free reinforcement learning whereas (Wang et al., 2019) proposed a Reinforced Cross-Modal (RCM) agent and a self-supervised imitation learning method to explore the unseen environment. Besides, (Li et al., 2019; Huo et al., 2021; Hao et al., 2020) applied pre-training skills towards better instruction understanding and generalization.
Another resolution to the paucity of training data is rather straightforward: generating more training data and annotations. For example, (Hong et al., 2020) split the instructions in R2R dataset into sub-instructions and annotated the corresponding sub-paths. (Ku et al., 2020) proposed Room Across Room (RxR) dataset, a multilingual VLN dataset with dense spatiotemporal grounding. Furthermore, like (Wang et al., 2020) one can use multitask learning framework to grasp common knowledge from other homologous datasets so as to better the agent’s performance on the current task.
These works are worthwhile but we observe that most of them ignore how much the model can learn from the dataset. In this paper, we focus to improve training methods to better exploit data. A better training paradigm can benefit both previous and future works on VLN tasks.
Curriculum Learning. Curriculum Learning (Bengio et al., 2009) is a learning paradigm that mimic the learning principle underlying the cognitive process of humans and animals, where a model is learned by gradually including from easy to complex samples in training. (Jiang et al., 2015) bridged the gap between curriculum learning and self-paced learning by proposing a novel self-paced curriculum learning method. (Graves et al., 2017) proposed an automated curriculum learning method that can automatically select the curriculum syllabus. Later, (Matiisen et al., 2017) introduced a teacher-student framework for automatic curriculum learning, where the student tries to learn a complex task and the teacher automatically chooses the task for student to train on.
Curriculum learning has been applied to several natural language processing tasks, such as question answering (Sachan and Xing, 2016; Liu et al., 2018), machine translation (Platanios et al., 2019) and so on. An empirical work done by (Hacohen and Weinshall, 2019) conclude that curriculum learning can effectively modify the optimization landscape and under mild conditions it does not change the corresponding global minimum of the optimization function. These works encourage us to apply the curriculum learning into the VLN tasks as it does not need any modification towards the navigation agent and is capable of improving the agent performance. To the best of our knowledge, this is the first work that introduces curriculum learning purely as a training paradigm for VLN tasks.
Babywalk (Zhu et al., 2020) is another work that applies the curriculum learning idea. It aims to enhance the transfer ability of navigation agents. Therefore, Babywalk uses dynamic programming to decompose the long instructions into shorter ones and then applies these sub-instructions for curriculum-based reinforcement learning. It does not pay much attention to how an agent performs within a dataset. Babywalk is a carefully designed VLN agent with complex model structure whereas our method is a easy, extendable model-agnostic training method. Our method will change neither the difficulty of training data nor the model complexity.

heading: 3 Curriculum Design for VLN
text: We observe that different samples in the dataset have different navigation difficulties. Our intuition is that, for human beings it is easy to find an object or a place within a small range. After exploration, it is natural for us to exploit the knowledge about the environment and complete the harder cross-room tasks. Hence, we hypothese that the number of rooms a path could cover (namely room length, R(p)) dominates the difficulty level of a navigation task. We therefore propose to re-arrange the R2R dataset based on R(p). Two examples with various R(p) are shown in Figure 2.
We firstly investigate the distributions of rooms that can be covered by a R2R path. Approximately 17% of the paths only pass through less than two rooms whereas about 15% of the paths span 5 rooms
or more. We then propose to split the train set of R2R dataset into 5 mutually exclusive subsets and each subset contains paths whose R(p) are constrained. We believe that the learning from simple to difficult is very similar to play arcade games, so the subsets are named round 1 to 5 according to its difficulty. To be specific, round i (i = 1, 2, 3, 4) subset contains samples whose ground truth path covers i rooms. Round 5 subset contains the left samples.
The basic statistics of the re-arranged R2R dataset is summarized in Table 1. As the construction of such dataset incorporate human prior knowledge and such dataset will be futher used for curriculum learning, we call it Room-to-Room for Curriculum Learning (CLR2R) dataset.
It should be noted that the definition of "difficulty" is open and task-specific, we throw a possibility here and encourage other attempts. Similar standards are easy to find in other datasets, for example, the number of dialogue rounds and the richness of dialogue phenomenon can both be taken as difficulty indicators on CVDN dataset(Thomason et al., 2019). Finally, we leave validation and test set of R2R dataset unchanged so as to have comparable experimental results.

heading: 4 Methods
text: There are multiple settings of curriculum learning. When a series of tasks is presented, we can apply automated curriculum learning (Graves et al., 2017) where agents are trained by tasks adapted to their capacities. While for single task setting, self-paced curriculum learning (Jiang et al., 2015) is more appropriate if the task dataset has samples with various difficulty levels. Our observations better fit the latter mode, hence we adopt self-paced curriculum learning as the training method.

heading: 4.1 General Setup
text: Self-paced curriculum learning (SPCL) is an “instructor-student collaborative” learning method as it considers both prior knowledge known before training and information learning during training in a
unified framework. To be specific, the objective loss function of SPCL is defined as
min w∈[0,1]n,θ∈Θ E(w,θ;λ,Ψ) = n∑ i=1 wiL (yi, f (xi,θ)) + g(w;λ)
s.t. w ∈ Φ (1)
where f (xi,θ) denotes the navigation agent, w = [w1, · · · , wn]T ∈ [0, 1]n is the weight variable reflecting the sample importance. g is called the self-paced function which controls the learning scheme, and λ is a hyper-parameter that limits the learning pace. Φ is a feasible region that encodes the information of a predetermined curriculum, defined below.
Definition 4.1 For training samplesX = {xi}ni=1, given a curriculum γ defined on it, the feasible region, defined by,
Φ = { w | aTw ≤ c } is a curriculum region of γ if it holds: 1) Φ ∧ w ∈ [0, 1]n is nonempty; 2) ai < aj for all γ (xi) < γ (xj) ; ai = aj for all γ (xi) = γ (xj) .
where vector a ∈ Rn that parameterizes a linear space is a function of curriculum γ. It is not hard to see that, prior human knowledge influence the feasible region Φ which constrains the weight parameter w. Hence, the update of w is subject to the learning progress L, self-paced function g and feasible region Φ. Based on our curriculum design, samples within each round should be given the same curriculum rank. Therefore, 5 scalars is enough to define a good vector a.
Compared with self-paced learning (SPL) (Kumar et al., 2010), SPCL is more general by introducing self-paced function g as a flexible regularization term. Self-paced functions are defined to be convex, which ensures that we can find good solutions of w within the linear curriculum region. Following (Jiang et al., 2014)’s work, (Jiang et al., 2015) further discussed several examples of self-paced functions. In this paper, we focus on the simplest two: (1) Binary scheme and (2) Linear scheme. As summarized in Table 2, the convex optimization problem
w∗ = argminw∈Φ ∑ wi`i + g(w;λ) (2)
have the closed form solution if Φ = [0, 1]n. Therefore, to solve the linear constrained convex optimization problem, we can simply apply a projection gradient descent method to obtain the optimal weight w∗.

heading: 4.2 Algorithm
text: (Jiang et al., 2015) proposed an alternative convex search (ACS) (M. Bazaraa and Shetty, 1993) algorithm to solve Equation 1. The main problem of the original algorithm is at step 4, where the optimal model parameter θ∗ is learned with most recent weight vectorw∗ fixed. In the training of navigation agents, it is impossible to compute the exact optimum of E(w∗,θ, λ,Ψ) due to both the time consumption and the lack of global convergence guarantees. We propose not to compute the exact minimum but replace step 4 with several gradient descent update steps. Doing so benefits the algorithm as the training is much faster then before.
According to (Gorski et al., 2007), the original SPCL algorithm proposed by (Jiang et al., 2015) is guaranteed to converge globally as the objective function is monotonically decreasing and bounded below. Our algorithm does not have the monotonically decreasing property as the stochastic gradient descent (SGD) update can not assure the continuous decrease of the function value. Alternatively,
our algorithm can be viewed as a naive version of mini-batch randomized block coordinate descent (MRBCD) method. (Zhao et al., 2014) claimed that MRBCD using semi-stochastic optimization scheme can attain linear rates of convergence, while the convergence of naive MRBCD is still left to be discussed. Further experiments show that our algorithm is empirically converged, which encourages the theoretical analysis in this direction.
Algorithm 1 takes the input of a predetermined curriculum γ, an instantiated self-paced function g, a stepsize parameter µ and an update interval T ; it outputs an optimal model parameter θ.
Algorithm 1 Self-paced Curriculum Learning Input: Input dataset D, predetermined curriculum γ, self-paced function g, stepsize µ and update
interval T . Output: Model paramater θ. 1: Derive the curriculum region Φ from γ. 2: Initialize w∗ and λ in the curriculum region. 3: while not converged do 4: for t = 0, 1, · · · , T − 1 do 5: Update θt+1 = SGD(E(w∗,θt;λ,Φ)) 6: end for 7: Record θ∗ = θT 8: Update w∗ = argminw E(w,θ
∗;λ,Φ) 9: If λ is small, then increase λ by stepsize µ.
10: end while 11: return θ∗.

heading: 5 Experiment Results and Analysis

heading: 5.1 Experiment Setup
text: Navigation agents. We experiment with three state-of-the-art VLN agents using different training paradigms and compare their performance on the original R2R validation set. We choose several state-of-the-art navigation agents including the Speaker-Follower agent (Fried et al., 2018) which applies the panoramic action space, the Self-Monitoring agent (Ma et al., 2019) which enforces crossmodel alignment and EnvDrop with Back Translation (Tan et al., 2019) that applies reinforcement learning. We reproduce these agents in a unified code framework and replicate the experimental results presented in original papers. We do not use any data augmentation tricks and all three agents are trained within the R2R dataset range.
Implementation details. We experiment with three training paradigms. They include the traditional machine learning (ML) strategy that training by uniformly sample mini-batches from R2R dataset, a naive curriculum learning (NCL) strategy that the agent is firstly trained on CLR2R round 1 split, then round 1~2 splits, and gradually trained on the whole CLR2R train set, and previously introduced self-paced curriculum learning (SPCL) strategy. As discussed in Section 4, we set ai = i for samples in CLR2R round i split. The constant c is chosen within range [0.95 ∗ ‖a‖1, ‖a‖1]. For self-paced functions, we choose the binary and linear scheme. For EnvDrop agent (Tan et al., 2019) which is trained by a mixed loss of imitation learning and reinforcement learning, we only use the ground truth trajectory-based loss to update the weight variable. More information is available in supplementary materials.
Evaluation metrics. We follow the standard metrics that (Fried et al., 2018) employed for evaluating the agent’s performance, including average Navigation Error (NE) which measures the distance between the end location and the target location, Success Rate (SR) which is the percentage of predicted end location within 3m of the target location, and Oracle Success Rate (OSR) which is the percentage of trajectory the shortest distance between which and the target location is within 3m. We also adopt success weighted by path length (SPL) as recommended by (Anderson et al., 2018). Furthermore, we consider the coverage weighted by length score (CLS) (Jain et al., 2019) which measures the overall correspondence between the predicted and ground truth trajectories. Actually,
nDTW and SDTW proposed by (Magalhães et al., 2019) are both useful metrics that capture path fidelity, hence we supplement a full-metric version of main results in supplementary materials.

heading: 5.2 Overall Performance
text: We first compare the performance of the different training paradigms on the CLR2R(R2R) validation set. Table 3 shows overall results. Experiments show that naïve curriculum learning is surprisingly effective. It improves the navigation outcomes of Follower and Self-Monitoring agent on validation unseen split. In contrast to machine learning where the whole training set is available, naïve curriculum learning exposes the training samples sequentially from easy to hard. Hence, follower agent trained by NCL is underperformed on the validation seen split due to its limited learning ability. For self-paced curriculum learning, we initialize the algorithm to pay more attention on round 1 and 2 splits, then allow it actively assigns higher weight for examples which the agent learns better. During training, the weight for each sample can finally converge nearly to one. Experiments show that agents trained by SPCL consistently achieve good performance on both validation seen and unseen split.
Furthermore, we compare the learning speed of agents trained by different training paradigms. As shown in Figure 3, there is obviously a gap between agents trained by machine learning and selfpaced curriculum learning. Agents trained by curriculum learning can learn fast and achieve better performance after the same number of iterations. This indicates that self-paced curriculum learning can not only improve the performance, but also optimize the training efficiency of the agents.

heading: 5.3 Further Analysis
text: We also investigate the hyperparameter robustness of self-paced curriculum learning. Moreover, we present the transfer learning outcomes of curriculum learning method on CLR2R and RxR dataset.
Hyperparameter robustness. To understand how the weight initialization and stepsize choice influence the training paradigm, we grid search these two hyperparameters and the result is shown in Figure 4. Following the core idea of curriculum learning by starting small, we enforce the initial weight for round 1 and 2 samples as one. Then w0 represents the initial weight for samples in round 3~5 splits. As shown in Figure 4, SPCL favors a smaller w0. This validates the principle of curriculum learning. Besides, SPCL is rather robust to weight initialization and the choice of stepsize as in most cases, the results in validation unseen split are better than the machine learning baseline. This also indicates a better generalizability of agents trained by SPCL. For validation seen split, the performance is usually no worse than the baseline. The overall outcomes prove that curriculum learning can improve the data exploitation efficiency and enhance the generalization ability.
Loss landscape. Following (Santurkar et al., 2018), we investigate the loss landscape during training by computing the distance between maximum and minimum loss. As shown in Figure 5, SPCL narrows the loss gap for Follower and Self-Monitoring agent. For EnvDrop agent, SPCL differs little from ML. We attribute this to the mixed learning strategy of EnvDrop, whose loss is a weighted sum of imitation learning(0.2) and reinforcement learning(0.8). Hence, the power of SPCL is not as
significant as in other agents. In general, our experimental results correspond to the theoretical result (Hacohen and Weinshall, 2019) that curriculum learning can effectively smooth the optimization landscape.
Transfer learning. To explore whether curriculum learning helps the agent to obtain better transferring ability, we conduct the following experiment as summarized in Table 5. As mentioned in Section 2, RxR dataset (Ku et al., 2020) is a large-scale multilingual VLN dataset built upon Matterport3D simulator (Chang et al., 2017). We select the RxR dataset as the target dataset of transfer learning since it shares the same environment with R2R dataset but is much harder in many aspects (see Table 4). As RxR dataset is multilingual, we employ its english subset, RxR-en, to avoid the language bias. For fairly comparison, the number of training iterations is fixed at 80,000 for line (3)~(5) in table. Specifically, for line (3) the agent is trained on a integrated R2R+RxR-en dataset for 80,000 iterations; for line (4), the agent is firstly trained on R2R trainset for 40,000 iterations and then transferred to RxR-en dataset for another 40,000 iterations; for line (5), the agent is firstly trained on R2R trainset and then trained on the integrated R2R+RxR-en dataset for another 40,000 iterations.
Line (1) and (2) indicates that transfer learning (R2R→RxR and vice-versa) performance is much weaker than the in-domain results. For line (3), the agent trained on an integrated dataset performs better on RxR-en validation split, but its performance on R2R validation split it not as good as line (1). The difficulty gap between samples from two data source confuses the agent. Line (3) and (4) indicates that starting from simple improves the agents performance on RxR dataset. Compared with line (4), curriculum learning in line (5) helps the agent to preserve its navigation ability on R2R dataset. Actually, line (4) can also be taken as a curriculum learning experiment whose target distribution is RxR dataset instead of R2R and RxR dataset.
This exploration experiment suggests that to transfer from an easier dataset towards a harder one, curriculum learning paradigm helps the agent to improve the performance on the harder dataset without degrading the performance on the easier dataset.
Combination with Pre-training. Pre-training methods are proved strong in many areas. We believe that navigation agents trained by our proposed curriculum training paradigm can also benefit from pre-training skills. That is, curriculum learning does not conflict with pre-training. To validate this, we combine agents trained by curriculum based methods with VLN-BERT (Majumdar et al., 2020), a visiolinguistic transformer-based model. VLN-BERT is a scoring function that evaluates the compatibility between path-instruction pairs, hence it can be easily applied to assist our navigation agents. Since VLN-BERT has to be used under beam search settings , we implement beam search only here. In (Majumdar et al., 2020), authors use a beam size as 30 to include as much path candidates as possible. While in this experiment, we aim to evaluate the usefulnees of combining curriculum and pre-training. Therefore, we restrict the beam size as 5 and purely use VLN-BERT model to scoring
and selecting the path-instruction pairs. We experiment with Follower agent under three different test modes, i.e. single-run, beam search with size 1, and vln-bert. Results are shown in Figure 6. Beam search and VLN-BERT scorer both can improve the agent performance. Navigation agents trained by curriculum-based methods obtain more improvements.

heading: 6 Conclusion and Future Works
text: We propose to apply curriculum learning as a useful training paradigm of VLN agents. We adapt the traditional self-paced curriculum learning and design the first curriculum for VLN task based on R2R dataset. Experiments illustrate that our method is model-agnostic and can improve both the performance and learning efficiency of the navigation agents. We verify the power of curriculum learning comes from its ability to smooth the loss landscape. Our further exploration indicates that curriculum learning is suitable for transfer learning and combination with pre-training methods.
In the future, we would like to explore the VLN task in two directions. Along the path of this paper, we would like to explore curriculum learning for vision-language navigation in a different scenario of city traveling where the agent needs to perform multi-scale navigation, i.e. city-level and building-level. Alternatively, inspired by the ampleness of VLN tasks and datasets, i.e. R2R (Anderson et al., 2018), CVDN (Thomason et al., 2019), HANNA (Nguyen and Daumé, 2019), TOUCHDOWN (Chen et al., 2019), REVERIE (Qi et al., 2020) and so on, it is possible to apply meta-learning on top of these datasets and get an adaptable navigation agent.

heading: Acknowledgements
text: This work is partially supported by Natural Science Foundation of China (No.71991471, No.6217020551), Science and Technology Commission of Shanghai Municipality Grant (No.20dz1200600, 21QA1400600) and Zhejiang Lab (No. 2019KD0AD01).

Review summaries:
This paper applies self-paced curriculum learning [1] to VLN. It presents two approaches: a naive approach where the agent is trained on data with increasing coverage over the environment, and a learning-based approach that dynamically adapts the weights of learning examples according to the agent's learning progress. On three types of VLN agents, these approaches yield moderate improvement `(~2-3% improvement in success rate on validation unseen environments). 

The paper proposes a curriculum learning (CL) based approach for the task of Vision-Language Navigation (VLN). The dataset chosen for experimentation is R2R which involves an agent to navigate from room to room to follow the provided instruction. The proposed method creates a curriculum for the learning agent based on the number of rooms a given path covers. The empirical results are reported using two methods -- (1) naive CL where the agent is trained on gradually harder tasks, (2) self paced curriculum learning (SPCL).  

This paper proposes a curriculum-based training paradigm for Vision-and-Language Navigation tasks. They focus on the aspect of whether the performance of a VLN agent can be improved without model structure change and data modification. They define the difficulty of the episode based on how many rooms it needs to cross and use this to divide the dataset into 5 parts. These parts are then used to form a curriculum learning training schedule. They use Self-Paced Curriculum Learning, and show that their method improves both the navigation performance and the training efficiency for the VLN task (R2R).