Curriculum Learning for Vision-and-Language Navigation

1 Introduction . Vision-and-Language ( VLN ) navigation task proposed recently by ( Anderson et al. , 2018 ) is a step towards building smart robots . It requires the agent to perceive the environment , understand human language instructions and finally unify the multi-modal information to make actions . Many stateof-the-art methods have been proposed . Some of them focus on the alignment between visual and textual inputs by improving model structure ( Ma et al. , 2019 ) or proposing novel auxiliary losses ( Zhu et al. , 2020 ) whereas others put their attention on the data augmentation ( Fried et al. , 2018 ; Tan et al. , 2019 ; Hong et al. , 2020 ; Ku et al. , 2020 ) . Large scale pre-training has also been employed to better generalize the downstream VLN tasks ( Li et al. , 2019 ; Majumdar et al. , 2020 ; Hao et al. , 2020 ; Huo et al. , 2021 ) . Despite great progress previous works have made , very few of them care about how much the agent learns from the dataset , i.e . is the agent a good student ? In computer vision , ( Hlynsson et al. , 2019 ) tries to answer this question by measuring the data efficiency — performance as a function of training set size — of deep learning methods . In vision-and-language navigation , ( Huang et al. , 2019 ) develops a discriminator that can filter the low quality instruction-path pairs to boost the learning efficiency . In this work , we focus on another aspect : can a VLN agent be further educated without model structure change and data modification ? We observe that many works neglect the internal distribution of the sample difficulty . For example , a navigation task within a single room should be considered easier than one that needs to travel two or more rooms . Current training methods simply flush in the data and do not distinguish difficulty levels among the training samples . As shown in Figure 1 ( a ) , navigation agents trained by such learning process do not perform well on those `` easy '' tasks even on the previously seen environments . We ∗Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . monitor the first error an agent makes during the navigation and demonstrate the ratio of different types of these errors in Figure 1 ( b ) . We find that when the navigation agent fails , about 50 % errors are caused by the agent wrongly predicts the next in-room direction . The ratio of this kind of error decreases as the navigation task spans more rooms , but still remains a relatively high level . These phenomenon indicates that the navigation agent is limited by its ability to navigate inside one room and cross two rooms . The poor performance of agents on those easy cases inspires us to borrow the idea from curriculum learning ( Bengio et al. , 2009 ) and propose a curriculum-based training paradigm . The basic idea of curriculum learning is to start small , learn easier aspects of the task and then gradually increase the difficulty level . The definition of `` difficulty '' is therefore worthwhile . We hypothesise that the difficulty of a navigation task is closely related to the rooms the agent needs to pass by towards the destination . On top of this , we introduce the curriculum design for VLN and re-arrange the benchmark Room-to-Room ( R2R ) dataset ( Anderson et al. , 2018 ) to make it suitable for curriculum learning ( Section 3 ) . We incorporate human prior knowledge about training samples into the training process of a navigation agent via self-paced curriculum learning ( SPCL ) ( Jiang et al. , 2015 ) . We adapt the traditional SPCL algorithm to efficiently train deep learning models ( Section 4 ) . Experiments show that our method can consistently improve both the navigation performance and the training efficiency of navigation agents ( Section 5 ) . In summary , our main contributions are : • We propose to explicitly incorporate human prior knowledge about training samples into navigation agent training process . • We design the curriculum for VLN and put forward the training paradigm of a navigation agent by curriculum learning without increasing the complexity of the model . • We empirically validate that the role of curriculum learning is to smooth loss landscape and hence find a better local optima . 2 Related Work . Vision-and-Language Navigation . Vision-and-language Navigation ( VLN ) is a task where an agent navigates to a goal location in a photo-realistic 3D environment under human instructions . ( Anderson et al. , 2018 ) formalized this task , proposed the benchmark Room-to-Room ( R2R ) dataset and set up an attention-based sequence-to-sequence baseline model . Other related VLN datasets include Touchdown dataset ( Chen et al. , 2019 ) , which is the first large-scale outdoor VLN dataset , and CVDN dataset ( Thomason et al. , 2019 ) that emphasizes on the robot-human dialogues during navigation . Embodied navigation tasks suffer from a limited size of training data , which results in several different research focuses . For data augmentation , ( Fried et al. , 2018 ) developed a speaker-follower model where the speaker model is usually used as a tool for instruction generation , and ( Tan et al. , 2019 ) proposed to apply a effective environmental dropout layer to mimic unseen environments during training . For better generalization , ( Wang et al. , 2018 ) introduced a hybrid model that integrated the model-based and model-free reinforcement learning whereas ( Wang et al. , 2019 ) proposed a Reinforced Cross-Modal ( RCM ) agent and a self-supervised imitation learning method to explore the unseen environment . Besides , ( Li et al. , 2019 ; Huo et al. , 2021 ; Hao et al. , 2020 ) applied pre-training skills towards better instruction understanding and generalization . Another resolution to the paucity of training data is rather straightforward : generating more training data and annotations . For example , ( Hong et al. , 2020 ) split the instructions in R2R dataset into sub-instructions and annotated the corresponding sub-paths . ( Ku et al. , 2020 ) proposed Room Across Room ( RxR ) dataset , a multilingual VLN dataset with dense spatiotemporal grounding . Furthermore , like ( Wang et al. , 2020 ) one can use multitask learning framework to grasp common knowledge from other homologous datasets so as to better the agent ’ s performance on the current task . These works are worthwhile but we observe that most of them ignore how much the model can learn from the dataset . In this paper , we focus to improve training methods to better exploit data . A better training paradigm can benefit both previous and future works on VLN tasks . Curriculum Learning . Curriculum Learning ( Bengio et al. , 2009 ) is a learning paradigm that mimic the learning principle underlying the cognitive process of humans and animals , where a model is learned by gradually including from easy to complex samples in training . ( Jiang et al. , 2015 ) bridged the gap between curriculum learning and self-paced learning by proposing a novel self-paced curriculum learning method . ( Graves et al. , 2017 ) proposed an automated curriculum learning method that can automatically select the curriculum syllabus . Later , ( Matiisen et al. , 2017 ) introduced a teacher-student framework for automatic curriculum learning , where the student tries to learn a complex task and the teacher automatically chooses the task for student to train on . Curriculum learning has been applied to several natural language processing tasks , such as question answering ( Sachan and Xing , 2016 ; Liu et al. , 2018 ) , machine translation ( Platanios et al. , 2019 ) and so on . An empirical work done by ( Hacohen and Weinshall , 2019 ) conclude that curriculum learning can effectively modify the optimization landscape and under mild conditions it does not change the corresponding global minimum of the optimization function . These works encourage us to apply the curriculum learning into the VLN tasks as it does not need any modification towards the navigation agent and is capable of improving the agent performance . To the best of our knowledge , this is the first work that introduces curriculum learning purely as a training paradigm for VLN tasks . Babywalk ( Zhu et al. , 2020 ) is another work that applies the curriculum learning idea . It aims to enhance the transfer ability of navigation agents . Therefore , Babywalk uses dynamic programming to decompose the long instructions into shorter ones and then applies these sub-instructions for curriculum-based reinforcement learning . It does not pay much attention to how an agent performs within a dataset . Babywalk is a carefully designed VLN agent with complex model structure whereas our method is a easy , extendable model-agnostic training method . Our method will change neither the difficulty of training data nor the model complexity . 3 Curriculum Design for VLN . We observe that different samples in the dataset have different navigation difficulties . Our intuition is that , for human beings it is easy to find an object or a place within a small range . After exploration , it is natural for us to exploit the knowledge about the environment and complete the harder cross-room tasks . Hence , we hypothese that the number of rooms a path could cover ( namely room length , R ( p ) ) dominates the difficulty level of a navigation task . We therefore propose to re-arrange the R2R dataset based on R ( p ) . Two examples with various R ( p ) are shown in Figure 2 . We firstly investigate the distributions of rooms that can be covered by a R2R path . Approximately 17 % of the paths only pass through less than two rooms whereas about 15 % of the paths span 5 rooms or more . We then propose to split the train set of R2R dataset into 5 mutually exclusive subsets and each subset contains paths whose R ( p ) are constrained . We believe that the learning from simple to difficult is very similar to play arcade games , so the subsets are named round 1 to 5 according to its difficulty . To be specific , round i ( i = 1 , 2 , 3 , 4 ) subset contains samples whose ground truth path covers i rooms . Round 5 subset contains the left samples . The basic statistics of the re-arranged R2R dataset is summarized in Table 1 . As the construction of such dataset incorporate human prior knowledge and such dataset will be futher used for curriculum learning , we call it Room-to-Room for Curriculum Learning ( CLR2R ) dataset . It should be noted that the definition of `` difficulty '' is open and task-specific , we throw a possibility here and encourage other attempts . Similar standards are easy to find in other datasets , for example , the number of dialogue rounds and the richness of dialogue phenomenon can both be taken as difficulty indicators on CVDN dataset ( Thomason et al. , 2019 ) . Finally , we leave validation and test set of R2R dataset unchanged so as to have comparable experimental results . 4 Methods . There are multiple settings of curriculum learning . When a series of tasks is presented , we can apply automated curriculum learning ( Graves et al. , 2017 ) where agents are trained by tasks adapted to their capacities . While for single task setting , self-paced curriculum learning ( Jiang et al. , 2015 ) is more appropriate if the task dataset has samples with various difficulty levels . Our observations better fit the latter mode , hence we adopt self-paced curriculum learning as the training method .

Summary: This paper applies self-paced curriculum learning [1] to VLN. It presents two approaches: a naive approach where the agent is trained on data with increasing coverage over the environment, and a learning-based approach that dynamically adapts the weights of learning examples according to the agent's learning progress. On three types of VLN agents, these approaches yield moderate improvement `(~2-3% improvement in success rate on validation unseen environments).