SOAT: A Scene- and Object-Aware Transformer for Vision-and-Language Navigation

1 Introduction . The vision-and-language navigation ( VLN ) task [ 1 ] requires an agent to follow a path through an environment that is specified by natural language navigation instructions . A central component of this task is associating ( or grounding ) the instruction to visual landmarks in the environment . Figure 1 provides an illustrative example from the Room-to-Room ( R2R ) dataset [ 1 ] : ‘ Exit the bedroom and turn left . Continue down the hall and into the room straight ahead and stop before the desk with two green chairs. ’ The visual landmarks in this instruction include scene descriptions ( e.g. , ‘ bedroom ’ and ‘ hall ’ ) and specific object references ( e.g. , ‘ desk ’ and ‘ two green chairs ’ ) . To be successful , a VLN agent should be able to ( a ) recognize and ( b ) ground both types of visual cues . Learning the appropriate grounding between referring expressions in an instruction and the corresponding visual regions is difficult in VLN due to the limited visual diversity seen in training . For example , the Room-to-Room ( R2R ) [ 1 ] and Room-Across-Room ( RxR ) [ 2 ] datasets only use 61 unique training environments , so models simply can not learn about the long-tail of visual cues that appear in new testing ( or validation ) scenes . To address this challenge , recent work has shown the promise of transferring visual grounding with multimodal representation models that are pretrained on a large amount of image-text web data before finetuning on the embodied VLN task [ 3 , 4 ] . In this work , we build on this general approach . For visual recognition , most VLN methods [ 1 , 4–14 ] first encode observations with a convolutional network that was trained to solve an image-level classification task – either using ImageNet [ 15 ] or the Places [ 16 ] scene recognition dataset . While ImageNet features may identify objects mentioned in the instructions and Places features might match the scene descriptions , neither solution was ˚Correspondence to abhinavmoudgil95 @ gmail.com 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . explicitly trained to recognize both types of visual cues , which is a limitation that we address in this paper . Furthermore , pretrained multimodal representation models have typically been trained with features from object detection networks rather than the image-level representations commonly used in VLN . Despite this , prior work leveraging these models [ 4 ] has continued to use the standard set of image-level features – leading to a significant domain shift between pretraining and VLN finetuning . In this work , we address these issues 1 ) by using multiple visual encoders to explicitly encode the inductive bias that the world is composed of objects and scenes and 2 ) by using object-level features from a detection network that match the features used to pretrained multimodal representation models . The starting point for our work is a VLNœ BERT [ 4 ] agent that processes scene-level features from a Places [ 16 ] CNN with a transformer-based [ 17 ] multimodal representation model that is modified with a recurrence mechanism designed for the VLN task . Our work extends this approach by including object features as an additional input to the multimodal processing . However , we find that simply adding object features to VLNœ BERT does not improve ( but rather slightly reduces ) performance . Thus , we propose architectural changes that allow the model to take better advantage of these two distinct types of visual information . Specifically , we change the attention pattern within the the transformer to effectively freeze the scene representations and focus the processing on the object-level inputs . The result is a new VLN agent that produces contextualized object representations by using scene features as high-level contextual cues . We experiment with our proposed approach on the Room-to-Room ( R2R ) [ 1 ] and Room-Across-Room ( RxR ) [ 2 ] datasets . Empirically , we find that our model substantially improves VLN performance over our VLNœ BERT baseline on R2R and outperforms state-of-the-art methods on English language instructions in RxR . Specifically , our proposed approach improves success weighted by path length ( SPL ) on the unseen validation split in R2R by 1.8 absolute percentage points . On RxR – a more challenging dataset due to indirect paths and greater variations in path length – we see even larger improvements . Success rate ( SR ) improves by 3.7 absolute percentage points , alongside a gain of 2.4 absolute percentage points on the normalized dynamic time warping ( NDTW ) metric . Through ablation experiments we find that ( consistent with the observations in [ 3 ] ) vision-and-language pretraining is vital to our approach , which suggests that strong visual grounding is key for using objectlevel features in VLN . Additionally , on RxR instructions that include six or more object references ( i.e. , object-heavy instructions ) , our method has even larger improvements over VLNœ BERT of 7.9 absolute percentage points in SR. To summarize , we make the following contributions : • We propose a scene- and object-aware transformer ( SOAT ) model for vision-and-language navigation that uses both scene-level and object-level features – explicitly encoding the inductive bias that the world is composed of objects and scenes . Our model uses a novel attention masking technique and view aggregation strategy , which both improve performance . • We demonstrate that our approach outperforms a strong baseline approach by 1.8 absolute percentage points in SPL on R2R and by 3.7 absolute percentage points in SR on RxR . • We show that our method has significantly stronger performance on instructions that mention six or more objects ( 7.9 absolute percentage points of improvement in SR ) , which further suggests that our model is better able to recognize and ground object references . 2 Related Work . Vision-and-Language Navigation . The Room-to-Room ( R2R ) [ 1 ] and Room-Across-Room ( RxR ) [ 2 ] datasets both situate the VLN task within Matterport3D [ 18 ] indoor environments . Since the release of R2R there has been steady improvement in VLN task performance [ 4–14 ] . Some of the key innovations include using instruction-generation via ‘ speaker ’ models for data augmentation [ 5 , 10 ] , combining imitation and reinforcement learning [ 6 ] , using auxiliary losses [ 7 , 13 ] , and different pretraining strategies [ 4 , 12 , 14 ] . All of these methods have one thing in common – they process visual observations with a single convolutional network pretrained to solve an classification task ( using either the ImageNet [ 15 ] or Places [ 16 ] datasets ) . In contrast , this work explores using a combination of features from visual encoders pretrained for scene classification and object detection . Object Detectors in VLN . Intuitively , object detections should naturally match the object cues ( e.g. , ‘ green chairs ’ ) mentioned in VLN instructions . Indeed , several recent studies [ 3 , 19–21 ] have demonstrated the utility of using object detectors for VLN . In [ 19 , 20 ] object classification labels from an object detector are encoded using a GLoVe [ 22 ] embedding . Similarly , [ 21 ] convert detections into a feature vector using the classification label , object area , and detector confidence . Unlike these methods , our model directly uses object features produced by a detector , which provide a richer , high-dimensional representation of each region . Additionally , our approach takes advantage of vision-and-language pretraining , which eases the burden of learning the grounding between natural language and object representations from scratch using only VLN data , as is done in these prior methods . In [ 3 ] , object features are used in a model that solves a path selection task in VLN , which requires pre-exploring an environment before executing the navigation task . By comparison , this work focuses on navigating without pre-exploration . 3 Preliminaries : VLN , Visual Encoders , Multimodal Transformers . This section reviews the vision-and-language navigation ( VLN ) task and describes how multimodal transformers are used in the recently proposed VLNœ BERT [ 4 ] model . Our approach builds on VLNœ BERT and is described in Section 4 . 3.1 Vision-and-Language Navigation . In VLN , agents are placed in a photo-realistic 3D environment and must navigate to a goal location that is specified through natural language navigational instructions I ( illustrated in Figure 1 ) . At each timestep t , the agent receives a set of panoramic observations Ot “ tot , iu36i “ 1 composed of RGB images from 36 viewing angles ( 12 headings ˆ 3 elevations ) . We follow the VLN with known navigation graph setting [ 1 , 2 ] – agents have access to a graph that specifies a set of navigable locations from each viewpoint in the environment . We agree with the limitations of this setting discussed in [ 23 ] . However , we report results for the nav-graph setting to be consistent with the long line of prior work in this area . We plan on generalizing to continuous environments in the future . Using the nav-graph , agents select an action from the set At “ tat , iuNti “ 0 consisting of Nt navigable locations and the stop action . The agent is successful if it calls stop within 3m of the goal location . 3.2 Multimodal Transformers . Here we provide a brief overview of multimodal transformers ( e.g. , OSCAR [ 24 ] ) , which provide a basis for the VLNœ BERT architecture . Multimodal transformers are an extension of transformerbased [ 17 ] language models such as BERT [ 25 ] that process image-text pairs . As in BERT , the text input is tokenized , encoded with a learned embedding , and then combined with positional information ( i.e. , word order ) . Commonly , the visual input ( i.e. , the image ) is first preprocessed by an object detector ( e.g. , a Faster R-CNN [ 26 ] model trained on Visual Genome [ 27 ] ) to produce a set of region features that are combined with spatial information ( e.g. , the offset to the bounding box in the image ) . To summarize , the multimodal input consisting of L word tokens tw1 , . . . , wLu andM image regions tr1 , . . . , rMu can be written as [ CLS ] , w1 , . . . , wL , [ SEP ] , r1 , . . . , rM where [ CLS ] is a special token used as a global representation of the input and the [ SEP ] token separates modalities . This set of multimodal inputs are fed to a series of transformer encoder layers [ 17 ] to produce contextualized representations for each input using attention-based processing . 3.3 Multimodal Transformers for VLN . The input interface for multimodal transformers can be easily adapted to process the language instructions and panoramic image from each timestep in VLN . However , VLN requires sequentially following navigation instructions ( e.g. , ‘ Exit the bedroom ... ’ then ‘ Continue down the hall ... ’ then ‘ stop before the desk with two green chairs. ’ ) . Accordingly , maintaining a history of the agent ’ s state is helpful for understanding which sub-instruction to follow at each timestep . Traditional VLN agents use recurrent neural networks ( e.g. , LSTM [ 28 ] ) to model state history . In contrast , VLNœ BERT [ 4 ] introduces a generic recurrence mechanism that can , in principle , be added to any multimodal transformer model to refashion it for the VLN task . In this work we extend the OSCAR [ 24 ] instantiation of VLNœ BERT . As shown in Figure 2a , VLNœ BERT [ 4 ] uses a multimodal transformer to process word tokens from the navigation instructions and scene features for a set of “ candidate ” views from a panoramic observation . There is one candidate view for each navigable action in At , corresponding with the views that best align with each navigable direction ( there are „ 4 such views on average ) . Accordingly , the output representation for each view is used to compute the probability of moving in that direction . The recurrence mechanism in VLNœ BERT is operationalized using a state token st , which is fed as an input to the multimodal transformer and is updated at each timestep . Initialization . The state token st is initialized by passing word tokens from the navigation instruction I along with special [ CLS ] and [ SEP ] tokens through the multimodal transformer . The output representation for the [ CLS ] token is used to set s0 and the outputs for the word tokens ( denoted as ψpIq ) are used as the language representation during navigation . Visual Inputs . For each RGB image in the Nt candidate views , VLNœ BERT runs a convolutional network trained on Places [ 16 ] to extract high-level scene features . To facilitate the stop action an all-zeros feature vector is added to the set of visual inputs . We denoted this set of visual inputs as V t. Navigation . At each timestep , the input to the multimodal transformer is composed of the state token st , the encoded instruction ψpIq , and the visual inputs V t. To reduce computation , the instruction representation is not updated during navigation ; the encoded word tokens simply serve as keys and values when processed by the multimodal transformer . Actions are predicted using “ state-conditioned ” attention scores for the scene features . These scores are produced at the final layer of the multimodal transformer . Concretely , let the final layer outputs for the state token st and scene features V t be denoted as ψpstq P Rd and ψpV tq P RpNt ` 1qˆd where Nt is the number of navigable locations and d is the model ’ s hidden dimension size . State-conditioned attention scores are calculate using a scaled dot-product as αpVtq “ ψpstqψpV tqT ? d ( 1 ) These scores αpVtq are normalized using the softmax function ãpVtq “ softmaxpαpVtqq and used as the probability of moving to one of theNt navigable locations or stopping . Finally , the state history is maintained by constructing the next state token st ` 1 with the output representation for the state token ψpstq using a “ refinement ” procedure detailed in the Appendix .

Summary: This paper proposes a transformer based VLN model based on VLNBERT, which leverages both object-level features (from a Visual Genome pre-trained object detector) and scene-level features (from a Places pre-trained detector). The intuition is that natural language instructions found in the VLN task will contain both references of specific objects in the scene (e.g. chair, sofa), as well as higher level references to entire rooms (e.g. kitchen, bedroom), and that using image features of both types will better the agent’s language grounding. They additionally propose a selective attention mechanism and a view-aggregation strategy. The selective attention only modifies the object-level features through the self-attention layers, keeping the scene-level and instruction features fixed, used for contextualizing the object-level features. The view-aggregation strategy, in my understanding, is a max-pool operation which uses the maximally scoring (compatibility score with the current state) object or scene feature from each view as the view’s representative for action selection. The paper presents a series of experiments comparing the proposed method to prior state-of-the-art in the R2R and RxR datasets, generally beating the presented baselines in the R2R validation-unseen and test splits (with the exception of VLNBERT pre-trained on VLN data) as well as the RxR validation-unseen split. A series of ablations are presented showing that the improved performance is due to the unison of the 3 proposed components rather than any single one alone, and that it rests on vision-language pretraining. Finally, they show that the performance improvement is even larger when evaluating on instances with more than 6 object references.