Spatially-Aware Speaker for Vision-and-Language Navigation Instruction Generation (2024)

Abstract: Embodied AI aims to develop robots that can understand and execute human language instructions, as well as communicate in natural languages. On this front, we study the task of generating highly detailed navigational instructions for the embodied robots to follow. Although recent studies have demonstrated significant leaps in the generation of step-by-step instructions from sequences of images, the generated instructions lack variety in terms of their referral to objects and landmarks. Existing speaker models learn strategies to evade the evaluation metrics and obtain higher scores even for low-quality sentences. In this work, we propose SAS (Spatially-Aware Speaker), an instruction generator or Speaker model that utilises both structural and semantic knowledge of the environment to produce richer instructions. For training, we employ a reward learning method in an adversarial setting to avoid systematic bias introduced by language evaluation metrics. Empirically, our method outperforms existing instruction generation models, evaluated using standard metrics. Our code is available at https://github. com/gmuraleekrishna/SAS.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13601–13614 August 11-16, 2024 ©2024 Association for Computational Linguistics
Embodied AI aims to develop robots that can understand and execute human language instructions, as well as communicate in natural languages. On this front, we study the task of generating highly detailed navigational instructions for the embodied robots to follow. Although recent studies have demonstrated significant leaps in the generation of step-by-step instructions from sequences of images, the generated instructions lack variety in terms of their referral to objects and landmarks. Existing speaker models learn strategies to evade the evaluation metrics and obtain higher scores even for low-quality sentences. In this work, we propose SAS (Spatially-Aware Speaker), an instruction generator or Speaker model that utilises both structural and semantic knowledge of the environment to produce richer instructions. For training, we employ a reward learning method in an adversarial setting to avoid systematic bias introduced by language evaluation metrics. Empirically, our method outperforms existing instruction generation models, evaluated using standard metrics. Our code is available at https://github. com/gmuraleekrishna/SAS.

heading: 1 Introduction
text: Incorporating language understanding in robots has been a long-standing goal of the NLP and robotic research community. Specifically, the VisionLanguage Navigation (VLN) task requires robots to follow natural language instructions grounded on vision to navigate in human living spaces. Although humans generally follow navigational instructions well, training robots to follow natural language instructions remains a challenging problem. Detailed navigation instructions may include landmarks, actions, and destinations. Recent work has succeeded in improving instruction understanding of robots by augmenting instruction and trajectory training data (Hong et al., 2020; Wang et al., 2022,
2023). They showed that using machine-generated instructions from a large number of navigational paths sampled from real houses helps robots navigate successfully even in previously unseen environments. However, there is still room for improvement, as the quality of machine-generated instructions is clearly lower compared to human annotations (Zhao et al., 2021).
In this work, we present a novel instruction generation model that can produce a variety of humanlike instructions using semantic and structural cues from the environment. Our method uses rooms, interesting landmarks, objects, inter-object relations, object locations, and spatial features to produce richer instructions that can be used by robots and humans alike (Fig. 1). Our Spatially-Aware Speaker (SAS) model generates information-rich instructions by leveraging expert demonstrations that map trajectories to verbal directions. Incorporating spatial references within these instructions is critical, as they convey the environmental layout, highlight key landmarks relevant to the actions taken, and gauge the progression of navigation. At its core, SAS employs a sequence learning framework that is fine-tuned through a combination of adversarial learning rewards and multiple objec-
13601
tives aimed at enhancing its linguistic generation capabilities.
The architecture of SAS is based on an EncoderDecoder model, which processes a sequence of viewpoints and corresponding actions that define a navigational path, subsequently generating a coherent set of instructions. During the encoding phase, the model extracts vital information from visual inputs, such as object categories (e.g., cupboard, bed), spatial relationships between objects (e.g., on top of, near, under), object placements, and significant landmarks within the viewpoint (e.g., bedroom, kitchen). These elements are combined with navigational actions to form a comprehensive vision-action representation that computes the temporal order.
The decoding phase acquires linguistic capabilities by linking this latent representation to the instructions encountered during training. An adversarial learning objective is introduced to encourage the generation of varied sentences, mitigating the potential biases that automatic evaluation metrics introduce. Through this novel approach, SAS outperforms existing instruction generation models on VLN datasets evaluated using standard language evaluation metrics.
Our contribution is as follows.
• We introduce a novel speaker model (SAS) that can incorporate semantic and structural viewpoint features into the instruction.
• We develop an adversarial reward learning strategy, that rewards diverse instructions, to train our SAS model.
• We introduce a large scale silver dataset for automatic data augmentation.

heading: 2 Related Work

heading: 2.1 Natural Language Navigational Guidance and Following
text: Methods for modelling human and robot behaviours for the generation and execution of natural language instructions span several disciplines, including cognitive psychology (Ward et al., 1986), sociology (Harrell et al., 2000), natural language processing (NLP) (Daniele et al., 2017), and robotics (Wang et al., 2023). Studies show that adequate navigational instructions have directions, landmarks, region descriptions and turn-by-turn actions (Look et al., 2005). These instructions are
also beneficial to human-machine interaction, particularly in embodied agents.
The embodied navigation problem has been receiving attention from multiple research domains such as robotics, NLP and scene understanding (Anderson et al., 2018; Dorbala et al., 2023). Recent studies have shown that VLN agents learn better on machine-generated examples (Fried et al., 2018; Tan et al., 2019; Wang et al., 2023). These methods, generally called Speaker models, are still far from generating human-like instructions, as exhibited by their lower machine-generation evaluation scores. Our work aims to improve the quality of the generated instructions over baseline models by including landmarks, actions, and directions.

heading: 2.2 Spatial and Semantic Scene Understanding for Embodied Navigation
text: Neuroscience has shown that humans, like others in the animal kingdom, use spatial and temporal cues to build a cognitive map of their surroundings (Kuipers, 1978). These cognitive maps are crucial for manipulating or navigating the environment. Inspired by this, recent studies in robotics and embodied navigation have used vision models to infer the structure of the environment (Kuo et al., 2023; Gopinathan et al., 2021), spatial relationships among objects in the scene (Qi et al., 2020; Moudgil et al., 2021) and environment layout (Gopinathan et al., 2023) to learn about the environment (Song et al., 2022). While these methods utilise structure, spatial and semantic knowledge in various combinations to learn vision-language association, they are not applied to instruction generation task. In this work, we use all four aspects of a satisfactory instruction - landmark through visual encoding, directions through directional encoding, turn-by-turn actions using action encoding and region descriptions as semantic encoding - to generate richer instructions.

heading: 2.3 Reinforcement learning for Instruction Generation
text: Reinforcement learning (RL) has been successful in machine generation tasks such as translation, instruction generation, captioning, and storytelling. In this paper, the primary objective is to maximise the expected return of a word-generating policy. RL instruction generation methods are found to learn the target distribution better than traditional maximum likelihood estimation (MLE) algorithms due to the inherent exposure bias (Arora et al., 2022).
Applying RL to learn the target distribution requires extensive feature and reward engineering. Instead, inverse reinforcement learning (IRL) is proposed to infer the expert’s reward function. Using an adversarial setting, IRL has been shown to improve visual story telling (Wang et al., 2018).
Existing work in VLN have studied natural language instruction generation (Fried et al., 2018; Wang et al., 2022, 2019) as a sequence generation problem, however, they focus on navigational success of the overall agent over instruction quality. Duo et al. Dou and Peng (2022) optimise their Speaker by using the similarity between itself and the gradient of the navigation model as a reward for RL. The authors evaluated their method on BLEU, a metric which does not guarantee high-quality instructions. Zhao et al. (2021) discovered that in the context of dialogue generation and navigational tasks, the majority of n-gram-based automatic language evaluation metrics show a weak correlation with human-annotated instructions.
Inspired by these studies, we adopt an IRL-based reward learning strategy to produce high-quality navigation instructions by indirectly learning the reward from language metrics. This mitigates exposure bias and avoids the model from gaming the metrics to achieve high evaluation scores - even with low-quality instructions.

heading: 3 Problem Definition
text: Here we present the task aimed at generating linguistic instructions from navigational demonstrations. The generation of instructions is posited as the converse operation to the standard VLN task, where an agent executes a navigation instruction in a house. For this, we develop a Speaker agent that synthesises a coherent set of natural language instructions from a sequence of navigational actions along a trajectory.
At each discrete time step t, the Speaker agent is presented with a panoramic visual observation Ot and a directional action at that signifies the transition to the next viewpoint within the trajectory. Upon completion of a navigational episode, the agent outputs the complete instruction of the traversed path, that is, X = {w0, . . . , wl}. Formally, the objective of the Speaker is to minimise the negative log-likelihood for ground-truth instruction X conditioned on trajectory T = {O1, . . . , ON} with parameter θ:
Lθ = − ∑
θ
log(p(X|T ; θ))) (1)

heading: 4 Preliminaries

heading: 4.1 Path Mixing
text: Prior work in VLN have shown that more instruction examples can improve an agent’s performance in previously unseen environments (Moudgil et al., 2021; Wang et al., 2022). Hence, to supplement training data, we mix parts of trajectories from the FGR2R dataset (Hong et al., 2020) (which is derived from the R2R (Anderson et al., 2018) dataset) to obtain additional instruction-trajectory pairs.
R2R dataset contains paths and humanannotated instructions for navigating inside 3D scanned house environments from the Matterport3D (MP3D) dataset (Chang et al., 2017). Further fine-grained (turn-by-turn) or microinstructions are available in the FGR2R dataset. We adopt FGR2R to enhance our training data by algorithmically combining parts of its trajectories, creating new instruction-trajectory pairs. Unlike REM (Moudgil et al., 2021; Liu et al., 2021), which randomly mixes data from different houses, we only mix trajectories from the same house to ensure visual and object referral consistency.
First, we identify key edges in the graph to mix trajectories, focusing on start edges εstart and end edges εend of navigation paths. These edges are crucial as they relate to start and end of instructions (e.g. "Walk away from the desk, Turn right"). Edges with micro-instructions1 that do not contain
1Part of the instruction pertinent to one edge of the path
a NOUN, VERB are ignored to avoid partial or inconsequential (wait there) actions. Then, we mix the remaining transition edges εtrans to form a trajectory from εstart to εend. Nodes that are too close to each other, leading to a lack of visual diversity or repetitive micro-instructions are also avoided. We connect edges based on the following criteria: (1) the distance between any two nodes (except the start and end nodes) should not exceed 3m, (2) the angles between edges should prevent looping; (3) the start and end nodes should not share an edge; (4) micro-instructions from shared edges of different trajectories are selected randomly.
The final instruction combines these microinstructions, and the trajectory is a sequence of edges (as shown in Fig. 2). Using this method, we generated 162k instruction-trajectory pairs with path lengths ranging from 5m to 20m. The dataset2 averages 7.27 views per trajectory, a mean path length of 14.4m, and about 82 words per instruction. More dataset statistics are added to the supplementary material.

heading: 4.2 Action Parsing
text: To allow the speaker to learn action phrases by associating them to the navigational actions, we automatically extract the action phrases from the instruction for training. For this, we use spaCy’s (Honnibal et al., 2020) dependency parsing and part-of-speech tagging to identify verb forms that are transitive, indirect transitive, direct transitive and also the part-of-speech forms such as nouns, adverbs, adpositions, interjections and determiners (refer Appendix B for more details). We use this algorithm to identify action phrases from each step of the instruction and classify parts of sentences to either actions or other phrases. The identified micro-instructions are used to train the decoder part of SAS (§6.2).

heading: 5 Spatially-Aware Speaker (SAS) Model
text: In this section, we present our Spatially-Aware Speaker model. SAS is an encoder-decoder model which generates an instruction for a trajectory when the sequence of viewpoints and actions are provided. The trajectory encoder produces visualaction context from the trajectory viewpoints and the corresponding actions. This context is used by instruction decoder to generate instruction.
2Available at https://zenodo.org/records/10396782
To incorporate spatial and semantic awareness, we provide three crucial pieces of information to the model, namely: Action Encoding, Structural Encoding and Semantic Encoding. These are explained in the following sections.
Action Encoding The action taken between viewpoints in a trajectory is represented by Action Encoding. The visual action encoding is the current heading and elevation of the agent with respect to the next view direction. The relative elevation (θ) and heading (ϕ) angles are encoded as Ea = [cos θ, sin θ, cosϕ, sinϕ].
Structural Encoding Structural encoding provides knowledge of the egocentric locations of objects with respect to the Speaker. In the panoramic view of each viewpoint, we extract an object’s location in the image frame, as well as its size and distance to the agent in order to represent a complete pose of the object relative to the agent. The image frame location is obtained from the location of the object’s bounding box detected by a Faster R-CNN (Ren et al., 2017) detector trained on the Visual Genome (Krishna et al., 2017) dataset. The size and distance of the object are obtained by projecting (inverse pinhole camera projection) the bounding box to the point cloud and measuring the centroid and volume of the contained point cloud. This provides an estimate of the size and distance to form the object descriptions. Effectively, the structural encoding is a combination of object features fo, object location (cx, cy), size so and distance do , respectively, i.e. Eso = [fo, (cx, cy), so, do].
Semantic Encoding To provide inter-object relationships we use 1600 object classes FasterRCNN and relationships extracted from ConceptNet (Speer et al., 2017). The object-to-objectroom semantic features are a combination of GloVe embedding G of the respective token, eobji,j,k= {G(<obji>);G(<relj>);G(<objk>)}.
Furthermore, we encode the relationship from room to object using the in the relation as erooml,m ={G(<objl>);G(<in>);G(<roomm>)}. For each viewpoint, we encode one eroom and one eobj per view direction with the highest detection confidence. In effect, we obtain the semantic encoding per viewpoint, Esm = {(eobj1 , eroom), . . . , (eobj36 , eroom)}. Panoramic Room-Object Attention Finally, we combine structural and semantic knowledge with panoramic view to obtain a panoramic knowledge
feature. For this we first concatenate the candidate feature fc with the Structural Encoding Eso and Semantic Encoding Esm:
fcs = W[fc;Esm;Eso] (2) where W is the trainable projection. We apply an attention module to make the model attend to information from different sub-spaces. The two projections Q(query) and K(key), which are from the action embedding and the candidate-semantic embedding, respectively, are applied to the attention as:
αk = softmax
( Q(hat )K
T (fcs)√ Dk
) (3)
ck = ∑
i,j
αi,jfpi,j (4)
gt = tanh(W[ck;hat ]) (5)
where cα is the context vector, Dk is the hidden size of the attention layer, hat is the hidden action state of the trajectory encoder and gt is the gated output. The affinity matrix fp governs the information flow between neighbouring view patches of the panorama. Finally, we capture the panoramic room-object feature using an LSTM:
hvt = LSTMv(gt, h v t−1), ∀t = 1, . . . , N (6)

heading: 5.1 Trajectory Encoder
text: The trajectory encoder consists of a multilayer bidirectional LSTM to summarise the input sequence at each time step conditioned on the navigational trajectory. This bidirectional approach ensures action context at each step is influenced by both the historic and the future actions in the sequence.
The first BiLSTMA encodes navigation actions from the ground truth action a. Scaled-dot attention is applied to the action hidden state ha and the panoramic visual features fv giving a context vector cw. A second LSTM encodes the change of the context vector as hv,at . This hidden state is used by the decoder to learn visual and language alignment. Formally:
hat = BiLSTMA(at, h a t−1) (7)
αw = softmax
( Qa(h a t )K T v (h
v t )√
Dk
) (8)
cw = ∑
i,j
αwh a t (9)
h̃ = tanh(W[cw;hvt ]) (10) ĥv,at = BiLSTMV A(cw, h̃) (11)
where a is the action embedding, h is the hidden state. Also, Qa = F aq (h a t ), Kv = F v k (fv), and Dk are query and key vectors, hidden dimension size of the soft attention, respectively.

heading: 5.2 Instruction Decoder
text: Our instruction decoder is guided by semantic and structural knowledge from the environment. The basic structure of the decoder is as follows. When the decoder is provided with the previous instruction token and the visual-action context hv,at , it applies an LSTM to encode the instruction token embedding from the previous time step:
wembt−1 = embedding(wt−1) (12)
hXt = LSTMX(w emb t−1 , h v,a t−1) (13)
we apply a scaled-dot attention on the projected instruction context QX = Fq(hXt ) and the visionaction context Kva = h v,a t and V = Fv(h X t ):
ĥvaXt = Attention(QX ,Kva, VX) (14)
Finally, the next predicted word is the token of maximum probability:
wt = argmax(WĥvaXt ) (15)

heading: 6 Training
text: The Encoder (§5.1) and Decoder (§5.2) modules of SAS are trained end-to-end. SAS model predicts the next token wt based on the complete trajectory T = {O1, . . . , ON} and all previous tokens w<t. The trajectory embedding from the encoder and w<t is fed to the decoder to produce a probability distribution pL over the next word token. This distribution is sampled as in (15) to predict the next token wt.
SAS model is trained using a mixture of a Teacher-Forcing (TF) (Williams and Zipser, 1989) and the ARL method (§6.1). In TF, the decoder generates the next token based on a ground truth token instead of using its predicted token as in the Student Forcing (SF) strategy. This method has been shown to improve the baseline methods (Anderson et al., 2018). Next, we describe reward learning in detail.

heading: 6.1 Reward Learning
text: Applying Reinforcement Learning (RL) for an instruction generation task using automatic evaluation metrics as reward functions, causes the model to game the metrics. Instead, in reward learning,
a reward model learns the best reward for humanannotated and speaker-generated instructions. In this strategy, we use a Generative Adversarial Network (GAN) (Goodfellow et al., 2020) architecture - with a Policy and Reward model - to learn an association between the instruction and reward distributions (Fig. 3). In an RL sense, this is a Markov decision process (MDP) with SAS as a policy Gβ(·), generating words XG and receiving a reward r(XG) score. The objective is to maximise the expected reward of Gβ , EXG∼Gβ [r(X
G)]. In reward learning, the reward distribution is learned from the demonstrations, rather than adopting a function to provide a reward.
We adapt the idea of Reward-Boltzmann distribution from Wang et al. (2018) to approximate a reward obtained for identifying fake (speakergenerated) or real (human-annotated) instructions. The approximate reward distribution Πα for an instruction X is defined as:
Πα(X) = eRα(X)∑ X eRα(X) (16)
where Rα is the reward function. The optimal reward function is achieved when (16) is equal to the distribution of human-annotated instructions. We optimise this using an adversarial two player min-max game between (1) Πα maximising its similarity (measured by KL-divergence) with the empirical distribution Πϵ(X) of the training dataset and minimising its similarity with the distribution of fake instructions from the policy Πβ and (2) maximising the similarity of the policy distribution Πβ with that of Πα. Formally, the objective is:
max β min α
KL(Πϵ||Πα)−KL(Πβ||Πα) (17)
This is optimised through a policy-gradient-based reinforcement learning method.
Reward Model We investigate two reward models Rω based on the CNN and RNN models. The CNN-based discriminator uses the GloVe embedded instruction Xemb and visual feature Oi to produce a reward score Rω after the activation function. Formally, the CNN and RNN rewards are, respectively:
RCNNω = WR(Conv(Xemb);WOOi) (18) RRNNω = WR(Xemb;WOAvgPool(Oi)) (19)
where WR and WO are linear learnable weights, Conv represents the convolution layers followed by the mean pooling operation, and [; ] is the feature concatenation. The final sigmoid activation is not shown for brevity.
Both the Speaker policy and the reward models are trained alternately using the Adversarial Reward Learning (ARL) algorithm (Algorithm A). Reward models are evaluated in our ablation studies (§8.3).

heading: 6.2 Supervised Learning
text: The supervised learning objectives used for both the teacher-forcing (used in the final model) and the student-forcing (for the ablation study) strategies are as follows.
Language modelling Conditioned on path T and linguistic embedding w<t, the probability of decoded words is optimised as a maximum likelihood estimation (MLE) problem:
LLM = − ∑
t∈{1:N} log(pL(wt|w<t, T )) (20)
where pL is the likelihood of a token given trajectory T and ground truth instruction tokens wGT<t .
Unlikelihood training Even low-perplexity machine generation models are prone to repeating tokens when presented with small examples (Holtzman et al., 2020). To mitigate this, we apply Sequence-Level unlikelihood loss (Welleck et al., 2020) on the decoded instruction that penalises repetition of word tokens w. The objective is to minimise the logarithmic likelihood of negative candidates (repeated tokens) Ct conditioned on previous tokens w<t:
LULS = − ∑
c∈Ct log(1− pµ(c|w<t)) (21)
Temporal Alignment Loss We introduce a temporal alignment loss (TAL) to train the decoder to attend between action phrases and visual-action context. The decoder’s attention matrix AD, which represents attention between word tokens and the panoramic action context, is compared to the ground truth vision language alignment scores (§4.2). Formally,
AGT = { 1, w ↔ oi;w ∈ X, oi ∈ O 0, otherwise
(22)
where↔ denotes action phrase-viewpoint alignment. LTAL is the binary cross-entropy loss between AGT and AD.
Total Objective The total training objective is the weighted sum of all losses, that is, L = λLMLLM + λULSLULS + λTALLTAL.

heading: 7 Experiments

heading: 7.1 Datasets
text: Our method is assessed using two datasets from the Vision-and-Language Navigation (VLN) field: R2R, which features brief trajectory paths and instructions for locating rooms, and R4R, an extension of R2R that links two adjacent tail-to-head trajectories along with their associated instructions to produce longer instructions. For training, we augment R2R dataset with the Path Mixing (PM) dataset §4.1.

heading: 7.2 Evaluation Metrics
text: We evaluate the performance of SAS instruction generation using standard language metrics such as SPICE (Anderson et al., 2016), CIDEr (Vedantam et al., 2015), ROUGE (Lin, 2004), METEOR (Denkowski and Lavie, 2014) and BLEU-4 (Papineni et al., 2001). SPICE is considered the main metric in navigational instruction generation tasks (Zhao et al., 2021; Wang et al., 2022). A high SPICE score indicates high lexical and semantic similarities of sentences and higher success of an embodied agent, which is important for navigational instructions (Zhao et al., 2021).

heading: 7.3 Implementation Details
text: We use Speaker-Follower (Fried et al., 2018), a popular baseline used in previous work, for our experiments. The model is trained for 100k epochs (≈14h) using NVIDIA RTX A6000 with batch size 8 and AdamW (Loshchilov and Hutter, 2019) optimiser with learning rate 5e-4. The visual feature and GloVe embedding sizes are 2048 and 300, respectively. The hidden size for the attention layers Dk is 512. The training objective weights are set in the ratio λLM : λULS : λTAL = 2 : 1 : 1 to prioritize language learning over repetition and alignment.. The hidden dimensions for the twolayer BiLSTM and the QKV sizes in Attention are set to 768d. We report the evaluation values of a single run.

heading: 8 Results
text: We evaluate two variations of SAS to measure the effectiveness of the proposed method (1) SAS with teacher-forced SASTF using PM augmentation and temporal alignment (TAL) and (2) SASARL+TF a mixture of TF and ARL using all augmentation and supervised learning objectives. From the results (Table 1), we see that our SAS method improves on the baseline by a large margin. Reward learning has absolute improvements of +5.9 (SPICE), +5.6 (CIDEr), +4 (METEOR), +8.5 (ROUGE), and +7.5 (BLEU-4) in the R2R ValUnseen split compared to the Speaker-Follower baseline. In the long instruction dataset R4R (Table 2), both variations show better scores on all metrics (SPICE: +7.4, METEOR: +8.6, ROGUE: +3.3 and CIDEr +8.6). In both ValUnseen splits, the CIDEr score is markedly impacted (R2R: -2.6 and R4R: -0.6) when comparing the overall best model to the baseline models.

heading: 8.1 Discussion
text: SAS shows a better instruction generation capability with TAL, which goes to show that supervising the speaker with action-only sentences is useful. In addition, our method outperforms the baseline in most of the major metrics, specifically in SPICE. This shows that spatial awareness is beneficial for Speaker models. In the R2R and R4R datasets, both SASTF and SASARL+TF have the best scores compared to the previous models, except for CIDEr. CIDEr rewards lexical similarity over semantic similarity. As temporal alignment of actions is pertinent in instruction generation and not in the lexical order, lower scores in these metrics definitely do not reflect wrong actions. Among these metrics, a higher SPICE score shows that our model generates temporally consistent instructions. Zhao et al. (2021) observe that SPICE correlates with human way-finding performance, VLN agent navigation performance, and subjective human judgements of instruction quality, when averaged over many instructions. This correlation is not observed at the instruction level due to the high variance between the words used in the instructions. Human evaluation of generated instructions should be performed to ensure the actual quality of the instruction. In our study, we consider the high SPICE score as an early indicator of robust pathfinding performance for both agents and humans. It also reflects that human judgement of the
quality of SAS-generated instructions is also high.

heading: 8.2 Qualitative Results
text: Our SAS model is able to generate meaningful instructions by including object and scene relevant tokens (such as "oven") as shown in Fig. 4 that are not referenced in the ground-truth instruction, while the Speaker-Follower baseline model produces shorter and action-focused sentences.

heading: 8.3 Ablation Studies
text: Here we ablate on different augmentation and training methods (Table 3) evaluated on R2R ValUnseen split. Method #1 uses the aforementioned Student forcing (SF) and #2 represents Teacher forcing (TF) for training. When Path Mixing is applied to the SAS model (#3), the Speaker learns the frequent object tokens in the instruction and how to correlate them with the visual features. Models #4 and #5 (SASTF ), trained using TAL, learn to associate the object tokens with the actions from the trajectory and inversely co-relate navigational actions with action phrases in the ground
truth instructions (Walk through the double door...). The models #6 and #7 (SASARL+TF ) are trained to improve the SPICE score using ARL. Using ARL and a GRU-based reward model (#7) has an advantage over using the CNN-based reward model (#6), which produces the highest scores.

heading: 8.4 Spatial and Semantic effectiveness
text: To study the effectiveness of the proposed spatial and semantic encoding, we measure the amount of object and spatial phrases mentioned in the generated instructions for the speakers evaluated in the R2R ValUnseen environment.
In Table 4, the values outside the parentheses represent the total counts of objects/landmarks (Obj.) i.e. chair, bathroom, etc., actions/direction phrases (Act.) i.e. turn left, top of, etc., and nonstopwords (NonStop). Meanwhile, the values in parentheses denote the average number of entities per instruction for each respective category. We observed that the SAS speaker includes 18.11% more objects and landmark entities and 12.03% more action/direction phrases compared to the LANA speaker. Furthermore, the average length of the instructions is also higher for SAS, indicating richer or more detailed instructions. Although the SAS model did not refer to all the objects or landmarks in the ground truth instructions (SAS: 4.31, Human: 6.01), it includes more action/direction phrases (SAS: 4.06, Human: 3.80). This suggests a better specificity for actions and spatial awareness.

heading: 8.5 Limitations
text: Large-scale datasets featuring a broad variety of human-annotated navigation instructions are rare, presenting a significant challenge in the field. Our approach seeks to navigate this obstacle by leveraging a small-scale dataset originally compiled with a different objective: to facilitate the learning of navigation from instructions. It is important to consider this context when evaluating our method’s performance, as it operates under the constraint of limited data diversity and volume. Next, we explain some of the challenges in the dataset.
R2R dataset exhibits a notable variation in unique tokens across its different splits: Train, ValSeen, ValUnseen, and TestUnseen. Figure 5 (a) underscores the differences in token commonality across these splits, the disparities being largely attributed to the frequency of tokens in each split. Figure 5 (b) shows the ratio of different words to that of common words in each split, revealing that the Train split, in particular, contains a significant number of unique word tokens compared to common words with respect to other splits. This variation poses a unique challenge for our SAS model, which aims to mimic the instruction distribution of the training set, but may diverge from the linguistic characteristics of the ValSeen and ValUnseen splits, potentially negatively impacting evaluation scores.

heading: 9 Conclusion
text: This work proposes a novel navigation instruction generation model that can produce diverse instructions by attending to several structural and semantic cues from the environment. By providing objects, their locations, and their relationships from the scene, Spatially-Aware Speaker can refer to important aspects of the scene in the instruction. An adversarial reward learning method encourages the model to generate diverse instructions. The results show that our method improves on the standard evaluation metrics and performs better than the baselines.
Future work In future work, integrating the SAS model with multimodal transformer architectures will be crucial for enhancing the generation of openvocabulary embodied instructions. This direction promises to overcome the limitations posed by current dataset constraints and improve performance in instruction generation tasks.

heading: Appendix A Adversarial Reward
text: Learning algorithm
We use Reward Learning (ARL) extended from (Wang et al., 2018). We alternatively train SAS policy and reward models for 100 epochs each. The speaker policy is optimised using the loss functions (§6) explained in the main text.
Algorithm 1 Adversarial Reward Learning Algorithm
for epoch← 1 to K do Obtain instruction X ← Πω ▷ SAS Speaker Policy if Train-Policy then
Obtain instruction X̃ ← D Update Speaker Policy gradient
else if Train-Reward then Update Reward gradient ▷ CNN or GRU
end if end for

heading: Appendix B Algorithm to extract action
text: phrases from instruction
Algorithm 2 Algorithm to extract action phrases Input: Navigation Instruction X Output: Action phrases lX Initialise the empty action phrase list lX . ltyp ← {TRANVERB,DITRANVERB,INTRANVERB, NOUN} lpos ← {ADV,ADP,INTJ,DET,INTRANVERB} for wj , wj+1 in X do
cwj ← check_verb(wj) cwj+1 ← check_verb(wj+1) if cwj in ltyp then
if cwj+1 in lpos then lX ← (wj ;wj+1) ▷ Join words as
a phrase else if cwj ̸= NOUN then
lX ← wj ▷ Verb as a phrase end if
end if end for

heading: Appendix C Implementation Details
text: We implement our method with PyTorch library. The instruction speaker is trained with AdamW Optimizer (Loshchilov and Hutter, 2019) for 100k iterations. Consistent with previous work, panoramic visual features are extracted using a ResNet-152 model, and the angle feature used is 128 dimensional. We retrieve K = 6 objects and relationships from Visual Genome for each view direction. The dimension of the hidden state is set to 512. All experiments were performed on an NVIDIA RTX A6000 GPU. SASTF model is trained to 100K iterations. The SASTF+ARL model uses pre-trained SASTF weights for initialisation. In ARL strategy, policy and reward models are trained alternatively every 100 iterations. All parameters were chosen based on the performance on the ValUnseen split. SAS model has 14.3M learnable parameters. The losses are mixed using the weights λLM : λULS : λTAL is 2 : 1 : 1. The implementation of the metrics is obtained from the publicly available COCO Caption Evaluation Toolkit 3. spaCy package uses the en_core_web_lg model.
C.1 Hyperparameter Search
The hyper parameters for the final model are selected from the highest SPICE score on the ValUnseen split. The learning rate is fixed based on observing the reward, IRL scores, and supervision losses to prevent model from over-fitting.
C.2 Dataset Statistics
R2R It has four splits, i.e., Train (61 scenes, 14k instructions), ValSeen (61 scenes, 1k instructions), ValUnseen (11 scenes, 2k instructions), and TestUnseen (18 scenes, 4k instructions). Train and ValSeen splits share house scenes but not with the unseen splits.
R4R It extends R2R and contains three sets, i.e., Train (61 scenes, 233k instructions), ValSeen (61 scenes, 1k instructions), and ValUnseen (11 scenes, 45k instructions).

heading: Appendix D Qualitative Examples
text: Here we list some successful instructions generated by the SAS model against the speaker follower (Fried et al., 2018) baseline.
3https://github.com/tylin/coco-caption
D.1 Successful Example 1
Baseline: Walk into the garage and turn right. Turn right and walk into the room. Walk into the bathroom. Stop. SASTF : Walk into the garage and turn right. Walk into the room with the two large chairs. Turn right and walk into the bathroom. Wait near the sink. SASARL+TF : Walk through the garage and turn right at the cupboard. Walk into the living room towards the couches. Walk into the bathroom and wait near the sink. GT: Go straight into the garage. Turn right and go into the room. Turn right and go into the bathroom. Wait near the sink.
D.2 Successful Example 2
Baseline: Walk the stairs. SASTF : Walk out of the bedroom. turn left and walk up the stairs . stop on the second step from the bottom . SASARL+TF : Walk past the television and out of the bedroom. Turn left and walk up the stairs. Stop on the third step from the bottom. GT: Walk through the bedroom and out into the hall way. Turn left and walk up to the stairs. Walk up to the first step and stop.

heading: Appendix E More visualisation of
text: trajectories
E.1 Successful Example 1
Figure 6 shows one of the success cases. Here, SASTF generated the correct transition phrases, but failed to mention any stop locations. On the other hand, SASARL+TF refers to objects outside of reference sentences and also identifies the correct stop location. This is a good example of Panoramic Room-Object Attention in working.
E.2 Failure Example 1
Figure 7 shows one of the failure cases. Here both SAS models fail to generate the correct instruction even though the rooms (living room, kitchen, dining room) and objects (table, chair) are identified. The models failed to align the objects, rooms and actions in the correct sequence.
E.3 Learning Curves
Figure 8 shows the average ARL rewards and SPICE scores from training the model with CNN and GRU reward models. Both models approach a reward close to 1, while the GRU reward model
GT: Continue up the stairs, walk towards the sitting area, go straight passed the table on the left. stop by the double doors. : Walk past the fireplace and up the stair. Walk past the dining table and through the doorway. Walk past the dining table and through the doorway. : Walk up the stairs and turn left. Walk past the dining room table and chairs . Stop in front of the door .
GT: Walk towards the ovens and take a left. Walk towards the fireplace and enter the dining room to the right of the fireplace. Stop in front of the white chair. : walk past the kitchen and into the kitchen . walk past the dining room table and chair . stop in front of the couch . : Walk through the kitchen and turn left. Walk through the dining room and into the living room. Stop in the doorway to the living room .
Figure 7: Example of a Failed Instruction Generation.
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
0
5
10
15
20
25
30
35
1 51 10 1 15 1 20 1 25 1 30 1 35 1 40 1 45 1 50 1 55 1 60 1 65 1 70 1 75 1 80 1 85 1 90 1
Iterations
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
0
5
10
15
20
25
30
35
1 51 10 1 15 1 20 1 25 1 30 1 35 1 40 1 45 1 50 1 55 1 60 1 65 1 70 1 75 1 80 1 85 1 90 1
Iterations
(a) (b)
Reward IRL Score Reward IRL Score
Figure 8: SPICE Scores and Rewards for (a) CNN-based and (b) GRU-based reward models.
obtains a higher SPICE score. This shows that recurrence can help the discriminator learn the difference between policy-generated and ground-truth instructions. A CNN can extract global information from the instruction but loses temporal information. As temporal information is crucial for navigational instructions and GRU can encode temporal aspects of the instruction as well as visual observations, it produces a higher reward for instructions closer to the ground truth.

heading: Appendix F Ethical Considerations
text: Embodied navigation stands as a promising frontier with the potential to revolutionise the landscape of language understanding for robots, thus facilitating their seamless integration into everyday human life. However, any effort that involves human-robotic interaction requires a steadfast commitment to upholding ethical, privacy, safety, and legal standards. Although the metrics employed to assess our work align with those commonly used in the machine generation domain, further investigations are imperative to ensure the ethical and safety considerations associated with the instructions generated using automatic methods and their use in the real world.
Our research draws on the R2R (Anderson et al., 2018), FGR2R (Hong et al., 2020) and R4R (Jain et al., 2020) datasets under the MIT licence, which feature an extensive collection of indoor photos captured from American houses, licenced by Matterport3D4. The Matterport3DSimulator, used in our experiments, is also under MIT license. To protect privacy and confidentiality, the providers of the datasets have anonymised both the houses and the associated photos. In addition, the navigational instructions derived from these datasets are devoid of explicit language. As a result, our work shows minimal ethical, privacy, or safety concerns.
4https://kaldir.vc.in.tum.de/matterport/MP_ TOS.pdf

heading: Future Work
text: Future work In future work, integrating the SAS model with multimodal transformer architectures will be crucial for enhancing the generation of openvocabulary embodied instructions. This direction promises to overcome the limitations posed by current dataset constraints and improve performance in instruction generation tasks