Scaling Vision-and-Language Navigation With Offline RL
Authors: Valay Bundele
Mahesh Bhupati
Biplab Banerjee
Aditya Grover
Venue: Accepted by TMLR ()

Abstract: The study of vision-and-language navigation (VLN) has typically relied on expert trajectories, which may not always be available in real-world situations due to the significant effort required to collect them. On the other hand, existing approaches to training VLN agents that go beyond available expert data involve data augmentations or online exploration which can be tedious and risky. In contrast, it is easy to access large repositories of suboptimal offline trajectories. Inspired by research in offline reinforcement learning (ORL), we introduce a new problem setup of VLN-ORL which studies VLN using suboptimal demonstration data. We introduce a simple and effective reward-conditioned approach that can account for dataset suboptimality for training VLN agents, as well as benchmarks to evaluate progress and promote research in this area. We empirically study various noise models for characterizing dataset suboptimality among other unique challenges in VLN-ORL and instantiate it for the VLN⟳BERT and MTVM architectures in the R2R and RxR environments. Our experiments demonstrate that the proposed reward-conditioned approach leads to significant performance improvements, even in complex and intricate environments.

Published in Transactions on Machine Learning Research (04/2024)
Scaling Vision-and-Language Navigation With Offline RL
Valay Bundele
valaghandele@ggaiil.com
University of Tübingen
Mahesh Bhupati
mail2hnahesh033770mmaaa...mmm
Indian Institute of Technology Bombay
Biplab Banerjee
bbanerjee
Indian Instiuute of Technology Bombay
Aditya Grover
adityag ues.ucla edu
University of California, Los Angeles
Reviewed on OpenReview: https: openreview. net/ forum? id- kPIU8Ph.Po
Abstract
navigation (VLN) has typically relied on expert trajecto- ries, which may always be available in real-world situations due to the significant effort required to collect them. On the other hand, existing approaches to training VLN agents that go beyond available expert data involve data augmentations or online exploration which can be tedious and risky. In contrast, it is easy to access large repositories of suboptimal offline trajectorics. Inspired by research offline reinforcement learning (ORL), we intro- duce new problem setup of VLN-ORL which studies VLN using suboptimal demonstration data. We introduce a simple and effective reward-conditioned approach that can account for dataset suboptimality for training VLN agents, well as benchmarks to evaluate progress and promote research in this area. We empirically study various noise models for character- izing dataset suboptimality among other unique challenges in VIN-ORL and instantiate it for the VLNOBERT and MTVM architectures in the R2R and RxR environments. Our ex- periments demonstrate that the proposed reward-conditionel approach leads to significant performance improvements, even in complex and intricate environments
The study of
1 Introduction
Vision-anddlangguage navigation (VLN) a complex task that requires agent to navigale through vironments based language instructions, involving visual perception, natural language processing, and sequential decision making. VLN has received considerable attention (Fu et al. 2020b Wang et al. 2019: Ma et al. 2019 Fried et al.) 2018) in recent years due to its polential applications in robolies, aulonomous vehicles, and virtual assistants. However, this task poses significant. challenges, such as dealing with am- biguity in language instructions, robust perception in dynamic environments, and efficient exploration in large-scale environments.
The key challenge in VLN is learning generalizable policy given diverse vision and language inputs, scaling training data is crucial for the performance of VLN agents. However, collecting and annotating VLN data is difficult, leading to a shortage of domain-specific training data that limits performance in unseen environments. To address this challenge, some works have performed data augmentation by syuthesizing new instructions FFied et al. 2018 or varying appearance of cxisting environments (Li et al. 2022). Further, many works use pre-trained models trained on large language and vision datasets using self-supervised
Work done when the author Indian Institute of Technology Bombay.
1 Code and datasets available

---

Published in Transactions on Machine Learning Research (04/2024)
objectives (Hao et al. 2020 Hong et al. 2021: Guhur et al.; 2021 Zhu et al, 2020 Some works have also acquired extra data through online exploration {Hong et al. 2021: Lin et al. 2022a Chen et al. 2021b). However, exploration can be dangerous in many real-world applications such as robots in safety- critical applications or healthcare problems. Specifically, in VLN, exploration can be dangerous if the agent misunderstande instructions or distribution shifts in visual inputs, leading to collisions with walls or humans. Given these limitations, our work centers on a fundamental question: How can we achieve effective data scaling without resorting to online exploration, a strategy that could pose safely concerns?
To this end, this work makes the observation that while expert data can be hard to acquire, often have access to large databases of subotimal offline trajectorics. Examples include: 1) Human Navigation Data: In congested urban settings, drivers often take suboptimal routes due to traffic, road closures, or parking constraints, providing a rich source of suboptimal navigation data. 2) Imperfect Simulated Environments: AT agents navigating simulated environments, like a shopping mall, may encounter dynamic obstacles and adjust their paths accordingly. These adaptations, driven by imperfoct navigation algorithms, could result in suboptimal trajectories. 3) Transfer Learning Scenarios: Consider robot model trained in laboratory setting and then deployed in a real-world hospital. Initially, the robot might follow suboptimal paths as it learns to navigate the unique challenges posed by the hospital's layout and unexpected obstacles before fine-tuning its navigation strategy. These are just a few illustrative examples, but in reality, a multitude of scenarios can provide suboptimal demonstration data with relative case. This accessibility to suboptimal data presents a crucial opportunity for advancing VLN research, surpassing the challenges associated with obtaining expert data and propelling progress in the field.
In recent years, learning from logged demonstration data has become an active area of research within the offline RL community. While many algorithms exist including value-based (Peng et al.)
2019 Wu et al. 2019 Siegel et al., 22200] Kumar et al.) 2020), model-basee (Kidambi et al.] 2020 Janner et al. 2019), and
model-free (Chen et al. 2021a Emmons et al. 2021) approaches for training agents from logged data, there has been limited work in extending these methods to challonging VLN domains. In this paper, we propose the selling of vision-language navigation with offine RL (VLN-ORL) which trains VLN agents efficiently using offine datasets. Given that most current offiine VLN algorithms rely on supervised learning from expert datasets, the key practical goal for VLN-ORL is to design algorithms that scamlessly align with the established VLN architectures and objectives in use today. Therefore, we propose to extend family of approaches based on RL. via supervised learning (RvS) (Fmmons et al. 2021
RyS approaches learn an agent policy from a sub-optimal dataset by conditioning the policy on a outcome variable, e.g., returns-tor go-go or the goal token. At test time, the agent is conditioned on the desired outcome, e.g., expert returns. For VL.N, we do not expect to know the goal coordinates at test-time and need to resolve the language instruction to a goal state. Prior work resolves this challenge by learning to map the agent state to reward function that can be maximized by the agent using off-the-shelf RL algorithm (Nair et al., 2022). However, this assumes the existence of a sufficient number of expert trajectories in the offline dataset, which might not be true. In contrast, we propose an alternate set of assumptions that require knowledge of the goal state only during training, even if it is not achieved by the agent in executing an instruction. We build on this assumption to design a novel reward token which allows flexible conditioning of VLN agents during training and testing.
Our next contribution is to establish the first set of oflline RL benchmarks for evaluating VLN algorithms. Specifically, we leverage rollouts of HAMT policy (Chen et al. 2021b) pre-trained on R2R (Anderson et al., 2018) and RxR (Ku et al. 2020) datasets and develop several noise models to perturb the agent rollouts and generate suboptimal datasets of varying difficulties. Furthermore, we employ the proposed reward token to condition two recent VIN agents, namely VEN and MTVM. Through the utilization of reward- conditioning, we observe a significant performance improvement, with an average increase of approximately 10-15% across most of the generated datasets. Notably, on the Random-policy R2R dataset, which features trajectories generated by a random policy, the performance of BEN&BERT exhibits a remarkable improve- ment of approximately 40%. In addition to these results, we evaluate our approach separately on subsets of varying difficulties of the RxR validation sets, revealing that our reward-conditioning technique consistently leads to performance improvements, even in complex and intricate environments, Moreover, our findings
2

---

Published in Transactions on Machine Learning Research (04/2024)
Train
Demos (Manually annotatec)
"Tum 180 degrees to face stairs. Walk upstairs. On landing, make slight left toward door corner. Stop in front of door."
Suboptimal Demos
Greedy Rewards
Test
"Exit room turn and go through doar on the left. Wait at the entrance."
Figure 1: Illustration of proposed setup and algorithmic framework for VLN-ORL. The proposed setup involves training the agent on dataset primarily comprised of suboptimal demonstrations of varying lengths (middle two rows). Unlike conventional VLN setups that rely solely on expert trajectories (top row), we condition our agent on greedy rewards during training to capture the degree of suboptimality without imposing excessive assumptions on the environment. Positive rewards guide the agent lowards the goal, negative rewards steer it away, and zero rewards prompt the agent to halt its movement. During testing, we condition the agent optimal greedy rewards for exccuting instructions successfully (last row).
indicate that reward-conditioning fosters the development of a more robust model, one that is less sonsitive variations in the training data.
To summarize, our major contributions are as follows:
We introduce novel problem setup, VLN-ORL, aimed at scaling VLN agents through the utilization of extensive suboptimal demonstration data.
We design various noise models for generating suboptimal datascts for R2R (Anderson et al. 2018 and RxR (Ku et al. 2020 environments.
We propose simple and effective approach for extending any existing VLN agent architecture to Irain on suboptimal demonstration data via reward-condittoningg
We empirically demonstrate that revard-condittoning can significantly improve the performance of VLN agents trained on suboptimal demonstrations, even in complex and intricate environments.
Since we propose using suboptimal datasets for effective data scaling, there may concerns about the potential introduction of unsafe actions. However, it's important note that while imperfect demonstrations may not follow optimal routes, not of them lead to unsafe actions. The offline RL setting assumes the existence of logged dataset of imperfect demonstrations. Notably, this dataset not collected by the
3

---

Published in Transactions on Machine Learning Research (04/2024)
learning agent itself; instead, it is assumed to be collected by other agents or behavioral policies. Hence, the agent we are trying to learn is not taking unsafe actions, since it is not even collecting the offline data. More generally, our main contribution in this work is not related to questioning how these imperfect demonstrations would be collected; it is more related to the question of how one can learn a policy from suboptimal offline datasets without any further exploration. This aligns with the common setup prior works on offiine RL (Fu et al. 2020a; Fujimoto et al. 2019), where the emphasis is placed on learning from historical data without requiring additional exploration during the training phase.
2 Related Works
Vision-language navigation: In recent years, there has been significant interest in developing visually grounded agents that can follow language instructions. One notable contribution is the introduction of R2R dataset (Anderson et al. 2018), where a seruuence-t--eqqueee
LSTM model was used to solve the VLN task. A critical challenge in VLN is to develop agents that can generalize well in unseen environments. Since training data is limited, many approaches use data augmentation or introduce auxiliary tasks for improving data efficiency. Speakor-follower (Fried ct al., 2018) models synthesize new instructions for data augmentation and implement pragmatic reasoning. Dnvprop (Tan et al., 2019) proposes an environmental dropout strategy to generate new environments. AuxRN (Zhu et al. 2020 trains the agent with self-supervised auxiliary reasoning tasks to leverage rich semantic information in the dataset. EnvEdit (Li et al., 2022) generates new environments by varying style, object appearance and object classes of existing envirouments. Our work is complementary to these works and con also benefit. from augmentation stralegies.
In related direction there has been an increase in the use of pre-trained vision-language models (Chen et al., 2020 Li al.] 2020 Su al. 2019 Li et al., 2019 for tasks related to VLN (Hao et al., 2020 Hong et al. (Hao al.) 2020 pre-trains vision-language encoder on a large number of image-text- action triplets in self-supervised manner and fine-tunes it downstream tasks. VLNCBERT (Hong et al.
2021 PREVALEN
2021b implements a recurrent function to store navigation history in state token. Follow-up works include ADAPT (Lin et al. 2022a) which provides the agent with act ion prompls to enable explicit learning of action- level modality alignment. HAMT (Chen et al. 2021b) uses long-horizon history encoded using a. hicrarchical vision transformer along with text and current observation to predict next action. AirBERT (Guhur et al., 2021) gathers vast amount of image-caption pairs to train for math-instruction matching task. These methods perform online exploration in addition to learning from manually annotated R2R and RrR datasets to train the agent effectively. However, we focus on developing techniques that can learn from offline sub- optimal dalasels reducing the need for exploration and developing maruualy annoloted dalasels.
Offline RL is a subfield of RL where the agent learns a policy from a fixed dataset of transitions without any interaction with the environment. Imitation-stylee approaches like behavioural cloning (BC), filtered BC, and weighted BC learn a policy that mimics actions of the offline dataset. A recent class of approaches (Chen et al.] 2021a| Kumar et al.) | 2019b) are based on conditional BC which involves conditioning the learned policy
Offline RL:
context variable such as the reward, the goal state or the expected return. We focus these methods here for their simplicity, effectiveness, and direct use in our paper, while deferring the survey of additional works in value and model-based methods for offline RL to Appendix C Kumar et al. (2019b) directly condition the agent's policy on external reward signals to optimize its behavior in RL tasks, demonstrating the potential of using any collected experience as optimal supervision when conditioned on policy quality. Decision Transformer (Chen et al., 2021a: Zheng et al.] 2022) parameterizes the conaltional BC policy via a transformer conditioned on rewards-to-go token. RvS (Emmons et al.} 2021) shows that the performance of such methods greatly depends on model capacity and conditioning information. Recent works (Janner et al.) 2022; Wang et al.) 2022, Ajay et al., 2022) have combined diffusion models with RL algorithms to solve sequential decision-making tasks. Our work sets itself apart from these conditioned BC approaches by introducing a novel reward token specifically tailored for flexible conditioning of VLN agents.
Several works have previously introduced tasks and datasets to benchmark progress in offline RL. They have mainly used pre-trained behavioural policics ranging from a random initial policy to a ncar-optimal policy for generating datasets. D4RL (Fu et al., (2020a) introduces range of dataset collection procedures based the replay experiences and rollouts of unconverted pre-trained policies for physical environments such
4

---

Published in Transactions on Machine Learning Research (04/2024)
MuJoCo. Singh et al. [002)) argue that practical applications of RL involve learning from datasets with high behaviour variability across the state space. [hhou et al. (2022) use a hand-designed expert policy to collect data for real-world robotics tasks. ReViND (Shah et al. 2022 introduces an offline RL system for long-horizon goal-conditioned robotic navigation where the goal is a 2D GPS coordinate expressed in the agent's ' frame of reference. Nair et al. (2022) learns vision-based language-conditiones
manipulation lasks from entirely offline datasets. While acknowledging the impact of these works, our work differs from them as we introduce the first offine RL system for vision- langugg navigation in 3D environments.
3 Preliminaries
Setup: We consider a multi-task extension of the Markov Decision Process (MDP), which can be represented M (S, A, P, (T)), H). Here, S represents the set of states, A refers to the discrete set of actions, P denotes transition dynamics, R represents the task-conditionedd reward function which can depend more states, p(T) denotes distribution over lasks specified by language instructions and H refers to the episode horizon. The term "episode horizon' refers to the predefined maximum duration or length of an episode during which an agent interacts with its environment to achieve a specific goal. Let I denote the set of language instructions and Im refer to the subset of instructions correspouding to a task Tm. We have access to an offline dataset of N trajectories, where trajectory Tn = [((#on: don); (Sin; aiz):.., (sfns afn)) in] consists of a sequence of state-action pairs along with the instruction ins having final state Snn for some positive integral <= N. The state refers to the tuple, 8 = {C,, d2, Φ1), where Ct represents the 3D position and Of and de represent the angles of heading and elevation respectively. Given the language instruction in, the agent observes a panoramic view of the environment at every timestep and takes an action at which takes it from state 8, to state 81+1. Specifically, the agent chooses one of the navigable locations returned by the environment at every timestep. The agent executes actions sequentially to navigate the environment and takes the stop action when it reaches the goal location.
one
Environment: We primarily work with the Matterport3D enviromment Chang et al. (2017) in this work. The Matterport3D simulator is a large-scale interactive RL environment constructed from the Matterport3D dataset. It consists of 10, 800 panoramic views constructed from 194, 400 RGB-D images of 90 building-scale scenes. The Room-to-Room (R2R) dataset contains 21, 567 open-vocabulary, crowd-sourced navigation instructions collected from Matterport3D. Each instruction provides a route that spans multiple rooms and consists of about 29 words on average. For instance, an instruction from the R2R dataset might read: "Proced upstairs, pass the piano through the archway straight ahead, turn right at the end of the hallway where the piclures and lable are located, and slop al the moose anllers on the wall." Successfully following such instructions requires a good language understanding of spatial concepts in the real world and the ability to interpret 3D visualizations.
The RxR dataset is also built upon the Matterport3D simulator. It comprises 33, 954 English instructions with an average length of around 100 words. When comparing paths in RxR and R2R, we note that RxR paths tend to be longer, spanning an average of 8 edges and 14.9 meters, whereas R2R paths span an average of edges and 9.4 meters. Additionally, variation in length between different RxR paths is much greater than that of R2R paths.
4 Methodology
Since VLN tasks involve navigating through large, complex environments, it can be difficult to collect sulficient amounts of high-quality data through online interactions alone. With oflline RL, however, agents can learn from diverse experiences that may not be feasible to collect through online interactions. In this regard, we propose new setup called VLN-ORL, where VLN agents are trained on suboptimal logged demonstrations.. Precisely, there are 2 key challenges that VLN-ORL seeks to combat: (a) choice of offline RL. algorithm, and (b) design of suitable reward token for conditioning.
Choice of offline RL algorithm: Valuo-basod (Peng et al. 2019 Wu al. 2019 Siegel et al.
2020 Kumar
et al. 2020) and model-based approaches (Kidambi et al.] |2200] Janner et al.) 2019 offine RL are either unstable to train suffer from compounding errors during planning (Chen et al. 2021a).
Incorporating
5

---

Published in Transactions on Machine Learning Research (04/2024)
such approaches in available VLN techniques to work with offline datasets would be cumbersome task. In contrast, conditioned BC approaches have shown performance at par with value-based approaches and are easier to scale with as they are trained using architectures and objectives similar to supervised learning (Reed ct al. 2022 Lee et al.) 2022). The key idea here is to condition the policy on a suitable variable, c.g., return or goal state. At test time, we set the conditioning to the desired value of the variable e-g., an expert relurn value. In this work, we will focus on extending conditional BC approaches for VLN-ORL through design of a practical reward token.
Designing reward tokens for conditional VLN agents: The key challenge for VLN-ORL is to charac- terize the degree of suboptimality in offline trajectorics. To this end, we need to define a reward token that can be computed easily during training and used to specify desired optimal behavior during testing. This requires assumptions on the observability structure of the environment. Guided by practical concerns, we propose a minimal set 3 assumptions. First, we note that general language instructions can be ambiguous (e.g. "go to the room" " can refer to any room in big house). To avoid unidentifiability in reward design, we assume that the language instruction for the training trajectories corresponds to single pair of 2D goal coordinates. Second, and goal coordinates. Third, we assume that at test time, the agent can detect complction of a language instruction, e.g., the agent might observe red sign at the goal location to indicate success.
assume that during training, we have access to the agent's current coordinates
We now make a few remarks to justify and motivate these assumptions. Note that the first two assumptions apply only for the training trajectories in our offline dataset. At test time, we allow the mapping wetween language instructions and goal states to be one-to-many, and do not assume any knowledge of the goal coordinates. During training, these assumptions are necessary to define suitable reward function (discussed below). The third assumption allows the agent to terminate the episode and avoid looping behavior, e.g., cycling through one or more goal room(s).
Given the above set of assumptions, we can now define a reward token for conditioning the agent during training and testing. The reward token relates to the presumned reward at each state, which can be either sparse or dense. Typically, sparse rewards for VLN indicate whether a task was accomplished (Nair et al., (022).. In offline RL, the majority of demonstrations are typically suboptimal and hence, relying on a sparse reward signal for conditioning the agent may not generalize well at test-time. However, if we have access to geometric map of the environment and coordinates of the agent state and the goal state, one can compute a dense reward function simply based on shortest path distances and optimize it via a path planning algorithm. However, we do not assume the map is known, as is typical for complex real-world environments, especially for multi-task open-world language guided scenarios at test-time.
Previous rescarch (gg et al., 1999) has shown that potential-based rewards, derived from potential function defined over the environment's 's states, can ensure the learned policy remains optimal relative the original reward function. These rewards are based the difference in potential function values between states, providing a way to shape the agent's 's behavior without altering its optimal policy. Leveraging the principles of potential-based rewards, we proposc to bridge the gap between sparse and densc rewards by conditioning the agent policy on a suitably shaped reward loken, referring to change in displacemenl across two consecutive states, denoted as åD. Let D(s, Gi) represent the Euclidean distance from an arbitrary state s to the goal state G. We define a dense version of reward token 8D for use in training as:
8 c, train (St. S1+1, G) = D(s,. G) - D(5++1, G)
for 12200 0. A positive value indicates that the agent gets closer to the goal whereas a negative value indicates that the agent is heading further away from the goal at time t + 1. A value of 0 indicates no displacement, which would be the reward when the agent has reached the goal. We make 2 remarks here: First, note that ò Delenne is a greedy measure of success, as it is possible that in some cases, the shortest path for an agent might involve navigating to locations that are further from the goal but open pathways that are significantly shorter in the long run. However, the positive side of this greedy approximation is that we do not require knowing full map of the environment during training. Second, evaluating 8 deenns requires knowledge of the next state st+1- While this is available during training in our offline trajectories, we cannot possibly know its value at test-time without executing an action at time t. We can handle this issue test-time by identically
6

---

Published in Transactions on Machine Learning Research (04/2024)
conditioning the model fixed positive value, without loss of generality, say +1, and 0 when the agent has reached the goal.
{
0 if G otherwise
8D,, test (Sts S141, G)
The justification for this test-time conditioning is that test time, we desire the agent to execute actions that move it closer to the goal. Again, there be false positives for this greedy conditioning but it allows to make lesser assumptions about knowing the environment map and does not require having predictor for next states. One issue that can occur in practice is that d and åDusst may not be on the same scale across different instructions and environments. This can lead to a distribution shift. To mitigate this issue, we propose a sparse scale-agnostic variant of the reward token that simply looks at the sign of the change in displacement:
1
if D(s,, G) > D(s G) if D(s., G) < D(s+11, otherwise
Deprrre train (5t, S1+1, G)
-1
As we will show our empirical ablations, while both Sparee and d can be used for conditioning during training, we find the former works better as does not suffer from scale mismatch at test-lime. The pseudocode for reward-condittoning algorithm is given in Appendix A
5 Experiments
We demonstrate the effectiveness of reward conditioning on two recent and representative VLN models: VLNOBERT and MTVM (Lin et al. 2022b Additionally, benchmark our approach against return- conditioning, which has demonstrated strong performance in offline RL tasks.
5.1 BERT-ORL
VLNOBERT is a multi-modal transformer model which consists of self-attention and cross-attention layers to fuse the information from visual and lext modalities. It implements a recurrent function in V&LBERT (Hao et al. 2020 to capture the entire navigational history the state token representation which then used for action prediction. We introduce VINOBERT-ORE which refers VINOBERT conditioned on the reward token. For reward conditioning, we add the reward token to the state token several steps in the pipeline (more details in Appendix B.. Since the state token in VLNØBERT captures navigation history, this allows the model to use the reward token along with navigation history for action prediction. We train the networks from scratch by minimising cross-entropy loss between predicted actions and ground-truth actions in the proposed offline datasets.
5.2 Return-conditioned VLNOBERT
In this approach, instead of conditioning VLNOBERT on the proposed reward token, we condition it on the returns-to-go token. The conditioning mechanism remains the same, involving addition of the returns-to-go token to the state token multiple steps in the pipeline. During training, each timestep, we define the returns-to-s token as the distance of the agent from the goal at that particular step. At test time, since we cannot assume knowledge of the agent's distance from the goal at each step, we initialize the returns-to- to-go token with the maximum length of trajectorics in the validation set. Subsequently, we continuously update it by subtracting the distance traveled by the agent at each step. To ensure fair comparison, we set the returns-to-ge token to zero when the agent reaches close to the goal, under the assumption that the agent can detect its proximity to the goal. We report all the results with the test-time initial return as the maximum length of trajectories in the validation set. However, we also include a performance comparison using the average length of trajectories in the validation set the initial return in Table g

---

Published in Transactions on Machine Learning Research (04/2024)
5.3 Offline RL Datasets for VLN
Given the lack of any existing studies in offline RL for VLN, we design datasets of varying difficulty levels so as to gencrate a challenging benchmark for VLN-ORI. We used a pro-trained HAMT agent to generate trajectories for instructions in train sets of R2R and RxR datasets. While it's true that the HAMT model is trained online, its role within our study pertains solely to the generation of an offline dataset. Tn the context of offline RL, the training process involves utilizing predetermined set of trajectories, without engaging in online exploration. Thus, our approach aligns with the principles of offiine RL, as we exclusively employ the offline dataset for agent training, rather than resorting to online exploration. This distinction remains consistent even in cases where trajectories are sourced from the HAMT model. It is important to emphasize that our core focus lies in advancing offline learning capabilities, irrespective of the trajectory generation method. We will open source our datasets for wider use by the community.
We have created two versions of cach datasct D which generated by rolling out HAMT: 1) D-R2R, generated using train set of R2R, and 2) D-RxR,, generated using Irain set of RxR. Given the instruction and visual state at any step, the model predicts an action which takes the agent from one state to the other. The trajectory ends when either stop action is chosen or maximum time limit is reached. For cach instruction in the train set, we generate a trajectory using the pre-trained HAMT model. In addition, we record distance of the agent from goal at every timestep. Specifically, we introduce these five datasets (D): Expert: This dataset consists of the trajectories obtained by rolling out pre-trained HAMT policy.
15%-noisy data: It includes trajectorics that have been perturbed with 15% noise. That is, cach step, we execute action from HAMT policy with 0.85 probability and a random action otherwise.
30%-noisy data: Same as above, except that we raise the noise probability to 30%.
Mixture: This dataset consists of a mixture of trajectories from expert and 15% noisy datasets.
Random: We also consider a challenging dataset generated by a completely random policy.
5.4 Experimental Setup
Datasets: We generate several offline datasets using instructions in train sets of R2R and RxR. The R2R dataset has 14,025 instructions in the train set and 4,173 instructions in the test set. The validation set is further divided into val-seen and val-unseen, having 1,020 and 2,349 instructions respectively. We use English subset of RxR which includes 26,464 path-instruction pairs in train set, 2,939 pairs in the val-seen set and 4,551 pairs in the val-unseen set.
The experiments were performed on NVIDIA A100 GPU, We used Adam optimizer with learning rate of 1e-5 to train the models. The batch size was kept as 64 and the models were trained for 500K iterations. We trained all the models from scratch in the offline RL setup. The image features were extracted using ResNet-152 (e et al. 2016) pre-trained on the Places365 dataset. The model with the best SR on validation unseen set was selected in each case.
Implementation details:
Evaluation Metrics: We use standard evaluation metrics: (1) Trajectory Length (TL): average length of agent's 's trajectory (in m) (2) Navigation Error (NE): average distance between the agent's 's final position and the target (in m); (3) Success Rate (SR): ratio of trajectories reaching the destination with maximum error of 3 m to the target; (4) Success rate weighted by Path Length (SPI...
5.5 Results and Discussion
The performance of VLN&BERT, renuiteconditioned and reward-conditionee VLNOBERT models was eval- uated validation sets of R2R and RxR datasets after being trained the proposed sub-optimal offline datasets. In particular, the model trained on R2R version of the generated datasct (D-R2R) was tested on the R2R validation sel, and likewise for RxR. In the proposed method, we assume that the agent can detect the completion of an instruction. To ensure a fair comparison, the episode of VLN&BERT is terminated upon reaching the goal state, while the returns-to-go token is set to zero for the return-conitioned agent. Additionally, since the goal states of trajectories in the test set are inaccessible, the reported results do not include performance metrics on the test set. Moving forward, we refer to VINGBERT as the baseline model.
8

---

Published in Transactions on Machine Learning Research (04/2024)
Table 1: Performance evaluation of different VLNOBERT agents on R2R and RxR ORL datasets.
RIR
RxR
Dara
Methoos
seem
Val
anen
Val
S
TL.
NE.
TL
SPL.
T1
NE.
SB.
Eappet
VENCBERT Retune BEST
11.35
00:10
35.10
13.23
10.31
31.35
15:42
$7.56
31:41
5.04
63.50
17.15
6.27
$3.00
23:29
18:79
23.70
22.39
37.07
31.85
VLN-- BERT
VINCHERRT RT--OR----- Dersss
15.24
66:91
66.00
00.75
15:17
16.55
19:37
10.02
65.25
15.75
15.05
01.01
VI MNN.BRRTT-OELE Spaae
15.71
65.09
68.46
16:39
08.54
63.00
19:05
10
10.76
47.25
43.11
15.36
45.15
35%
VENCHERR
11:00
01.61
32.14
90.35
11:00
7:01
16,102
16.21
11:17
35.76
15.36
11.00
01:33
Remmm VLN: RESET
0.56
31.05
13.35
17.33
6.06
56.34
42.3
23:22
15.25
20.57
21.00
11.42
37.00
31.35
VEN.BERT T-ORL Desso
17:35
7.55
63.04
51.05
15.00
0.79
00.02
57.50
23:33
11:00
41.82
38.57
15:00
9:56
18.46
43.00
VENUBERT SO--RR------
15.55
7.55
63.00
57.50
15.08
0.55
07:48
00:44
33.51
10.75
46.75
12.64
18:13
0.18
51.40
46.74
Mixtume
VUNCHERT
10:43
0.91
61.31
70:17
10:97
0:45
57.17
10. 12.
17:01
10.08
17.50
30.07
17.76
9.35
17.11
01.85
Retummt VIL ViNea-ae
13.03
5.38
61:12
56-31
15:00
5,50
11:30
55. 139
11 85.85
10:57
31:33
30:76
9:36
35.06
27:73
VLN- BERT--RR- M-R---------aa Dense
14.76
7.02
66.33
00.61
16.90
6.72
07:35
$3.85
17:45
10.75
43.86
41.35
15.44
3.56
50.36
41:27
YLN: BERT--RRRRRLR Sporee
15.77
6.74
66.81
00.29
15.00
6.65
08.92
01.73
33.05
10.05
50.3 00.80
40.56
15.58
8,51
54.25
40.3
YUN BERR
30.36
0..53
20.15
15:03
56.78
3.35
20123
17.57
35:37
15.00
25.35
52:12
21.05
11:20
15.00
23:65
Returet VINCHMEET
35.56
0.76
16.36
13:64
87.30
9:37
16:13
11.0 07
95 86
13.00 30
23.
19.13
24.77
12.41
25.00
21.70
10005 uokey
VLN-BER BEE---OLL--
7.05
18.81
06.24
18.04
7.87
16.99
45.05
23003
12:09
38.05
S1.84
21.07
10.05
30.26
94.85
BERR--RL-Spanee
7.71
$7.88
50.79
18.
00.60
58.11
22.54
12:25
38.82
94.59
20.57
43.95
The results, which are showcased in Table 1 indicate that reward-conditioned BER outperforms both the baseline and the relurn-conditioned model on all oflline datasets, including Expert, Noisy, Random, and Mixture. The reward-conditioned model achieved success rate that was at least 5% higher than the baseline on both R2R and RxR validation sets, even when trained on the Expert dataset. Moreover, the reward- conditioned model demonstrated a siguificant improvement on R2R validation unseen set, achieving around 13% higher success rate compared to the baseline. Furthermore, the reward-conditioned model achieved around 20% higher success rate than the return- return-conditioned model on the validation seen set of RxR. The improvement due to reward-conditioning was even more pronounced when the models were trained on Noisy datasets. Specifically, the reward-condiitoned model achieved a success rale about 15% higher than the baseline on the R2R. validation sets and around 5% higher on the RxR validation sets when trained on the 15% Noisy dataset. Compared to the return-conditions model, the improvement was around 10% on the R2R validation sets and around 15% on the ExR validation sets.
Notably, when the models were trained on 30% Noisy datasets, the reward-conditioned agent achieved a 20% higher success rate as compared to the baseline on the validation unseen set of R2R. On the RxR validation sets, the seward-conditiones agent achieved around 15% higher success rates than the return-condee ceuurn-condttonee agent. Furthermore, when trained on the Random dataset, the reward-conditioned model achieved roughly 40% higher succoss rate on the R2R validation sets compared to both the bascline and the return-conditioned agents. This demonstrates that the reward-conditionel model is robust lo sub-oplinal training datasets.
Moreover, the results show that addition of noise negatively impacted the performance of both VLNÖBERT and retumncondiiioned models, while the performance of reward-conditioned model was not significantly affected. This also evident from Fig. (d), which clearly illustrates that as noise is introduced, the performance gap between the reward-conditioned model and the baseline widens. The model performance the Mixture dataset was slightly better than that on the Expert datasct, possibly due to inclusion of 15%-noisy trajectories along with expert trajectories, providing more diverse set of experiences for training. Furthermore, we note that the reward-conditioned model occasionally demonstrates higher navigation error than the baseline. This discrepancy can be attributed to our model being conditioned on a zero reward only when it reaches proximity to the goal. In cases where the agent fails to reach the goal, the conditioning on a +1 reward incentivizes continuous movement, leading to both higher navigation errors and increased trajectory lengths in certain instances. Overall, our cvaluation clcarly shows that offline RL via reward conditioning is an effective approach to improve the performance of VLN agents on suboptimal datasets. It enables the model to learn from noisy datasets and achieve superior results, especially in challenging scenarios such as on the Random dataset.
Interestingly, while the return-conditioned model outperforms the baseline on most of the R2R datasets, it demonstrates less effectiveness on the RxR datasets. This discrepancy may be attributed to the signifi- cant variance in trajectory lengths within the RxR datasct, which hinders generalization when initializing the test-time returns based on the maximum trajectory length. Consequently, this results in a decline in performance, highlighting the limitations of return-conditioning in scenarios where test-time returns are un-
9

---

Published in Transactions on Machine Learning Research (04/2024)
known. Furthermore, as previously discussed, present the performance comparison of return-conditioncl VENT on RxR datasets with different test-time initial returns in Table 9 The results illustrate that
using the maximum length of trajectories as the initial return yields superior performance compared to using the average length.
Why is the reward-conditioned model able to learn from suboptimal datasets?
We have a reward ra associated with each action or in the dataset, defined as:
train (St, 1,+11 G) = D(s,, G) D(s+11,))
triin Here, I. represents the time step, and 2 O. The resulting value of D ( 81+1, G) provides valuable insights: a positive value signifies that the agent is making progress towards the goal, while a negative value indicates movement away from the goal. A value of zero represents no change, which occurs when the agent has successfully reached the goal.
Now, in the context of the suboptimal dataset, certain actions contribute to the agent's progression towards the goal (yielding positive rewards), while others cause the agent to move away from the goal (leading to negative rewards). At a given time step t, let's say a specific trajectory involves action associated with reward rg. The model M which predicts actions would be conditioned on this reward rt at that timestep. Consoquently, the model M knows that it must generate an action that would lead to an change in distance indicated by Tt- If the action at is goal, even when the optimal choice is move towards it the corresponding reward It would be negative, Conditioning model M on this negative reward effectively instructs it to generate action that aligns with the suboptimal behavior demonstrated in the dataset. Paradoxically, this doesn't 'tt hinder the model's learning as it already knows that it needs to output an suboptimal action (indicated by a negative reward) at that timestep. Thus, minimizing the disparity between predicted and actual actions makes the model grasp the connection between actions and their associated rewards.
causing the agent to move away from the
In essence, through this reward-guided training process, the model acquires the ability to predict actions contingent upon reward information. This dynamie becomes particularly advantageous during testing. At test time, we can condition the model with positive rewards each timestep. As result, the model is primed to consistently generate actions that propel the agent closer to the goal. This predictive behavior is a direct result of the model's training, as it has learned to associate positive rewards with actions that effectively drive progress lowards the goal.
5.6 Ablation studies
Is the proposed reward token too greedy?: It might seem that the reward token used for conditioning the model is too greedy and can lead the agents to get stuck in cases where they need to consider the long-term impact of their actions. To address this concern, we conducted comprehensive analysis of the evaluation sets to gain deeper insights into the effectiveness of our approach. More specifically, we examined the trajectories within our evaluation sets to determine the percentage of cases where the agent must deviate from the intended goal at least once to eventually reach it, indicating a capacity to reason about the long-term consequences of its actions. We observed that within the R2R validation sets, this scenario is relatively infrequent. However, within the RxR validation seen and unseen sets, a substantial Pertion-54..72 and 55.11% respectively-of trajectories fall under this category. We undertook separate evaluation of our approach specifically on these types of trajectories. To achieve this, we partitioned each of the RxR validation sets (both the "val seen' and "val unseen' sets) into two distinct subsets. The first subset, referred to as "val tough', comprises trajectories that involve deviations from the goal at least once. On the other hand, the second subset, termed as 'val easy", includes trajectories that do not involve any such deviations. In order assess the effectiveness of our approach, we mcasured success rates (SR) and success rates weighted by path length (SPL) on all four of these subsets: "val tough' and "val easy" within both the val seen' and 'val unseen" sets of RxR. The results are shown in Table 2
10

---

Published in Transactions on Machine Learning Research (04/2024)
Table 2: Performance evaluation of VLNOBERT agents on different subsets of RxR. valiclation sets.
Datasets
Methods
Val longh
Val
Vidl
tongh
Vai unseen easy
SK T
SPL T
SRT
SPLT
SKT
SPL T
SRT
SPLT
Expert
VENCBERT
36.72
33.15
53.31
52.01
36.12
33.21
62,31
60.28
ReturnC VENOBERT
21.91
17.31
30.00
33.59
29.35
24.38
46.74
41.14
41.12
36.42
61.07
57.47
41.63
36.33
68.38
64.45
15% noiey
VLNCBERT
31.58
25.5
52.23
30.63
33.53
30.55
58.30
56.77
ReturnC VINOBERT
22.91
18. 6G
41.53
35.07
26.59
00
49.44
43.16
BLN::BRT--RR-SSEEE
38.92
34.76
56,98
52.83
39.95
35.14
66.62
61.89
30% hoisy
VENCBERT
26.49
24.67
44.21
42.73
28.51
25.93
52.72
51.03
ReturCC VLNOBERT
21.85
18.02
39.38
35.38
28.35
23.38
47.72
41.68
33.71
30.14
47.88
37.4
32.23
58.56
Handow
VLNCBERC
14.79
13.99
29.84
27.08
23.17
20.24
38.4
36.59
ReturnC VENOBERT
17.45
13.44
30.16
25. 25.28
18.06
14.79
36.81
31.01
VLNÇBBRTT BERR-BRR--L-LSaee ORL--Sprse
28.69
25.04
46.06
11.96
31.66
26.75
55.41
50.14
Table 3: K-fold validation results for VLNOBBERT agents trained on different subsets of the 30% Noisy R2R dataset.
R2lI vul seen Average Accuracy
I2R val unseen
%llatasot
Methods
Average Accuracy
25%
VLNOBERT
33.23
37.36
2.88
VLNOBERT ORL-
54.64
1.71
$4.87
2.32
50%
VINOBERT
44.10
2.61
42.02
2.35
VLNOBERT-ORE SELIT-OR---PPHH Sparse
59.97
60.17
75%
VLNGBERT
48.58
1.74
45.28
2.1
"LNOBERT-ORLE
63.01
0.91
64.31
1.35
It is evident from the results that reward conditioning improves performance across all four subsets, notably even in more challenging "val tough" scenarios. When trained on the Expert dataset, the reward conditioning approach yields a 5% higher success rate for both seen and unseen "val tough' subsets compared to the baseline. Additionally, it demonstrates a significant. improvement of around 20% compared to the return- conditioned agent on most of the subsets. The improvement was even more pronounced when the models were trained on the Noisy datasets. When trained on the 15% Noisy dataset, reward conditioning led to roughly a 7% enhancement compared to the baseline and approximately a 15% improvement compared to the return-conditioned agent the 'val tough" subsets of both when the model was trained on the 30% Noisy dataset, the reward-conditioned agent achieved a 9% higher success rate on the 'val unseen tough' set compared to the baseline. Even when trained on the Random dataset, the reward-conditioned model showed exceptional performance gains, achieving a 14% success rate improvement for the 'val seen tough' set and an 8% improvement for the "val unscen tough' set compared to the baseline. Additionally, a noteworthy boost of approximately 17% in performance observed for the "val easy" subsets of both 'val seen' and 'val unseen' sets compared the baseline. Moreover, compared to the return-conditioned agent, the reward-conditionel agent demonstrated 15% higher success rate the "val seen easy" set. Although the proposed reward token seems to be greedy, it effectively enhances performance across diverse scenarios, including those requiring deviations from the intended goal.
and unseen validation sets. Furthermore,
Is it really not too greedy?: While 'val-tough" accounts for trajectories with at least one deviation from the goal, it remains uncertain whether the proposed reward-conditionee agent will excel with a higher number of consecutive deviations from the goal. To assoss the robustness of our model, we extended our analysis explore trajectories where the agent consecutively deviates from the goal at least twice, denoted as N2. Additionally, we examined cases of consecutive deviations denoted N3, N4, and N5, where Ni represents the number of trajectories where the agent moves away from the goal consecutively at least times. The analysis, shown in Table g reveals a substantial number of trajectorics involving consecutive deviations of the agent from the goal. In this table, NO represents nunber of trajectories which do not involve any deviation from the goal. Let Ti denote the set of trajectories where the agent moves away from the goal atleast times consecutively. We assessed the performance of the proposed mothod, the bascline agent and the ceturn-onddiioned model on each of these sets (Ti) across both validation seen and unseen sets of RxR. We report the SR and SPL. for each set in the Tables and
11

---

Published in Transactions on Machine Learning Research (04/2024)
Table 4: Impact of training seed variation on performance of VLNOBERT agents on 30% Noisy R2R dataset.
R2R val seen
R2R val unseon
%daatset
Methods
Average Accuracy
Average Accuracy
0
VLNOBERT
39.13
1.03
37 97.93
0.17
25%
MENNOBERTTRRE Opaase
53.94
0.34
55.39
0.51
VLNO FINOBERT
43.8
0.64
41.47
0.11
50%
59.05
1.01
59.18
0.51
VLNOBERS
47.41
0.2
44.81
0.15
75%
BERT-ORI-----
62.53
0.3
63.87
0.26
Duectt Roner
125
(a) Validation Seen
(b) Validation Unseen
(c)
Validation Seen
(d) Validation Unseen
Figure 2:
Performance comparison of VLNOBERT and reward-conditioned VINOBERT on the validation
sets (a, b) based on varying sizes of the training subset from the 30% Noisy R2R datasot (c, d) as a function of the level of noise in the proposed R2R training datasel.
As evident from the data presented in the tables, employing reward-condine conditiioniiiiningg...ooo.. yiclds performance enhance- ments, even in scenarios involving consecutive deviations from the goal. When the models are trained on the Expert dataset, the utilization of reward-conditioning results in performance boost of around 5% the T2 set within the validation unseen set compared to the baseline. Moreover, this enhancement extends to other sets (Ti) within the validation seen and unseen sets, with performance improvements ranging from 3% to 5%. Compared to the return-conditioned model, the reward-conditioned model exhibits approximately a 15% higher success rate on most of the Ti sets within validation seen. When trained on 15% Noisy dataset, roward-conditioning enhances performance by around 4-5% across most sets compared to the baseline.
Upon training on the 30% Noisy dataset, the efficacy of reward-conditioning becomes more pronounced, yielding approximate 7% improvement in performance on the T2 sets in both validation seen and unseen sets. Across other sets (Ti), performance enhancements of approximately 3% to 7% are observed. Moreover, compared to the return- return-conditinn conditionnd agent, the performance improves by around 10% on most subsets of the RxR validation seen set. When trained on the Random dataset, the roward-conditioned agent achioves a 14% higher success rate compared to baseline on the T4 set of validation unseen. Substantial improvements, around 10%, are observed on most other sets as well. Additionally, the performance improves by around 12% on most subsets of validation unseen set compared to the redurn-conditooned agent. Despite the inherent greediness of the proposed reward token, it remarkably boosts performance across various scenarios, including those necessitating consecutive deviations from the intended goal.
However, it's crucial to acknowledge that our reward-conditioned model exhibits a decline in performance when faced with sets characterized by a higher number of consecutive deviations. This could be attributed to several factors. Primarily, these sets inherently comprises more intricate and challenging scenarios, thereby presenting a more formidable testing ground. Additionally, the model's inherent greediness might occasion- ally pose limitations, especially in scenarios where optimal navigation involves substantial deviations from the goal before convergence. However, the empirical evidence have presented in our analysis conclusively demonstrates the efficacy of our approach in improving performance even within complex scenarios, without entailing the risk of agents getting stuck. Thus, we feel it is reasonable to anticipate that the efficacy of the proposed greedy reward conditioning technique would persist even in intricate and complex environments.
Size of the datasct: We conducted an evaluation of the performance of VLNOBERT and reward- conditioned VLNCBERT trained on different subsets of the 30% Noisy R2R dataset. The results of our evaluation, shown in Fig. ((a)(b), indicate that the performance of both models improves the dataset
12

---

Published in Transactions on Machine Learning Research (04/2024)
Table Performance evaluation of VLNOBERT agents on different subsets of RxR validation seen set.
T2
T3
T4
T5
Datasets Expert
Methods
SR
SPL
SR
SPL
5R
SPL
SR
SPL
VLNGBERT
31.82
28.06
25.45
25.51
27.11
21.77
23.52
22.39
ReturnC VEN
16.19
12.4
16,9
13.77
14.21
10.82
16.14
14.35
VLN-HERT--RR--SPE
34.31
28.48
28.
38.09
27.16
27.71
15% BONY
VENCBBRT
26.21
22.87
21.78
19007
17.15
15:41
17.05
15.91
ReturnC DENT
16.43
13.36
17.86
14.54
14.08
11. 46
11.43
10.17
30% money
VENOBERT
23.93
21.08
20.35
15:12
15.9
15:24
16:14
14.30
ReturnC RENT
15.71
13.22
13.48
12.37
9.63
5.4
8.71
8.71
VLNO BERT BERT-ORL--Saas
17.58
Random
VENOBERT
10.14
9.57
8.57
5.12
8.45
8.01
6.33
6.6
ReturnC VLNOBERT
12.38
9.24
11.9
9.24
8.40
6.4
10.19
9.75
VIN--BERT--RR Sparsee
19.05
Table 7: Performance evaluation of VLNCBERT agents on different subsets of RxR validation unseen set.
T5
Datasets
Metthods
SI
SPL
5.1
5P"
511
SPL
51t
STL
Expert
VINCCRETT
ReturnC VENCBERT
90.55
22.85
19.71
26.47
20.85
18.7
15.34
13.94
1.2
13.04
12:19
14:36
VLN BEN---ER-LLLLSp
35.16
30.59
29.36
26.
26.55
24.74
21.01
19.27
VINGRRET
28.48
25.75
23.67
moday
ReturnC VLNGBBEIT
20.81
17.75
17.42
15.67
12.08
11.0 1..04
10.87
10.23
VLN DERT--DEE---SSS Spars
33.48
29.97
29.9
26.53
24.78
22.24
18.12
15.55
30% swiey
ReturnC VLNOBERT
VENGBERT
20.75
39.77
18.42
20.77
17.58
11.53
11:62
10.56
111.22
10.75
VINNGRERT-ORE BERT--RL--SSorr
31.43
26.88
25.79
22.42
20.47
18.46
13.77
12.87
VLNGBERT
0.82
Roniom
ReturnC VENOBERT BERT
12:23
10.41
9.6
8.37
7.67
6.30
6.52
6.32
F.N.BEETT-OLL-S.
24.62
20.36
22.77
19.2
20.06
16.93
13.04
10.16
size increases. Initially, the improvement is substantial, but it decreases gradually increase the subset size of the dataset. This can be attributed to the diminishing marginal benefit of adding new trajectories to the dataset. Since the model may have already encountered similar trajectories before, the value of adding new ones decreases over time.
We further run k-fold validation by randomly selecting x% of the dataset, k=3 times. In Table 3 we report the average accuracy and standard deviation (r). Notably, the prediction variance, indicating that reward-conditioning contributes to the development of more robust model, one that is less susceptible to the specific data partitions.
model demonstrates lower
We assessed the performance of VLNÖBERT and reward-conte corard-conditioned VENOBERT across different subsets of the 30% Noisy R2R dataset using three distinct model sceds. In cach instance, we maintain the data subset fixed and train the model using three different seeds on that particular dataset. The results are shown in Table Notably, the low standard deviations suggest that the model's 's performance exhibits minimal variation with changes in the random seed, highlighting its robustness and stability across different initial conditions.
Training seed variation:
Sparse vs Dense Rewards: To gain further insights into our model's performance, we conducted more detailed analysis by considering dense and sparse reward tokens during training. At test time, we condition the model on sparse reward tokens only. Table ! shows that the sparse-reward conditioned model performs slightly better than the dense-reward conditioned model. One possible explanation for this is that the sparse reward case emulates the test conditions during training, which aids the model performing better at test time. This experiment highlights that reward-conditioning substantially euhances performance, even when distances from the goal location are not known at every step. It is only necessary to have knowledge of goal when the agent reaches close to it. We propose that this information can be readily provided to the agent by placing a beeper at the goal.
How do we condition?: We explore two conditioning approaches for VLNÖBERT: concatenation and addition of reward token to state representation. Section 5.1 discusscs addition technique. In concatenation approach, we project reward token into an embedding space of size one-fourth the dimension of common embedding space. This projected embedding is then concatenated with state representation, followed by
13

---

Published in Transactions on Machine Learning Research (04/2024)
linear layer to restore original embedding size. This form of reward concatenation is applied at every step where reward addition is performed in Section 5.1 We train both models on 30% Noisy R2R dataset and observe that they perform similarly on validation sets, as depicted in Table These results indicate that both conditioning tochniques can provide necessary conditional information.
Alternative baseline model: We further extend our analysis in- clude the conditioning of a more recent VLN model, MTVM (Lin et al, 2022b). The architectural details are discussed in Appendix sults presented in Table I0 clearly demonstrate the effectiveness of reward- conditioning in enhancing performance across all proposed datasets. Even on the Expert dataset, reward-conditionedd model showcases a remark- able improvement approximately 20%. Furthermore, the performance gap between the reward-conditioned model and the baseline consistently widens datasct's noise level increascs. Notably, on the challenging Ran- dom dataset, the rewarl-conditioned model exhibits success rale that is approximately 30% higher than the baseline. These findings underscore the significant performance hancement achieved by swward-oonditionin in MTVM on suboptimal datasets. Since the proposed offline RL method also yielded positive results for VENTBERT, its generalizability holds great promise for improving performance of VLN agents.
Table 5: Performance of different reward conditioning schemes; the 30% Noisy R2R dataset.
B The re-
Methods
Val seen
TL
NE
SR
SPL
Concetenation
16.81
6.91
64.64
69.68
Addition
65396
R2R
Methouls
Val
umsown
TL
NE
SR
SPL f
Concelenation
6.76
05.07
55:13
Addiicion
Table 8: Analysis of the trajectories in the validation sets of RxR.
Set
Total
No
N1
N2
N3
N4
N5
Vol
1098
Val unseen
4551
2043
2508
1366
725
339
1.8
Table 9: Performance comparison of return-conditioned VINO on RxR datasets with different test- time initial returns.
Val
Val
Data
Initial return
TL.
NE
SR
SPL
TL
NE
SR
SPL
Expert
Average
14.17
9.55
30.83
26.93
11.06
8.92
35.68
31.65
Maximun
24.74
37.07
30% Noisy
Average
16.11
9,91
28.62
26.04
15.85
9.12
35.79
30.93
Maximum
23.22
13:25
29.87
25.97
21.99
11.42
37.02
31.58
Random
Average
23.92
13.8
9.49
6.85
23.49
12.36
15.58
12.25
Maximun
21.77
25.99
21.7
Table 10: Performance evaluation of different MTVM agents on ORL datasets for R2R.
Data
Mothods
Val seen
Val unseem
NE
SPL
TL
Expert
MTVV
12.77
6.59
46.82
38.32
12.92
6.01
43.59
35.52
MTVM-ORI--D
16.98
ATVM--RRL Spaase
16.39
6.73
65.23
60.91
15.91
0.75
66.84
59.66
15% moice
ATTVM
11.12
6.41
40.74
95
11.25
6.45
41.
38.22
UTVM-ORI-Dema
17.51
7.05
61.12
56.67
16.79
6.87
62
54.67
KTVM-ORL--Spare
16.66
6.65
66.21
60.43
16:22
0.77
66.84
58.66
30% noise
MTVM
9.00
7.70
36.74
35.9
9.01
28.5
36.57
39.6
MTVM--RL- -easeeeeee
Dease
17.89
7.61
58.18
52.80 22.89
16.63
7.23
61.6
53.97
MTVM-ORLE
Sparse
17.51
6.54
65.13
59.15
10.79
6.24
67.18
58,25
Mixtare
MTVV
9.35
6.04
52.69
51.78
9.23
6.29
52.
541.94
MTVM-ORI-
16.81
6.61
30.61
16.32
65-56
MTVM-ORI-Spann
16.60
6:28
68.46
62.92
16..39
6.79
67.18
59.94
Randon
21.36
MTVM--ORI. Dense
20.58
7.77
53.87
45.33
20.57
7.50
54.96
44.35
Sparee
47.75
19.41
6 Conclusion and Limitations
We introduced the setting of offiine RL for VLN. We showed that conditioning the model on suitably shaped reward token can significantly improve performance of VLN agents trained large suboptimal
14

---

Published in Transactions on Machine Learning Research (04/2024)
VLN: BERT
VLN:BERT-ORL-Sparse
Panoramic view
Heading
Panoramic view
Heading
Instruction
"With the sink to your left and the stove to your right, start walking forward until you exit the kitchen, veering right and entering the second bedroom your left."
Figure 3: Visualisation of panoramic views and headings (view in heading direction) at every step for agents trained the 30% Noisy R2R dataset. The seward-conditiones agent correctly follows the instruction to reach the goal whereas the bascline agent takes the wrong path and reaches elsewhere. Instead of moving forward with the sink to the left and stove to right (indicated by green box) the baseline agent goes in the other direction to the hall (indicated by red box). continues in that direction and eventually exits the house. Conversely, the reward-conditioned agent correctly navigates the route, exiting the kitchen and entering the bedroom instructed.
datasets. Further, we introduced several benchmarks for VLN-ORL to promote research in this area and conducted extensive ablations on the design of reward tokens. While our method has exhibited superior performance on various suboptimal datasets, it is important to acknowledge some limitations. Firstly, our approach relics on prior knowledge of stop state to terminate opisodes, and we plan to address this limitation in our future work. Secondly, encouraging the agent to move closer the goal at each step may result. longer paths in certain scenarios, and we will investigate strategies mitigate this effect. Furthermore, we aim explore alternative offline RL algorithms and architectures, including value-based methods and biffe policies, to broaden the scope of our research and enhance performance in VLN tasks.
References
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is condi- tional generative modeling all you need for decision-makinee arXiv preprint orXiv:2211.1500 2022.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and-language navigation: Interpreting visually-grounder nav- igation instructions in real environments. In Proceedings of the IEEE conference on computer vision and pallern recognition, pp.
2018.
15

---

Published in Transactions on Machine Learning Research (04/2024)
Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. 2017.
arXiv preprint Xie:
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Ar- avind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084 150997, 2021a.
Shizhe Chen, Pierre-Louis Guhur, Cordelia Schmid, and Ivan Laptev. History aware multimodal transformer for vision-and-language navigation. Advances in neural information processing systems, 34:5834-5847, 2021b.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Luu. Uniter: Universal image-text representation learning. In Computer Vision Cicio-FCCCV 2020: 16th Eu- ropean Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part XXX, pp. 104 120. Springer, 2020.
Scolt Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rys: What is essential for oflline rl via supervised learning? arXiv preprint arXiv:2112.10751, 2021.
Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg- Kirk-aikpa,riik, Kate Saenko, Dan Klein, and Trevor Darrell. Speaker-follower models for vision-and- language navigation. Advonces in Neural Information Processing Systems, 31, 2018.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data- driven reinforcement learning. arXiv preprint arXiv:2004.079.11 2020a.
Tsu-Jui Fu, Xin Eric Wang, Matthew F Peterson, Scott T Grafton, Miguel P Eckstein, and William Yang Wang. Counterfactual vision-and-languuage navigation via adversarial path sampler. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part VI 16, pp. 71-86. Springer, 2020b.
Scott Fujimoto, Edoardo Conti, Mohammad Ghavamzadeh, and Joelle Pineau. Benchmarking batch deep reinforcement learning algorithms. arXiv preprint ar Xix1119110001 708, 2019.
Picrre-Louis Guhur, Makarand Tapaswi, Shizhe Chen, Ivan Laptev, and Cordelia Schmid. Airbert: In- domain pretraining for vision-and--anguaen navigation. In Proceedings of the IEEE/CVP International Conference on Computer Vision, pp. 1634-1643, 2021.
Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, and Jianfeng Gao. Towards learning a generic agent for vision-and-languase navigation via pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13137-13146, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Yicong Hong, Qi Wu, Yuankai Qi, Cristian Pedriiguz--Opzzo and Stephen Gould. Vin bert: A recurrent rision-and-language bert for navigation. In Proceedings of the IEEE/CVE conference on Computer Vision and Pallern Recognition, pp. 1643 1643-1653, 1653, 2021.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32, 2019.
Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint XrXiv:2205. 09991, 2022.
Rahul Kidambi, Aravind Rajeswaran, Pranceth Netrapalli, and Thorsten Joachims. Morel: Model-basse based offline reinforcement learning. Advances in neural information processing systems, 33:21810 2221810-2182,, 21823, 2020.
16

---

Published in Transactions on Machine Learning Research (04/2024)
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint or arXi:2110.0000 06669, 2021.
Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. Room-across com-across-room: Multilingual vision-and-langunge navigation with dense spatiotemporal grounding. arXiv preprint WVXiv2201000995. 2020.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019a.
Aviral Kumar, Xue Bin Peng, and Sergey Levine. 191iv 13465, 2019b.
Reward-conditioned policies.
arXiv preprint
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-lcarning for offline reinforce- ment learning. Advances in Newral Information Processing Systems, 33:1179 1191, 2020.
Kuang-Huei Lee, Ofir Nachum, Mengjiao Yang, Lisa Lee, Daniel Freeman, Winnie Xu, Sergio Guadarrama, Tan Fischer, Eric Jang, Henryk Michalewski, et al. Multi-game decision transformers. orXiv preprinl arXiv:2205.155: 15241, 2022.
Jialu Li, Hao Tan, and Mohit Bansal. Envedit: Environment, editing for vision-and-languuaee navigation. Tn Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. -14777 2022.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. orXiv preprint arXix:190 03557, 2019.
Xiujun Li, Xi Yin, Chunyuan Li, Pengehuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Com- puter Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 28, 2020, Proceedings, Part XXX 16, pD. 121 137. Springer, 2020.
Bingqian Lin, Yi Zhu, Zicong Chen, Xiwen Liang, Jianzhuang Liu, and Xiaodan Liang. Adapt: Vision- language navigation with modallty-aligged action prompts. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pp. 14396-15406,, 2022a.
Chuang Lin, Yi Jiang, Jianfei Cai, Lizhen Qu, Gholamreza Hallari, and Zehuan Yuan. Multimodal trans- former with variable-length memory for aooon--nd-language navigation. In Computer Vision-ECCT 2022: 17th European Conference, Tel Aviv, Israel, October 23 27, 2022, Proceedings, Part XXXVI, pp. 380 397. Springer, 2022b.
Chih- Yao Ma, Jiasen Lu, Zuxuan Wu, Ghassan AlRegib, Zsolt Kira, Richard Socher, and Caiming Xiong, Self-monitoring navigation agent via auxiliary progress estimation. arXiv preprint wxXiv1901.0800:. 2019.
Suraj Nair, Eric Mitchell, Kevin Chen, Silvio Savarese, Chelsea Finn, et al. Learning language-condtionned robot behavior from offline data and crowd-sourced annotation. In Conference on Robot Learning, pp. 1303-1315. PMLR, 2022.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Ieml, volume 99, pp. 278-287 Citescor, 1999.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage- regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00117 2019.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth- Maron, Mai Gimenez, Yury Sulsky, Jackic Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprinl ar Xrx::2205.000 06175, 2022.
Dhruv Shah, Arjun Bhorkar, Hrish Leen, Ilya Kostrikov, Nick Rhinchart, and Sergey Levine. Offline rein- forcement learning for visual navigation. Xie preprint or orXin:2212 08244, 2022.
17

---

Published in Transactions on Machine Learning Research (04/2024)
Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.08390 2020.
Anikait Singh, Aviral Kumar, Quan Vuong, Yevgen Chebotar, and Sergey Levine. Offline rl with realistic datascts: Heteroskodasticity and support constraints. arXiv preprint arXiu:2211.0102, 2022.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VI-bert: Pre-training of generic visual-linguisiic representations. orXiv preprint Xrx::11908.. 08530, 2019.
Hao Tan, Licheng Yu, and Mohit Bansal. Learning to navigate unseen environments: Back translation with environmental dropout. arXiv preprinl Arxio:1904.. 01004.441199 04195, 2019.
Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. Reinforced cross-modal matching and sell-supervised imitation learning for vision- language navigation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nilion, pp. 6629 6638, 2019.
Zhendong Wang. Jonathan J Hunt, and Mingyuan Zhou. Diffusion policics as expressive policy class for oflline reinforcement learning. Xiv preprinl arXic:2208.06190 06193, 2022.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint crXfe:1911.11361, 2019.
Qinqing Zheng, Amy Zhang, and Adilya Grover. Online decision transformer. In International Conference on Machine Learning, pp. 27042-27059 27059. PMLR, 2022.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions pattern analysis and machine intelligence, 40(6): 1452 1464, 2017.
Gaoyue Zhou, Liyiming Ke, Siddhartha Srinivasa, Abhinav Gupta, Aravind Rajeswaran, and Vikash Kumar. Real world offline reinforcement learning with realistic data source. arXiv preprint arXiv:2210.06,73 2022.
Fengda Zhu, Yi Zhu, Xiaojun Chang, and Xiaodan Liang. Vision-language navigation with self-supervised auxiliary reasoning tasks. In Proceedings of the LEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020.
18

---

Published in Transactions on Machine Learning Research (04/2024)
A
Methodology
Algorithm
!
describes reward-token conditioning in detail.
Algorithm 1: Reward token conditioning
Training Phase:
Input: Instruction I, Visual features Vi, State token q--1, Ground-ruth action de, Current state St,
Next state 8t+1, Goal location G
Output: Trained policy model M parameters
At any timestep t, prodict action by conditioning M on the reward token
a, M T, V,, 5 T (St, $141, G))
Take ground-truth truth action a₂ to go to next state 51+1
Compute cross-entropy loss between the predicted action a, and the ground-truth action at Optimize loss until convergence to train M
Testing Phase:
Input: Instruction I, Visual state V,, Current state st, State token @t-1
Output: Next state $t+1
At any timestep t, use the trained model M to predict action
4 = M(1, & DDpaas)
Execute the action 0e to move to the new state 81+1
B Experiments
B.1 Model Details
1. BER: We use the architecture of VLNOBERT Hong et al. (2021) as our baseline agent policy, VINOBERT is a multi-modal transformer model which consists of self-attention and cross-attention layers to fuse the information from visual and text modalities. It implements a recurrent function in V&LBERT Hao et al. (2020) capture the entire navigational history in the state representation. At cach timestep, the agent receives three inputs, the previous state token q-1,, the language instruction and the visual features VA for the current scene. VINOBERT employs self-attention mechanism that operates the pross-modall tokens, allowing it to capture the correlation between the textual and visual inputs. This enables the model to determine the action probabilities pf accurately at each timestep t.
tt P = VLN O BERT(Q-1.1 VV))
Initially, the language instruction is fed into BENTBERT and the linguistic features are extracted. The final hidden state corresponding to the predefined [CLS] token is taken as the aggregate sequence representation. It is also used as the initial state representation %0- At every timestep, the agent observes a panoramic view of the environment consisting of 36 images, {- The visual features V2, are extracted using ResNet-152 He et al.] (2016) pre-trained on the Places365 dataset ZZhou et al.] (2017). vision encoder F, is then used project the extracted features into the common embedding space.
V = F.(V)
The visual tokens Vi are concatenated with the state token g:-1 to obtain the state-visual features He. The state-visual representation is then updated by calculating cross- modal attention between the state-visual features H, and the instruction features f..
H Cross Crosstnttntion((FA) He, Sú, 0c)
19

---

Published in Transactions on Machine Learning Research (04/2024)
where de the parameters of the cross-attention module. The updated state-visual features He are then fed to a self-attention module to capture the relationship between the state and the visual features.
H Pi = SelfAttention(Hz (a)
where 0, represents the self-attention module parameters. The state-visual features and the instruction features are passed through the cross-attention and self-attention modules multiple times refine the rep- resentations and capture state-visual relationships. The similarity scores pf computed between the state reprosentation and the visual features in the last sclf-attention module used as the action prediction probabilities. The action predicted with the highest probabilily is chosen at each timestep.
To condition VLNOBERT on rewards, we add the reward token to the state token at several steps. Since the state token captures navigation history, this allows the model to use the reward token along with the navigation history to correctly predict the action. We firstly extract the instruction features fi, the visual features Vú and the state representation 9--1- Thereafter, we add the reward token Daparee to the state representation and concatenate the updated state representation with the visual tokens to obtain the state-visual features Hi Next, cross-attention is performed between these features, with the state-visual features serving as querics and the instruction features serving as both keys and values.
2.
a- 9t-1 + sparse
train,
H. Concat(gi
H CrossAAtennion
-: ==1==1100) [0] Vi" [1
"
In order to strengthen the incorporation of conditioning information by the model, further add the reward token å Deparre to the state token 4-11 The updated state-visual representation is then fed to the self-attention module. This approach ensures that the reward token is given significant consideration by the model during the decision-making process.
" 4-1 9h-1 + Sparee train, H" Concat((*"11 V,,))
"
H P SelfAttention(HH, ..))
The VINOBERT architecture is composed of several blocks, each of which includes the cross-aliention and self-attention modules. We incorporate the conditioning information into the state token in each block. In the final block, action prediction probabilities pt are obtained by normalizing the attention scores between the state token and the visual features. Fig. p depicts the reward token conditioned model pipeline.
Formally, we minimise the navigation loss, which is,
L -=-Σai logri
where of refers to the teacher action and př denotes the action prediction probabilities at timestep t.
3. M MTVM consists of language encoder, a vision encoder and a cross-modalityy encoder. Additionally, also stores activations of the previous time steps a memory bank and uses memory- aware consistency loss help learn a better joint representation of temporal context with random masked instructions.
20

---

Published in Transactions on Machine Learning Research (04/2024)
Initialization
(CLS), [SEP)
Instruction
Self-Attention
70, f
Nav igatiionn
Tross-Attention
Self-Attention
H Pr
Figure 4: Reward token conditioning in BENNBORTT- Initially, the language instruction is encoded by the self-attention module. During navigation, the sequence of state and visual tokens along with instruction features is passed multiple times through the cross-attention and self-attention modules infer the action prediction probabilitics.
Initially, the instruction fed to the multi-layer transformer to get the language representation X. The convolutional vision encoder is then used to encode the image observations. At each step, the vision repre- sentation Ve consists of the encodings of all the candidate image observations.
In order to learn modality representations, cross-modality encoder consisting of self-attention and cross-attention layers used. Similar to VLNOBERT, we add the reward token 8D to the [CLS] token embedding (X(0)) before we give as input to the cross-modality encoder. The language representation X, vision representation Ve, and previous aclivations M, are then fed to the cross-modality encoder as:
x X Sparee
train,
X = [x10]: X(1 :]]
x, M, V2 = C(X, [M; Vi)
where [:] denotes concatenation. The output V. is then given as input the action prediction head to make the action decision for this step:
= MLP(V.)
The memory bank is updated at the end of cach step by reusing the object activations 1 according to the current agent action decision as
Mr (M-11, [V+;;
21

---

Published in Transactions on Machine Learning Research (04/2024)
where k refers to the index of the selected vision output and di is the corresponding directional feature of action at step t. A mixture of imitation learning loss and memory-aware consistency loss was used to train the network from scratch.
B.2 Visualisations
Fig. and show the visualisation of the R2R trajectories of VLNOBERII-RL and VLNGBERT trained on 30% noisy R2R. dataset. Fig. 7 and 8 show the RxR trajectory visualisation for the VLNOBERT-ONE and VINOOBRRT agents trained on 30% noisy RxR dataset. The panoramic view and the heading (view in the heading direction) are shown at each navigation step. In each trajectory, the reward-conditioned model correctly follows the instruction to reach the goal state whereas the bascline model gets distracted between and doesn't 't reach the goal location. For example, in Fig. [ the agent was supposed go straight in the first navigation step but the VLNOBERT agent took the other gate. Additionally, in Fig. 7 the VLNOBERT agent initially got confused but was later able to exit the room. However, it kept walking and finally stopped at wrong location.
VLN BERT-ORL-Sparre
VLNOBER
Panoramic view
Heading
Panoramic view
Heading
Instruction
"Walk straight then turn right and walk down the hallway until you get near the large painting of man."
Figure 5: Visualisation of panoramic views and headings of VIN:BERT-ORI, and VLNOBERT at every step in the trajectory.
22

---

Published in Transactions on Machine Learning Research (04/2024)
BLN-
VLNOBERT
Panoramic view
Heading
Panoramic view
Heading
Instruction
"Go down the hallway, turn left in the circle room, go into the doorway before the mirror and take a few steps in and stop."
Figure Visualisation of panoramic views and headings of VINOBERT-ORR, and VLNOBERT at every step in the trajectory.
23

---

Published in Transactions on Machine Learning Research (04/2024)
VLN:BERT-ORL-Sparse
VLNOBERT
Panoramic view
Heading
Panoramic view
Heading
Instruction
"You are standing facing towards the sofa. Turn right and exit the room through the open entrance the front. Slightly turn right. There is big clock towards your left side. Walk straight till you reach the open entrance towards your left side. Turn left and enter into the open entrance. Slightly turn left and stand in front of the closed door. You have reached your point."
Figure 7: Visualisation of panoramic views and headings of VLNOBERT-ORI and VLNOBERT at every step in the trajectory.
24

---

Published in Transactions on Machine Learning Research (04/2024)
BERR
VLNOBERT
Panoramic view
Heading
Panoramic view
Heading
Instruction
"For your start point you are at doorway facing room that has red printed carpet and a fire place across the room with a picture over the mantel. On either side the of the fire place there are book selves or cases. Turn to your right and exit this room you now enter into a vintage study room you will see table in front of you that has a lamp and a statue to the foot or the legs the table. Turn to the left and move forward behind that table until you reach what it looks like, it could be fire place to your left with portrait above it of man. Once you are in front of that fireplace you have reached your end point."
Figure 8: Visualisation of panoramic views and headings of VINOBERT-ORL. and VINOBERT at every step in the trajectory.
25

---

Published in Transactions on Machine Learning Research (04/2024)
C Additional related works on Offline RL
A common approach to offline RL is to use value-based methods where the goal is to learn policy that maximizes notion of expected cumulative reward, given the fixed dataset of interactions. However, they suffer from issues such as value overestimation and error propagation. Policy constraint methods Peng et al. (2019) ; Wu et al. (2019); Siegel et al. (2020); Kumar et al. (2019a) try to address this issue by constraining the learned policy to stay close to the behavioural policy. Regularization based methods penalize the large Q values to avoid issues pertaining to the OOD actions. Specifically, CQL Kumar et al.. (0200) penalizes the large Q-values for state-action pairs not observed in the dataset. Implicit Q-learning Kostrikov et al. (2021) estimates the Q-values using density ratio function and trains the policy network using importance-wiightee regression.
26


Reviews:
[{"review_id": "TmbFKLT5fn", "reviewer": "Reviewer_5e26", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n\n**Writing:** Overall this is a well-written, clear, and easy to understand paper.\n\n**Dataset:** The authors craft a new dataset/benchmark for their VLN which includes suboptimal trajectories. I think this is a valuable contribution.\n\n**Experiments:** Ablation studies are comprehensive for the most part. \n\n### **Weaknesses:**\n\n**Clarity:**\n\n- Figure 1 isn’t very attractive, and more importantly, isn’t that helpful currently:\n    - It’s very hard to realize what’s going on by looking at the figure alone, and then if I read the caption by itself, it’s a simple enough description to not need the figure at all. I would recommend trying to make the figure itself more clear and make a larger distinction between trajectories and images, what the rewards are doing, arrows pointing out what comes from the what dataset, replacing text with shapes (e.g.: expert demos being shaped as a square is not informative, if it’s shaped as a dataset icon then it’s easy to understand what it is without *needing* to read the label. And there’s no distinction between “expert demos” square and the “VLN-ORL” square but one is referring to a dataset and another is referring to your method.).\n    - It may also be good to simply show how the reward token is used in a generic (transformer) model architecture in this figure next to VLN-ORL, rather than just labeling the rewarded trajectories with VLN-ORL\n\n**Claims/Framing:** \n\n- “However, we focus on developing techniques that can learn from offline suboptimal datasets without the need for exploration..” I would tone this statement down quite a bit, it’s debatable if there will ever be methods which do not require *any* exploration to do unseen VLN, especially given that this paper is only evaluated in simulation and still getting ~50% success rates. I believe that VLN-ORL can actually be complementary to the related works this statement is in relation to, so it should just be framed that way.\n\n**Method Novelty/Technical Contribution:**\n\n- The method is to add a reward conditioning token, just like Decision Transformers and other return-conditioned RL approaches, but which is densified by using reward *differences.* I think this kind of reward densifying is usually treated as an engineering trick and it is definitely not a novel one. I’m not sure of any major technical contribution here, especially given the experiments do not compare against any other offline RL approaches to demonstrate that this specific reward-conditioning works better.\n- In addition, this reward-conditioning has clear problems that actual, non-conditioned offline RL wouldn’t have. If you force the model to be conditioned on reward differences and always condition it on a positive reward difference at test time, then it will be difficult for it to deviate from this behavior. Return-conditioning solves this problem, but the authors noted it’s hard to predict returns at test time. This still warrants a fair comparison, as return-conditioned offline RL has been shown to have some success even when the test-time return is unknown.\n    - Even if you don’t condition on returns, regular Offline RL will still be able to stitch together trajectories and overcome suboptimal actions within a single trajectory,. I don’t see any evidence from the proposed method or the experiments that VLN-ORL can avoid this issue.\n\n**Experiments**\n\n- Due to the above weaknesses in the technical contributions, I think comparing against return-conditioning is required for acceptance.\n- In addition, there should be comparisons against regular offline RL, because the authors are proposing an offline RL method. You could modify the existing VLN architectures with Q-transformer: [Q-Transformer (qtransformer.github.io)](https://qtransformer.github.io/), or follow this for implementing IQL on transformers [[2206.11871] Offline RL for Natural Language Generation with Implicit Language Q Learning (arxiv.org)](https://arxiv.org/abs/2206.11871), for example.\n- Ablation study: The ablation study claims to show that the greedy reward maximization objective prevents the agent from getting stuck. But inclusion criteria for the “val-tough” dataset is just a minimum of one action deviation from the optimal goal trajectory. This doesn’t conclusively demonstrate much, because a deviation of 1 action from the optimal goal trajectory is likely easy to overcome, when actions are learned and averaged over a large dataset, even with a greedy objective.\n    - To prove this is actually true, it authors should partition “val-tough” into multiple subsets of various numbers of action deviations, and compare performance improvement over baselines at each # of action deviations.\n- Standard deviation: I know it’s not that common to have standard deviations reported in these supervised learning settings, but given Table 3 showing varying levels of action prediction accuracy deviation based on small dataset differences, the authors should report numbers averaged over multiple model training seeds too.\n\n**Minor Issues:**\n\n- Page 5: wrong closing quotation markers for “go to the room”\n- Bottom of methods section: Appendix A link doesn’t work\n- Table 3 is too small", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present VLN-ORL, a simple reward-conditioning objective for VLN that adds a dense L2-distance based reward signal to condition models on so that they can learn from suboptimal data. They also introduce a new dataset/benchmark to evaluate VLN agents in simulation with suboptimal datasets.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Unfortunately, while the paper is well-written and interesting, I am not convinced that the reward-conditioning is more effective than existing methods due to the greediness of the reward conditioning. I would like to see every experimental concern addressed before acceptance, namely a comparison with return-conditioned offline RL (with the same models), regular offline RL methods (e.g., with a Q-transformer inspired model), and an ablation study which conclusively demonstrates that the reward mechanism being greedy is fine.\n\nThis reward-conditioning isn’t really a novel idea either, but I think that is OK given they show it works in VLN specifically and that the reward requires L2 distance, which is specific to navigation tasks.\n\nEvery other concern is minor and would strengthen the work if improved upon.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "N0NFqjFEDy", "reviewer": "Reviewer_mjiP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. Please see the above contribution list. In general, I like this novel problem and the efforts behind to make it trackable. \n\n**Weakness** \n1. The concern regarding the high-quality data scarcity is valid to me. However, the simulator indoor-room navigation might not be the optimal setup to study it. Since we actually can construct good navigation data under this setup, and we are not sure about whether the \"\nsimulated\" imperfect data reflects the actual noise (as listed in the intro of the paper). The presentation of the method can be better with a real inaccurate data setup and also with the proposed method.\n2. The design of the reward highly depends on well topology of the. It generally assumes that the graph of the environments has strong connectivity. For sparse-connected network, e.g., \n    ```\n   A1 -- A2 --- A3 .... -- An\n                           |\n   B1 -- B2 --- B3 .... -- Bn\n     ```\n   drawn in geometric distance. Most of the paths (e.g., from A2 --> B1) would involve a long path of negative reward -1 (i.e., from A2 to An). I think that the current reward design is not robust to such sparse graph. The illustration of the above happens in real life as well, e.g., for building with two floors where the ladders are only available at corners. Under this env, the navigation goal across floors can possibly trigger the long-path negative rewards. \n3. The dataset is constructed from a pre-trained VLN model HAMT. I have a little bit trouble in persuading myself regarding the effectiveness of such dataset. The subpoints are: (a) as stated in the paper, the actions are predicted with model that has been trained on the given dataset; if the model is capable enough, the generated paths should reveal the training path? (b) The noise takes random actions; would it be better to still sample from the policy but with higher temperature (i.e., a smoothed distribution)? (c) What's the accuracy of the HAMT model under `Expert', `15%-noisy data', and `30%-noisy data'? I would like to understand how \"noisy\" these data are given whether the final state of the paths reach the target goal. \n4. I also got a little bit confused on interpreting the results. Since the training path are generated by the HAMT model, this strategy is similar to a distillation approach. Thus, the confusion to me is whether the HAMT-generated data is actually noisy, instead of some reasonable data can be helpful to training (as distillation). \n\n\nOverall, I think that it is a new problem formulation to me and I would like to see these work come to the community. However, I would like to convince myself that the experimental setup is somewhat reasonable and the improvement over this experimental setup (by following work) can faithfully reflect the research progress.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "1. The paper proposed to study the problem for learning from noisy data (without active exploration). This is a realistic consideration as (a) the real data captured from world can be noisy. (b) active exploration by agents can lead to destructive affects.\n2. The paper built up a dataset and evaluation protocol to study the effectiveness of this method. \n3. The paper used an offline reinforcement learning algorithm to solve the proposed problem. The offline RL takes the expected local reward as a condition. This reward is available during training and set to be constant during inference. \n4. The paper shows improvement over the proposed experimental setup and provide comprehensive analysis.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Question:\n1. The paper claims that it would like to solve the questions \"How can we achieve effective data scaling without resorting to online exploration, a strategy that could pose safety concerns?\". However, it is not clear to me why the proposed method is free of this risky exploration. Since the offline RL needs some non-perfect routes for the training and it can introduce unsafe actions. Thus I would like a paragraph which precisely clarify the difference of the approches (previous RL and the offline RL) in the paper. \n2. The paper assumes \"training trajectories corresponds to a single pair of 2D goal coordinates\" because of the \"instructions\" are ambiguous. Given that, is it better to consider some more concrete tasks such as \"image-goal navigation\"? It can intrinsically resolve this concern. \n3. Please saw the weakness as well.\n\nIssues:\n1. It seems that \"episode horizon\" in Sec 3.Setup is not a well-defined terminology. The meaning is not clear to me and also it's hard to find on Internet.", "broader_impact_concerns": "n.a.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2o2oQHn392", "reviewer": "Reviewer_vWx3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- VLN-BERT-ORL and MTVM-ORL achieve consistently better SR and SRL than naive VLN-BERT and MTVM without reward token. The results are empirically strong.\n- VLN-BERT-ORL and MTVM-ORL can learn from the suboptimal (noise-added, random) data, while naive VLN-BERT and MTVM face significant performance degradation. This might be a promising approach for vision-and-language navigation, since we do not always need to collect gold trajectories.\n\n### Weaknesses\n- In Table 1 & 5, naive VLN-BERT and MTVM sometimes achieves better navigation error than proposed methods, but I think the explanation / analysis are limited.\n- It might be seen as naive adaptation of the reward-conditioned policies in offline RL, like Decision Transformer, to vision-and-language navigation domain. \n- Offline RL requires positional information to compute reward. This means VLN-ORL can learn from sub-optimal data, but requires additional annotation about positions. There are some trade-offs. If positional information are available, VLN-BERT with position conditioning might be a possible baseline (I know this will face the issue in test time, but I'm not fully sure why positional information is available in training time, but unavailable in test time).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies reward-conditioned vision-and-language navigation problems, called VLN-ORL (vision-and-language navigation with offline RL). Previous vision-and-language navigation works often leverage expert demonstrations only, and employ data augmentation strategy or online exploration to tackle data shortage issue. By leveraging offline RL paradigm, the model could learn from sub-optimal demonstration data.\n\nThe proposed methods, VLN-BERT-ORL and MTVM-ORL, trains policies conditioned on reward tokens, and tests policies conditioned on  reward = +1. Such RL-based approaches achieve consistently better success rate (SR) and success rate weighted by path length (SRL), compared to Behavior Cloning (w/o any conditioning) baseline.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Figure 1 is a bit confusing. I think step 3 in train data is shared among expert / suboptimal demos, but labeled with -1. Expert demonstrations also have sub-optimal steps?\n- In addition, sub-optimal demonstrations finally reach goal (r=0), but they seem to be different from the goal in expert demonstration. Is it OK to be labeled with 0 even if the final destination is different from the oracle? \n- In Table 5, some numbers are not bolded. I don't think this is intended.\n- In Figure 3, RewardC VLN-BERT should be VLN-BERT-ORL for consistency? Moreover, it would be nice to add some short notes around each step (e.g. explaining why this step is OK/NG). The reader should infer the situation from those small pictures.", "broader_impact_concerns": "I think broader Impact concerns are not applicable or are sufficiently addressed in this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers all agree that the paper is of interest to the TMLR audience and provides clear and convincing evidence for its claims.

One reviewer was left with a small concern: "the reward is designed to overfit the natural of RxR and not proven to be generalizable yet; adding noise to distillation is still kind of distillation; I still can not convince myself that HAMT resembles the actual noise in offline data".  The authors might want to consider this in their final revision.
Decision: Accept (poster)