Vision-Language Navigation with Energy-Based Policy
Authors: Rui Liu
Wenguan Wang
Yi Yang
Venue: NeurIPS 2024 poster (2024)
Keywords: Vision-Language Navigation; Vision-Language; Embodied vision
Primary area: robotics

Abstract: Vision-language navigation (VLN) requires an agent to execute actions following human instructions. Existing VLN models are optimized through expert demonstrations by supervised behavioural cloning or incorporating manual reward engineering. While straightforward, these efforts overlook the accumulation of errors in the Markov decision process, and struggle to match the distribution of the expert policy. Going beyond this, we propose an Energy-based Navigation Policy (ENP) to model the joint state-action distribution using an energy-based model. At each step, low energy values correspond to the state-action pairs that the expert is most likely to perform, and vice versa. Theoretically, the optimization objective is equivalent to minimizing the forward divergence between the occupancy measure of the expert and ours. Consequently, ENP learns to globally align with the expert policy by maximizing the likelihood of the actions and modeling the dynamics of the navigation states in a collaborative manner. With a variety of VLN architectures, ENP achieves promising performances on R2R, REVERIE, RxR, and R2R-CE, unleashing the power of existing VLN models.

Vision-Language Navigation with Energy-Based Policy
Rui Liu
Wenguan Wang?
Yi Yang 1.2.+
The State Key Lab of Brain Macinne Intelligence, Zhejiang University, Hangzhou, China College Computer Science and Technology, Zhejiang University, Hangzhou, China https: /gi (iggituut thub. com/ www.
Abstract
Vision Vision-language navigation (VLN) requires an agent to execute actions following human instructions. Existing VLN models are optimized through expert demon- strations by supervised behavioural cloning or incorporating manual reward engi- neering. While straightforward, these efforts overlook the accumulation of errors in the Markov decision process, and struggle to match the distribution of the expert policy. Going beyond this, we propose an Energy-based Navigation Policy (ENP) to model the joint state-action distribution using energy- model. At each step, low energy values correspond to the state-action pairs that the expert most likely to perform, and vice versa. Theoretically, the optimization objective is equiv- alent to minimizing the forward divergence between the occupancy measure of the expert and ours. Consequently, ENP learns globally align with the expert policy by maximizing the likelihood of the actions and modeling the dynamics the navigation states in collaborative manner. With variety of VLN architectures, ENP achieves promising performances on R2R, REVERIE, RxR, and R2R-CE, unleashing the power of existing VLN models.
Introduction
Vision laiiunlanguuae navigation (VLN) [1 entails an embodied agent that executes a sequence of actions traverse complex environments based on natural language instructions. VLN is typically considered as Partially Observable Markov Decision Process (POMDP) (2). where future states are determined only by the current state and current action without explicit rewards. Therefore, the central challenge for VLN is to learn an effective navigation policy. In this context, existing neural agents [ 3
6 are naturally constructed as data-driven policy networks by imitation learning (IL. bootstrapping from expert demonstrations, i.e., ground-truth trajectories ①.
and
Most VLN models 13 6 utilize behavioural cloning (BC), a classic approach for IL. through supervised training. While conceptually simple, BC suffers seriously from quadratic accumulation errors 9 10 along the trajectory due covariate shift, especially in partially observable settings. [I] and 'pscudo interactive demonstrator" (6), which
Several efforts introduce 'student-forcing essentially online versions of DAGGER (12), to alleviate this distribution mismatch. DAGGER assumes interactive access to expert policies (i.e., the shortest path from current state to the goa and reduces linear expected errors 12 10 Some other studies 13
5 combine reinforce- ment learning (RL) [1] and IL. algorithms to mitigate the limitations of BC. Though effective, it presents challenges in designing an optimal reward function [15] and demands careful manual reward engineering. Such reward functions may not be robust to changes in the environment dynamics and
"Corresponding author: Yi Yang.
"These approaches require a slightly more interactive setting than traditional imitation learning, allowing the learner to query the expert at any navigation state during training. Previous studies [ have adopted this setting, and this assume is feasible for many real-world imitation learning tasks 12
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Trors rightt snd wabhe through the sitzhen, 6e0 right isto the living moo..
Echavioural cloning
Disorininative
Actiow, Classifter
Cenditional Distribucion
(Rotion/ staate)
Energy-bbsed Navigation. Poliey
Eneray based Mootel
overey
Aggst Actipu
Johne Distrituctins.
Step
Export bewandbettioo
Havigptll Wilgeeli
Weysttit
1 dn Vidgin
Plstate, action)
Figure 1: Comparison of behavioural cloning (BC) and ENP for VLN. Previous methods use BC optimize the conditional action distribution directly. ENP models the joint state-action distribution through an energy- model. The low energy values correspond to state-action pairs that the expert most likely to perform.
struggle to generalize well across diverse scenes (16). In addition, recent research 6 has revealed that training transformer 17 -based VLN models using RL would introduce instability.
4 essentially train a discriminative action classifier 8 3 4], and model the action distribution conditioned on current navigation state P(a|s) at each time step (Fig. Tp. This classifier is supervised by the cross-entropy loss minimize 1-step deviation error of single-step decision along trajectories [18) However, the prediction of current action affects future states during the global execution of the learned policy, potentially leading to
From probabilistic view, most VLN agents
catastrophic failures [ [2]. In other words, solely optimizing the conditional action probabilities (i.e., policy) remains unclear for the underlying reasons of the expert behaviour, yet fails exploit complete information in the state-action distribution P(8, a) induced by the expert 14]
In light of this, propose Energy-based Navigation Policy (ENP) framework that addresses the limitations of current policy learning in VLN from an energy perspective (Fig. D Energy-based models (EBMs) [99 flexible and capable of modeling expressive distributions, they impose minimal restrictions on the tractability of the normalizing constant (partition function) (20]. We reinterpret the VLN model demonstrations. Specifically, ENP models the joint distribution of state-action pairs, i.e., P(s, a), by energy function, and assigns low energy values to the state-action pairs that the expert mostly performs. For optimization, this energy function is regarded as the unnormalized log-density of joint distribution, which can be decomposed into the standard cross-entropy of action prediction and marginal distribution of the states P(s). During training, ENP introduces persistent contrastive to estimate the expectation of P(s), and samples by Stochastic Gradient Langevin Dynamics 23 In this way, ENP simulates the expert by maximizing the likelihood of the action and
mergy-basee model to learn the expert policy from the expert
divergence 21 [22
incorporates prior knowledge of VLN by optimizing state marginal distribution in a collaborative manner, thereby leveraging the benefits of both energy-based based and discriminative approaches.
Theoretically, the training objective of ENP is maximize the expected log- likelihood function over the joint distribution P(s,a) a). This is equivalent to estimating the unnormalized probability density Therefore, ENP performs prioritized optimization for entire trajectories rather than the single-time step decisions in BC, and achieves a global alignment with the expert policy. Furthermore, we realize that ENP is closely related Inverse For the optimized objective, ENP shares similarities with
(i.e., energy) of the expert's occupancy measure [15]
24 255
Reinforcement Learning (IRL) 7 26 [5].
IRL but minimizes the forward KL divergence 27 28
between the occupancy measures.
For thorough examination ($4), we explore ENP across several representative VLN architectures B
4 6 29). Experimental results demonstrate that ExP outperforms the counterparts, e.g., 2% SR and 1 1% SPL gains over R2R (. 1.22% RGS on REVERIE [30]. 2% SR on R2R-CE [II. and 1.07% NDTW on RxR-CE (32) respectively. ENP only helps release the power of existing VLN models, but also evidences the merits of energy-based approaches in the challenging VI.N decision-making.
2 Related Work
Vision-Language Navigation (VLN), Early VI.N agents neural policy network within sequence-to-sequence framework (3 35 to predict action distribution.
3 [K are built upon a recurrent

---

As compressing all history into hidden vector might be sub-optimal for state tracking across extended trajectories, later attempts [36 37 long-term planning. Recent efforts
incorporate memory buffer (e.g., topoloiiall graph) for are devoted to encode complete episode history of 39 5
38
navigation states and actions by transformer
and optimize the whole model in end-to-end training. Later, various strategies have been proposed to improve the generalization of policies in both seen and unseen environments [ (1), such as progress estimation [42], instruction generation [33 43 44
[6). backtracking [47, cross-modal matching [48 49), self-supervised pre-training [50 5 52].
environmental augmentation 3
53 54 (55, visual landmarks [56] [5]. exploration [58] 59 map
building [60] 61 62 633, waypoint prediction [64], and foundation models [65 66].
Policy Learning in VLN. VLN can be viewed as a POMDP [2]. providing several expert demon-
strations sourced from ground-truth shortest-path trajectories [ 30 or human demonstrations [32]. Behavioural cloning (BC)
66
68 a typical approach of imitation learning (II.) 7 15],
widely 6 with supervised policy learning. Nevertheless, it suffers from
used in current VI.N agents 1 33
4
distribution shifts between training and testing [2 69 I introduces 'student- forcing'
1 training
regime from sequence prediction to mitigate this limitation. Later agents 3 8 S combine both IL
and model-free RL [14, 70] for policy learning, where the reward function is manually designed on the distance metric. Instead of directly mapping visual observations into actions or state-action values, [3 [ investigate model- -based RL [72 73] for look-ahead planning, exploring the potential future states. As reward engineering requires careful manual tuning. [16] proposes to learn reward functions from the expert distribution directly. Although RL is an effective approach in principle, DUET [6] finds it difficult to train large-scale transformers with inaccurate and sparse RL rewards [74] in VLN. Existing studies [61
use an interactive demonstrator (similar to DAGGER 12 to generate additional pseudo demonstrations and achieve promising performance.
62 63
In this paper, ENP learns to align with the expert policy by matching the joint state-action distri- bution for entire trajectories from the occupancy measure view. Furthermore, ENP reinterprets the optimization policy into an energy-based model, representing an expressive probability distribution.
Energy-based Model. Energy-based models (EBMs) [19 are non-normleieed probabilistic models that represent data using Boltzmann distribution. In EBMs, each sample associated with an energy value, where high-density regions correspond to lower energy [75]. A range of MCMC-based [76] 21
and MCMC-free 77 78] approaches can adopted to estimate the gradient of the log-likelihood.
EBMs have been widely utilized in generation [22] perception 77]. and policy learning 80 81).
The most related work 25
24 22 will be discussed below. (25 employs EBM for distribution
matching, strictly prohibiting interaction with the environment. Moreover, this approach is applied in
an offline setting, specifically for healthcare. [24] considers the expert energy as the reward function and leverages this reward for reinforcement learning. [22] adopts parameterized energy functions to
model the conditional action distribution, optimizing for actions that minimize the energy landscape.
In contrast, ENP devotes to learning the joint distribution of state-action pairs from the expert demonstrations online. An external marginal state memory introduced to store the historical information for initializing the samples for next iterations. ENP demonstrates the potential of EBMs in VIN decision making, and improves the existing VLN agents across various frameworks.
3 Method
In this section, we first formalize the VIN task and diseuss the limitations of existing work from probabilistic view (...T). Then we introduce Energy- Navigation Policy learning framework ENP (3.2)) Furthermore, ENP compared with other imitation learning methods based divergence minimization ($ (.3). Finally, we provide the implementation details ($3.4).
3.1 Problem Formulation
VLN is typically formulated as POMDP 2 51]
with finite horizon T, defined as the tuple AS,A, T, R. pn>.. S the navigation state space. AA is the action space. Note that we use panoramic action space 33 which encodes high-level behaviour. The transition distribution T(* |st, ax) (Markov property) of the environment, the reward function R(st, a)), and the initial state distribution Po unknown in VLN and T can only be queried through interaction with the environment.
3

---

Marginal State Memory
Instruction
O Ossevaatio
History
A Atiio
guisode
24 M.MM
Go stright past the lunge tile and chems turn left and walt near the lazp.
ac
H
SGLD (Eq
Cross- Modal State Encoder Fee,
Dc 0
action logit
State
Lg (Eig
Represectation"
Action Classifier Fe..
Figure 2: Overview of ENP. At each step t. the agent acquires a series of observations. and predicts the next step based on the instruction and navigation history, ENP optimizes the marginal state matching loss Ls through SGLDD sampling from Marginal State Memory (Fq. 9. and minimizes the cross-entropy loss C. jointly (Eq.
In the standard VLN setting (1 training dataset D {xx, +)} consists of pairs of the language instructions æ and corresponding expert trajectories т. At step t, the agent acquires an egocentric observation Ot, and then takes an action ae A from Kr candidates based on the instruction x. The agent needs to learn a navigation policy π (a|s) and predict the action distribution P(ae|s)).
transformer- based networks 4 6 combination of a cross- state encoder Fe, and a conditional action classifier Fe. for action prediction. Hence they are usually built composition of Fe. o Fe.. There are different state representations in previous work [] here we define a more general one, The cross-modal state encoder extracts joint state representation 8, Fo, (a, O,, H,) E TR K. by the visual embedding O₂, the instruction embedding æ, and history embedding He along the trajectory т. Concretely, H, comes from recurrent hidden units of Seq2Seq framework or external memory module (e.g., topological graph (6)) Then, the action classifier maps the state representation 8, into an action-related vector a [a](+))]= E RKK i.e., ar = Fo., (s+), and oz [k] is termed as the logit for k-th action candidate. Then the probability of dr is computed by softmax, i.e., P(ar,x(s)) E exp(a,t)
Recent VLN agents employ Seq2Seq 8
expla: (k)
The mooss- state encoder Fe. and action classifier Fo., usually employ a joint end-to-end training. Imitation learning (IL) aims learn the policy from expert demonstrations D without any reward signals. Behaviour cloning (BC) (68) is a straightforward approach of IL by maximizing likelihood estimation of x (als), L.e., minimizing the cross-entropy loss in supervised way:
arg min -Fllog x (a]ss) + C. E (arT)w E. log Po(arla:),
(1)
where 0= = U. BC suffers from the covariate shift problem for the crucial i.i.d. assumption, as
the execution of at will affect the distribution of future state 8,41 [9]. Inspired by DAGGER [12).
some agents [T 6 query the shortest paths (by expert policy π²) during navigation, explore more trajectories
by interacting with the environments, and aggregate these trajectories as D+
((x.T)) U (xxr)()) r t. Then, the policy is updated by mixing iterative learning [12]. However, these traditional approaches ignore the changes in distribution and train policy that performs well under the distribution of states encountered by the expert. When the agent navigates in the unseen environment and receives different observations than expert demonstrations, it will lead to a compounding of
errors, as they provide no way to understand the underlying reasons for the expert behaviour 22
3.2 Ensrgy-based Expert Policy Learning
Instead of solely training the discriminative action classifier each step, want to learn a policy n² by exploiting the complete information in the state-action distribution from the expert policy x². To facilitate later analysis, the occupancy measure (s,, a) of the policy introduced [14]. p= a) denotes the stationary distribution of state-action pairs (s, a) when an agent navigates with policy π in the environment, defined as:
pT (s, A <(als) Σ, P(s. E, P(se 5,a a),
(2)
and p² (s, a) has one-to-one relationship with policy m: p² p" = nt 15 Therefore, induce a policy nt with p'(ss a) that matches the expert (3 a) based on the joint distribution of state-action pairs ((s, a). omitted for brevity. The matching loss Cp is designed as:
arg min [es.a)) [log p" (s, a)] x L, Σ (a.1)~ log Po(s.
(3)
Then we reformulate it from an energy view, as energy- models (EBMs) (19 77 parameterize an unnormalized data density and can represent a more expressive probability distribution. Specifically,

---

EBMs are first introduced to model P(s,a) of the agent Boltzmann distribution:
Po(e,a) exp( (o(ssa))
(4)
Z
where Eg(s, a)) : SxAA+RR is the energy function, and Ze is the partition function (normalizing constant). As it is difficult to optimize the joint distribution directly [79), we factorize it (Fig. 2
llog Po(s,a) log Po(a)s) log Po(s) L.p = Lx Lis-
The first term, log log P(]ss), is typically from a discriminative classifier and can be optimized by minimizing the cross-entropy loss Ln as in Eq. I As stated in [82 83 the marginal state distribution matching loss Ls provides an explicit objective for navigation exploration and incorporates prior knowledge from the expert demonstrations. P(s) is computed by marginalizing out as:
Σ exp( exp(-Ee(s; )+))
Po(s)
Po(s)= = Σ Po(s,ax)
(6)
Zo
It is worth noting that log P(s) is the likelihood of EBM, which cannot be computed directly due to the intractable partition function Ze- A common approach to estimate the gradient of the log-likelihood for likelihood maximization with gradient ascent (75)
veLs + Vo log Ps(s)) = V log Σ exp(-Eelo [o ax)) + V log Za
(7)
= Vebe(s) E. (8) Vati (5)].
Considering the low energy value corresponds to the state-action pairs that the expert mostly perform, the action logits or [k] of the agent can be used to describe such energy value as Eo(st, ar.;) x) -ur,k
formally. Then, the first gradient term, - -V (s),, is calculated by automatic differentiation in the deep network of the neural agent. Estimating Ve log Zo requires Markov chain Monte Carlo (MCMC) [84] sampling from the Boltzmann distribution within the inner loop of learning. Specifically, a sampler based Stochastic Gradient Langevin Dynamics [ (SGLD), practical method of Langevin MCMC
(55 can effectively draw samples as:
~ U,
V=E)())++ V: + E,
(8)
where UI denotes the uniform distribution, i is the sampling iteration with step size & is Gaussian
noise, and 3 will converge to (S( when e-0 and i-oo. While training on new state, this MCMC
chain will be reset. In practice, adopting standard MCMC sampling suffers from slow convergence and requires expensive computation. Recent persistent contrastive divergence approaches [76] 21
maintain single MCMC chain, and use it to initialize a new chain for next sampling iteration (2]
Inspired by this, we introduce a Marginal State Memory M served as an online replay buffer [22
collecting historical information of the MCMC chain during training (Fig. 2) To initialize the inner loop of SGLD, the state 8 sampled randomly from M. This helps continue refine the previous samples, further accelerating the convergence. After several iterations 1, 8f is stored in M:
HUM, - = VoE' E, s MM
(9)
where / is fewer than the number of iterations required for MCMC convergence. The training details are illustrated in Algorithm [
3.3 Comparison with Other Imitation Learning Methods
In VLN, ENP aims to leam a policy n² from expert demonstrations (T²)) without access to the explicit reward, which is analogous to the most common approaches of IL, e.g., BC and inverse reinforcement learning (IRL). For in-depth analysis, we explore the relation between ENP and other IL Methods, and provide a general understanding of them from a divergence viewpoint (28].
Standard BC is treated supervised learning problem (Eq. D. and the policy is required to match the conditional distribution of n² under Leiblerr (KL. divergence distance DKL. as:
arg min - EEloo T" (a]s)) = ( ([ - (()ss))s)),
(10)
where H((*)) = ([ (a|=)) is constant, a.k.a. the causal entropy 86 H(*²) is considered as the
expected number of options at each state for Th (als) 28 15].
5

---

Algorithm Energy- Navigation Policy (ENP) Learning Algorithm
1: Input: The training dataset 2> = {(x,T)) Cross- State Encoder Fe., Action Classifier Fe.. Visual embedding O, Instruction emhedding æ: History embedding H, 2: Initialize: 0 0g a, and Marginal State Memory M- Mat-MMre
3: for training step n = to N do
4: Aggragate : = {(I, n.. of visited states by ni and actions from n². Sample (a, r) from D UD
5: for navigation step / = 1 to T' do
6: S: Fe. (a. O., H.). d; Fa. (s:). Sample si from UUM
Defined in 3.
7: for SGLD iteration
to do
8:
+++1 = 8; 7 Vo (6i) + 5- C~N, N. 8/ -M
Defined in Eq.
9: E. f- log Pa(a:)s)) Ls +- (Fa((.)) ) Fa()
Defined in Eq. and
10: Backpropagate VeL, VeC, + VeLs. Update Fe, and Fe.
11:
Return: Optimized Parameters 0 = 0, U 0o of Fo, and Fo.
For ENP, the optimization objective in Eq. B can be written in form of KL distance minimization:
arg min (( log (s, = Dx (p' a)lle" (s) + H(p (s,a)),
(11)
where (s a)) is the entropy of the occupancy measure, which is a constant. IRL. aims to infer the reward function of the expert, and learn policy optimize this reward. A representative work, AIRL 27), adopts an adversarial reward learning, which is equivalent to matching p (f(s, a) [15] as:
arg min DKL(p" (s, as,a) ()) -EE (..a) (s, (s) H(pp(s, (s)
(12)
Hence, our objective is similar to the forward KL version of AIRL (28 27 except for the entropy term. Both AIRL and ExP perform prioritized optimization for entire trajectories rather than the single-time step decisions in BC, so the compounding error can be mitigated [15] 27 As DxL is non-symmetric metric, the forward KL divergence results in distributions with mode-covering behaviour, while using the reverse KL results in mode-seeking behaviour (18). In addition, ENP directly models the joint distribution instead of recovering the reward function. We compare them in the experiment ( ($4.3) and find ENP is more applicable to VLN (Table
3.4 Implementation Details
Network Architecture. ENP is general framework that can be built upon existing VLN models. We investigate ENP on several representative architectures, including: i) EnvDrop B adopts a Seq2Seq model with a bidirectional LSTM-RNN and an attentive LSTM-RNN. ii) VLNOBERT [4] is built upon multi-layer transformers with recurrent states. iii) DUET 6 is a dual-scale transformer-based model with topological map for global action space, enabling long-term decision-making. iv) ETPNav 229 utilizes a transformer- based framework with an online topological map. For continuous environments, both BERT 4 and ETPNav (29 employ a waypoint predictor (64). K, dimension of the state representations 8, E IR KD is different due to the local or global action spaces of these methods. In $4, we follow the same settings of these models, replace their policy learning as ENP, and maintain a Marginal State Memory M of 100 buffer-size.
Training, For fairesss, the hyper-parameter (e.g., batch size, optimizers, maximal iterations, learning rates) of these models are kept the original setup. Back translation is employed to train EnvDrop 3 The parameters of VLNOBBETT 4 is initialized from PREVALENT (51) The pre-training and
finetuning paradigm is used for DUET 6 and ETPNay (2 For SGLD, 3 is sampled from M with a probability of 95% (Eq. 9 In Env EnvD
and VER [4, the number of SGLD iterations is set as t= 15. Lore = 20 and ln = of the pre-training and finetuning stages are used for DUET [6) and ETPNav 29) Although the step size and the Gaussian noise E in the original transition kernel of SGLD are related by Var(E) = 23 most studies [2, 79 choose to set e and & separately in practice. The step size c= 1.5 and &~~ 0.01) are set ENP ($4.3)
Inference. During the testing phase, the agent uses the learned Fe. for cross moddll state encoding, and predicts the next step via Fo., until it chooses [STOP] action reaches the maximum step limit.
Reproducibility, All experiments are conducted on a single NVIDIA 4090 GPU with 24GB memory in Porch. Testing is conducted on the same machine.

---

4
Experiment
We examine the efficacy of ENP for VLN in discrete environments ($4.)) and more challenging continuous environments Then we provide diagnostic analysis on core model design ($4.3)
4.1
VLN in Discrete Environments
Datasets. R2R contains 7, 189 shortest-path trajectories captured from 90 real-works building- scale scenes [87] It consists of 22K step-by-step navigation instructions. REVERIE (30] contains 21, 702 high-level instructions, which requires an agent to reach the correct location and localize a remote target object. All these datasets
devided into train, val seen, val unseen, and test unseen splits, which mainly focus on the generalization capability in unseen environments. Both R2R and REVERIE [30] are built upon Matterport3D Simulator [.]
Evaluation Metrics. As previous work [1 (0), Success Rate (SR), Trajectory Length (TL), Oracle Success Rate (OSR), Success rate weighted by Path Length (SPL.,, and Navigation Error (NE) are
used for R2R. In addition, Remote Grounding Success rate (RGS) and Remote Grounding Success weighted by Path Length (RGSPL) are adopted for object grounding in REVERIE.
Performance on R2R. Table demonstrates the numerical results of ENP with different frameworks on R2R. With Seq2Seq-style neu- ral architectures. ENP provides 1% SR and 1% SPL gains over En- vrrpp 3 on val unseen. Adopt- ing ENP in transformer-based mod- els leads to promising performance gains on test unseen, e.g., 2% on SR with VLNOBERT [4), and 2% on SR with DUET 6). The performance is still lower than BEVBert [6]] and BSG [62] due to the additional seman- tic maps. Moreover, we provide the average success rate across different path lengths in Appendix (Fig. 4
Table 1: Quantitative comparison results on R2R I ($4.1
Models
R2R vol unseen
R2R test wnseew
TL.
NEj
SR SPL.
T1_1
NEL
SR1
1PL
Seq2Seq
JCVRR00
8.39
7,81
22
8.13
7.85
20
18
SF
33
6.62
35
14.82
6.62
35
AuxRN
48
5.28
55
Active
33
20.60
4.36
58
40
21.60
4.33
60
41
HAMT
5
11.46
2.29
66
61
12.27
3.93
65
SSM
18
CVP322211
20.4
HOP
89
CVP22220
12:27
3,80
64
57
12.68
3.83
64
59
TD-STP
W0
pe0mm
3.22
3.73
67
LANA
91
12,00
62
12.60
65
BSG 62
pcev2221
14.90
2.89
74
62
14.86
3.19
73
62
BEVBet
6
14.55
2.81
75
3.13
73
VER
92
14.83
2,80
76
15.23
2.74
76
DavDrop
3
10,70
5.22
52
48
11.66
5.23
51
47
ENP-
EnyDrg
11.17
4.69
53
49
10.78
5.30
82
48
BERT
12.01
"6"9"""3"""
'5
12.35
4.09
63
"5""
ENP-VLNC
O
12.17
3.81
65
57
13.12
3.91
65
59
DUET x eoperten
13.94
3.31
72
60
14.73
3.65
69
59
ENY-DUET
14.23
3.00
14.27
3.39
71
presents the results of ENP with different transformer-based frameworks on REVERIE. ENP surpasses all the counterparts on both val unseen and test unseen by large margins, e.g., 1.34% on SR and 0.64% on RGS over VLNOBERT 44. 0.68% on SR and 1.22% on RGS over DUET 6). verifying the efficacy of ENP. Inspired by recent efforts [6]]. potential improvement on object grounding can be achieved by introducing CLIP features 93
Performance on REVERIE.
Table 2: Quantitative comparison results on REVERIE 30
($4.1)
unavailable statistics.
REVERIE val unsee
REVERIE test anzzen
Models
TL.
OSRT
SKT
SPL.1
RUST
RUSPLT
TL.
OSR?
SR1
SPL.
RGIST
RESPL.
SequSeq
1
11.07
8.07
4.20
2.84
2.16
1.63
10.89
0.88
3.59
3.09
2.00
1.58
RCM
11.98
14.23
9.29
6.97
4.89
3.89
10.60
11.68
7.84
6.67
3.67
3.14
INP
30
45.28
28.20
14.40
7.19
7.84
4.67
39.05
30.63
19.88
11.61
11.28
6.08
Airbert
18:71
34.51
27.89
21.88
18:23
14.18
17.91
34.20
30.28
23.61
16.83
13.28
HAMT
3
--0000001
14.08
36.84
32.95
30.20
18:92
17.28
13.62
33.41
30.40
26.67
14.88
13.08
HOP
39
16.46
26.11
18.85
15.73
16.38
30.17
14.34
TD-STP
53
39.48
34,88
27.32
21.16
16.56
40.26
35.89
27. 51
19.88
15.40
23.18
52.97
32.86
22.77
18.83
51.72
32.95
BSG
12
24.71
58.05
52.12
35.59
35.36
24,24
22.90
62.83
56.45
38.70
33.15
22.34
56.40
51.78
36.37
34.71
24.44
52.81
36.41
32,06
VER
92
23.03
61.09
55.98
39.56
33.71
23.70
24.74
62.22
56.82
38.76
33.88
23.19
VENGHERT
16.78
35.02
30.67
24 90
18.77
15.27
86
32.91
29 29.61
23.99
16.50
13.51
Exp-
M..--VLNBERRT BERT
16.90
35.80
31.27
25.22
18.78
15.80
15.90
34,00
30.95
24.46
17.14
14.12
DCET
[
22111
51337
16.58
33.75
"52115"
22.00
211.0
3631
31.31
36606
'3..8
221.6
ENI-DUET
25.76
84.70
48.90
33.78
34,74
23.39
22.70
59.38
53.19
36.26
33.10
22.14
Qualitative Results In Fig. (a), we illustrate the qualitative comparisons of ENP against DUET on R2R val unseen. Based on ENP, our agent yields more accurate predictions than DUET. It verifies that ENP leads to better decision-making when facing challenging scenes. In Fig. (b), our agent may fail due to the serious occlusion, and introducing map prediction (95] may alleviate this problem.
7

---

Nalk forword right stales. Walk to wall windows and right into the recreation room and stop before you reach the pool table.
Walk into the door left enter the sitting Wall acmes the sittingg room Stop before you reach steps.
G3
EVP
DUET
O Failure
Figure 3: Qualitative results on R2R I ($4 (a) DUET 6 arrives in the wrong room instead of 'recreation room' since the scene contains multiple rooms. Our agent reaches the goal successfully, demonstrating better decision-mann making ability. (b) Failure case: Due to partial observability and occlusion of the environment, it is hard find 'kitchen" at some positions. Thus our agent goes the wrong way and ends failure ($4.1).
4.2
VLN in Continuous Environments
Datasets. R2R-CE 33 and RxR-CE 32 are more practical yet challenging, where the agent can navigate to any unobstructed point instead of teleporting between fixed nodes of a pre-defined naviga- tion graph in R2R. R2R-CE comprises a total of 5, 611 shortest-path trajectories with significantly longer time horizons than R2R. RxR-CE presents more instructions with longer annotated paths, including multilingual descriptions. In addition, the agents in RxR-CE are forbidden to slide along obstacles. Both of are R2R-CE and RxR-CE ARRCE driven by Habitat Simulator (96)
Evaluation Metrics. The metrics of R2R-CE [31] are similar to R2R [. For RxR-CE [32],
Normalize Dynamic Time Wrapping (NDTW) and NDTW penalized by SR (SDTW) are further used
to evaluate the fidelity between the predicted and annotated paths.
Performance on R2.-CE. As shown in Table 3 ENP provides a considerable performance gain against VENTBERT 44, e.g., 1% SR and 1% SPL on val unseen. Concretely, it outperforms the top-leading ETPNav 29 by 1% and 1 1% in terms of OSR and SR on test unseen, respectively. These
quantitative results substantiate our motivation of empowering VLN agents with ENP.
Table 3: Quantitative comparison results on R2R-CE
31
4.2
'-': unavailable statistics.
RZR-CE
Models
vol seen
vol unseen
test unseen
T1.
NEL
OSR1
SR1
SPI.1
T1.
NEI
OSR
SRT
SPL.7
T1.
NEL
SR1
SPI.1
VLN-CE 31
9.26
7.12
46
37
35
8.64
7.37
40
32
30
8.85
7.91
36
28
25
CMTP
7.10
7.90
IIPN
7
8.54
5.48
53
46
43
7.62
6.31
36
34
8.02
6.65
37
32
30
SASRA
33
8.89
7.71
36
34
7,89
8.32
24
22
CM2
39
CVRR2002
12.05
6.10
51
43
11.54
7.02
42
34
28
13.90
7.70
39
31
24
WSMGMap
3.65
6.28
12,30
7.11
35
28
Sim2Sim
10
11.18
4.67
61
52
44
10.69
6.07
52
43
36
11.43
6.17
52
44
37
BEVBert
[CC12003)
4.57
67
59
50
4.70
67
59
50
DREAM
11.60
4.09
11,30
5.53
1,,80
5.48
49
HNR
[01
11.79
3.67
76
12.64
4.42
61
13.03
4.81
58
50
VINOBERT 4
(29992001)
12,50
5.02
59
50)
44
12:23
5.74
53
44
39
13.51
5.89
51
42
36
ENP-VLNOBERE
12.08
4.89
60
12.37
5.81
45
40
13,21
5.68
52
44
36
ETPNNN
29
T1.78
'3.95"
7/-
66
39'
11.90
4,71
68
57
49
12.87
5.12
03
38
48"
ENP-ET
11.82
3,90
73
68
59
11.45
4.69
65
58
50
12.71
5.08
64
56
48
Performance on RxR-CE. Table reports the results with Marky-m'55 instructions (56) on RxR-CE. ENP achieves competitive results on longer trajectory navigation with multilingual instructions, e.g., 55.27% vs. 54.79% SR and 62.97% vs. 61.90% NDTW for ETPNav (29] on val unseen. We attribute this to the global expert policy matching, which promotes the path fidelity. This indicates ENP enables the agent to make long-term plans, resulting in better NDTW and SDTW.
Table 4: Quantitative comparison results on RxR-CF 32
($4.2)
unavailable statistics.
RxR-CE val
RxR CF. vo!
Models
NEJ
SRI
SPL
NDTW
SDTW
NEL
SR1
SPL
NDTW
SDTW1
CWP-CMA
[4
JUVPRSR211
8.76
26.59
22,16
47.05
VIRT.BERT
[eV9a2a01)
8.98
27.08
22.65
46.71
Reborn
20
5.69
52.43
45.46
66.27
44.47
5.98
48.60
42.05
63,35
41.82
HNR
101
4.85
63.72
53.17
68.81
52.78
5.51
56.39
46.73
63,56
47.24
ETPNaY
29
5:03
61.46
50.83
66.41
51.28
5.64
54.79
44.89
61.90
45.33
ENP ENP-ETPNav
5.10
62.01
51.18
67.22
51.90
5.51
55.27
45.11
62.97
45.83
8

---

4.3 Diagnostic Experiment
Joint Distribution Matching. We first investigate the joint distribution matching loss Cp (Eq. 33 in ENP, which is factored as the discriminative action loss - (Eq. D and the marginal state distribution matching loss L.s (Eq. Both of f and Ls are online optimized iteratively. A clear performance
Table 5: Ablation study of objective loss (
4.3
R2R vol R2R-CE vol
Models
Objecrive
SR1
SPL,1
SR1
SPL-
ENP ENP-VLNC E
2
61
55
44
39
ENR-DUETRETINE
72
60
57
49
Er 2
58
50
drop is observed in Table e.g. SR from 74% to 72% for DUET 6 on R2R, as only optimizing the conditional action distribution with C.: will degenerate into BC (or DAGGER) with cross-entropy loss. This reveals the appealing effectiveness of the joint distribution matching strategy in ENP.
Step Size and Guassian Noise in SGLD. In ENP, SGLD [23] is used to draw samples from
Table 6: Ablation study of step size and noise ($4.3)
Models
ASGLD
R2R vol R2R-CE vo!
Boltzmann distribution (Eq. 99) and further op- timize Ls. Following recent EBM training, we relax the restriction on the relation between the step size and Gaussian noise Ç. ie.. Var(E) =
SRT
SPL"
Var(E) 0:01
39
BNI-
1.5.
Var(E) 0.01
Varl() 0.01
64
65
57
45
40
40
15, Varl()
= 0.1
63 55
41
38
shhles the results with different step sizes (when I = 15), and changing c in small range has little effects on the performance. In addition, different noise variances Var(E) are also explored. Our ENP is insensitive to parameter changes. We find that = 1.5 and ~ NN00, 0.01) for the transition kernel of SGLD works well across a variety of datasets and frameworks in VLN.
Number of SGLD Loop per Training Step. For the SGLD sampler, we study the influence of the number of inner loop iterations per training step.
Table 7: Ablation study of SGLD iterations ($4.3)
RZR RZR-CE
Models
#Loop
SRT
SPLt
SR"
SPLT
5
64
56
44
39
In Table 7 we can find that = 15 iterations is enough to sample the recurrent states for En-
15
57
20
65
57
45
40
vDrop 3 and VLNGBERT 44. Slightly more iterations are required for the models [6] [29 with
&EXT- ETPNaY 20
ENI-DUET K
20,
In
In
5
74
60
58
56
50
20,
In
60 58
additional topological graph memory due to the global action space. This can be mitigated by pre-storing the samples in the Marginal State Memory .M during pre-training. As these frame- works 66 29] employ pre-training and finetuning paradigm for VLN, we design a different number of iterations, i.e., Lpre 20 and In = 5. It is worth noting that the state memory Mppc can acquire the samples while performing single-step action prediction in the VLN pre-training Thus, the SGLD iterations in finetuning based on MareMpre can be further reduced.
ENP Ps. AIRL. In Table g we compare ENP against AIRL (27. which is representative work of adversarial imitation learning. The objective function for ENP is equiv- alent to the forward KL divergence minimization, where
Table 8: ENP vs. AIRL ($4.3).
R2R vaf R. vol
Models
Policy
SR1
SPL. SR1
SPL
VENOBERT
4
AIRI 22
63
ENp
65
57
45
40
AIRL 27 makes use of the mackward KL divergence. By modifying the original policy learning in VLN [4] (a mixture of RL and IL) an adversarial reward learning formulation, AIRL achieves the comparable performance. ENP yields better scores on R2R-CE with 45% SR and 40% SPL. Meanwhile, ENP avoids the potential influence from the adversarial training.
Runtime Analysis. The additional computational complexity of ENP from the iterations of SGLD inner loop. Towards this, we make some specific designs, such as Marginal State Memory .M and pre-storing in pre-training. During inference, there no additional computational consumption in executing the strategy compared to existing VLN agents.
5 Conclusion
In this paper, we propose ENP, an Energy- Navigation Policy learning framework. By explicitly modeling the joint state-action distribution using an energy- based model, ENP shows promise to solve the intrinsic limitations of supervised behavioural cloning. Through maximizing the likelihood of the action prediction and modeling the dynamics of the navigation states jointly, ENP learns to minimize the divergence distance from the expert policy in a collaborative manner. Experimental results on gold-standard VIN datasets in both discrete (R2R and REVERIE) and continuous environments (R2R-CE and RxR-CE) demonstrate the superiority against existing methods. In the future, we will explore more tasks and frameworks with the energy-based policy.

---

Acknowledgment. This work was supported by the National Science and Technology Major Project (No. 20232D0121300), the National Natural Science Foundation of China (No. 62372405). the Fundamental Research Funds for the Central Universities 226-2024-00058, the National Key Human-Machine
Laboratory of Human-N Machine Hybrid Augmented Intelligence, Xi' an Jiaotong University (No. HMHAI-202403),, CIPSC-SMP-Zhij Large Model Cross-Disciplinar Fund, and the Earth System Big Data Platform of the School of Earth Sciences, Zhejiang University.
References
[1] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian D. Reid, Stephen Gould, and Anton van den Hengel. Vision-and-langune lan-lang navigation: Interpreting visually-groudded navigation instructions real environments. In CVPR, 2018.
[2] Karl Johan Aström. Optimal control of markov processes with incomplete state information Journal of mathematical analysis and applications, 10:174-205, 1965.
[3] Hao Tan, Licheng Yu, and Mohit Bansal, Learning to navigate unseen environments: Back translation with environmental dropout. In NAACI, 2019.
[4] Yicong Hong. Qi Wu, Yuankai Qi, Cristian Todriguez-Opazz, and Stephen Gould, VIn bert: A recurrett vision-and-language bert for navigation. In CVPR, 2021.
[5] Shizhe Chen, Pierre-Louis Guhur, Cordelia Schmid, and Ivan Laptev. History aware multimodal trans- former for vision-and- lainn-and-langguge navigation. NeurIPS, 2021.
[6] Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Think global, act local: Dual-scale graph transformer for vision-and-languagge navigation. In CVPR, 2022.
[7] Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In ICML., 2004.
[8] Xin Wang. Qiuyuan Huang, Asli Celikyilmaz. Jianfeng Gao, Dinghan Shen. Yuan-Fang Wang. William Yang Wang, and Lei Zhang. Reinforeed cross-modal matching and self-supervised imitation learning for vision-langee asioonlanguage navigation. In CVPR, 2019.
[9] Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In AISTATS, 2010.
[10] Tian Xu, Ziniu Li, and Yang Yu. Error bounds of imitating policies and environments. In NeurfPS, 2020.
[11] Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In NeurlPS, 2015.
[12] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online lcarning. In AISTATS, 2011.
[13] Xin Wang, Wenhan Xiong, Hongmin Wang, and William Yang Wang. Look before you leap: Bridging model-free and model-based reinforcement learning for planned-ahead vision-andd-lnguagge navigation. ECCV, 2018.
[14] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[15] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In NewrIPS, 2016.
[16] Hu Wang. Qi Wu, and Chunhua Shen. Soft expert reward learning for dision-and-languaee navigation. In ECCV, 2020.
[17] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.
[18] Christopher M Bishop. Pattern recognition and machine learning. Springer, 2:645-6755 2006.
[19] Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Fujie Huang. A tutorial on energy-based learning, Predicting structured data, 100), 2006.
[20] Fredrik K Gustafisonn, Martin Danelljan. Radu Timofte, and Thomas B Schön. How to train your energy-based model for regression. In BMVC, 2020.
10

---

[21] Tijmen Tieleman. Training restricted boltzmann machines using approximations to the likelihood gradient. In ICML, 2008.
[22] Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. In NeurlPS, 2019.
[23] Max Welling and Yec W Tch. Bayesian learning via stochastic gradient langevin dynamics. In ICML, 2011.
[24] Minghuan Liu, Tairan He, Minkai Xu, and Weinan Zhang. Energy-based imitation learning. In AAMAS. 2021.
[25] Daniel Jarrett, Ioana Bica, and Mihaela van der Schaar. Strictly batch imitation learning by energy-bas based distribution matching. In NeurIPS, 2020.
[26] Brian D Ziebart, Andrew L Maas. J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In AAAI, 2008
[27] Justin Fu, Katie Luo, and Sergey Levine. Learning rohust rewards with adverserial inverse reinforcement learning. In ICLR, 2018.
[28] Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A divergence minimization perspective on imitation learning methods. In CoRL, 2020.
[29] Dong An, Hanqing Wang, Wenguan Wang, Zun Wang, Yan Huang. Keji He, and Liang Wang. Etpnav: Evolving topological planning for vision-language navigation continuous environments. IEEE TPAMI, 2024.
[30] Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, and Anton van den Hengel. Reverie: Remote embodied visual referring expression in real indoor environments. In CVPR, 2020.
[31] Jacob Krantz, Erik Wijmans, Arjun Majumdar. Dhruv Batra, and Stefan Lee. Beyond the nav-graph: Vision-and-language navigation in continuous environments. In ECCV, 2020.
[32] Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. Multilingual ssiionand--anguunn navigation with dense spatiotemporal grounding. In EMNLP, 2020.
room:
[33] Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg- Kirkpaikk,, Kate Saenko, Dan Klein, and Trevor Darrell. Speaker-follower models for vision-and-lange ann-gaggaggeeeeeeeeaeaa navigation. In NeurlPS, 2018
[34] Ilya Sutskever, Oriol Vinyals, and Quoc V Le.. Sequence to sequence learning with neural networks. In NeurIPS, 2014.
[35] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 98))1133- 1780, 1997.
[36] Zhiwei Deng, Karthik Narasimhan, and Olga Russakovsky. Evolving graphical planner: Contextual global planning for vision-and aadn--nn-lanngaae navigation. In NewriPS, 2020.
[37] Kevin Chen, Junshen Chen, Jo Chuang, Marynel Vázquez, Silvio Savarese. Topological planning with transformers for vision-and and-laagage navigation. In CVPR, 2021.
[38] Hanqing Wang. Wenguan Wang. Wei Liang, Caiming Xiong, and Jianbing Shen. Structured scene memory for vision-language navigation. In CVPR, 2021.
[39] Alexander Pashevich, Cordelia Schmid, and Chen Sun. Episadic transformer for vision-and- laiiaaand-- navigation. In ICCV, 2021.
[40] Aming Wu, Rui Liu, Yahong Han, Linchao Zhu, and Yi Yang. Vector-decomposed disentanglement for domain-invariant object detection. In ICCV, 2021.
[41] Yi Yang, Yucting Zhuang, and Yunhe Pan. Multiple knowledge representation for big data artificial intelligence: framework, applications, and case studies. Frontiers of Information Technology & Electronic Engineering. 22(12):
2021.
[42] Chih- Yao Ma. Jiasen Lu, Zuxuan Wu, Ghassan AlRegib, Zsolt Kira, Richard Socher, and Caiming Xiong Self-monitoring navigation agent via auxiliary progress estimation. In ICLR, 2019.

---

[43] Hanqing Wang. Wei Liang, Jianbing Shen, Luc Van Gool, and Wenguan Wang. Counterfactual cycle- consistent learning for instruction following and generation in vision-language navigation. In CVPR, 2022.
[44] Xiaohan Wang, Wenguan Wang. Jiayi Shao, and Yi Yang. Learning to follow and generate instructions for kanguage-capable navigation. IEEE Transactions on PAMI, 2023.
[45] Xianghao Kong, Jinyu Chen, Wenguan Wang, Hang Su, Xiaolin Hu, Yi Yang, and Si Liu. Controllable navigation instruction generation with chain of thought prompting. In ECCV, 2024.
[46] Sheng Fan, Rui Liu, Wenguan Wang, and Yi Yang. Navigation instruction generation with bev perception and large language models. In ECCV, 2024.
[47] Liyiming Ke, Xiujun Li, Yonatan Bisk, Ari Holtzman, Zhe Gan, Jingjing Liu, Jianfeng Gao, Yejin Choi and Siddhartha Srinivasa. Tactical rewind: Self-correction via backtracking in vision-and-language navigation. CVPR, 2019.
[48] Fengda Zhu, Zhu, Xiaojun Chang, and Xiaodan Liang. Vision langualan navigation with self- supervsuee auxiliary reasoning tasks. In CVPR, 2020.
[49] Hanqing Wang, Wei Liang, Lue V Gool, and Wenguan Wang. Towards versatile embodicd navigation. In NeurIPS, 2022.
[50] Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh, and Dhruv Batra, Improv- ing iision-add-languag mavigation with image-text pairs from the web. In ECCV, 2020.
[51] Weituo Hao, Chunyuan Li, Xiujun Li, Lawrenee Carin, and Jianfeng Gao. Towards learning generic agent for vision-and-language navigation via pre-training. In CVPR, 2020.
[52] Kunyang Lin, Peihao Chen, Diwei Huang, Thomas H Li, Mingkui Tan, and Chuang Gan. Learning vision-and aad---nd--laaa language navigation from youtube videos. ICCV, 2023.
[53] Chong Liu, Fengda Zhu, Xiaojun Chang, Xiaodan Liang, Zongyuan Ge, and Yi-Dong Shen. Vision- language navigation with random environmental mixup. In ICCV, 2021.
[54] Jialu Li, Hao Tan, and Mohit Bansal. Envedit: Environment editing for vision-and sasion-an--langee and-language navigation. In CVPR, 2022.
[55] Zun Wang, Jialu Li, Yicong Hong, Yi Wang, Qi Wu, Mohit Bansal, Stephen Gould, Hao Tan, and Yu Qiao. Scaling data generation in iision-and-bl and-language navigation. In ICCV, 2023.
[56] Su Wang, Ceslee Montgomeyy, Jordi Orbay, Vighnesh Birodkar, Aleksandra Faust, Izzeddin Gur, Natasha Jaques, Austin Waters, Jason Baldridge, and Peter Anderson. Less more: Generating grounded navigation instructions from landmarks. In CVPR, 2022.
[57] Yibo Cui, Liang Xie, Yakun Zhang, Meishan Zhang. Ye Yan, and Erwei Yin. Grounded entity-landman adaptive pre-training for vision-and-languuae navigation. In ICCV, 2023.
[58] Hanqing Wang, Wenguan Wang, Wei Liang. Steven CH Hoi, Jianbing Shen, and Luc Van Gool. Active perception for disual-language navigation. IJCV, (33):67-6255) 2023.
[59] Jinyu Chen, Wenguan Wang, Si Liu, Hongsheng Li, and Yi Yang. Onnidirectional information gathering for knowledge transfer-based audio-visual navigation. In ICCV, 2023.
[60] Peihao Chen, Dongyu Ji, Kunyang Lin, Runhao Zeng, Thomas Li, Mingkui Tan, and Chuang Gan. Weakly-supervised multi-gramularity map learning for vision-and-language navigation. In NeurlPS, 2022.
[61] Dong An, Yuankai Qi, Yangguang Li, Yan Huang, Liang Wang, Tieniu Tan, and Jing Shao. Bevbert: Multimodal map pre-training for language-guided navigation. In ICCV, 2023.
[62] Rui Liu, Xiaohan Wang, Wenguan Wang, and Yi Yang. Bird' seed's-eye-view scene graph for vision-language navigation. In ICCV, 2023.
[63] Zihan Wang, Xiangyang Li, Jiahao Yang, Yeyi Liu, and Shuqiang Jiang. Gridmm: Grid memory map for vision- aodoo--nn-language navigation. In ICCV, 2023.
[64] Yicong Hong, Zun Wang, Qi Wu, and Stephen Gould, Bridging the gap between learning in discrete and continuous environments for vision-and and lan-uag navigation. In CVPR, 2022.
12

---

[65] Gengze Zhou, Yicong Hong, and Qi Wu. Navgpt: Explicit reasoning in vision-and-language navigation with large language models. In AAAI, 2024.
[66] Duo Zheng. Shijia Huang. Lin Zhao, Yiwu Zhong, and Liwei Wang. Towards learning generalist model for embodied navigation. CVPR, 2024.
[67] Dean Pomerlcau. Efficient training of artificial neural networks for autonomous navigation. Neural computation, 3(1) 1991.
[68] Mariusz, Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Fimer, Beat Flepp. Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self-driving cars. arXiv preprint 160. 2016.
[69] Pete Florence. Corey Lynch, Andy Zeng, Oscar A Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lec, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning, In CoRL, 2022.
[70] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillierap. Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In ICMI., 2016.
[71] Hanqing Wang, Wei Liang, Luc Van Gool, and Wenguan Wang. Dreamwalker: Mental planning for continuous vision-language navigation. In ICCV, 2023.
[72] Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blažej Osiński, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model based reinforcement learning for atari. In ICLR, 2019.
[73] Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. In NeurlPS, 2015.
[74] Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, et al. Stabilizing transformers for reinforcement learning, In JCML, 2020.
[75] Yang Song and Diederik Kingma. How to train your energy-based models. arXiv:2101.03284 2021.
arXiv preprint
[76] Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computa- tion, 177111777-8880 2002.
[77] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In NeurlPS, 2019.
[78] Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A estimation principle for unnormalized statistical models. In AISTATS. 2010.
[79] Will Grathwohl, Chan-Chich Wang, Been-Herrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. In ICLR, 2019.
[80] Chelsea Finn, Paul Christiano, Pieter Abbcel, and Sergey Levine. A connection between genera- tive adversarial networks, inverse reinforcement learning, and energy-based models. arXiv preprint arXiv:1611.03853 2016.
[81] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. ICML, 2017.
[82] Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. /CML, 2019.
[83] Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdinov. Efficient exploration via state marginal matching. arXiv preprint arXiv: 1906.0524, 2019.
[84] Radford M Neal et al. Meme using hamiltonian dynamics. Handbook of markov chain monte carlo, 2(11):2, 1.
[85] Giorgio Parisi. Correlation functions and computer simulations. Nuclear Physics B, (88(3))378-384) 1981.
13

---

[86] Brian D Ziebart, J Andrew Bagnell, and Anind K Dey. Modeling interaction via the principle of maximum causal entropy. In ICML, 2010.
[87] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niebner, Manolis Savva, Shuran Song, Andy Zeng. and Yinda Zhang. Matterportád: Learning from rgb-d data in indoor environments. In 3DV, 2017.
[88] Hanqing Wang, Wenguan Wang, Tianmin Shu, Wei Liang, and Jianbing Shen. Active visual information gathering for vision-language navigation. In ECCV, 2020.
[89] Yanyuan Qiao, Yuankai Qi, Yicong Hong, Zheng Yu, Peng Wang, and Qi Wu. Hop: history-and-order aware pre-training for vision-and and aud-language navigation. In CVPR, 2022.
[90] Yusheng Zhao, Jinyu Chen, Chen Gao, Wenguan Wang, Lirong Yang, Haibing Ren, Huaxia Xia, and Si Liu. Target-driven structured transformer planner for vision-language navigation. In ACM MM, 2022.
[91] Xiaohan Wang. Wenguan Wang, Jiayi Shao, and Yi Yang. Lana: A langunge-capable navigator for instruction following and generation. In CVPR, 2023.
[92] Rui Liu, Wenguan Wang, and Yi Yang. Volumetric environment representation for vision-language navigation. CVPR, 2024.
[93] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[94]
Locrre-Louis Guhur, Makarand Tapaswi, Shizhe Chen, Ivan Laptev, and Cordelia Schmid. Airbert: In-domain pretraining for vision-and- lisiou--nd-language navigation. In ICCV, 2021.
[95] Devendra Singh Chaplot, Dhiraj Gandhi, Saurahh Gupta, Abhinav Gupta, and Ruslan Salakhutdinov. Learning explore using active neural slam. In ICLR, 2019.
[96] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied research. In ICCV, 2019.
[97] Jacob Krantz, Aaron Gokaslan, Dhruv Batra, Stefan Lee, and Oleksandr Maksymets. Waypoint models for instruction-guided navigation in continuous environments. In ICCV, 2021.
[98] Muhammad Zubair Irshad, Niluthpol Chowdhury Mithun, Zachary Seymour, Han-Pang Chiu, Supun Samarasekera, and Rakesh Kumar. Semantically-aweer spatio-temporal reasoning agent for vision-and- language navigation continuous environments. In ICPR, 2022.
[99] Georgios Georgakis, Karl Schmeckpeper. Karan Wanchoo, Soham Dan, Eleni Miltsakaki. Dan Roth, and Kostas Daniilidis. Cross-modal map learning for vision and language navigation. CVPR, 2022.
[100] Jacob Krantz and Stefan Lee, Sim-2 2-sim transfer for vision-an lan-uang navigation continuous environments. In ECCV, 2022.
[101] Zihan Wang, Xiangyang Li, Jiahao Yang, Yeqi Liu, Junjie Hu, Ming Jiang, and Shugiang Jiang. Lookahead exploration with neural radiance representation for continuous vision-language navigation. In CVPR, 2024.
[102] Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky, Neural networks for machine learning lecture 6a overview of mini-batch gradient descent.
[103] Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. EMNLP, 2019.
[104] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2018.
[105] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 123:32-73, 2017.
[106] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer. Georg Heigold, Sylvain Gelly. et al. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR, 2020.
14

---

[107] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[108] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
[109] Yang Song, Sahaj Garg. Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and estimation. In Uncertainty in Artificial Intelligence, 2020.
[110] Weijian Luo, Boya Zhang, and 7hihua Zhang. Entropy- training methods for scalable neural implicit samplers. In NeurIPS, 2023.
15

---

A Appendix
A.1 Model Details
EnyDrop [3] adopts an encoder-decoder model. The encoder is a bidirectional LSTM-RNN with an embedding layer. The decoder an attentive LSTM-RNN. EnvDrop uses the -048-ddmmnsiooal image features from ResNet-152 pretrained on ImageNet. The model is first trained with supervised learning via the mixture of imitation and reinforcement learning. Then, it is finetuned by back translation with environmental dropout. The word embedding size is 256 and the dimension of the action embedding 64. The size of the LSTM units is 512 while 256 is set for the bidirectional LSTM. RMSprop [02] is employed to optimize the loss with a learning rate 1x 1x10 and the batch size is set as 64.
VINO
built upon LXMERT-like transformer 103 It is trained directly on the mixture of the original training data and the augmented data from PREVALENT [5 for R2R ①. Adam optimizer [104] is used and the batch size is set to 16. For REVERIE [30], the image features are extracted by ResNet-152 pre-trained on Places365 [48), and the object features are encoded by Faster-RCNN pre-trained on Visual Genome 105 The batch size is set as 8.
VEN for CE 64 proposes a waypoint predictor generate set of candidate waypoints during navigation. The training objective is the cross-entropy loss between the ground-trun truth and action predictions. It is minimized by AdamW optimizer [04] with the learning rate 1 x 10 during
training. The decay ratio is set as 0.50 for R2R-CE [3 and 0.75 for RxR-CE [32
The hidden layer size is 768. The image features are extracted by ViT-B/16 [106) pretrained on ImageNet [107]. On REVERIE [30], DUET is first pre-trained with the batch size of 32. Some auxiliary tasks are employed, such as single-step action prediction, object grounding, masked language modeling, and masked region classification [5]. Then it is finetuned with the batch size of 8.
DUET 6 uses dual-scale graph transformer initialized from LXMERT 103
ETPNav 29
contains topological mapping module and waypoint prediction module [64] is
pre-traing t-atrained using proxy tasks following existing transformer-basee VLN models [4 with a batch size of 64 and a learning rate x 10 5 During finetuning, the AdamW optimizer [104] is utilized.
The batch size is set as 16 with learning rate of x 10-5. The decay ratio is 0.75.
A.2
Comparison Results across Different Path Lengths
In Fig- 4 ENP demonstrates notable performance improvements over BC 66) across both seen and unseen splits of R2R []. When the length of ground-truth paths exceeds 16 meters, our ENP outperforms BC by significant margin in both the val seen and val unseen splits. In addition, the performance gains are relatively more significant in the unseen split compared to the seen split. For example, ENP achieves approximately 5% improvement of SR on the val seen and 6% improvement on the val unseen for navigation paths longer than 16 meters.
75
BC-DUET
ENP-DUET
BC-DUET
ENP-DUET
Average Success Rtte (%)
Average Staccess (%)
75
30
50
11:15
21.25
Lengil (m)
(b)
Length (m)
Figure 4: The average success rate of ENP and BC 6 across diff'erent trajectory lengths.
A.3 Discussion
Terms use, Privacy, and License. R2R D, REVERIE 30
R2R-CE 31,
RxR-CE
datasets utilized in our work contain large number of indoor photos from Matterport3D 887
16

---

which are freely available for non-commercial research purposes. No identifiable individuals are present in any of the photographs, All experiments are conducted on the Matterport3D and Habitat Simulators [ 96 which are released under the MIT license.
Linitations. Existing studies in VLN mainly focus improving the success rate of navigation, with limited attention given to collision avoidance and navigation safety. In addition, current agents are typically designed for static environments within simulators. To mitigate the risk of collisions, is essential for robots to operate under specific security constraints. Therefore, further advancements are necessary to ensure safe deployment in real-world dynamic environments.
Future Direction. i) We explore ENP on several representative VLN architectures 3
and
the generalization of ENP to other architectures [63 61] and more tasks [T08] is not clear. We will verify it in future work. ii) Efficiently sampling from un-normalized distributions for high-dimensional data is fundamental challenge in machine learning. Utilizing MCMC-free approaches [75] [09] and neural implicit samplers [110] may offer valuable insights to enhance our approach. One limitation of ENP that the optimization through the SGLD inner loop each training iteration (Algorithm [ which would reduce the training efficiency (about 9% delay for 5 SGLD iterations). In practice, ENP balances performance and training time with a limited number of iterations. iii) In the future, agents should be able to actively acquire the skills by watching videos similar to humans [52] thereby reducing high training data (expert demonstrations) requirements.
Broader Impacts. We introduce an energy based policy learning for VLN. Equipped with ENP, exist- ing agents are able to perform comprehensive decision-making, showeasing promising improvements across various VIN benchmarks. In real-world deployment, the agent holds potential to significantly benefit society by providing assistance individuals with specific needs, e.g., blind and disabled, enabling them to accomplish daily tasks more independently. In addition, we encourage more efforts devoted policy learning for future research in the community.
baasey
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See $1 We include the paper's contributions and scope in the abstract and
introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
In Appendix A.3 we discuss the limitations of the work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
18

---

Justification: In 3.2 we provide the details of our method.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: In Algorithm T and 3.4
We provide the implementation details of our method.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer:
[No]
Justification:
As we promised, the code will be released upon the publication of our paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
public caalc/guiieee quw.g/udeeeeeeeeeeo Sewedidde/eoeeeuuineem CodeSulmmeeeeeeeeeeeoeee ss onPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https?
//nips. httpsscccpccccccccceeeeeeeee cetsc///ui////eeeeeeoeoo.sscsses ies CodeeSumm ss iiiiooollii ooliccyy icy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: In $4 and Appendix A.I we provide the training and test details for the experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: In Table 2 and we report error bars on VLN benchmarks.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
20

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
In 334 we provide the details of the computer resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We acknowledge the NeurlPS Code of Ethics and obey them in our paper.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
In Appendix A.3 discuss the broader impacts of our work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper is based on publicly available datasets and models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: In Appendix A.3 we provide the details of the license and terms of use. We also use existing models components and cite appropriately.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paaperswis wwwoosocittcceeeeoo com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "ipD8tgc1yu", "reviewer": "Reviewer_UeCa", "summary": "This paper proposes to learn a better Vision-and-Language Navigation agent by jointly optimizing the action and the state. Specifically, they propose an energy-based expert policy, where they adds an explicit objective for navigation exploration and incorporates prior knowledge from the expert demonstrations. Empirical results on multiple VLN benchmarks (e.g., R2R, REVERIE, R2R-CE) demonstrates the effectiveness of the proposed approach.", "questions": "Please refer to weakness.", "limitations": "N/A", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed approach is supported with good theoretical foundations, and this paper provides a detailed derivation of the objective, as well as providing explanation on how to place the approach in previous literature with imitation learning approaches. \n2. Empirical results on multiple VLN benchmarks demonstrate the effectiveness of the approach, and also demonstrates this approach's generalization to different base VLN agents. \n3. Detailed ablations to investigate different components of the approach (e.g., step size, gaussian noise, SGLD loop).", "weaknesses": "1. The proposed approaches work pretty well on sota agents like DUET, but it's underinvestigated whether the proposed approach also benefits VLN agents when having large-scale data. Specifically, it's important to have experiments on ScaleVLN or HM3D-AutoVLN.\n2. The approach might be sensitive to hyperparameters as suggested in Table 6 and Table 7, which might need additional efforts in hyperparameter tunning when generalized to different environments/agents/tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: *\"... whether the proposed approach also benefits VLN agents when having large-scale data.\"***\\\n**A1:** Thank you for your comments. The proposed ENP is a general framework to mitigate the accumulation of errors in the sequential decision-making process of VLN. To demonstrate the effectiveness of our ENP on large-scale data, we conduct experiments on ScaleVLN [49], without image augmentation [48] as follows.\n\n|Models|Training Data|R2R *val unseen* SR|R2R *val unseen* SPL|R2R *test unseen* SR|R2R *test unseen* SPL|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|DUET [6]|Original Data [1]|71.52|60.02|69|59|\n|**ENP-DUET (ours)**|Original Data [1]|**74.39**|**60.24**|**71.43**|**60.31**|\n|DUET [6]|ScaleVLN [49]|79|70|77|68|\n|**ENP-DUET (ours)**|ScaleVLN [49]|**80.40**|**70.46**|**79.25**|**69.39**|\n\nAs seen, ENP gives consistent improvement with large-scale training data (*i.e.*, **1.40**% and **2.25**% SR on val unseen and test unseen, respectively). Please note that all the experiments above utilize the original hyperparameters of ENP (L226-228). We will update this result in Table 1 of the final version.\n\n**Q2: *\"The approach might be sensitive to hyperparameters as suggested in Table 6 and Table 7 ...\"***\\\n**A2:** We clarify the robustness of ENP to hyperparameters based on the following aspects. **First**, as shown in Table 6 and Table 7, ENP achieves superior performance with **consistent hyperparameters for the same model across different tasks** (e.g., R2R [1] and R2R-CE [32]). **Second**, ENP is **insensitive to parameter changes within adjacent ranges**. Due to limited space in the paper, we reported only a subset of the results in Tables 6 and 7. For Table 6, we provide detailed results of $\\epsilon$ as follows. We find that it has minimal impact on the performance when $\\epsilon$ varies between 1.3 and 1.9. \n\n|Models|#Loop|R2R *val unseen* SR|R2R *val unseen* SPL|R2R-CE *val unseen* SR|R2R-CE *val unseen* SPL|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|ENP-VLNBERT|$\\epsilon=1.0$, $Var(\\xi)=0.01$|64|**57**|**45**|39|\n|ENP-VLNBERT|$\\epsilon=1.1$, $Var(\\xi)=0.01$|**65**|**57**|**45**|39|\n|ENP-VLNBERT|$\\epsilon=1.3$, $Var(\\xi)=0.01$|**65**|**57**|**45**|**40**|\n|ENP-VLNBERT|**$\\epsilon=1.5$, $Var(\\xi)=0.01$**|**65**|**57**|**45**|**40**|\n|ENP-VLNBERT|$\\epsilon=1.7$, $Var(\\xi)=0.01$|**65**|**57**|**45**|**40**|\n|ENP-VLNBERT|$\\epsilon=1.9$, $Var(\\xi)=0.01$|**65**|**57**|44|**40**|\n\n**In addition**, more experimental results related to Table 7 are presented below. Obviously, when the hyperparameter $I$ is between 15 and 19, the performance of ENP-VLNBERT remains stable.\n\n|Models|#Loop|R2R *val unseen* SR|R2R *val unseen* SPL|R2R-CE *val unseen* SR|R2R-CE *val unseen* SPL|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|ENP-VLNBERT|$I=14$|**65**|56|**45**|39|\n|ENP-VLNBERT|**$I=15$**|**65**|**57**|**45**|**40**|\n|ENP-VLNBERT|$I=16$|**65**|**57**|**45**|**40**|\n|ENP-VLNBERT|$I=17$|**65**|**57**|**45**|**40**|\n|ENP-VLNBERT|$I=18$|**65**|**57**|**45**|**40**|\n|ENP-VLNBERT|$I=19$|**65**|56|44|**40**|\n|ENP-VLNBERT|$I=20$|64|56|44|39|\n\nWe will definitely add these detailed tables to the Appendix in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C473cWn17h", "reviewer": "Reviewer_Ngjf", "summary": "This paper studies vision-and-language navigation (VLN) by addressing the problems of error accumulation in the Markov decision process and the difficulties in designing an optimal reward function, respectively, in the commonly applied behavioral cloning (BC) and reinforcement learning (RL) approaches. It proposes an energy-based policy (ENP) for training the agent by estimating the unnormalized probability density of the expert’s occupancy measure, optimizing the entire trajectories rather than the single-time step decisions in BC. Meanwhile, to facilitate the MCMC sampling in ENP, a Marginal State Memory is introduced for the agent as an online reply buffer. The proposed method has been evaluated on multiple popular VLN agents and several benchmarking datasets and environments, together with detailed analyses of the ENP and design choices.", "questions": "No other questions; I wish the authors could kindly respond to my concerns in Weaknesses.", "limitations": "The authors have discussed some limitations of the proposed method and its potential societal impact. As for suggestions, mainly the two points in Weaknesses – test on large-scale training data and run thorough result analysis.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.\tThe energy-based navigation policy proposed in this paper introduces a new training perspective for VLN and potentially for broader visual navigation problems. The idea is very well-motivated as it addresses the limitations of commonly applied BC and RL approaches. As a researcher in visual navigation, I am happy to see the exploration of such a fundamentally different training approach and the fact that the method works well across models and settings.\n2.\tArguments and design choices made in this paper are well-justified by analyses and experiments (e.g., Section 3.3 and Section 4.3).\n3.\tThe proposed idea has been evaluated across various popular VLN baselines (e.g., EnvDrop, VLN-BERT, and DUET), different datasets (R2R, REVERIE, and RxR), and different environments (e.g., discrete and continuous), achieving obvious improvements and showing the effectiveness of the method.\n4.\tOverall, the paper is nicely written. Contents, tables, and figures are clearly presented.", "weaknesses": "1.\tThe proposed ENP has only been evaluated on a limited scale of VLN datasets and models. It is unclear if the method will still benefit larger models, or the case where more sufficient training data is available to learn a very generalizable policy through IL/BC (e.g., 4.9M ScaleVLN data [49] vs. the original 14K R2R data). I am concerned that IL/BC is much more friendly and stable than ENP to scale with data.\n2.\tThere is no comprehensive successful/failure case analysis of ENP versus existing training methods. It is unclear exactly in what scenarios an agent trained with ENP shows an advantage.\n\nI need to mention that I have very little knowledge about energy-based models, so I might have missed some key details in mathematics. I will rely more (and probably adjust my review based) on the comments from the other reviewers on the theoretical part of this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Evaluation on ScaleVLN.**\\\n**A1:** Per your request, we conduct additional experiments on ScaleVLN [49] using the original hyperparameters of ENP (L226-228).\n\n|Models|Training Data of R2R|*val unseen* SR|*val unseen* SPL|*test unseen* SR|*test unseen* SPL|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|DUET [6]|Original Data [1]|71.52|60.02|69|59|\n|**ENP-DUET (ours)**|Original Data [1]|**74.39**|**60.24**|**71.43**|**60.31**|\n|DUET [6]|ScaleVLN [49]|79|70|77|68|\n|**ENP-DUET (ours)**|ScaleVLN [49]|**80.40**|**70.46**|**79.25**|**69.39**|\n\nAs seen, ENP outperforms DUET with the same training data from ScaleVLN, *e.g.*, **1.40**% and **2.25**% SR gains over *val unseen* and *test unseen*, respectively.\n\n**Q2: *\"IL/BC is much more friendly and stable than ENP to scale with data.\"***\\\n**A2:** **First**, we have validated the effectiveness of ENP on large-scale training data in **Q1**. We agree that classic behaviour cloning (BC) is straightforward and easy to train for sequential decision-making in VLN. However, as mentioned in many studies [13,16,29], BC suffers from accumulation errors during both training and inference due to covariate shift (L25-35). **Second**, ENP utilizes persistent contrastive divergence (PCD) [73] to train the energy-based models (L182-190). The stability of PCD has been widely demonstrated in various tasks, such as classification [76], generation [23], reinforcement learning [77,78], and natural language processing [ref-1]. **Third**, the training curves provided in the PDF of \"global\" response (Fig. S3) clearly show that our ENP maintains a stable training process on both the original R2R data (14K) and ScaleVLN (4.9M). We will clarify this in the Experiment of the final version.\n\n[ref-1] Residual energy-based models for text generation. ICLR 2020.\n\n**Q3: *\"It is unclear if the method will still benefit larger models ...\"***\\\n**A3:** It is interesting to investigate ENP on larger models. We noted that NaviLLM [62], which is built updon a large language model (Vicuna-7B [ref-2]), uses the similar pre-training and finetuning paradigm [6] with BC, so ENP can be seamlessly integrated into this framework. Due to the limited time, we incorporate ENP ($I=15$, $\\epsilon=1.3$, $Var(\\xi)=0.01$) during the finetuning stage. From the following results, we observe that ENP gives **2.43**% and **2.78**% improvements on SR of R2R *val unseen* and SR of REVERIE *val unseen*, respectively. More results will be provided in the Experiment of the final version.\n\n|Models|R2R *val unseen* SR|R2R *val unseen* SPL|REVERIE *val unseen* SR|REVERIE *val unseen* SPL|\n|:-:|:-:|:-:|:-:|:-:|\n|NaviLLM [62]|67|59|42.15|35.68|\n|ENP-NaviLLM|**69.43**|**60.47**|**44.93**|**36.67**|\n\n[ref-2] Instruction tuning with GPT-4.\n\n\n**Q4: More analysis on successful/failure cases.**\\\n**A4:** We provide more analysis of ENP *vs.* DUET on success/failure cases in the PDF material of \"global\" response (Fig. S2). As illustrated in Fig. S2 (a), our agent with ENP successfully reaches the final location, whereas DUET fails due to the accumulation of errors over long paths. In Fig. (b), both our agent and DUET struggle to find the target landmark due to the lack of a specific landmark recognition module like [50,51]. We will incorporate this figure into Fig. 3 in the final revision. \n\n**Q5: *\"... in what scenarios an agent trained with ENP shows an advantage.\"***\\\n**A5:** **First**, in Table 3 and 4, ENP achieves consistent performance gains in both *seen* and *unseen* environments across different datasets (R2R-CE [33] and RxR-CE[37]). **Second**, the gains in *unseen* are relatively more significant than that in *seen*. For example, ENP achieves **5.18**% gains of SR on R2R *val seen* (>10m) and **9.35**% gains on *val unseen* (>10m) as follows. **Third**, we further observe that our ENP gives more improvements on challenging cases with long trajectories as follows. ENP demonstrates its capability on R4R [ref-3], where the average path length is about twice that of R2R. The average success rate of ENP exceeds that of BC by **6.40**% on *val seen* (>20m). The performance gain can also significantly reach up to **9.03**% on R4R *val unseen* (>20m). \n\n|Models|R2R *val seen* (<10m)|R2R *val seen* (>10m)|R2R *val unseen* (<10m)|R2R *val unseen* (>10m)|R4R *val seen* (<20m)|R4R *val seen* (>20m)|R4R *val unseen* (<20m)|R4R *val unseen* (>20m)|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|BC-DUET|76.33|65.50|72.51|46.08|70.88|57.25|52.52|43.71|\n|ENP-DUET|77.69|70.68|74.63|55.43|72.67|63.65|54.91|52.74|\n|$\\Delta$|+1.36|+**5.18**|+2.12|+**9.35**|+1.79|+**6.40**|+2.39|+**9.03**|\n\nThis indicates that ENP alleviates the cumulative errors in sequential decision-making (L200-210). In the PDF of \"global\" response, we provide a detailed analysis of success rate across different path lengths (Fig. S1). We will include it in the Appendix of the final version.\n\n[ref-3] Stay on the Path: Instruction Fidelity in Vision-and-Language Navigation. ACL 2019.\n\n**Q6: More explanation about energy-based models (EBMs).**\\\n**A6:** Here, we analyze the EBMs from the perspective of sequential prediction [ref-4]. Existing VLN models are based on locally normalized models which optimize the prediction through maximum likelihood at each step. To multigate the exposure bias [ref-4], EBMs define a global energy function at the sequence level, where lower energy values correspond to higher probabilities [21]. Then the normalized probability density can be obtained through dividing by the partition function (Eq. 4). Thus, estimating the partition function is crucial for training EBMs. There is a long history in machine learning [20] for training EBMs. We adopt persistent contrastive divergence [22,73] to sample the partition function and optimize the EBMs (L182-190). We hope it will help you understand more clearly. We will add a detailed background of the EBMs in the Related Work. \n\n[ref-4] Sequence level training with recurrent neural networks. ICLR2016.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ynvmdHYrEg", "reviewer": "Reviewer_dJ3y", "summary": "To attempt to solve the catastrophic failure issues that arise from behavioral cloning, the authors introduce Energy-based Navigation Policy (Enp) which uses an energy-based model to learn the expert policy from demonstrations. The authors claim that by doing so Enp prioritizes learning the full trajectory instead of single-time step decisions in BC.", "questions": "**Q1.** Why are the DUET or, e.g., BEVBert/GridMM, paths not shown in Figure 3b too? If multiple agents face the same issue, that would show this is the problem of observability.\n\n**Q2.** What are the $\\pm$ values in Table 3, is it the standard deviation? If so, why is it not reported for the other methods?\n\n**Q3.** Do the authors have any intuition on why increasing $I_{\\text{pre}}$ and $I_{\\text{ft}}$ could result in a lower SR/SPL (Table 7)?", "limitations": "The authors do not discuss the limitations of their method. The limitations paragraph in Appendix A.2. does not reflect the particular ENP method they propose and is instead generic to the problem. Training time is one potential limitation that should be studied and discussed in further detail.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "**S1.** The issue of catastrophic failure from BC is an important one that which is likely applicable to the setting of VLN — this justifies the importance of the problem.\n\n**S2.** The method is simple and described appropriately, both of which are strengths of the work.", "weaknesses": "**W1.** *Overall improvements are low.* The performance improvements provided by ENP are small compared to the BC baselines, often around $1-2\\%$ in terms of SR and similar in SPL. This is particularly poignant when compared to the improvements provided by semantic grid maps. By itself, this would not necessarily be a major issue, if it was clear where ENP was helping (W2).\n\n**W2.** *It is unclear when ENP is better than BC, and if the intuitive justification of catastrophic failure applies.* A more in-depth analysis of the success/failure cases of using ENP vs BC is warranted. Metrics like SR suggest that ENP can be helpful in avoiding the failure that BC can induce, but it is not entirely clear this is the case — when is ENP helping? Given the justification of the catastrophic failure of BC, you would expect this would be in longer trajectories, but the example in Figure 3a does not justify that given its short trajectory length. Could you create a benchmark specifically to test this? Including this analysis could boost the case for the motivation behind ENP significantly.\n\n**W3.** *ENP is likely more expensive to train than BC.* The authors should include a compute comparison for the training of ENP vs BC. While the inference runtime is the same, training time for ENP will likely be significantly influenced by $I$.\n\nThe combination of W1 and W2 severely hinder the practical motivation of the method.\n\nMinor comments:\n\n- It’s not really required to include all baselines in the main results tables. E.g., Seq2Seq, SF and RCM are pretty outdated at this point and always beaten by other methods — it’s better to focus the comparison with the most recent methods and leave the full tables to the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The improvement of ENP is lower than that of semantic grid maps.**\\\n**A1:** **First**, respectfully, the improvement of 1-2% on both SR and SPL is **non-trival** in VLN. This point was acknowledged by the other reviewers, such as \"work pretty well\" (Reviewer UeCa) and \"achieving obvious improvements\" (Reviewer Ngjf). **Second**, although we agree that semantic maps [54-57] are powerful tools, ENP improves the performance from different perspectives. ENP learns an unbiased expert policy by exploiting information from the training data (Eq. 3), while semantic maps introduce additional scene information, *e.g.*, [55] encodes spatial priors by grid maps. **Third**, the below table shows ENP requires significantly less GPU memory than [55] (DUET-based [6] framework). \n\n|Models|GPU Memory on NVIDIA RTX 4090|\n|:-:|:-:|\n|DUET [6]|14G|\n|BEVBert [55]|20G|\n|ENP-DUET|**15G**|\n\nTherefore, it is **unfair** to compare ENP directly against the semantic maps. More importantly, ENP is a flexible solution and can be seamlessly integrated into existing methods without extra inference time. The following table indicates that ENP is **complementary** to [55] and further enhance the performance.\n\n|Models|R2R *val unseen* SR|R2R *val unseen* SPL|REVERIE *val unseen* SR|REVERIE *val unseen* RGS|\n|:-:|:-:|:-:|:-:|:-:|\n|BEVBert|75|64|51.78|34.71|\n|ENP-BEVBert|**76**|**65**|**52.71**|**34.80**|\n\n**Q2: *\"... when ENP is better than BC ...\"***\\\n**A2:** **First**, in Tables 1-4, ENP can achieve consistent performance gains across different datasets. **Second**, we explore ENP and BC across different trajectory lengths in Fig. S1 of the \"global\" response PDF. In Fig. S1 (a) and (b), ENP exceeds BC by **6**% and **9**% on R2R *val seen* and *val unseen* (16-20m). **Third**, we further validate the effectiveness of ENP on R4R [ref-3] with the longer paths in the following table (also in Fig. S1 (c) and (d)). ENP surpasses BC by **6.40**% and **9.03**% on R4R *val seen* and *val unseen* (>20m), and the gains in *unseen* environments are relatively more significant.\n\n|Models on R4R *val*|*seen* (<20m)|*seen* (>20m)|*unseen* (<20m)|*unseen* (>20m)|\n|:-:|:-:|:-:|:-:|:-:|\n|BC-DUET|70.88|57.25|52.52|43.71|\n|ENP-DUET|72.67|63.65|54.91|52.74|\n|$\\Delta$|+1.79|+**6.40**|+2.39|+**9.03**|\n\nWe will add these figures and tables to the Appendix.\n\n[ref-3] Stay on the Path: Instruction Fidelity in Vision-and-Language Navigation. ACL 2019.\n\n**Q3: A benchmark with longer trajectories.**\\\n**A3:** We clarify that we have conducted experiments on R2R-CE [32] and RxR-CE [33] in Tables 3 and 4 (L283-292), which involve **significantly longer action trajectories** (56 vs. 5 steps in R2R). **In addition**, per your request, we report the results on R4R [ref-3] with longer paths. CLS [ref-3] measures how closely the trajectory aligns with the ground truth. ENP achieves **3.52**% and **3.11**% gains on SR and CLS of *val unseen*.\n\n|Models on R4R *val unseen*|SR|CLS|\n|:-:|:-:|:-:|\n|DUET|50.18|40.54|\n|ENP-DUET|**53.70**|**43.65**|\n\n**Q4: A detailed analysis of success/failure cases.**\\\n**A4:** Thanks for your feedback. We include more success/failure cases in the PDF material of \"global\" response (Fig. S2). In Fig. S2 (a), our agent with ENP successfully reaches the final location, whereas DUET fails due to the accumulation of errors over long paths. In (b), both our agent and DUET struggle to locate the target due to the lack of a specific landmark recognition module like [50,51]. This figure will be added to Fig. 3. \n\n**Q5: Training time for ENP.**\\\n**A5:** We report the training time for ENP with different SGLD iterations ***I*** on a NVIDIA RTX 4090. In practice, we find that ***$I_{ft}=5$*** SGLD loops per training iteration is good enough for model convergence (Algorithm 1), introducing a minor computational overhead, approximately 4 minutes delay for 1K training iterations.\n\n|Models|minutes/1K training iterations|SGLD *$I_{ft}$*|\n|:-:|:-:|:-:|\n|DUET|40|--|\n|ENP|42|3|\n|ENP|44|5|\n|ENP|54|10|\n\n**Q6: The comparison with recent methods.**\\\n**A6:** Sure, we will reorganize Tables 1-3 and highlight the comparison with the most recent methods. The complete tables will be included in the Appendix.\n\n**Q7: *\"... the DUET paths not shown in Figure 3b ...\"***\\\n**A7:** To clearly present the comparison, we only include two navigation trajectories in each figure for better understanding (Fig. 3). We will include the comparison results of these methods in the Appendix and provide further analysis. \n\n**Q8: The standard deviation ($\\pm$) in tables.**\\\n**A8:** Our apologies. \"$\\pm$\" in Tables 1-4 denotes the standard deviation after repeating the experiments with different random seeds. Unfortunately, the corresponding standard deviations for other methods were not provided in their original papers.\n\n**Q9: *\"... increasing $I_{pre}$ and $I_{ft}$ results in a lower SR/SPL (Table 7)\"***\\\n**A9:** Since there is a domain gap between *seen* and *unseen* environments [3,9], increasing SGLD iterations ($I_{ft}$) may lead to overfitting on *val seen*, resulting a lower SR/SPL on *val unseen*. Additional results on *val seen* for Table 7 are provided in the following. \n\n|Models|#Loop|R2R *val seen* SR|R2R *seen* SPL|R2R *val unseen* SR|R2R *unseen* SPL|R2R-CE *val seen* SR|R2R-CE *seen* SPL|R2R-CE *val unseen* SR|R2R-CE *unseen* SPL|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|DUET [6,30]|--|77|68|72|60|66|59|57|49|\n|ENP|$I_{pre}$=20, $I_{ft}$=5|78|68|**74**|**60**|68|59|**58**|**50**|\n|ENP|$I_{pre}$=20, $I_{ft}$=10|**81**|**70**|73|60|**69**|**60**|58|49|\n\n\n**Q10: More discussion about the limitations.**\\\n**A10:** One limitation of ENP is that the optimization through the SGLD inner loop at each training iteration (Algorithm 1). It would reduce the training efficiency (about 9% delay for 5 SGLD iterations). In practice, ENP balances performance and training time with a limited number of SGLD iterations. We will provide more analysis in the Discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**To all reviewers:** 

Thank you very much for your valuable time and suggestive comments. In response, we have meticulously addressed each point raised and provided point-to-point clarifications. Also, the final version of our manuscript will be updated accordingly. 

We appreciate the positive feedback from **Reviewer UeCa** and **Ngjf**. Their main concern is about the performance of our ENP on larger-scale datasets, such as ScaleVLN [49]. In our response, we report the results and demonstrate the effectiveness of our ENP on large-scale training data.

We are very grateful for the constructive comments from  **Reviewer dJ3y**. To address your concerns, we provide a detailed and thorough analysis for comparing ENP and BC. In addition, we include **supplementary figures in the PDF of this "global" response** for the following aspects:
1. A comparison of navigation accuracy between ENP and BC across different path lengths and environments, as shown in Figure S1.
2. More success/failure cases of our ENP *vs.* BC in Figure S2 (also mentioned by **Reviewer Ngjf**).
3. The training curves for ENP on R2R and ScaleVLN in Figure S3 (also noted by **Reviewer Ngjf**).

For more details, please refer to our responses to each reviewer. We have strived to address each of your concerns and welcome further discussions and insights.

Sincerely yours,\
Authors

Meta-review: This paper tackles vision-language navigation through a perspective that differs from the traditional behavioral cloning methods. Specifically, the authors propose an energy-based method for joint optimization of the feature-action space. Theoretical motivation for this approach is discussed, and strong results are shown across a number of VLN settings. 

  Overall, the reviewers appreciated the strong novel contribution in terms of both theoretical foundations and method which differs from traditional approaches, as well as the empirical results and ablations. A number of joint concerns were raised by a number of reviewers such as the lack of validation on larger-scale VLN settings as well as models, as well as success/failure analysis to show exactly where/how this method improves performance. Other weaknesses such as an apparent sensitivity across the hyper-parameters were mentioned. The authors provided a strong rebuttal with new results on larger-scale settings (ScaleVLN) and models (NaviLLM with a Vicuna-7B model). Importantly, the original hyper-parameters were used, showing both good scaling as well as some hyper-parameter insensitivity. Reviewers were overall satisfied with raised scores based on the rebuttal. 

  After considering all of the materials for this paper, I recommend acceptance. It provides an interesting novel approach that is not typical in the community, with strong results, potentially paving the way for new exploration in this area. Further, the method seems applicable to other IL problems where BC is common, and I hope that this method can be explored within those contexts and settings as well. I highly encourage the authors to include the rebuttal results and discussions in the main paper, as I believe it significantly strengthens the paper's claims and impact.
Decision: Accept (poster)