Vision-Language Navigation with Energy-Based Policy

Authors: rui  liu
wenguan  wang
yi  yang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-01 07:41:17

Abstract: Vision-language navigation (VLN) requires an agent to execute actions following human instructions. Existing VLN models are optimized through expert demonstrations by supervised behavioural cloning or incorporating manual reward engineering. While straightforward, these efforts overlook the accumulation of errors in the Markov decision process, and struggle to match the distribution of the expert policy. Going beyond this, we propose an Energy-based Navigation Policy (ENP) to model the joint state-action distribution using an energy-based model. At each step, low energy values correspond to the state-action pairs that the expert is most likely to perform, and vice versa. Theoretically, the optimization objective is equivalent to minimizing the forward divergence between the occupancy measure of the expert and ours. Consequently, ENP learns to globally align with the expert policy by maximizing the likelihood of the actions and modeling the dynamics of the navigation states in a collaborative manner. With a variety of VLN architectures, ENP achieves promising performances on R2R, REVERIE, RxR, and R2R-CE, unleashing the power of existing VLN models.

Main paper: 
 introduction: 
 Vision-language navigation (VLN) [1] entails an embodied agent that executes a sequence of actions to traverse complex environments based on natural language instructions. VLN is typically considered as a Partially Observable Markov Decision Process (POMDP) [2] , where future states are determined only by the current state and current action without explicit rewards. Therefore, the central challenge for VLN is to learn an effective navigation policy. In this context, existing neural agents [1, 3, 4, 5, 6] are naturally constructed as data-driven policy networks by imitation learning (IL) [7] and bootstrapping from expert demonstrations, i.e., ground-truth trajectories [1] .Most VLN models [1, 3, 8, 4, 5, 6] utilize behavioural cloning (BC), a classic approach for IL, through supervised training. While conceptually simple, BC suffers seriously from quadratic accumulation errors [9, 10] along the trajectory due to covariate shift, especially in partially observable settings. Several efforts introduce 'student-forcing' [1, 11] and 'pseudo interactive demonstrator' [6] , which are essentially online versions of DAGGER [12] , to alleviate this distribution mismatch. DAGGER assumes interactive access to expert policies (i.e., the shortest path from current state to the goal 2 ) and reduces to linear expected errors [12, 10] . Some other studies [13, 3, 8, 4, 5] combine reinforcement learning (RL) [14] and IL algorithms to mitigate the limitations of BC. Though effective, it presents challenges in designing an optimal reward function [15] and demands careful manual reward engineering. Such reward functions may not be robust to changes in the environment dynamics and struggle to generalize well across diverse scenes [16] . In addition, recent research [6] has revealed that training transformer [17] -based VLN models using RL would introduce instability.From a probabilistic view, most VLN agents [1, 3, 4, 6] essentially train a discriminative action classifier [8, 3, 4] , and model the action distribution conditioned on current navigation state P (a|s) at each time step (Fig. 1 ). This classifier is supervised by the cross-entropy loss to minimize 1-step deviation error of single-step decision along trajectories [18] . However, the prediction of current action affects future states during the global execution of the learned policy, potentially leading to catastrophic failures [9, 12] . In other words, solely optimizing the conditional action probabilities (i.e., policy) remains unclear for the underlying reasons of the expert behaviour, yet fails to exploit complete information in the state-action distribution P (s, a) induced by the expert [14] .In light of this, we propose an Energy-based Navigation Policy (ENP) framework that addresses the limitations of current policy learning in VLN from an energy perspective (Fig. 1 ). Energy-based models (EBMs) [19] are flexible and capable of modeling expressive distributions, as they impose minimal restrictions on the tractability of the normalizing constant (partition function) [20] . We reinterpret the VLN model as an energy-based model to learn the expert policy from the expert demonstrations. Specifically, ENP models the joint distribution of state-action pairs, i.e., P (s, a), by energy function, and assigns low energy values to the state-action pairs that the expert mostly performs. For optimization, this energy function is regarded as the unnormalized log-density of joint distribution, which can be decomposed into the standard cross-entropy of action prediction and marginal distribution of the states P (s). During training, ENP introduces persistent contrastive divergence [21, 22] to estimate the expectation of P (s), and samples by Stochastic Gradient Langevin Dynamics [23] . In this way, ENP simulates the expert by maximizing the likelihood of the action and incorporates prior knowledge of VLN by optimizing state marginal distribution in a collaborative manner, thereby leveraging the benefits of both energy-based and discriminative approaches.Theoretically, the training objective of ENP is to maximize the expected log-likelihood function over the joint distribution P (s, a). This is equivalent to estimating the unnormalized probability density (i.e., energy) of the expert's occupancy measure [15, 24, 25] . Therefore, ENP performs prioritized optimization for entire trajectories rather than the single-time step decisions in BC, and achieves a global alignment with the expert policy. Furthermore, we realize that ENP is closely related to Inverse Reinforcement Learning (IRL) [7, 26, 15] . For the optimized objective, ENP shares similarities with IRL but minimizes the forward KL divergence [27, 28] between the occupancy measures.For thorough examination ( §4), we explore ENP across several representative VLN architectures [3, 4, 6, 29] . Experimental results demonstrate that ENP outperforms the counterparts, e.g., 2% SR and 1% SPL gains over R2R [1] , 1.22% RGS on REVERIE [30] , 2% SR on R2R-CE [31] , and 1.07% NDTW on RxR-CE [32] , respectively. ENP not only helps release the power of existing VLN models, but also evidences the merits of energy-based approaches in the challenging VLN decision-making. 
related work: 
 Vision-Language Navigation (VLN). Early VLN agents [1, 13, 3, 33] are built upon a recurrent neural policy network within a sequence-to-sequence framework [34, 35] to predict action distribution.As compressing all history into a hidden vector might be sub-optimal for state tracking across extended trajectories, later attempts [36, 37, 6, 38] incorporate a memory buffer (e.g., topological graph) for long-term planning. Recent efforts [39, 5, 4] are devoted to encode complete episode history of navigation states and actions by transformer [17] and optimize the whole model in end-to-end training. Later, various strategies have been proposed to improve the generalization of policies in both seen and unseen environments [3, 8, 40] , such as progress estimation [41] , instruction generation [33, 42, 43, 44, 45] , backtracking [46] , cross-modal matching [47, 48] , self-supervised pre-training [49, 50, 51] , environmental augmentation [3, 52, 53, 54] , visual landmarks [55, 56] , exploration [57, 58] , map building [59, 60, 61, 62] , waypoint prediction [63] , and foundation models [64, 65] .Policy Learning in VLN. VLN can be viewed as a POMDP [2] , providing several expert demonstrations sourced from ground-truth shortest-path trajectories [1, 30] or human demonstrations [32] . Behavioural cloning (BC) [66, 67] , a typical approach of imitation learning (IL) [7, 15] , is widely used in current VLN agents [1, 33, 4, 6] with supervised policy learning. Nevertheless, it suffers from distribution shifts between training and testing [12, 68] . [1] introduces 'student-forcing' [11] training regime from sequence prediction to mitigate this limitation. Later agents [3, 8, 4, 5] combine both IL and model-free RL [14, 69] for policy learning, where the reward function is manually designed on the distance metric. Instead of directly mapping visual observations into actions or state-action values, [13, 70] investigate model-based RL [71, 72] for look-ahead planning, exploring the potential future states. As reward engineering requires careful manual tuning, [16] proposes to learn reward functions from the expert distribution directly. Although RL is an effective approach in principle, DUET [6] finds it difficult to train large-scale transformers with inaccurate and sparse RL rewards [73] in VLN. Existing studies [60, 61, 62] use an interactive demonstrator (similar to DAGGER [12] ) to generate additional pseudo demonstrations and achieve promising performance.In this paper, ENP learns to align with the expert policy by matching the joint state-action distribution for entire trajectories from the occupancy measure view. Furthermore, ENP reinterprets the optimization of policy into an energy-based model, representing an expressive probability distribution.Energy-based Model. Energy-based models (EBMs) [19] are non-normalized probabilistic models that represent data using a Boltzmann distribution. In EBMs, each sample is associated with an energy value, where high-density regions correspond to lower energy [74] . A range of MCMC-based [75, 21] and MCMC-free [76, 77] approaches can be adopted to estimate the gradient of the log-likelihood. EBMs have been widely utilized in generation [22] , perception [78] , and policy learning [79, 80] . The most related work [25, 24, 22] will be discussed below. [25] employs EBM for distribution matching, strictly prohibiting interaction with the environment. Moreover, this approach is applied in an offline setting, specifically for healthcare. [24] considers the expert energy as the reward function and leverages this reward for reinforcement learning. [22] adopts parameterized energy functions to model the conditional action distribution, optimizing for actions that minimize the energy landscape.In contrast, ENP devotes to learning the joint distribution of state-action pairs from the expert demonstrations online. An external marginal state memory is introduced to store the historical information for initializing the samples for next iterations. ENP demonstrates the potential of EBMs in VLN decision-making, and improves the existing VLN agents across various frameworks. 
method: 
 In this section, we first formalize the VLN task and discuss the limitations of existing work from a probabilistic view ( §3.1). Then we introduce Energy-based Navigation Policy learning framework -ENP ( §3.2). Furthermore, ENP is compared with other imitation learning methods based on divergence minimization ( §3.3). Finally, we provide the implementation details ( §3.4).VLN is typically formulated as a POMDP [2, 50] with a finite horizon T , defined as the tuple <S, A, T , R, ρ 0 >. S is the navigation state space. A is the action space. Note that we use panoramic action space [33] , which encodes high-level behaviour. The transition distribution T (s t+1 |s t , a t ) (Markov property) of the environment, the reward function R(s t , a t ), and the initial state distribution ρ 0 are unknown in VLN and T can only be queried through interaction with the environment. Lπ (Eq. 1) at action logit In the standard VLN setting [1] , a training dataset D = {(x, τ )} consists of pairs of the language instructions x and corresponding expert trajectories τ . At step t, the agent acquires an egocentric observation O t , and then takes an action a t ∈ A from K t candidates based on the instruction x. The agent needs to learn a navigation policy π * (a|s) and predict the action distribution P (a t |s t ).F θaRecent VLN agents employ Seq2Seq [8, 3] or transformer-based networks [4, 6] as a combination of a cross-modal state encoder F θs and a conditional action classifier F θa for action prediction. Hence they are usually built as a composition of F θs ⋄ F θa . There are different state representations in previous work [50, 4] , here we define a more general one. The cross-modal state encoder extracts a joint state representation s t = F θs (x, O t , H t ) ∈ R Kt×D by the visual embedding O t , the instruction embedding x, and history embedding H t along the trajectory τ . Concretely, H t comes from recurrent hidden units of Seq2Seq framework or an external memory module (e.g., topological graph [6] ). Then, the action classifier maps the state representation s t into an action-related vectora t = {a t [k]} Kt k=1 ∈ R Kt , i.e., a t = F θa (s t ), and a t [k] is termed as the logit for k-th action candidate. Then the probability of a t,k is computed by softmax, i.e., P (a t,k |s t ) =exp(at[k]) K k=1 exp(at[k]) .The cross-modal state encoder F θs and action classifier F θa usually employ a joint end-to-end training. Imitation learning (IL) aims to learn the policy from expert demonstrations D without any reward signals. Behaviour cloning (BC) [67] is a straightforward approach of IL by maximizing likelihood estimation of π * (a|s), i.e., minimizing the cross-entropy loss in a supervised way:EQUATIONwhere θ = θ s ∪ θ a . BC suffers from the covariate shift problem for the crucial i.i.d. assumption, as the execution of a t will affect the distribution of future state s t+1 [9] . Inspired by DAGGER [12] , some agents [1, 6] query the shortest paths (by expert policy π e ) during navigation, explore more trajectories {τ + } by interacting with the environments, and aggregate these trajectories asD + = {(x, τ )} ∪ {(x, τ + )}.Then, the policy is updated by mixing iterative learning [12] . However, these traditional approaches ignore the changes in distribution and train a policy that performs well under the distribution of states encountered by the expert. When the agent navigates in the unseen environment and receives different observations than expert demonstrations, it will lead to a compounding of errors, as they provide no way to understand the underlying reasons for the expert behaviour [27] .Instead of solely training the discriminative action classifier at each step, we want to learn a policy π * by exploiting the complete information in the state-action distribution from the expert policy π e . To facilitate later analysis, the occupancy measure ρ π (s, a) of the policy π is introduced [14] . ρ π (s, a) denotes the stationary distribution of state-action pairs (s, a) when an agent navigates with policy π in the environment, defined as:EQUATIONand ρ π (s, a) has a one-to-one relationship with policy π:ρ π = ρ π * ⇔ π = π * [15]. Therefore, we induce a policy π * with ρ * (s, a) that matches the expert ρ e (s, a) based on the joint distribution of state-action pairs P (s, a). t is omitted for brevity. The matching loss L ρ is designed as:EQUATIONThen we reformulate it from an energy view, as energy-based models (EBMs) [19, 78] parameterize an unnormalized data density and can represent a more expressive probability distribution. Specifically, EBMs are first introduced to model P (s, a) of the agent as a Boltzmann distribution:EQUATIONwhere E θ (s, a) : S ×A → R is the energy function, and Z θ is the partition function (normalizing constant). As it is difficult to optimize the joint distribution directly [78] , we factorize it as (Fig. 2 ):EQUATIONThe first term, log P (a|s), is typically from a discriminative classifier and can be optimized by minimizing the cross-entropy loss L π as in Eq. 1. As stated in [81, 82] , the marginal state distribution matching loss L S provides an explicit objective for navigation exploration and incorporates prior knowledge from the expert demonstrations. P (s) is computed by marginalizing out a as:EQUATIONIt is worth noting that log P (s) is the likelihood of an EBM, which cannot be computed directly due to the intractable partition function Z θ . A common approach is to estimate the gradient of the log-likelihood for likelihood maximization with gradient ascent [74] :EQUATIONConsidering the low energy value corresponds to the state-action pairs that the expert mostly perform, the action logits a t [k] of the agent can be used to describe such energy value as E θ (s t , a t,k ) ≜ -a t,k formally. Then, the first gradient term, -∇ θ E θ (s), is calculated by automatic differentiation in the deep network of the neural agent. Estimating ∇ θ log Z θ requires Markov chain Monte Carlo (MCMC) [83] sampling from the Boltzmann distribution within the inner loop of learning. Specifically, a sampler based on Stochastic Gradient Langevin Dynamics [23] (SGLD), a practical method of Langevin MCMC [84] ) can effectively draw samples as:EQUATIONwhere U denotes the uniform distribution, i is the sampling iteration with step size ϵ, ξ is Gaussian noise, and ŝi will converge to P θ (ŝ) when ϵ→0 and i→∞. While training on a new state, this MCMC chain will be reset. In practice, adopting standard MCMC sampling suffers from slow convergence and requires expensive computation. Recent persistent contrastive divergence approaches [75, 21] maintain a single MCMC chain, and use it to initialize a new chain for next sampling iteration [22] .Inspired by this, we introduce a Marginal State Memory M served as an online replay buffer [22] , collecting historical information of the MCMC chain during training (Fig. 2 ). To initialize the inner loop of SGLD, the state ŝ0 is sampled randomly from U ∪ M. This helps continue to refine the previous samples, further accelerating the convergence. After several iterations I, ŝI is stored in M:EQUATIONwhere I is fewer than the number of iterations required for MCMC convergence. The training details are illustrated in Algorithm 1.In VLN, ENP aims to learn a policy π * from expert demonstrations (π e ) without access to the explicit reward, which is analogous to the most common approaches of IL, e.g., BC and inverse reinforcement learning (IRL). For in-depth analysis, we explore the relation between ENP and other IL Methods, and provide a general understanding of them from a divergence viewpoint [28] .Standard BC is treated as a supervised learning problem (Eq. 1), and the policy is required to match the conditional distribution of π e under Kullback-Leibler (KL) divergence distance D KL as:EQUATIONwhere H(π e ) = E(π e (a|s)) is a constant, a.k.a. the causal entropy [85] . H(π e ) is considered as the expected number of options at each state for π e (a|s) [28, 15] . for SGLD iteration i = 1 to I do 8:ŝi+1 t = ŝi t -ϵ 2 2 ∇ ŝE θ (ŝ i t ) + ξ, ξ ∼ N , ŝI t → M▷ Defined in Eq. ( 9). 9:Lπ ← -log P θ (at|st), LS ← E θ (st) -E θ (ŝ I t )▷ Defined in Eq. ( 1) and ( 7). 10:Backpropagate ∇ θ Lρ = ∇ θ Lπ + ∇ θ LS. Update F θs and F θa 11: Return: Optimized Parameters θ = θs ∪ θa of F θs and F θa For ENP, the optimization objective in Eq. 3 can be written in a form of KL distance minimization: where H(ρ e (s, a)) is the entropy of the occupancy measure, which is a constant. IRL aims to infer the reward function of the expert, and learn a policy to optimize this reward. A representative work, AIRL [27] , adopts an adversarial reward learning, which is equivalent to matching ρ e (s, a) [15] as:EQUATIONHence, our objective is similar to the forward KL version of AIRL [28, 27] except for the entropy term. Both AIRL and ENP perform prioritized optimization for entire trajectories rather than the single-time step decisions in BC, so the compounding error can be mitigated [15, 27] . As D KL is a non-symmetric metric, the forward KL divergence results in distributions with a mode-covering behaviour, while using the reverse KL results in mode-seeking behaviour [18] . In addition, ENP directly models the joint distribution instead of recovering the reward function. We compare them in the experiment ( §4.3) and find ENP is more applicable to VLN (Table 8 ).Network Architecture. ENP is a general framework that can be built upon existing VLN models. We investigate ENP on several representative architectures, including: i) EnvDrop [3] adopts a Seq2Seq model with a bidirectional LSTM-RNN and an attentive LSTM-RNN. ii) VLN⟳BERT [4] is built upon multi-layer transformers with recurrent states. iii) DUET [6] is a dual-scale transformer-based model with a topological map for global action space, enabling long-term decision-making. iv) ETPNav [29] utilizes a transformer-based framework with an online topological map. For continuous environments, both VLN⟳BERT [4] and ETPNav [29] employ a waypoint predictor [63] . K t dimension of the state representations s t ∈ R Kt×D is different due to the local or global action spaces of these methods. In §4, we follow the same settings of these models, replace their policy learning as ENP, and maintain a Marginal State Memory M of 100 buffer-size.Training. For fairness, the hyper-parameters (e.g., batch size, optimizers, maximal iterations, learning rates) of these models are kept the original setup. Back translation is employed to train EnvDrop [3] . The parameters of VLN⟳BERT [4] is initialized from PREVALENT [50] . The pre-training and finetuning paradigm is used for DUET [6] and ETPNav [29] . For SGLD, ŝ0 is sampled from M with a probability of 95% (Eq. 9). In EnvDrop [3] and VLN⟳BERT [4] , the number of SGLD iterations is set as I = 15. I pre = 20 and I ft = 5 of the pre-training and finetuning stages are used for DUET [6] and ETPNav [29] . Although the step size ϵ and the Gaussian noise ξ in the original transition kernel of SGLD are related by Var(ξ) = ϵ [23] , most studies [22, 78] choose to set ϵ and ξ separately in practice. The step size ϵ = 1.5 and ξ ∼ N (0, 0.01) are set in ENP ( §4.3).Inference. During the testing phase, the agent uses the learned F θs for cross-modal state encoding, and predicts the next step via F θa , until it chooses [STOP] action or reaches the maximum step limit.Reproducibility. All experiments are conducted on a single NVIDIA 4090 GPU with 24GB memory in PyTorch. Testing is conducted on the same machine.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes]Answer: [No] Justification: As we promised, the code will be released upon the publication of our paper.Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment: 
 We examine the efficacy of ENP for VLN in discrete environments ( §4.1), and more challenging continuous environments ( §4.2). Then we provide diagnostic analysis on core model design ( §4.3).Datasets. R2R [1] contains 7, 189 shortest-path trajectories captured from 90 real-world buildingscale scenes [86] . It consists of 22K step-by-step navigation instructions. REVERIE [30] contains 21, 702 high-level instructions, which requires an agent to reach the correct location and localize a remote target object. All these datasets are devided into train, val seen, val unseen, and test unseen splits, which mainly focus on the generalization capability in unseen environments. Both R2R [1] and REVERIE [30] are built upon Matterport3D Simulator [1] .Evaluation Metrics. As in previous work [1, 30] [4] , and 2% on SR with DUET [6] . The performance is still lower than BEVBert [60] and BSG [61] due to the additional semantic maps. Moreover, we provide the average success rate across different path lengths in Appendix (Fig. 4 ).Performance on REVERIE. Table 2 presents the results of ENP with different transformer-based frameworks on REVERIE. ENP surpasses all the counterparts on both val unseen and test unseen by large margins, e.g., 1.34% on SR and 0.64% on RGS over VLN⟳BERT [4] , 0.68% on SR and 1.22% on RGS over DUET [6] , verifying the efficacy of ENP. Inspired by recent efforts [60] , potential improvement on object grounding can be achieved by introducing CLIP features [92] . Qualitative Results. In Fig. 3 (a), we illustrate the qualitative comparisons of ENP against DUET [6] on R2R val unseen. Based on ENP, our agent yields more accurate predictions than DUET. It verifies that ENP leads to better decision-making when facing challenging scenes. In Fig. 3 (b), our agent may fail due to the serious occlusion, and introducing map prediction [94] may alleviate this problem. Datasets. R2R-CE [31] and RxR-CE [32] are more practical yet challenging, where the agent can navigate to any unobstructed point instead of teleporting between fixed nodes of a pre-defined navigation graph in R2R. R2R-CE comprises a total of 5, 611 shortest-path trajectories with significantly longer time horizons than R2R. RxR-CE presents more instructions with longer annotated paths, including multilingual descriptions. In addition, the agents in RxR-CE are forbidden to slide along obstacles. Both of are R2R-CE and RxR-CE driven by Habitat Simulator [95] .Evaluation Metrics. The metrics of R2R-CE [31] are similar to R2R [1] . For RxR-CE [32] , Normalize Dynamic Time Wrapping (NDTW) and NDTW penalized by SR (SDTW) are further used to evaluate the fidelity between the predicted and annotated paths.Performance on R2R-CE. As shown in Table 3 , ENP provides a considerable performance gain against VLN⟳BERT [4] , e.g., 1% SR and 1% SPL on val unseen. Concretely, it outperforms the top-leading ETPNav [29] by 1% and 1% in terms of OSR and SR on test unseen, respectively. These quantitative results substantiate our motivation of empowering VLN agents with ENP. [29] on val unseen. We attribute this to the global expert policy matching, which promotes the path fidelity. This indicates ENP enables the agent to make long-term plans, resulting in better NDTW and SDTW. Joint Distribution Matching. We first investigate the joint distribution matching loss L ρ (Eq. 3) in ENP, which is factored as the discriminative action loss L π (Eq. 1) and the marginal state distribution matching loss L S (Eq. 7). Both of L π and L S are online optimized iteratively. A clear performance drop is observed in Table 5 , e.g., SR from 74% to 72% for DUET [6] on R2R, as only optimizing the conditional action distribution with L π will degenerate into BC (or DAGGER) with cross-entropy loss. This reveals the appealing effectiveness of the joint distribution matching strategy in ENP. Step Size and Guassian Noise in SGLD. In ENP, SGLD [23] is used to draw samples from Boltzmann distribution (Eq. 9) and further optimize L S . Following recent EBM training, we relax the restriction on the relation between the step size ϵ and Gaussian noise ξ, i.e., Var(ξ) = ϵ.Table 6 shows the results with different step sizes (when I = 15), and changing ϵ in a small range has little effects on the performance. In addition, different noise variances Var(ξ) are also explored. Our ENP is insensitive to parameter changes. We find that ϵ = 1.5 and ξ ∼ N (0, 0.01) for the transition kernel of SGLD works well across a variety of datasets and frameworks in VLN. In Table 7 , we can find that I = 15 iterations is enough to sample the recurrent states for En-vDrop [3] and VLN⟳BERT [4] . Slightly more iterations are required for the models [6, 29] with additional topological graph memory due to the global action space. This can be mitigated by pre-storing the samples in the Marginal State Memory M during pre-training. As these frameworks [6, 29] employ pre-training and finetuning paradigm for VLN, we design a different number of iterations, i.e., I pre = 20 and I ft = 5. It is worth noting that the state memory M pre can acquire the samples while performing single-step action prediction in the VLN pre-training. Thus, the SGLD iterations in finetuning based on M ft ←M pre can be further reduced. ENP vs. AIRL. In Table 8 , we compare ENP against AIRL [27] , which is a representative work of adversarial imitation learning. The objective function for ENP is equivalent to the forward KL divergence minimization, where AIRL [27] makes use of the backward KL divergence. By modifying the original policy learning in VLN⟳BERT [4] (a mixture of RL and IL) as an adversarial reward learning formulation, AIRL achieves the comparable performance. ENP yields better scores on R2R-CE with 45% SR and 40% SPL. Meanwhile, ENP avoids the potential influence from the adversarial training.Runtime Analysis. The additional computational complexity of ENP is from the iterations of SGLD inner loop. Towards this, we make some specific designs, such as Marginal State Memory M and pre-storing in pre-training. During inference, there is no additional computational consumption in executing the strategy compared to existing VLN agents. 
conclusion: 
 In this paper, we propose ENP, an Energy-based Navigation Policy learning framework. By explicitly modeling the joint state-action distribution using an energy-based model, ENP shows promise to solve the intrinsic limitations of supervised behavioural cloning. Through maximizing the likelihood of the action prediction and modeling the dynamics of the navigation states jointly, ENP learns to minimize the divergence distance from the expert policy in a collaborative manner. Experimental results on gold-standard VLN datasets in both discrete (R2R and REVERIE) and continuous environments (R2R-CE and RxR-CE) demonstrate the superiority against existing methods. In the future, we will explore more tasks and frameworks with the energy-based policy. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: In Table 1 , 2, 3, and 4, we report error bars on VLN benchmarks.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We acknowledge the NeurIPS Code of Ethics and obey them in our paper.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: In Appendix A.3, we discuss the broader impacts of our work.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper is based on publicly available datasets and models.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: In Appendix A.3, we provide the details of the license and terms of use. We also use existing models as components and cite appropriately.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: A.1 Model Details EnvDrop [3] adopts an encoder-decoder model. The encoder is a bidirectional LSTM-RNN with an embedding layer. The decoder is an attentive LSTM-RNN. EnvDrop uses the 2048-dimensional image features from ResNet-152 pretrained on ImageNet. The model is first trained with supervised learning via the mixture of imitation and reinforcement learning. Then, it is finetuned by back translation with environmental dropout. The word embedding size is 256 and the dimension of the action embedding is 64. The size of the LSTM units is set to 512 while 256 is set for the bidirectional LSTM. RMSprop [101] is employed to optimize the loss with a learning rate 1×10 -4 and the batch size is set as 64.VLN⟳BERT [4] is built upon LXMERT-like transformer [102] . It is trained directly on the mixture of the original training data and the augmented data from PREVALENT [50] for R2R [1] . AdamW optimizer [103] is used and the batch size is set to 16. For REVERIE [30] , the image features are extracted by ResNet-152 pre-trained on Places365 [47] , and the object features are encoded by Faster-RCNN pre-trained on Visual Genome [104] .The batch size is set as 8.VLN⟳BERT for CE [63] proposes a waypoint predictor to generate a set of candidate waypoints during navigation. The training objective is the cross-entropy loss between the ground-truth and action predictions. It is minimized by AdamW optimizer [103] with the learning rate 1×10 -5 during training. The decay ratio is set as 0.50 for R2R-CE [31] and 0.75 for RxR-CE [32] .DUET [6] uses a dual-scale graph transformer initialized from LXMERT [102] . The hidden layer size is 768. The image features are extracted by ViT-B/16 [105] pretrained on ImageNet [106] .On REVERIE [30] , DUET is first pre-trained with the batch size of 32. Some auxiliary tasks are employed, such as single-step action prediction, object grounding, masked language modeling, and masked region classification [5] . Then it is finetuned with the batch size of 8.ETPNav [29] contains a topological mapping module and a waypoint prediction module [63] . It is pre-trained using proxy tasks following existing transformer-based VLN models [4, 6] with a batch size of 64 and a learning rate of 5×10 -5 . During finetuning, the AdamW optimizer [103] is utilized. The batch size is set as 16 with a learning rate of 1×10 -5 . The decay ratio is 0.75.In Fig. 4 , ENP demonstrates notable performance improvements over BC [6] across both seen and unseen splits of R2R [1] . When the length of ground-truth paths exceeds 16 meters, our ENP outperforms BC by a significant margin in both the val seen and val unseen splits. In addition, the performance gains are relatively more significant in the unseen split compared to the seen split. For example, ENP achieves approximately 5% improvement of SR on the val seen and 6% improvement on the val unseen for navigation paths longer than 16 meters. Terms of use, Privacy, and License. R2R [1] , REVERIE [30] , R2R-CE [31] , and RxR-CE [32] datasets utilized in our work contain a large number of indoor photos from Matterport3D [86] ,which are freely available for non-commercial research purposes. No identifiable individuals are present in any of the photographs. All experiments are conducted on the Matterport3D and Habitat Simulators [1, 95] , which are released under the MIT license.Limitations. Existing studies in VLN mainly focus on improving the success rate of navigation, with limited attention given to collision avoidance and navigation safety. In addition, current agents are typically designed for static environments within simulators. To mitigate the risk of collisions, it is essential for robots to operate under specific security constraints. Therefore, further advancements are necessary to ensure safe deployment in real-world dynamic environments.Future Direction. i) We explore ENP on several representative VLN architectures [3, 4, 6, 29] , and the generalization of ENP to other architectures [62, 60] and more tasks [107] is not clear. We will verify it in future work. ii) Efficiently sampling from un-normalized distributions for high-dimensional data is a fundamental challenge in machine learning. Utilizing MCMC-free approaches [74, 108] and neural implicit samplers [109] may offer valuable insights to enhance our approach. One limitation of ENP is that the optimization through the SGLD inner loop at each training iteration (Algorithm 1), which would reduce the training efficiency (about 9% delay for 5 SGLD iterations). In practice, ENP balances performance and training time with a limited number of iterations. iii) In the future, agents should be able to actively acquire the skills by watching videos similar to humans [51] , thereby reducing high training data (expert demonstrations) requirements.Broader Impacts. We introduce an energy-based policy learning for VLN. Equipped with ENP, existing agents are able to perform comprehensive decision-making, showcasing promising improvements across various VLN benchmarks. In real-world deployment, the agent holds potential to significantly benefit society by providing assistance to individuals with specific needs, e.g., blind and disabled, enabling them to accomplish daily tasks more independently. In addition, we encourage more efforts devoted to policy learning for future research in the community.NeurIPS Paper Checklist Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.These approaches require a slightly more interactive setting than traditional imitation learning, allowing the learner to query the expert at any navigation state during training. Previous studies[1,6] have adopted this setting, and this assume is feasible for many real-world imitation learning tasks[12]. 


Decision: This paper tackles vision-language navigation through a perspective that differs from the traditional behavioral cloning methods. Specifically, the authors propose an energy-based method for joint optimization of the feature-action space. Theoretical motivation for this approach is discussed, and strong results are shown across a number of VLN settings. 

  Overall, the reviewers appreciated the strong novel contribution in terms of both theoretical foundations and method which differs from traditional approaches, as well as the empirical results and ablations. A number of joint concerns were raised by a number of reviewers such as the lack of validation on larger-scale VLN settings as well as models, as well as success/failure analysis to show exactly where/how this method improves performance. Other weaknesses such as an apparent sensitivity across the hyper-parameters were mentioned. The authors provided a strong rebuttal with new results on larger-scale settings (ScaleVLN) and models (NaviLLM with a Vicuna-7B model). Importantly, the original hyper-parameters were used, showing both good scaling as well as some hyper-parameter insensitivity. Reviewers were overall satisfied with raised scores based on the rebuttal. 

  After considering all of the materials for this paper, I recommend acceptance. It provides an interesting novel approach that is not typical in the community, with strong results, potentially paving the way for new exploration in this area. Further, the method seems applicable to other IL problems where BC is common, and I hope that this method can be explored within those contexts and settings as well. I highly encourage the authors to include the rebuttal results and discussions in the main paper, as I believe it significantly strengthens the paper's claims and impact.

Reviews: review: paper_summary: paper_summary: This paper proposes to learn a better Vision-and-Language Navigation agent by jointly optimizing the action and the state. Specifically, they propose an energy-based expert policy, where they adds an explicit objective for navigation exploration and incorporates prior knowledge from the expert demonstrations. Empirical results on multiple VLN benchmarks (e.g., R2R, REVERIE, R2R-CE) demonstrates the effectiveness of the proposed approach.
strength_weakness: strength_weakness: 1. The proposed approaches work pretty well on sota agents like DUET, but it's underinvestigated whether the proposed approach also benefits VLN agents when having large-scale data. Specifically, it's important to have experiments on ScaleVLN or HM3D-AutoVLN.
2. The approach might be sensitive to hyperparameters as suggested in Table 6 and Table 7, which might need additional efforts in hyperparameter tunning when generalized to different environments/agents/tasks.
questions: questions: Please refer to weakness.
limitations: limitations: N/A
paper_summary: paper_summary: This paper studies vision-and-language navigation (VLN) by addressing the problems of error accumulation in the Markov decision process and the difficulties in designing an optimal reward function, respectively, in the commonly applied behavioral cloning (BC) and reinforcement learning (RL) approaches. It proposes an energy-based policy (ENP) for training the agent by estimating the unnormalized probability density of the expert’s occupancy measure, optimizing the entire trajectories rather than the single-time step decisions in BC. Meanwhile, to facilitate the MCMC sampling in ENP, a Marginal State Memory is introduced for the agent as an online reply buffer. The proposed method has been evaluated on multiple popular VLN agents and several benchmarking datasets and environments, together with detailed analyses of the ENP and design choices.
strength_weakness: strength_weakness: 1.	The proposed ENP has only been evaluated on a limited scale of VLN datasets and models. It is unclear if the method will still benefit larger models, or the case where more sufficient training data is available to learn a very generalizable policy through IL/BC (e.g., 4.9M ScaleVLN data [49] vs. the original 14K R2R data). I am concerned that IL/BC is much more friendly and stable than ENP to scale with data.
2.	There is no comprehensive successful/failure case analysis of ENP versus existing training methods. It is unclear exactly in what scenarios an agent trained with ENP shows an advantage.

I need to mention that I have very little knowledge about energy-based models, so I might have missed some key details in mathematics. I will rely more (and probably adjust my review based) on the comments from the other reviewers on the theoretical part of this paper.
questions: questions: No other questions; I wish the authors could kindly respond to my concerns in Weaknesses.
limitations: limitations: The authors have discussed some limitations of the proposed method and its potential societal impact. As for suggestions, mainly the two points in Weaknesses – test on large-scale training data and run thorough result analysis.
paper_summary: paper_summary: To attempt to solve the catastrophic failure issues that arise from behavioral cloning, the authors introduce Energy-based Navigation Policy (Enp) which uses an energy-based model to learn the expert policy from demonstrations. The authors claim that by doing so Enp prioritizes learning the full trajectory instead of single-time step decisions in BC.
strength_weakness: strength_weakness: **W1.** *Overall improvements are low.* The performance improvements provided by ENP are small compared to the BC baselines, often around $1-2\%$ in terms of SR and similar in SPL. This is particularly poignant when compared to the improvements provided by semantic grid maps. By itself, this would not necessarily be a major issue, if it was clear where ENP was helping (W2).

**W2.** *It is unclear when ENP is better than BC, and if the intuitive justification of catastrophic failure applies.* A more in-depth analysis of the success/failure cases of using ENP vs BC is warranted. Metrics like SR suggest that ENP can be helpful in avoiding the failure that BC can induce, but it is not entirely clear this is the case — when is ENP helping? Given the justification of the catastrophic failure of BC, you would expect this would be in longer trajectories, but the example in Figure 3a does not justify that given its short trajectory length. Could you create a benchmark specifically to test this? Including this analysis could boost the case for the motivation behind ENP significantly.

**W3.** *ENP is likely more expensive to train than BC.* The authors should include a compute comparison for the training of ENP vs BC. While the inference runtime is the same, training time for ENP will likely be significantly influenced by $I$.

The combination of W1 and W2 severely hinder the practical motivation of the method.

Minor comments:

- It’s not really required to include all baselines in the main results tables. E.g., Seq2Seq, SF and RCM are pretty outdated at this point and always beaten by other methods — it’s better to focus the comparison with the most recent methods and leave the full tables to the appendix.
questions: questions: **Q1.** Why are the DUET or, e.g., BEVBert/GridMM, paths not shown in Figure 3b too? If multiple agents face the same issue, that would show this is the problem of observability.

**Q2.** What are the $\pm$ values in Table 3, is it the standard deviation? If so, why is it not reported for the other methods?

**Q3.** Do the authors have any intuition on why increasing $I_{\text{pre}}$ and $I_{\text{ft}}$ could result in a lower SR/SPL (Table 7)?
limitations: limitations: The authors do not discuss the limitations of their method. The limitations paragraph in Appendix A.2. does not reflect the particular ENP method they propose and is instead generic to the problem. Training time is one potential limitation that should be studied and discussed in further detail.

Comments: title: Final Rating
Final Rating
comment: Dear Reviewer UeCa,

Thank you again for your positive feedback and for considering our rebuttal. We will include these experimental results in the final version of our paper. Please do not hesitate to post comments if there are any further questions we can address.

Thank you,\
Authors
Dear Reviewer dJ3y,

We greatly appreciate your valuable time and constructive feedback. We will incorporate your suggestions into the revised version of the paper. If you have any further questions or suggestions, please feel free to share them.

Thanks again,\
Authors
Thanks for addressing my concerns. The additional experiments are very strong. I will raise my rating to weak accept (6).
I thank the authors for their clear and detailed rebuttal. I believe the extra results provided make the paper much stronger than the original submission, particularly the ones on the success rate as a function of path length for ENP compared to BC and the training time table + discussion. It is key that these are incorporated in the paper to provide a full picture of the method's advantages and drawbacks. I will raise my original score.
Dear Reviewer Ngjf,

Thank you very much for your valuable time and thoughtful review. We are pleased to hear that our rebuttal addressed your concerns, prompting a rise in the rating. We will revise our paper according to your comments and incorporate all suggestions into the final version.

Thanks again,\
Authors
Thanks to the authors for providing a clear and solid response to my concerns. I am happy with the additional results that justify the data and model scalability and successful/failure cases (questions raised by more than one reviewer), given the limited rebuttal time. I decided to raise the rating from Weak Accept (6) to Accept (7). I sincerely hope that the authors can further extend and incorporate all responses into the final version of their paper.