Test-Time Adaptation for Online Vision-Language Navigation with Feedback-based Reinforcement Learning
Authors: Sungjune Kim
Gyeongrok Oh
Heeju Ko
Daehyun Ji
Dongwook Lee
Byung-Jun Lee
Sujin Jang
Sangpil Kim
Venue: ICML 2025 poster (2025)
Keywords: vision-language navigation
test-time adaptation
Primary area: applications->robotics
TL;DR: We introduce a novel test-time adaptation framework for online vision-language navigation using feedback-based reinforcement learning.

Abstract: Navigating in an unfamiliar environment during deployment poses a critical challenge for a vision-language navigation (VLN) agent. Yet, test-time adaptation (TTA) remains relatively underexplored in robotic navigation, leading us to the fundamental question: what are the key properties of TTA for online VLN? In our view, effective adaptation requires three qualities: 1) flexibility in handling different navigation outcomes, 2) interactivity with external environment, and 3) maintaining a harmony between plasticity and stability. To address this, we introduce FeedTTA, a novel TTA framework for online VLN utilizing feedback-based reinforcement learning. Specifically, FeedTTA learns by maximizing binary episodic feedback, a practical setup in which the agent receives a binary scalar after each episode that indicates the success or failure of the navigation. Additionally, we propose a gradient regularization technique that leverages the binary structure of FeedTTA to achieve a balance between plasticity and stability during adaptation. Our extensive experiments on challenging VLN benchmarks demonstrate the superior adaptability of FeedTTA, even outperforming the state-of-the-art offline training methods in REVERIE benchmark with a single stream of learning.

Test-Time Adaptation for Online Vision-Language Navigation with Feedback-based Reinforcement Learning
Sung June Kim 1
Gyeongrok Oh 1 Heeju Ko
Daehyun Ji Sangpil Kim
Dongwook Lee²
Byung-Jun Lee"
Sujin Jang?
Abstract
Online Dala
Navigating in unfamiliar environment dur- ing deployment poses critical challenge for a vision-language navigation (VLN) agent. Yet, test-time adaptation (TTA) remains relatively un- derexplored in robotic navigation, leading us to the fundamental question: what are the key prop- erties of TTA for online VIN?? In our view, effec- tive adaptation requires three qualities: 1) flexi- bility in handling different navigation outcomes, 2) interactivity with external environment, and 3) maintaining a harmony between plasticity and stability. To address this, we introduce FEEDTTA, novel TTA framework for online VLN utilizing feedback-based reinforcement learning. Specif- ically, FEEDTTA learns by maximizing binary episodic feedback, practical setup in which the agent receives a binary scalar after each episode that indicates the success or failure of the naviga- tion. Additionally, we propose a gradient reg- ularization technique that leverages the binary structure of FEEDTTA achieve balance be- tween plasticity and stability during adaptation. Our extensive experiments on challenging VI.N benchmarks demonstrate the superior adaptabil- ity of FEEDTTA, even outperforming the state- of-the-arr art offline training methods in REVERIE benchmark with a single stream of learning.
"o to the dlining rion on level and pull out the white chair cleames to the Living rocm."
Correct
Wrong
Binary Epicodic Faadback
Environment
Figure I. Illustration of the learning paradigm of FEEDTTA. The navigation agent adapts to streaming online test data by learning to maximize the cumulative binary episodic feedback, which in- dicates navigation success or failure. The simplicity and efficacy of the FeedTTA framework demonstrate its potential for practical applications in real-world navigation scenarios.
via imitation learning vast collection of annotated ex- pert demonstrations, aiming to translate human behavior into generalized robotic actions (Hao et al., 2020; Chen et al., 2022c). However, it is inevitable for a trained policy to encounter unseen environments during online deploy- ment, leading to compromised reliability. Therefore, the ability of instantly adapting test-time environment and performing beyond its trained capabilities, i.e., test-time adaptation (TTA), is crucial in real-world robot navigation.
Despite its potential benefits, the application of TTA on on- line robotic navigation remains underexplored. One existing approach (Gao et al., 2024a) relies on the widely adopted TTA paradigm of entropy minimization ( (Wang et al., 2020a; Zhang et al., 2022), where we identify several limitations of its usage navigational policies. First, entropy minimiza- tion reduces policy resilience on failed trials. That is, the adaptations derived from failed samples attempt to improve general predictive accuracy, but induce overfitting on similar failure patterns. For example, when the initial navigation fails, entropy minimization intensifies the probabilities of the actions that lead to failure in repeated episodes. Second, entropy minimization limits exploration. The sequential Dlecision-makng nature of VL.N requires careful balance between exploitation and exploration. By prioritizing en- tropy minimization, the approach overly focuses on exploit- ing existing knowledge while neglecting the opportunity to learn from new, unfamiliar scenarios.
1. Introduction
Vision- Vision-Language Navigation (VLN) is a fundamental task of connecting human interactions with robotic Al systems (Wu et al., 2024). The navigation policies are typically trained
'Equal contribution 'Department of AI, Korea University, Seoul, K.rea 'Samsung Al Center, DS Division, Suwon, S. K.reaa. Correspondence to: Sujin Jang <s.steve samsung.com>> Sangpil Kim <spk7 ac....
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

TTA for Online VLN with Feedback- Reinforcement Learning
This naturally leads us to address the important research question: What are the key properties of TTA for online VLN? Our analysis focuses on the following perspectives:
gradient updates, ensuring the policy retains essential prior knowledge and avoids catastrophic forgetting.
We empirically demonstrate the effectiveness of the proposed method through extensive experiments on REVERIE (Qi et al., 2020), R2R (Anderson et al., 2018), and R2R-CE (Krantz et al., 2020) benchmark. FEEDTTA successfully overeomes test-time distribution shifts, show- ing substantial performance gains in classical evaluation protocol of VLN. However, existing metrics primarily focus on calculating overall averages across test-time samples, making them inadequate for analyzing sample-wise adapt- ability. Hence, we propose adaptive success rate (ASR), which measures the sample-wise transition of results before and after adaptation. The results confirm that FEEDTTA also outperforms the compared baselines in ASR, showcas- ing its advanced adaptability to test-time distribution shifts and enhanced resilience in online VI.N.
Flexibility. The adaptation should be made in response to the navigation outcome. This ensures that the policy can adjust dynamically to different outcomes without overfit- ting to specific failure patterns.
Interactivity. The adaptation should be able to incor- porate external signal from the end user, enabling more natural and prompt adaptation to unforeseen situations by learning human-like -ike behavior.
Plasticity & Stability. The adaptation should be versa- tile in learning new information, while preventing catas- trophic forgetting of previously acquired knowledge.
Based on this analysis, we introduce FEEDTTA, a novel TTA framework for online VLN using feedback-based rein- forcement learning (RL). Specifically, this work studies highly practical setting of binary episodic feedback, where after each episode, the oracle provides the agent with a binary scalar of +1 or -1, indicating whether the given in- struction was successfully completed or not. The agent adapts to new environment by attempting to maximize the cumulative feedback throughout iterations. Suppose you are an end user of a trained navigation agent. The agent carries out an instruction, and determines to stop at a certain point, presuming it is the desired destination. Now, you simply inform the agent whether it is correct or wrong, which is a rather trivial and inexpensive interaction. In an occasional case where human feedback is unavailable, it is also feasible for an agent to inquire AI systems (i.e, Large Language Models (Achiam et al., 2023; Liu et al., 2024a)) for judg- ment. Regardless of the feedback oracle, we show that the policy adaptation is possible, even with small amount of streaming test data.
In summary, the contributions of this work are as follows.
We introduce FEEDTTA, novel TTA framework for on- line VLN utilizing feedback-based RL. FEEDTTA learns from user feedback the end of each test-time episode (in- teractivity), where feedback is given depending on the navigation outcome (flexibility..
We propose SGR as a gradient regularization technique to mitigate non-stationary learning, thereby enhancing both plasticity and stability of FEEDTTA.
Experiments on challenging VLN benchmarks demon- strate the superiority of FEEDTTA not only in classical metrics, but also in our proposed sample-wise metric ASR. Furthermore, FEEDTTA even outperforms the state-of- the-art offline training methods in REVERIE benchmark.
2. Related Works
While the feedback offers clear and explicit guidance for adaptation in unfamiliar environment, the binary nature of stationnton into
2.1. Vision-Language Navigation
the feedback system may introduce the adaptation process, leading to loss of plasticity (Dohare et al., 2024). For example, unlike conventional optimization signals, FEEDTTA estimates gradients at two distinct ex- tremes (i.e., +1 for success and 11 for failure). We exploit this property and develop a gradient regularization technique named stochastic gradient reversion (SGR) to alleviate po- tential stasstationaity. First, for each episode, SGR ran- domly selects a subset of parameters to apply regularization. Then, SGR modifies the direction of the estimated gradients by reversing the derivatives of the score function w.r.t the selected parameters. Incorporating this counterfactual rea- soning results in a smoother gradient distribution throughout the learning process, thereby improving plasticity. Further- more this enhances stability by regulating abrupt shifts in
The goal of Vision-Language Navigation (VLN) is to fol- low natural language instructions to reach at a designated position by using visual cues from the camera sensors (Wu et al., 2024; Gu et al., 2022). In terms of model architec- ture, carly works focuses on modeling the sequential action prediction nature of VLN with recurrent neural network (An- derson et al., 2018; Fried et al., 2018). Later, multimodal pre-training with transformers (Vaswani, 2017) emerges as mainstream learning paradigm (Hao al., 2020; Hong et al., 2021; Li et al., 2019), enabling fast optimization of the policy for multiple downstream navigation tasks. Re- garding model learning strategy, imitation learning is most widely adopted to translate expert behavior into robotic ac- tion (Chen et al., 2022c; Pashevich et al., 2021; An et al., 2023; Liu et al., 2024d). Many works also incorporate
2

---

TTA for Online VLN with Feedback- Reinforcement Learning
reinforcement learning to refine the policy beyond super- vised trajectories (Chen al., 2021; Tan et al., 2019; Wang et al., 2020b; Chen et al., 2022a). With the advent of Large Language Models (LLMs), most recent works (Zhou et al., 2024; 2025; Long et al., 2024; Yu et al., 2023; Zheng et al., 2024) utilize human-like reasoning capabilities of LLMs to accomplish the navigational task. Despite these attempts, online VLN agents remain vulnerable when facing environ- ments beyond the training set, as existing approaches rely on offline learning strategies.
3. Method
3.1. Task Description
Suppose we have a pre-trained VLN policy xe. parameter- ized by A. At test time, x0 is exposed N continuously streaming test data X' [X1, .... Xy Each element X, consists of natural language instruction I, and an initial visual state s% which is a 360 panoramic view of the surrounding environment. To accomplish the given in- struction, the agent starts from 8 and predicts next action at each time step using To, until it decides to stop. This pro- duces a trajectory T = (8,, as )=)(=0 where a, is the selected action at step and T is the total number of steps taken by the agent.
2.2. Test-time Adaptation
Test-time adaptation (TTA) has emerged as a practical so- lution for handling distribution shifts, by directly adapting a pre-trained model to unlabeled stream of test data (Liang et al., 2024). One research direction focuses on adjusting the pre-trained normalization statistics with the estimated ones from the test batch (Nado et al., 2020; Schneider et al., 2020; You al., 2021: Zhao et al., 2023a). Entropy minimization is also extensively studied, aiming to reduce prediction certainty in test domain (Wang et al., 2020a; Zhang et al., 2022; Niu et al., 2022; Gao et al., 2024b). Recently, adap- tation in continuously evolving environments is explored to address real-world challenges (Liu et al., 2024c; Wang et al., 2022; Liu et al., 2024b). Despite its practical neces- sity, TTA is still in the early stages of research within the VLN field. FSTTA (Gao al., 2024a) initiated the study by applying entropy minimization while considering the episodic structure of VIN. However, we observe several limitations (see Section 1) and develop flexible, interac- tive, and well-balanced TTA framework for online VLN by leveraging feedback-based reinforcement learning.
3.2. Binary Episodic Feedback
Feedback Mechanism. We assume the presence of an oracle O (e.g. human AT system) test time to assess real-time navigation result. Once the agent determines to stop, the oracle provides the agent with a binary feedback F, where +1 is given if the predicted trajectory T successfully followed the given instruction 7, and -1 otherwise. Formally, we consider as a function of r and X, which formulates the feedback mechanism as:
(1 lr F= O(,
(1)
if
Unlike step-wise feedback which requires tracking through- out the whole episode, is trivial to simply evaluate whether the complete trajectory was success or failure at the end of the episode, making it highly practical and feasible for online environment. In this study, we focus on the most practical setting of binary feedback, leaving the exploration of more advanced feedback systems for future research.
2.3. Feedback-basee Reinforcement Learning
Reinforcement Learning with Human Feedback (RLHF) (Ouyang et al., 2022) and its variants (e.g., RLAIF (Lee et al., 2023a) and DPO (Rafailov et al., 2024)) burst onto the field of the Large Language Model, integrating human preference into output generation (Achiam et al., 2023; Tou- vron et al., 2023; Liu et al., 2024a). Inspired by this suc- cess, many works integrate the feedback system into vari- ous downstream tasks (Lee et al., 2023b; Pinto et al., 2023; Black et al., 2023). In the context of TTA, RLCF (Zhao et al., 2023b) utilizes CLIP (Radford et al., 2021) feedback for improving the zero-shot generalization capacity of vision- language models. Similar to our work, DFA (Peng et al., 2023) uses human feedback for adapting control policy, but requires multiple steps to generate counterfactual demon- strations, which is an infeasible setup in online Instead, we consider a binary episodic feedback, which is a highly practical interaction with the external environment, making it suitable for online navigation.
Feedback-based Policy Gradient. FEEDTTA lever- ages a Monte Carlo policy gradient algorithm REIN- FORCE (Williams, 1992) to learn from the received feed- back at the end of each navigation episode. A general RE- INFORCE algorithm aims at optimizing the parameter 0 of policy to maximize the score function of the expected return G, = E 7d R where R is the reward and y is the discount factor. In FEEDTTA, the rewards are as- signed as for < 1, and a binary episodic feedback F for T 1, giving us the score function as:
navigation.
T-1 Σ t=
((0) = Ey
G.
E ETTWTY
Ext--1
(2)
Then, according to the policy gradient theorem, the approxi-
3

---

TTA for Online VLN with Feedback-basee Reinforcement Learning
Feedback based
Counterfactual Gradient Dittibution
Algorithm 1 Online Learning Process of FEEDTTA
Gradiet Button
Require: Pre-trained VLN policy тe. Online data stream
Coeine Smitrty
X' {X1, ...,, }. Feedback oracle O:
1: 01 initialized policy parameters, N data count,
T. : steps count, 77 learning rate, state, a action;
Gradiant Distibution
J-tmitrtty betfon retultatrationn
2: for each step n E (.. do
Gradient Distribution
900
1000
1200
Iterations
2200
2000
2002
3: Follow instruction In E Xn and generate trajectory:
T E (st, ae) ~ mp:
Figure 2. Conceptual Illustration of SGR. (Leff) By reversing the gradients, SGR reduces the distribution gap between the two extreme cases that may cause non stationaar learning. (Right) Specifically, among the variants of a, the negative value (reversion) shifts the original gradient closest to the counterfactual distribution.
Binary Episodic Feedback:
4: Receive binary feedback = O(T, Xn)) (Eq.1);
Define a score function J (TTp) to maximize (Eq.2);
Approximate policy gradient ( w.r.t 8 (Eq.3); // Stochastic Gradient Reversion:
mated gradient of the policy Te is:
7: Sample reversion candidate G S VoJ(O) (Eq,4);
8: Reverse ge... E E and obtain (.((T))' (Eq.5);
9: Update parameter 9 through gradient ascent (Eq.6):
(e. = Bar
Σ Ve mola si) s1) -
0x+1 f- 02 + ((a))
10: end for
(3)
output Policy parameter H" adapted to X
where (a(ac st) is the probability of taking action at in state 8t under the policy parameterized by 8. Here, the pa- rameter update directly depends on the navigation outcome F and the log probability for each selected action, imply- ing that the policy flexibly adopts different strategies for different results.
each dimension of the parameter space:
VoJ(O) {
aJ(0)
0J(0)) )
30 ] [,00 ... ... 90
00,
Analysis 3.1: LLMs as Oracle. Although episodic feed- back is an inexpensive interaction, human involvement may not always be possible in real-world enviromments. In such cases, the agent can leverage the commonsense reason- ing capability of LLMs (Achiam et al., 2023; Liu et al., 2024a) for judgment. In this research, we utilize the GPT-4 model (Achiam et al., 2023) as the LLM oracle. We observe that while LLMs can also provide beneficial feedback for adaptation with reduced burden of human labeling, their reliability remains a concern, requiring a careful prompting for accurate judgments. Please refer to Appendix A for prompting details and Exp. 5.3 for our empirical analysis on using LLMs as feedback oracle.
where M denotes the number of dimensions that forms the parameter space. First, SGR randomly samples subset of dimensions from (o-( where the elements drawn from Bernoulli distribution with probability p:
g = (90.. bm = 1, ~ Bernoull(y)) S (O),
(4)
where is the Bernoulli random variable. Then, SGR re- verses the elements of G by multiplying a negative coeffi- cient, modifying the gradients as:
VoJ(ØY = =(6b J.M
aggem. :
if gam. E G ap+(1-p) 90m if gom.. $ g
(5)
3.3. Stochastic Gradient Reversion
where a 0 is the reversion magnitude. The 1 p seg- ments of the derivatives gem. & G are proportionally scaled to preserve consistency in the expected magnitude (i.e. E Ls ge..: Vm E {1..... M)). We provide the deriva- tion in Appendix B. 1. Utilizing the modified gradient, the parameter update at the th iteration becomes:
The binary feedback provides the agent with straightfor- ward direction for achieving navigation success in unfamil- iar test-time environments. However, the estimated gradi- ents from the binary signals point toward extreme ends in the parameter space, potentially causing non-stationarity. There- fore, we propose Stochastic Gradient Reversion (SGR), a gradient regularization method for FEEDTTA maintain plasticity and stability during adaptation.
Bxt1 + 0, + J(Y,,
(6)
where 17 > 0 is the learning rate. The conceptual illustration of SGR and the overall learning process of FEEDTTA are summarized in Figure 2 and Algorithm 1, respectively.
Regularization Method. SGR utilizes the binary nature of the feedback mechanism, and learns the 'what if' scenario, instead of focusing exclusively on immediate feedback. To simplify the explanation. we reformulate Eq. 3 as a set of partial derivatives of the score function ((] with respect to
Analysis 3.2: Alleviating Non-stationarity. Due to the
binary feedback system, the score function to be maximized from each feedback negatively related (i.e. -J(0))== =
4

---

TTA for Online VLN with Feedback- Reinforcement Learning
=(0)==-1). Therefore, by reversing the gradient direction for some dimensions, SGR can partially simulate counter- factual scenario. This mechanism allows for more flexible and dynamic adaptation, taking both possible outcomes into consideration rather than limiting updates to a single ex- treme. As a result, SGR smoothens the abrupt changes in gradient distributions, thereby alleviating the non-stationary learning environment with enhanced plasticity and stability.
4. Experimental Setup
4.1. Dataset Description
For evaluation, we use three representative VLN bench- marks: REVERIE (Qi et al., 2020), R2R (Anderson et al., 2018), and R2R-CE (Krantz et al., 2020). REVERIE is a goal-oriented task, focusing on locating remote objects with high-level instructions. The navigation is considered successful when the agent stops within 3m radius of the target object and selects the correct bounding box from the panoramic view. R2R and R2R-CE contains fine-grained navigation instructions. Similarly, the agent should stop within 3m from the target. R2R-CE is a variant of R2R in a continuous environment.
Analysis 3.3: Catastrophic Forgetting. We analyze the expected absolute value (EAV) of the gradients to support the above claim. EAV quantifies the deviation from the case where neither forgetting nor adaptation occurs, indicating the extent of policy forgetting and adaptation. For brevity, we omit the dimension index m in subsequent derivations. In a standard gradient update, the EAV is given by:
4.2. Evaluation Metries
We follow the standard evaluation protocol from the pre- vious works (Chen et al., 2021; 2022c; Gao et al., 2024a) and report Trajectory Length (TL), Navigation Error (NE), Success Rate (SR), Oracle Success Rate (OSR), Suc- cess Penalized by Length (SPL), Remote Grounding Suc- cess (RGS) and Remote Grounding Success Penalized by Length (RGSPL). Please refer Appendix C for details of each metrie. In addition to these metrics, we propose the 'Adapted Success Rate (ASR)' metric to accurately measure sample-wis wise transition of results before and after adaptation. ASR can be formulated as:
Σ [[[VoJ(O)]] = Σ lgll-
For small p and such that |a| < p. the EAV for the SGR- modified gradients is:
Σ - Σ plagol+ (11gol+(1-p))
90
ap + (1 p)
(8)
Using a first-order approximation:
E [ x= Σ [p lagel + (1 p) (11 + p)
Σ (1 - 1² - -ap) lgol-
1 ASR = (P(STTA) | |Saaee) + P(STTA (F
2
(9)
Therefore, applying SGR scales the EAV by a factor of (1 p² - aaa) S 1, reducing the gradient magnitude compared to the standard gradient update. If a = 0, this corresponds to gradient dropout, where the scaling factor is fixed at 1 p=p². If a > 0, the scaling factor is controlled by a, but remains bounded above as 1 - p² ap < 1-p?. However, when a < O, the scaling factor is also controlled by a, but bounded both above and below as 1 1-p² < 11 -ap S This result demonstrates that reversing a subset of gradients as proposed in SGR provides a strategic way to balance plasticity and stability in adapting to unseen environments.
where P(STTA SBaee) is the preserved SR (PSR), measur- ing how much the policy succeeds in the samples that would have also succeeded in the base policy before adaptation. P(STIA FB represents the converted SR (CSR), indicat- ing how much the policy succeeds in samples that would have previously failed before adaptation. By averaging the two, ASR can comprehensively assess both the plasticity and stability of the adaptation process.
4.3. Implementation Details
Pre-trained Navigation Policies. We select HAMT (Chen et al., 2021), DUET (Chen et al., 2022c), BEVBert (An et al., 2023) and EPTNav (An et al., 2024) as the target policy to perform test-time adaptation. HAMT is fully transformer- based VLN network trained via reinforcement learning. DUET combines the global map encoding and the local visual encoding through graph transformer. BEVBert improves the spatial awareness of VLN with bird' veyw map representation. EPTNav focuses agent's long-range goal planning in continuous environments. Our proposed FEEDTTA is applied at the inference time of these offline trained VLN policies. Specifically, we freeze the language and visual encoders, updating the parameters starting from the cross-modae encoder onward.
Analysis 3.4: Reversion Magnitude. In practice, can take on any value from the set of real numbers, which can result in different interpretations. When = O, the formula- tion becomes equivalent to gradient dropout (GD) (Tseng et al., 2020). While GD can bring robustness in the learning process to some extent, disregarding the updates in certain dimensions as whole causes loss of plasticity in the zeroed- out dimensions. When 0, it simply scales the gradient while keeping the direction unchanged. This equivalent to adjusting the learning rate for the selected dimensions. However, our empirical observations indicate that reversing the gradient with a negative a yields better performance than when a 2 0 (see Exp. 5.4).
5

---

TTA for Online VLN with Feedback-base Reinforcement Learning
Table Experimental results on the REVERIE dataset. f implies that the results are obtained from our re- re-implementation (same for Table 2 and Table 3). In the last column, we report the average inference time per 4 episodes. measured in milliseconds.
Methods
Val Scen SPT.
Val Unseen SPL.
Test Unsoon
Inf. Time
OSR
SRC
RGSPI.
OSR
SR1
RGSPI.
OSR
SR
SPI-1
RGSPI.1
Offime-training
BERTBERT (Hong et al., 2021)
$3.90
51.79
47.96
35.61
35 02
30.67
24.90
15.27
32.91
29.61
23.99
13.51
HOP+ (Qino et al, 2022)
54.88
53.76
47.19
33.85
36.24
31.78
26.11
15.73
33,06
30.17
24.34
14.34
BEVBerl (An el al., 2023)
76.18
73.12
65.32
51.73
56.40
51.78
36.37
24.44
57.26
52.81
36.41
22.09
KERM (Li al., 2023)
76.88
70.45
56.07
55.21
50.44
35.28
24.45
57.58
52.43
39.21
23.64
NaviLLM (Zheng et al., 2024)
53.74
44.56
36.64
56.21
43.49
34,45
VLN-VER (Liu ot al., 2024d)
80.49
75.83
66.19
56.20
61.09
55.98
39.66
23.70
62.22
56.82
33.88
23.19
Test-time Adaptation
HAMT (Chen et al., 2021)
47.65
43.29
40.19
25.18
36.84
32.95
30.20
17.28
33.41
30.40
26.67
13.08
85,4
Temt! (Wang et al., 2020s)
40.78
25.81
32,60
30.56
28.23
14.48
25.06
23.73
21.78
10.82
200.3
w/ ESTIA' (Gao 2024aa)
48.21
42.87
39.56
24.58
36.78
32.89
30.51
17.20
33.39
30.39
26.65
13.61
460.1
w/ FEEDTTA (Ours)
62.97
55,80
49,70
31,80
40.73
35.05
31.60
17.83
38.62
34.14
29:07
14.36
384.2
DUET (Chen et al., 2022c)
73.86
71.75
63.94
51.14
51.07
46.98
33.73
23.03
56.91
52.51
36.06
22.06
176.8
Tent
73.72
71.89
64.06
50.41
51.43
47.55
33.99
23,32
57.12
52.61
36.17
12.16
515.8
w/ FSTTA
75.59
75.48
65.84
52.23
56.26
54.15
36.41
23,56
58.44
53.40
36.43
22.40
868.0
W/ TTEDTTA (Ours)
86.16
84.19
75.54
60.32
71.60
66.49
45.38
30,75
58.76
53.58
37.66
24.10
672.0
TTA Baselines. To date, FSTTA (Gao al., 2024a) is the only existing baseline that shares the task objective of TTA for online VLN with our FEEDTTA. However, due to identified issue in the official code', we re-implement the
RQ2: How sensitive the performance to the quality and the quantity of the feedback provided?
RQ3: Can LLMs replace human as the feedback oracle?
RQ4: How does SGR enhance plasticity and stability and out the experiments, a t mark to indicate that the results
method to ensure proper functionality. We denote through-
alleviate non-stationarity during adaptation?
are obtained from our re-implemeenttion. Furthermore, we
RQ5: How does FEEDTTA compare to approaches that use dense reward signals?
include Tent (Wang et al., 2020a) as our comparison to thor- oughly contrast TTA against the entropy minimization paradigm. For comparable evaluation with our approach,
5.1. Main Navigation Results
Tent is applied on a per-episode basis.
The experiments in this section address RQ1 by comparing the adaptability of FREDTTA against TTA baselines while also comparing its performance to recent state-of-the-ee offline training methods in three datasets.
Hyperparameter and GPU Settings. We use a batch size
of 1 to properly simulate the online environment. Then, we search the best-performing values for the reversion rate p and the reversion magnitude a within {0.01, 0.05, 0.1, 0.2, 0.3) and (-0.01, -0.025, -0.05, -0.075, -0.1, -0.2, 0.33, respectively. For the REVERIE dataset, the results in the paper are obtained with p 0.01 and a = -02 for the validation seen split, and p = 0.05 and a = -0. for the validation unseen split. For R2R and R2R-CE, we use p = 0.05 and a = 0.1 for both splits. We report the performance variations for the combinations of the p and in Appendix B.2. The learning rate 17 is set as 5e-6. All other hyperparameters adhere to the default configuration of the target policy. Lastly, all experiments are conducted on a single NVIDIA Tesla A100 GPU. However, FEEDTTA does not require high-end server-grasd grade GPUs and can be efficiently deployed on practical hardware (c.g.. GTX 1080).
Table I reports the experimental results on the REVERIE dataset, where FEEDTTA is applied to HAMT and DUET. First, we observe that FEEDTTA brings signif- icant performance increase across all data splits and eval- uation metrics. Specifically, our method improves SR and OSR of DUET up to 41.53% and 40.20% on the validation unseen split, respectively. For the test unseen split, we uti- lize LLMs as the feedback oracle due to the unavailability of viel-iie data, yet the results remain promising compared other baselines in both HAMT and DUET. An- other noticeable aspect is that only with single stream of online learning, FEEDTTA on DUET outperforms recent tiite-of-thh-aat offline training methods. This highlights the efficiency of actively adapting to domain shifts on-the-fly, rather than relying on passive strategies that aim for general- ized performance. Lastly, we compare the average inference time per episodes. The parameter update for the adaptation brings inevitable overhead for all TTA methods. However, considering the substantial performance increases and that FEEDTTA does not hinder latency during navigation, the additional overhead is negligible.
REVERIE.
5. Experiments
In this section, we present experimental results of our study. Specifically, the experiments are conducted with a focus on answering the following research questions:
RQ1:
How well does FEEDTTA perform when compared other TTA and offline training baselines?
R2R & R2R-CE. Table 2 and Table 3 shows the navigation results for R2R and R2R-CE -CE dataset, respectively. Here, we
ht gin of /gggt /tgitttttt hulb con/ /fe to pileyyes 20242004
PSTTA/ sspex/1
6

---

TTA for Online VLN with Feedback-basee Reinforcement Learning
DUET
FSTTA
FeedTTA (Ours)
Table 2. Experimental results on the R2R dataset.
Val Seen
80
Val Unseen
Methools
[
NE
Val Soca
SRC
SPI
T1:
NE.
Vail Unsees
SR1
SPL.1
maning
SeepSeen donderon ata 2018)
10.33
6.01
19
5.39
7.81
22
PREVALENT (Sto et 2020)
10.32
3.67
09
65
10.19
4.71
58
53
HAMT (Chen et 3021)
18.15
2.51
76
72
11.46
3.65
66
61
HOP (200 2012))
11.26
75
12.27
64
5-10
10-15
15-20
5-10
10-15
15-20
DAVI5 al.,
12:45
80
12.65
67
DUET (Chen 20044)
79
13.94
72
Figure 3. Trajectory Length Analysis. We visualize the relation between the ground truth TL (x-axis) and the SR (y-axis) for the REVERIE datasct. The length are measured in metric.
75
Ne PREOTTA (Ours)
11.49
800
13.52
75
RRVWle 2028))
75
FSTTA!
12.28
2.31
80
13.96
2.80
74
we PESSTTTAA (Ours)
11.88
2,87
82
77
12:24
2.77
75
66
ment, episodes are randomly selected to receive accurate feedback, while the remaining episodes are given inaccurate feedback. Then, we obtain results with feedback accura- cies varying from 50% to 100%. Feedback accuracies less than 50% leads to obvious adaptation failure. Furthermore, the overall result suggests that the SR and the SPL metries are proportional to the quality of the feedback. However, FEEDTTA outperforms the baseline in SR with from 50%- 60% of the accuracy, implying that the method is robust to noisy or inaccurate feedback.
Table 3. Experimental results on the R2R CE dataset.
Michods
T1.
NE
Val Sem OSR1
SR1
SPA.1
T
NE
1
Via Unooee Oskt
skt
SPL.
SeitSing (Auderoce 2014
4.76
7.12
1:55
SASRA (hikra mual 2053:
0.59
7.71
36
94
7:84
6:32
51
22
CM 10oegala et al. 20220
12:35
6.10
51
43
35
11.54
1.02
42
34
25
WS KSKMMMMPPP en al, 302200
16.13
565
15
2003)
E17har las 3030g
11:75
533
72
66
30
11.00
4.71
65
37
49
PSTIA
ai0kel
11:38
3.93
72
11.57
BUVIBee (Ax et S002
13:94
5.7
73
68
13.2
4.57
67
99
50
WINNTAA
14:07
6.1
34
4.30
51
discover that FEEDTTA also adapts well on fine- tne-grrineed instructions and in continuous environment. For instance, on the R2R validation unseen split, FEEDTTA improved 8.33% on SPL for DUET, while reducing 10.88% in NE. Similarly, on the validation seen split, FEEDTTA enhanced 4.05% in SPI. with 1 12.39% shorter TL. for BEVBert. We observe consistent results in the R2R-CE dataset. For example, on the validation seen split, with reductions of 18.30% in NE and 3.15% in TL, FEED improves BEVBert by 8.22%, 7.35%, and 5.00% in OSR, SR, and SPL., respectively.
First-K-Sample -Sample. Providing feedback for every navigation episode may not be feasible in real-world scenarios. In Fig- ure 4-(b), we report the performance changes w.r.t. number of feedback provided. Specifically, the x-axis denotes the percentage of first K episodes that receive feedback, where we report results for every 10%. Here, we observe that FEEDTTA surpasses the baseline results with only using 20% of the total episodes, showcasing its high efficiency, The performance improves further in proportion the in- crease in the percentage of episodes receiving feedback.
Performance w.r.f. Ground Truth TLs. We further cate- gorize the navigation tasks of REVERIE based on ground truth TLs and evaluate the SR within each category. Fig- ure 3 illustrates the results, where we derive two major insights. First, FSTTA exhibits only minimal performance improvement over the baseline and even shows a decline in scenes with short TLs. However, our FEEDTTA brings solid performance gains across all categories. Furthermore, in the validation unseen split, both the baseline and FSTTA experience a reduction in performance as the navigation instructions require covering longer distances. Unlike this, FEEDTTA demonstrates a relatively consistent SR regard- less of TLs, highlighting the method's robustness across diverse scenes and instructions.
Interval- Update. Another strategy to measure sen- sitivity on feedhack quantity is to modify update intervals. Figure 4-(c) illustrates the changes in performance with re- spect to update intervals, where feedback is provided after every 1, 2, 4. 10, 20, and 100 iterations. For both data splits, frequent update generally produces better navigation results. Although the two splits differ in total amount of data, set- ting the update interval greater than 10 and using less than 20% of data commonly hinders the adaptation in both splits. This implies that maintaining a balance between feedback frequency and the amount of data utilized is critical for effective adaptation of FEEDTTA.
5.3. L.Mss as Feedback Oracle
5.2. Quality and Quantity of Feedback
Before utilizing the LLMs for the evaluation of the test un- seen split in Table 1, we first validate their feasibility as the oracle on the REVERIE validation unseen split. We leverage two-step LLM architecture for determining the navigation success or failure. First, we ask the LLM to identify a target goal from the instruction. We then provide the goal and the panoramic image from the last step of navigation, asking whether the navigation was successful. The details of the prompts for the experiments are provided in Appendix A.
The following experiments address RQ2 by studying the sensitivity of FEEDTTA on the quality (e.g., based on accu- racy) and the quantity (e.g., based on first K samples and update interval) of the feedback. We use the REVERIE dataset and DUET as the baseline for this experiment.
Feedback Accuracy. Figure 4-(a) illustrates the perfor- mance changes w.r.t. feedback accuracies. In this experi-
7

---

TTA for Online VLN with Feedback-based Reinforcement Learning
SR (Ours)
SPL (Ours)
Baseline SR Val usseen
Baseline SPL
seen
scen
Vall scon
Val unszen
65
60
60
Meric (h)
Merro
Moric
60
12-
50
60
03
30
33 P.eeeeddok Accuncc
120
70
80
90
320
23
60
130
20
40
62
02
100
20
60
100
20
40
60
10
100
(3) Percenaaee Feedback giver Episodes (based first updates)
Percentage Feedbock given Lipioodes (based update intervals)
Figure 4. Feedback Analysis. We study the sensitivity of our method on (a) foodback accuracy, (b) number of first K feedback samples, and (c) update interval. The plotted results represent the average of experiments conducted with different sceds.
ReatTh
Table 4. Experiments on Large Language Model Oracle.
Cumelatie Success Rote
Woight Magnitud:
Peeeita wrttin
L.Mss
Feedback Accuracy
Navigation
Performance
Accuracy
Recall
Precision
OSRT
SR1
SPL1
RGSPLI
3
GPT-4o-iin
0.65
0.62
0.73
59.61
49.90
32.50
23.27
1
GPT-4o
0.72
0.84
0.68
58.12
52.29
33.56
22.59
Helt
m5
Table 5. Effects of different gradient regularization variants on a. FEEDTTA wfo reg. denotes a variant of FEEDTTA without any
500 2000
1250
2000
2503
3030
3500
200
3030
1002 Iteratioss
2200
3000
2006 3000
Benations
regularization techniques applied.
Figure 5. Plasticity Analysis. We illustrate (left) cumulative suc- cess rates and (right) changes in weight magnitude over iterations. The results are averaged across 3 experiments with diffcrent seeds.
Mechods
SRR
SPL,
ROSPL
PSR1
CSR1
DUBT (Chen 2023c)
71.75
63.94
PEEDTTA wlr
17.18
81:80
10
56.33
21.88
94:72
49.04
PREDTTA GD to o)
15.04
83.06
73.68
59.27
73.07
96.12
20.03
TEEDTTA GS in By
17.36
84.14
72.94
55.39
73.31
96.72
49.90
FEEDTTA SGR
15.17
84.19
75.54
60.32
26.67
97:33
56.01
policy's 's performance, its effectiveness is further amplified with the addition of gradient regularization. Among them, reversing the gradient directions and enabling counterfac- tual reasoning as in SGR yields superior result in the binary feedback environment. Specifically, for both data splits, SGR brings 14.21% % and 10.28% improvements in CSR, re- spectively, indicating the flexibility of FEEDTTA in dealing with failure scenarios. In the validation unseen split, GD shows the highest result in PSR, but rather decreases CSR, hindering the balance of the two metrics.
Unneen
DUET
22.11
46.98
33.73
23.03
FESOTTA wore
40.55
83.1
FRIDTTA (o- Oi
35.20
63.36
39.81
27.48
64.38
41.65
+ PEEDTTA G/ Oy
31.04
61.63
41.47
27.61
65.13
83.91
46.35
+ FEEDTTA w/ SGR (o <0)
31.83
66 49
45.38
30.75
67.00 69
85. 18
50.21
For the experiment, we utilize the GPT-4o (Achiam al., 2023) and its smaller variant the oracle for DUET, and report the results in Table 4. With 65% and 72% of feed- back accuracies, respectively, the LLM oracles generally enhance the baseline performance, which corresponds to our experiment in Figure 4a)). Furthermore, the larger model outperforms the smaller variant in predicting navigation outcomes, suggesting a correlation between commonsense reasoning capabilities and navigation reasoning. Therefore, as LLMs advance further, their reliability feedback ora- cles will also improve, making them an efficient alternative to human feedback.
Weight Magnitude Analysis. As the agent repeatedly per- forms the online navigation task in a non-stationary environ- ment, it tends to experience a loss of plasticity. We relate this phenomenon with the increase in the weight magni- tude, where larger magnitudes imply a potential for over- fitting (Dohare et al., 2024). To analyze our method in this perspective, we visualize the cumulative success rates and changes in the LI weight magnitude on the validation unseen split of REVERIE in Figure 3. Here, observe that without any regularization or with simple scaling method, the policy encounters the loss of plasticity and re- sults in a gradual performance drop. This corresponds to the changes in weight magnitude, where the two variants exhibits the largest scale. Unlike these methods, SGR stands out with the lowest scale in weight magnitude, resulting in stable increase throughout iterations. This is attributed to its counterfactual reasoning strategy that addresses the non- non-stationary nature of the binary learning environment.
5.4. Effects of Stochastic Gradient Reversion
In this section, we address RQ4 by comparing the overall navigation performance between different gradient regular- ization methods, analyzing weight magnitude for plasticity, and analyzing catastrophic forgetting for stability.
Gradient Regularization Comparison. Table presents the navigation results of FEEDTTA using different gradi- ent regularization methods from Analysis 3.3, with a fo- cus on the ASR metric. We denote the regularization with = 0 as GD (i.e., gradient dropout) and the regularization with a > O as GS (i.e., gradient scaling), where we set a = 0.05 to ensure a valid comparison with our method.
Catastrophic Forgetting Analysis. Preserving the trained knowledge is as much important as acquiring new knowl- edge. Table 6 reports the results on the validation seen split,
While FEEDTTA alone significantly enhances the target
8

---

TTA for Online VLN with Feedback-based Reinforcement Learning
ages feedback- based reinforcement learning. The proposed adaptation strategy utilizing binary episodic feedback en- ables agents to dynamically interact with their external en- vironment by providing them with a notion of success and faiture. Additionally, we develop gradient regularization method, SGR, to robustly alleviate non-stationarity during adaptation. Through extensive experiments on challenging VLN benchmarks, our FEEDTTA showcases its superiority not only in traditional metries but also in the proposed ASR metric, which evaluates the sample-wise transition of results before and after adaptation.
Table 6. Experiments on Catastrophic Forgetring.
Val Unseen (L) Val Seen (f)
Methods
TL4
OSRt
SK1
SPL?
RGST
RGSPLT
DUET (Chen et al., 2022c)
13.86
73,86
71.15
63,94
57,41
51.44
+ FSTTA (Gao et al,, 202
13:40
73.16
71.78
64.18
$7.05
51.18
+ FEEDTTA reg.
19:43
73.72
61.96
59.24
+ FEEDTTA w/ GD
22.01
72:24
71.05
57.19
57.20
46.72
4 PREETTA w/ GS
20:11
72,80
71.68
58.90
57.55
47.72
FEEDTTA w/ SGR
18.45
76.04
73.86
62.74
59.31
49.93
Table 7. Comparison of Feedback Strategies
Feedback Strategy
SR
SPL
RGSPL
Distance-based (Dense)
63.25
42.89
28.46
Limitation and Future Work. While LLMs show substan- tial potential as a replacement for human feedback, concerns about their reliability as oracles remain unresolved. The dis- parity between the accuracy of human judgment and LLM judgment must be minimized ensure safer real-world ap- plications. Since this work successfully incorporated the concept of beedback-based RI. for the VLN task, we propose exploring more advanced applications of LLMs as feedback oracles as a promising direction for future research.
Goal-based (Sparse)
66.49
45.38
30.75
re-evaluated after the TTA the validation unseen split to measure catastrophic forgetting, First, our FEEDTTA, without gradient regularization, enhances the OSR, SR and RGS metric after adaptation on the validation unseen dataset. This suggests that, beyond adaptation to a specific domain, the proposed feedback-based RI. framework broadly en- hances navigation success. We interpret the increase in TL as representing the minimal additional exploration re- quired to achieve navigation success. Furthermore, while GD and GS exhibit catastrophic forgetting, the proposed SGR rather brings substantial improvements in the success rates, strengthening the policy's generalizability as well as adaptability on specific domain.
Impact Statement
Although our FREDTTA leads significant performance im- provements, it does not guarantee perfect prediction across the diverse environment. Therefore, significant attention must be paid for rigorous verification processes, prior to integrating into the embodied AI system.
5.5. Comparison with Different Feedback Strategies
The rationale behind choosing a simple binary episodic feedback mechanism stems from the practical limitations of the online test-time navigation environment: (1) Human involvement should be minimal, as following every navi- gation steps to provide rewards is infeasible in real-world environment; and (2) Reward systems used in offline learn ing (e.g. step-wise distance-based rewards) ) are infeasible at test-time, as we assume no access to ground-truth goal position or pre-defined maps. We empirically evaluate the efficiency of the feedback system by comparing our method with the step-wise distance-based reward system used in HAMT, where the feedback is defined as the reduction in distance to the target at each step. Additionally, if the agents successfully arrives at the goal positions, 2 is given as a success signal and otherwise -2 as a penalty. As we observe from Table 7, our binary episodic feedback surpasses the distance-based dense reward system, even without access to ground- -tuth information. This clearly demonstrates that the proposed feedback mechanism appears be simple, yet efficient and effective in improving navigation performance.
A Acknowledgement
This work was supported by Samsung Advanced Insti- tute of Technology (SAIT, 50%). Culture, Sports and Tourism R&D Program through the Korea Creative Content Agency grant funded by the Ministry of Culture, Sports and Tourism (International Collaborative Research and Global Talent Development for the Development of Copy- right Management and Protection Technologies for Gen- erative AI, RS- ASS2002-003345025, 13%; Research neural watermark technology for copyright protection of gener- ative AI 3D content, RS RS-2024-00348469, 25%), the Na- tional Research Foundation of Korea(NRF) grant funded by the Korea govemmen((MSII) 200200000262202200222220000 10%), Institute of Information communications Technology Planning & Evaluation (UTP) & ITRC(Information Tech- nology Research Center) grant funded by the Korea gov- crmmen((MSIT)
12-099-11 1911999079, Artificial Intelli- gence Graduate School Program(Korea University), 1%; 1 1%), and Artificial intelli- gence industrial convergence cluster development project funded by the Ministry of Science and ICT(MSIT, Ko- rea)&&Gaangju Metropolitan City.
6. Conclusion
In this work, we introduce FEEDTTA, an effective TTA paradigm for online vision-language navigation that lever-
9

---

TTA for Online VLN with Feedback-based Reinforcement Learning
References
Fried, D., Hu, R., Cirik, V., Rohrbach, A., Andreas, J., Moreney, L.-P., Birr-Kiikpatrrck, T., Saenko, K., Klein, D., and Darrell, T. Speaker-follower models for vision- and-language navigation. Advances in neural information processing systems, 31, 2018.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Gao, J., Yao. X., and Xu, C. Fast-slow test-time adaptation for online saiion-and-lagguage navigation. In Forty-first International Conference on Machine Learning, 2024a.
An, D., Qi, Li, Y., Huang, Y., Wang. L.., Tan, T., and Shao, Bevbert: Multimodal map pre-tree pre-training for language- guided navigation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 2737- 2748, 2023.
Gao, Z., Zhang, X.-Y., and Liu, C.-L. Unified entropy op- timization for open-set test-time adaptation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23975-23984, 2024b.
An, D., Wang, H., Wang, W., Wang, Z., Huang, Y., He, K., and Wang. L. Etpnav: Evolving topologiaal planning for lisoon-aagguage navigation in continuous environments. IEEE Transactions on Pattern Analysis and Machine In- felligence, 2024.
Georgakis, G., Schmeckpeper, K., Wanchoo, K., Dan, S., Miltsakaki, E., Roth, D., and Daniilidis, K. Cross-modal map learning for vision and language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15460-15470, 2022.
Anderson, P., Wu, Q.. Teney, D., Bruce, J., Johnson, M., Sünderhauf, N., Reid, I., Gould, S., and Van Den Hengel, A. Vision-andd-anguage navigation: Interpreting visually- grounded navigation instructions in real environments. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3674-3683, 2018.
Gu, J., Stefani, E., Wu, Q. Thomason, J., and Wang, X. E. Vision-and-language navigation: A survey of tasks, methods, and future directions. arXiv preprint arXiv:2203. S2ii:2203./2667, 2022.
Black, K., Janner, M., Du, Y., Kostrikov, I., and Levine, S. Training diffusion models with reinforcement learning. arXiv preprint 1xXXi:23305133011 2023.
Hao, W., Li, C., Li, X., Carin, L., and Gao, J. Towards learning generic agent for vision-and-language naviga- tion via pre-training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13137-13146, 2020.
Chen, J., Gao, C., Meng, E., Zhang, Q., and Liu, S. Re- inforced structured state-evolution for vision-language navigation. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp- 15450-15459, 2022a.
Hong, Y., Wu, Q., Qi, Y., Modrriuez-Opaze C., and Gould, S. Vin bert: A recurrent vision-and-lanee language bert for navigation. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp. 1643- 1653, 2021.
Chen, P., Ji, D., Lin, K., Zeng, Li, T., Tan, M., and Gan, C. Seaely-- multi-granularity map learning for vision-and-language navigation. Advances in Neu- ral Information Processing Systems, 35:38149-38161, 2022b.
Irshad, M. Z., Mithun, N. C., Seymour, Z., Chiu, H.-P., Samarasekera, S., and Kumar, R. Semantically spatio-temporal reasoning agent for uusion-and-langee navigation in continuous environments. In 2022 26th International Conference on Pattern Recognition (ICPR), pp. 4065 4071. IEEE, 2022.
Chen, S., Guhur, P.-L.,, Schmid, C., and Laptev, I. History aware multimodal transformer for vision-and-langee language navigation. Advances in neural information processing systems, 34:5834-5847, 2021.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.- et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4026, 2023.
Chen, S., Guhur, P.-L., Tapaswi, M., Schmid, C., and Laptev, Think global, act local: Dual-scale graph transformer for iision-and-language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16537-16547, 2022c.
Krantz, J., Wijmans, E., Majumdar, A., Batra, D., and Lee, S. Beyond the nav-graph: Vision-anddlaanguage navigation in continuous environments. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVIII 16, pp. 104 120. Springer, 2020.
Dohare, S., Hernandez- Garcia. J. F., Lan, Q., Rahman, P., Mahmood, A. R., and Sutton, R. S. Loss of plasticity in deep continual learning. Nature, 632(8026):768-777 2024.
10

---

TTA for Online VLN with Feedback-basee Reinforcement Learning
Lee, H., Phatale, S., Mansoor, H., Lu, K. R., Mesnard, T., Ferret, J., Bishop, C., Hall, E., Carbune, V., and Rastogi, A. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267, 2023a.
Lu, Y., Zhang, H., Nie, P., Feng, W., Xu, W., Wang, X. E., and Wang, W. Y. Anticipating the unseen discrep- ancy for vision and language navigation. arXiv preprint arXiv:2209.04725, 2022.
Nado, Z., Padhy, S., Sculley, D.. D'Amuur, A., Lakshmi- narayanan, B., and Snoek, J. Evaluating prediction-time batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006. 10963, 2020.
Lee, K., Liu, H., Ryu, M., Watkins, O., Du, Y., Boutilier, C., Abbeel, P., Ghavamzadeh, M., and Gu. S. S. Aligning text- to-image models using human feedback. arXiv preprint arXiv: 12192, 2023b.
Niu, S., Wu, J., Zhang, Y., Chen, Y., Zheng, S., Zhao, P., and Tan, M. Efficient test-time model adaptation with- out forgetting. In International conference on machine learning, pp. 16888-16905, PMLR, 2022.
Li, x., Li, C., Xia, Q. Bisk, Y., Celikyilmaz, A., Gao, J., Smith, N., and Choi, Y. Robust navigation with lan- guage pretraining and stochastic sampling. arXiv preprint arXiv: 0244444,,222,,4... 2019.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 11-27730-277444 2022.
Li, X., Wang, Z., Yang, J., Wang, Y., and Jiang, S. Kerm: Knowledge enhanced reasoning for saion-and-language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2583- 2592, 2023.
Liang, J., He, R., and Tan, T. A comprehensive survey test-time adaptation under distribution shifts. Interna- tional Journal of Computer Vision, pp. 1-34, 2024.
Pashevich, A., Schmid, C., and Sun, C. Episodic transformer for aaaoo--nndlanggaage navigation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 15942-15952, 2021.
Liu, H., Li, C., Wu, Q. and Lee, J. Visual instruction tun- ing. Advances in neural information processing systems, 36, 2024a.
Peng, A., Netanyahu, A., Ho, M. K., Shu, T., Bobu, A., Shah, J., and Agrawal, Diagnosis, feedback, adaptation: human comman-nn-the-beee in-the-loop framework for test-time policy adapta- tion. In International Conference on Machine Learning. pp. 27630-27641 PMLR, 2023.
Liu, J., Xu, R., Yang, S., Zhang. R., Zhang, Q., Chen, Z., Guo, Y.. and Zhang. Continual-nan mac: Adaptive distribution masked autoencoders for continual test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 28653- 28663, 2024b.
Pinto, A. S., Kolesnikov, A., Shi, Y., Beyer, L., and Zhai X. Tuning computer vision models with task rewards. International Conference on Machine Learning, pp. 33229-3220 -33299. PMLR, 2023.
Liu, J.. Yang, S., Jia, P., Zhang, R., Lu, M., Guo, Y., Xue, W., and Zhang, S. Vida: Homeostatic visual domain adapter for continual test time adaptation. In The Twelfth International Conference on Learning Representations, 2024c.
Qi, Y., Wu, Q., Anderson, P., Wang, X., Wang. W. Y., Shen, C., and Hengel, A. v. d. Reverie: Remote embodied visual referring expression in real indoor environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 9982-9991, 2020.
Liu, R., Wang, W., and Yang. Y. Volumetric environment representation for vision-language navigation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pp. 16317-16328, 2024d.
Qiao, Y.. Qi, Y., Hong, Y., Yu, Z.. Wang, P., and Wu, Q. Hop: History- and---nddddedd aware pre- pre-training for vision-and. language navigation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 15418-154271 2022.
Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Jiang, Q. Li, C., Yang, J., Su, H., et al. Grounding dino: Marry- ing dino with grounded pre-training for open-set object detection. In European Conference Computer Vision, pp. 35-55. Springer, 2025.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8888 8763. PMLR, 2021.
Long, Y., Li, X., Cai, W., and Dong. H. Discuss before moving: Visual language navigation via multi-expert discussions. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 17380 17387. IEFE, 2024.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. C. Er- mon, S., and Finn, C. Direct preference optimization:

---

TTA for Online VLN with Feedback-basee Reinforcement Learning
Your language model is secretly reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
Yu, B., Kasaei, H., and Cao, M. Co-navgpt: Multi-robot co- operative visual semantic navigation using large language models. arXiv preprint arXiv:2310.07937, 2023.
Schneider, S., Rusak, E.,, Eck, I.., Bringmann, O., Bren- del, W., and Bethge, M. Improving robustness against common corruptions by covariate shift adaptation. Ad- vances in neural information processing systems, 33: 11539-11551, 2020.
Zhang, M., Levine, S., and Finn, C. Memo: Test time
robustness via adaptation and augmentation. Advances in neural information processing systems, 35:38629-38642, 2022.
Zhao, B., Chen, C., and Xia, S.-T. Delta: degradation- free fully test-time adaptation. arXiv preprint arXiv:2301.33018, 2023a.
Tan, H., Yu, I.., and Bansal, M. Learning to navigate un- seen environments: Back translation with environmental dropout. arXiv preprint arXiv: 1904.04994 2019.
Zhao, S., Wang. X., Zhu, L.,, and Yang, Y. Test-time adap- tation with clip reward for zero-shot generalization in languae models. In The Twelfth International Conference on Learning Representations, 2023b.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
Zheng, D., Huang, S., Zhao, L.., Zhong, Y., and Wang, Towards learning a generalist model for embodied navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13624- 13634, 2024.
Tseng, H.-Y., Chen, Y.-W., Tsai, Y.-H., Liu, S., Lin, Y.-Y., and Yang, M.-H. Regularizing meta-learning via gradient dropout. In Proceedings of the Asian Conference on Computer Vision, 2020.
Zhou, G., Hong, Y., and Wu, Q. Navgpt: Explicit reasoning in vision-ande and-language navigation with large language models, In Proceedings of the AAAI Conference on Artifi- cial Intelligence, pp. 7641-7649, 2024.
Vaswani, Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Wang, D., Shelhamer, E.,, Liu, S., Olshausen, B., and Darrell, T. Tent: Fully test-time adaptation by entropy minimiza- tion. arXiv preprint arXiv:2006. 200..0026. 2020a.
Zhou, G., Hong, Y., Wang, Z., Wang. X. E., and Wu, Q. Navgpt-2: Unleashing navigational reasoning capability for large vision- languaee models. In European Confer- ence Computer Vision, pp. 260-278. Springer, 2025.
Wang, H., Wu, Q.. and Shen, C. Soft expert reward learning for vision-and- aoooo--nd-language navigation. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IX 16, pp. 126- 141. Springer, 2020b.
Wang, Q., Fink, O., Van Gool, L., and Dai, D. Continual test- time domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7201-7211 2022.
2022.
Wang, 7., Li, X., Yang. J., Liu, Y., and Jiang, S. Gridmm: Grid memory map for vision-and-languue languuge navigation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 15625-15636, 2023.
Williams, R. J. Simple statistical gradient-following algo- rithms for connectionist reinforcement learning. Machine learning, 8:229-256, 1992.
Wu, W., Chang, T. Li, X., Yin, Q. and Hu, Y. Vision- language navigation: a survey and taxonomy. Neural Computing and Applications, 36(7):3291-3316 2024.
You, F., Li, J., and Zhao, Z. Test-time batch statistics calibra- tion for covariate shift, arXiv preprint 1rX0422110.00055, 2021.
12

---

TTA for Online VLN with Feedback-basee Reinforcement Learning
A. Details of Large Language Model Oracles
Figure 6, we provide the prompts we utilized for the experiments in a dialogue format. As mentioned in Sec 5.3, process leverages a two-step architecture, where the LLM first identifies the target goal from the given instruction, and then determines the navigation success/failure based on the alignment of the goal with the panoramic image from the last navigation step. The system offers an effective solution in scenarios where human feedback is unavailable.
Example dialogue for the REVERIE dataset
You are indoor novigation robot executing the following instruction.
Instruction: (instruction_ txt] Task: Considering this instruction, what is the place or object you should ultimately look for? Answer simple word or phrase. include any additional text in your response.
Answer {response_goal}
Your Task:
Analyze the given panoramic view image together to identify if the image generally describe scenario where the {response_god) } is visible. Conclude whether the navigation is success or failure based on broad iggment between instruction and image.
Evaluation Criteria:
1. Yes: If the fresponse_godd can be reasonably found with the given image.
2. No: If the {response_goal)- inconsistent wi the given image, wi no reasonable alignment or evidence.
Important Notes:
1. Use reasonable inference to find the fresponse goal] in the given image. If the image broadly describe the target object(e.g.. "armchair" can be replaced with "chair"), treat it as a success. 2. Partial Alignment: If the mojority the details in the given image align with the instruction, consi der it "Yes".
Exact positioning minor missing details should not override clear overall match.
Input:
natural language instruction: finstruction_txt?
Panorami Image:
Output: Your response should only be "Yes" or "No".
Answer
"Yes"
"No
Figure 6. Overall pipeline of LLMs as an oracle.
B. Details of Stochastic Gradient Reversion
B.I. Derivation of the Scaling Factor in Eq.5
We provide mathematical derivation of how the scaling factor of ap-11-p3 in Eq. can ensure consistency expectation.
Step Expectation the Modified Gradient: The modified gradient can written as:
Dem. = 90m. (a bm + (1 bm))).
13

---

TTA for Online VLN with Feedback-basee Reinforcement Learning
Taking the expectation over bm where E [(bm] p. have:
E. Elg go... Elar bm + (1 bm)]-
Substituting Eb p, the expectation becomes:
Elge 90... (op + (1 (p)))
Step 2. Scaling the Modified Gradient: To ensure consistency in expectation, scale so.. by apt(1-p)- The scaled gradient is:
Do op + p)
9o..
Step 3. Expectation of the Scaled Gradient: Taking the expectation of He... we get:
gb
E Elg
Bl
E
E
op + p)
op p)
From Step 1, E[gg,.. = 90... (app + (1 p)). Substituting this gives us:
no.. (ap + (1 p))
9o
ap (1 p)
90m
B.2. Reversion Rate and Reversion Magnitude a
SR
SPI
RGSPL
Reversion Rate p
0.3
0.2
8.05
0.01
63.14
21,94
Reversion Magnitude &
Reversion Magnitude a All reversion magnitudes are negative, with minus signs omitted for simplicity.
Reversion Magnitude a
Figure Hyperparameter Analysis of SGR. We illustrate the performance variations w.r.t. the reversion rate p and the reversion magnitude a. The red markers in the color bar indicate the performances of DUET w/ FEEDTTA hefore applying the SGR regularization and the yellow-boxed cells are the reported combination throughout the manuscript.
In this section, analyze the performance variations for the combinations of the reversion rate p and the reversion magnitude a in SGR. Figure illustrates the results of FEED the validation unseen split of REVERIE, measured on three metrics SR, SPL, and RGSPL. DUET (Chen al., 2022c) is utilized as the target policy. Here, observe three insights: (1) SGR generally brings improvements in the performance, demonstrating its robustness on various configurations of p and a. (2) A reversion rate of p = 0.05 generally yields decent navigation performances. (3) Regularizing an excessively small number parameters with p = 0.01 has minimal effect on performance. (4) Reversing gradients with magnitudes exceeding 0.3 results lower SPL and RGSPL, suggesting increased exploration during navigation.
C. Details of Evaluation Metrics
Below, we provide the details of the evaluation metrics that are used throughout our experiments.
Trajectory Length (TL) measures the average distance the agent traveled from the starting point to the endpoint in metric units. Lower value typically indicates efficient navigation.
14

---

TTA for Online VLN with Feedback-basce Reinforcement Learning
Table 8 General TTA sequence ordering.
Table 9. Continual TTA sequence ordering.
Method
SR
SPL
RGSPL
Method
SR
SPL
RGSPL
DUET
46.98 65.33 10)
33.73 42.63 (+9)
23.03 28.71 (#:1.45)
DUET w/ FEEDTTA
46,98 54.81 (+-1.89)
33.73 36.70 (+0.877)
23.03
w/ FEEDTTA
23.74 (=-0.47)
Table Per-Scene TTA sequence ordering
SR
68
72.22/27.16
32.55/7251
51.75/51.75
43.34/
30,30/ 3946
46.4/56.75
5060/64.57
45.89/71.54
55.67/ 69.47
(1:1.65)
(:00,77)
(:10.96)
(411.00)
(1:3.25)
(1:4.49)
(4:3.92)
(::7.48)
(4:2.73)
(2:1.53)
SPT.
30.76/40.30
/6.1/
19.88/22.7
/8.97/38.9
25.55/29.31
19.1//2473
35.21 30,53
34.36/38.59
99,54//50.20
44.91/
(4:0.96)
(#:1.73)
(4:049)
(d11330)
(4:3.14)
(4:3.57)
(4:3.12)
(#5.551)
(1:6.74)
(#:2.87)
Navigation Error (NE) measures the average distance between the ground truth endpoint and the predicted endpoint in metric units. Lower value indicates that the agent closely followed the given instruction.
Success Rate (SR) calculates the fraction of successful navigation over total navigation attempts, where NE considered as success.
Oracle Success Rate (OSR) calculates the fraction of successful navigation over total navigation attempts, where it is considered as success if one of the navigation points in the trajectory contains ground truth endpoint.
Success penalized by Path Length (SPL. evaluates the weighed trajectory efficiency for navigation success, where score closer to SR indicates that the trajectory closely followed the shortest path. The equation is formulated as SPL
Σ n= Sn The where S is the binary indicator for success and SP denotes the shortest path.
Remote Grounding Success (RGS) measures the portion of navigation attempts that successfully grounded the target object required from the instruction, determined by bounding box prediction with IoU (intersection over union) > 0.5 compared to the ground truth.
Remote Grounding Success penalized by Path Length (RGSPL) calculates the weighed trajectory efficiency for remote grounding success, similar to the SPL metric.
D. Effects of Different Sequence Ordering
Online learning is inherently sequence- deceedee. However, we show that the benefits of FEEDTTA is invariant to sequence ordering through the following experiments with three different configurations. We use the 'validation unseen' split of the REVERIE dataset and compare with the DUET policy. For all configurations, the reported numbers of FEEDTTA are the average of the results from different seeds, with standard deviation reported in brackets.
General TTA: In this configuration, all episodes are randomly ordered regardless of scene IDs, which corresponds the experimental setting reported in Table of our paper. The results are reported in Table 8.
Continual TTA: For this configuration, fix the episode orders for each scene ID, and set the adaptation sequence based on mixed scene ID orders, evaluating continual adaptation performances across different scenes. The results are reported in Table 9.
Per-Scene TTA: Here, we analyze the effects of random episode orders for each scene ID. Note that in this setting, the adaptation is performed per-scene, and not throughout the entire validation set. The results are reported in Table 10 in the form of (DUET/ +FeedTTA).
These experiments confirm that sequence ordering does influence navigation outcomes; however, the benefits of FEEDTTA remain consistent, as evidenced by superior performances with low variations across different seeds.
E. Trajectory Visualization
In this section, we analyze the trajectories of DUET before and after applying our FEEDTTA with visual illustrations. For this study, we select two episodes that initially fail under the base DUET policy but achieve success following a one-step parameter update with our FEEDTTA. The visualized results in Figure 8 provides the following insights.
15

---

TTA for Online VLN with Feedback-based Reinforcement Learning
(I) The base DUET policy navigates to destination that closely resembles the instructed location but often fails due to missing critical details. For instance, in the first sample shown the left of Figure 8, the agent following the DUET policy navigates to the bathroom, as specified by the instruction. However, the predicted destination does not fully align with the specific details provided in the instruction. This can be also observed in the second example, where the agent stops near the kitchen near the family room, but fails to reach the precise location described in the instruction.
(2) Upon receiving negative feedback via FEEDTTA after a navigation failure, the agent adjusts its trajectory at a certain point that we refer to as an uncertainty zone. Uncertainty zone is a visual state with multiple feasible navigable locations, resulting in high uncertainty (i.e., entropy). In both examples, the measured entropy at the uncertainty zone is the highest observed along the DUET trajectory. The updated policy from FEEDTTA chooses an alternative path within the uncertainty zone, enabling exploration of new possibilities.
(3) Only with a single-step parameter update with feedback, the agent successfully arrives at the desired destination. Multiple iterations from the online data stream further enhance adaptability, and generalizability by learning the notion of success and failure, which is demonstrated throughout our experiments.
eask and keat Now pichure
Unde BaP Zone
chote cleane
the Thetng Foom
Uecartion Zone
Ward!
DULY
CUCY
Caraac
!!!
Figure& 8. Visual Analysis of FEEDTTA. We illustrate two examples of episodes that initially fail under the base policy but achieve success after applying FEEDTTA. We provide a top-down view of the trajectories, along with a panoramic image of the uncertainty zone and the endpoints reached by DUET and FEEDTTA.
F. FAQ& & Discussions
In this section, we share some of the notable questions and discussions that emerged throughout the research process.
Question 1: How is learning from online feedback different from learning from ground truths?
Answer: Online feedback and ground truth are inherently different in the VI.N task. First, ground truth consists of offline collected state-action pairs for each step, whereas online feedback can be any scalar value based the oracle's objective. Accordingly, ground truths directly force the policy to learn optimal action, which guarantees performance when strictly followed. However, feedback provides indirect guidance by encouraging actions that maximize the reward, which may vary depending on the oracle's preferences. Considering these aspects, learning from ground truths is infeasible and impractical in the online TTA setup, whereas feedback-base learning offers a more adaptable framework by enabling policies to iteratively improve through interaction with the environment and alignment with the oracle's 's objectives.
Why is the performance improvement smaller in the R2R datasets compared to that of REVERIE's? Answer: We assume this is due to differences in the instructions between the two datasets and their alignment with the binary episodic feedback mechanism of FEEDTTA. While FEEDTTA demonstrates its superior rest-time adaptability in both datasets, the R2R-trained policy and the REVERIE-trained policy inherently require different guidance for performance improvements. Given that the former relies on dense, step-wise guidance during training, the binary episodic feedback provided by FEEDTTA might be relatively sparse to drive significant performance enhancements. However, the latter, which trained operate under less structured and more abstract instructions, is better suited benefit from the sparse binary episodic feedback of FEEDTTA, allowing it to adapt more effectively during test-time.
Question 2:
Question 3: Can FEEDTTA adapt to novel navigation tasks at test time?
Answer: Continuing from the previous analysis, we empirically address this question by conducting experiments on the R2R dataset (i.e., step-by- bt-step instruction following task) with policy trained on the REVERIE dataset (i.e., goal-oriented task), and similarly, on the REVERIE dataset using a policy trained on R2R. Evaluations are carried out on the validation unseen split of both dataset. Table 11 shows that while the absence of fine-grained trajectory details in REVERIE instructions leads
16

---

TTA for Online VLN with Feedback-basee Reinforcement Learning
to longer TL for R2R-trained policies identify the goal point, both training brings improvement in the navigation success of each task. This suggests that FEEDTTA effectively leverages the shared knowledge underlying both tasks, which involves reaching a target point based on given instructions. However, the results from the RGSPL metric suggest that adapting and improving on an untrained task in real-time using a single-stream online learning approach is challenging.
Table I1. Experimental results on the cross cross-task TTA.
REVERIE R2R
NE +
SR
SPL
R2R REVERIE
TL. +
SR
RGSPL
DUET (Chen et al., 2022c)
9.78
17.33
4.82
DUET
14.67
24.91
3.47
w/ FEEDTTA
8.94
18.35
6.51
w/ FEEDTTA
22.50
28.86
3.49
Question 4:
Can LLM--genrrted counterfactual evaluations replace SGR?
Answer: The counterfactual reasoning of SGR is a regularization technique applied on limited number of parameters. which means that the large portion of parameters should be updated based proper feedback for intended functionality. Furthermore, while LLMS can indeed reason counterfactual scenarios, their reliability on predicting navigation outcomes itself still remains as challenge, making them unsuitable as a direct replacement for SGR.
Question 5:
Can FeedTTA be applied to Visual Navigation tasks?
Answer: Yes, FEEDTTA can be applied to Visual Navigation (VN) tasks even in the absence of complex language instructions, as it only requires determining success or failure within the navigation system. To identify the dominant modality influencing navigation outcomes, we analyze navigation consistency for each trajectory in the REVERIE dataset, where each trajectory is paired with multiple language instructions. Specifically, compute the average success rate across different instructions for each trajectory. We then identify trajectories with consistent outcomes-definn high (> 0.8) or low (< 0.2) average success rate-and and calculate their proportion within the validation set. Our experiment yields a ratio of 0.72, suggesting that visual observations are a key factor not only in VN tasks but also in VLN, where they play a more decisive role compared to language variations.
as those with a
Question 6: Does increased trajectory length in some episodes represent beneficial exploration?
Answer: We justify that the increased trajectory length (TL.) indicates beneficial exploration by empirically testing the hypothesis: "The overall increase in TL primarily results from episodes that would have failed in the origi- nal navigation but succeeded after applying FeedTTA". In Table 12, we compare the increase in TLs for the suc- cessful navigation episodes after adaptation, categorized based on the pre-tested results before applying FEEDTTA. For this experiment, we use the 'validation unseen' split of the REVERIE dataset with DUET as the base policy. Here, we discover that the average TL increase is significantly larger for fail-to- tousccessusssssse cases than for success-to totttoo-sscssss cases. This clearly demonstrates the important role of FEEDTTA in overcoming failure cases through extended exploration in unseen navigation environment.
Table 12. Trajectory length and exploration
Success->uucee Fail-SSuccess
Inereased TL
3.54 (+1.45)
10.65 (+3775
Question 6: What new directions for research does FEEDTTA suggest?
Answer: FEEDTTA serves as an exemplar of incorporating the recent advancements of the beedback-based RL into the robotic navigation task. Considering the contributions and limitations of FEEDTTA, we suggest the following topics as prospective future research directions:
Advanced application of LLMs as navigation oracles. As highlighted in Sec.5.3 and Sec.6, enhancing the accuracy of navigation outcome predictions is essential to ensure safer deployment of LLMs as navigation oracles. One approach is to develop a more advanced LLM architecture and prompting system capable of capturing the complex reasoning underlying its predictions. Another way is to incorporate visual foundation models (Radford et al., 2021; Kirillov et al., 2023; Liu et al., 2025) to provide LLMs with more spatial contexts. Both approaches will enhance the reliability of LLMs as oracles, benefiting not only the TTA of VLN but also zero-shot VLNs (Zhou et al., 2024; Long et al., 2024).
Test-time Adaptation on untrained navigation tasks. In real-world scenarios, the given instructions and tasks may differ from the trained navigation tasks, leading to the outcomes shown in Table 11. Therefore, it is crucial to develop a generalized TTA for diverse navigation tasks to ensure the versatility of embodied agents in real-world applications.
bredback-based RL for offline VLN training. RL is accommodated in several previous literature (Chen et al., 2021; 2022a) with heuristic reward shaping. However, with the success of feedback-based RL. in online navigation with
17

---

TTA for Online VLN with Feedback-basee Reinforcement Learning
FEEDTTA, training the notion of success and failure through binary episodic feedback can alleviate the burden of manual reward shaping. We speculate that combining policies trained with offlin feedhack-based RI. and online TTA using feedback-based RL techniques, such as FEEDTTA, could yield substantial synergy.
18


Reviews:
[{"review_id": "H249mrC6qd", "reviewer": "Reviewer_vESv", "summary": "The paper introduces FEEDTTA, a test-time adaptation (TTA) framework for vision-language navigation (VLN) that uses binary episodic feedback to adapt navigation policies in unfamiliar environments. To maintain stability during learning from binary signals, the authors propose Stochastic Gradient Reversion (SGR), a technique that reverses gradient directions for randomly selected parameters. Experiments on REVERIE, R2R, and R2R-CE benchmarks show FEEDTTA outperforms other TTA methods and sometimes even surpasses offline training approaches.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your **positive evaluation** of our work. We hope our response fully addresses all concerns and demonstrates the strength of our contributions.\n\n### Q1. How does different sequence orderings affect adaptation?\n> We agree that online learning is sequence-dependent. However, we show that the benefits of FeedTTA is invariant to sequence ordering through the following experiments with three different configurations. We use  the 'validation unseen' split of the REVERIE dataset and compare with the DUET policy. For all configurations, the reported numbers of FeedTTA are the average of the results from 3 different seeds, with standard deviation reported in brackets.\n\n> **1. General TTA**\n> - In this configuration, all episodes are randomly ordered regardless of scene IDs, which corresponds to the experimental setting reported in Table 1 of our paper.\n\n|Method|SR|SPL|RGSPL|\n|---|---|---|---|\n|DUET|46.98|33.73|23.03|\n|FeedTTA|65.33 $(\\pm1.10)$|42.63 $(\\pm1.98)$|28.71 $(\\pm1.45)$|\n\n> **2. Per-Scene TTA**\n> - Here, we analyze the effects of random episode orders for each scene ID. Note that in this setting, the adaptation is performed per-scene, and not throughout the entire validation set. The results below are in the form of (DUET / +FeedTTA)\n\n|Scene ID | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |\n|---|---|---|---|---|---|---|---|---|---|---|\n|SR       | 48.63 / 60.78 $(\\pm1.65)$ | 72.22 / 77.16 $(\\pm0.87)$ | 32.65 / 37.51 $(\\pm0.96)$ | 46.85 / 51.75 $(\\pm1.00)$ | 43.34 / 51.09 $(\\pm3.25)$ | 30.30 / 39.46 $(\\pm4.49)$ | 44.84 / 56.76 $(\\pm5.92)$ | 50.60 / 64.57 $(\\pm7.48)$ | 45.89 / 71.54 $(\\pm2.74)$ | 55.67 / 69.47 $(\\pm1.53)$ |\n|SPL      | 30.74 / 40.80 $(\\pm0.96)$ | 56.61 / 61.51 $(\\pm1.73)$ | 19.88 / 22.79 $(\\pm0.49)$ | 37.27 / 38.98 $(\\pm1.30)$ | 25.55 / 29.31 $(\\pm3.14)$ | 19.81 / 24.73 $(\\pm3.57)$ | 35.21 / 30.53 $(\\pm3.12)$ | 34.36 / 38.59 $(\\pm5.51)$ | 29.54 / 50.29 $(\\pm6.74)$ | 44.91 / 56.42 $(\\pm2.87)$ |\n\n> **3. Continual TTA**\n> - For this configuration, we fix the episode orders for each scene ID, and set the adaptation sequence based on mixed scene ID orders, evaluating continual adaptation performances across different scenes.\n\n|Method|SR|SPL|RGSPL|\n|---|---|---|---|\n|DUET|46.98|33.73|23.03|\n|FeedTTA|54.81 $(\\pm1.89)$|36.70 $(\\pm0.87)$|23.74 $(\\pm0.47)$|\n\n> These experiments confirm that **sequence ordering does influence navigation outcomes**; however, the **benefits of FeedTTA remain consistent**, as evidenced by superior performances with low variations across different seeds.\n\n---\n### Q2. Does increased trajectory length represent beneficial exploration?\n> A : We justify that the increased trajectory length (TL) indicates beneficial exploration by empirically testing the hypothesis: ***“The overall increase in TL primarily results from episodes that would have failed in the original navigation but succeeded after applying FeedTTA”***. In the table below, we compare the increase in TLs for the successful navigation episodes after adaptation, categorized based on the pre-tested results before applying FeedTTA. For this experiment, we use the 'validation unseen' split of the REVERIE dataset with DUET as the base policy. Here, we discover that **the average TL increase is significantly larger for fail-to-success cases than success-to-success cases**. This clearly highlights the **role of FeedTTA in overcoming failure cases through extended exploration in unseen environment.**\n\n||Success->Success|Fail -> Success|\n|---|---|---|\n|Increased TL|3.54 $(\\pm1.45)$|10.65 $(\\pm3.75)$|\n\n---\n### Q3. How does FeedTTA compare to approaches that use more informative feedback (beyond binary signals)?\n> A : The rationale behind choosing a simple binary episodic feedback mechanism stems from the practical limitations of the online test-time navigation environment:\n> - Human involvement should be minimal, as following every navigation steps to provide rewards is infeasible in real-world environment.\n> - Reward systems used in offline learning (e.g. step-wise distance-based rewards ) are infeasible at test-time, as we assume no access to ground-truth goal position or pre-defined maps.\n>\n> We empirically evaluate the efficiency of the feedback system by comparing our method with the step-wise distance-based reward system used in HAMT, where the feedback is defined as the reduction in distance to the target at each step. Additionally, if the agents successfully arrives at the goal positions, 2 is given as a success signal and otherwise -2 as a penalty. As we observe from the table below, **our binary episodic feedback surpasses the distance-based dense reward system, even without access to ground-truth information**. This clearly demonstrates that the **proposed feedback mechanism appears to be simple, yet efficient and effective in improving navigation performance**. \n\n|Feedback Strategy|SR|SPL|RGSPL|\n|---|---|---|---|\n|Distance-based (Dense)|63.25|42.89|28.46| \n|Goal-based (Sparse)|66.49|45.38|30.75|", "claims_and_evidence": "Most claims are well-supported by empirical evidence:\n\n- FEEDTTA's superior performance over other TTA methods is demonstrated across multiple datasets and metrics.\n- The effectiveness of SGR compared to gradient dropout and scaling is shown through ablation studies.\n- The claim about LLMs as potential feedback oracles is supported, though with acknowledged limitations in reliability.\n\nHowever, some claims require closer scrutiny:\n\n- The claim of outperforming state-of-the-art offline methods is limited to specific configurations (primarily REVERIE with DUET) rather than being a general finding.\n- The interpretation of increased trajectory length as beneficial \"exploration\" lacks rigorous justification.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are appropriate for online VLN:\n\n- Binary episodic feedback is a practical choice for real-world deployment scenarios.\n- The introduction of ASR effectively measures adaptation capability by examining both preserved and converted success rates.\n- The three benchmark datasets (REVERIE, R2R, R2R-CE) represent diverse VLN tasks.\n\nThe evaluation could be strengthened by examining how the sequence of navigation episodes affects adaptation performance, as online learning is inherently sequence-dependent.", "theoretical_claims": "The theoretical claims are limited and mostly sound. The derivation of the scaling factor in SGR (in supplementary material) ensures consistency in expectation during gradient updates. The analysis of how SGR alleviates non-stationarity is reasonable but not formally proven.", "experimental_designs_or_analyses": "The experimental designs are generally sound:\n\n- Comparisons with other TTA methods (Tent and FSTTA) establish clear baselines.\n- Ablation studies on feedback quality and quantity provide useful insights.\n- The catastrophic forgetting analysis appropriately measures stability.\n\nOne limitation is the lack of analysis regarding sequence effects in online learning - different orderings of test examples might yield different adaptation results.", "relation_to_broader_scientific_literature": "The paper effectively bridges three research areas:\n\n- VLN: It addresses a gap in online adaptation where previous works focused primarily on offline training.\n- TTA: It identifies limitations of entropy minimization approaches for sequential decision-making tasks.\n- Feedback-based RL: It adapts concepts from RLHF literature to navigation tasks.\n\nThe binary feedback mechanism builds on established sparse reward RL approaches, though the paper positions this in the novel context of test-time adaptation for VLN.", "essential_references_not_discussed": "The paper would benefit from discussing:\n\n- Connections to continual reinforcement learning literature, particularly works addressing non-stationarity in online learning environments.\n- Prior work on sample ordering effects in online RL, such as curriculum learning approaches.\n- Research on uncertainty-aware navigation that could provide context for understanding where adaptation is most effective.", "other_strengths_and_weaknesses": "Strengths: Practical approach that requires minimal feedback, making it feasible for real-world deployment\nWeaknesses: Limited novelty in the basic approach, as online RL with sparse rewards is well-established", "other_comments_or_suggestions": "N/A", "questions_for_authors": "- How would different orderings of test examples affect FEEDTTA's performance? Have you experimented with different sequence orderings to quantify this effect?\n- The trajectory length often increases after adaptation. Could you provide more evidence that this represents beneficial exploration rather than inefficient navigation?\n- How does FEEDTTA compare to approaches that use more informative feedback (beyond binary signals) in terms of adaptation efficiency? Is there a trade-off between feedback simplicity and adaptation speed?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DYkQ7Af9Qp", "reviewer": "Reviewer_gT5s", "summary": "FeedTTA is a test-time adaptation (TTA) framework for online vision-language navigation (VLN). It utilizes LLM for external interaction, providing binary feedback to the deployed navigation algorithm and establishing a feedback-based online reinforcement learning mechanism. By leveraging binary episodic feedback and gradient regularization, FeedTTA enhances adaptability while balancing plasticity and stability. Experiments demonstrate that FeedTTA outperforms state-of-the-art methods, excelling in unfamiliar environments.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed feedback. Below, we provide our responses to each comment and hope they contribute to a better evaluation of our work.\n\n### Q1. Computational feasibility of FeedTTA\n> A : We first clarify that **FeedTTA does not require high-end server-grade GPUs and can be efficiently deployed on practical hardware (e.g., GTX 1080)**. The trained DUET policy consists of 181.08M parameters, whereas FeedTTA trains only 78.67M parameters for adaptation—just 43% of the total—making adaptation highly efficient. Furthermore, FeedTTA increases memory usage by only 0.67%, requiring 4.42 GB compared to the 4.39 GB used by baseline VLN models during inference. Lastly, while FeedTTA introduces some latency due to backpropagation, its episodic updates do not impact real-time navigation performance, making it **a practical and feasible solution for real-world deployment**.\n\n---\n### Q2. Can we leverage LLMs to provide more detailed feedback instead of simple binary feedback?\n> A : As discussed in Section 5.3, LLMs achieve at most 73% accuracy in predicting simple navigational outcomes. While this is sufficient to guide baseline policies, errors can accumulate, hindering stable adaptation. In our response to Q3 of reviewer vESv, we showed that dense step-wise rewards, though less efficient than sparse goal-based feedback at test time, still significantly improve performance. As LLMs advance in navigational reasoning, their potential for richer feedback remains an exciting research direction.\n\n---\n### Q3. Could LLM-generated counterfactual evaluations replace SGR?\n> A : The counterfactual reasoning of SGR is a regularization technique applied on a limited number of parameters, which means that the large portion of parameters should be updated based on proper feedback for intended functionality. Furthermore, while LLMs can indeed reason counterfactual scenarios, their reliability on predicting navigation outcomes itself still remains as a challenge, making them unsuitable as a direct replacement for SGR.\n\n---\n### Q4. Can FeedTTA be applied to VN tasks?\n> A : **Yes, FeedTTA can be applied to VN tasks even in the absence of complex language instructions**, as it only requires determining success or failure within the navigation system. To identify the dominant modality influencing navigation outcomes, we analyze navigation consistency for each trajectory in the REVERIE dataset, where each trajectory is paired with multiple language instructions. Specifically, we compute the average success rate across different instructions for each trajectory. We then identify trajectories with consistent outcomes—defined as those with a high (> 0.8) or low (< 0.2) average success rate—and calculate their proportion within the validation set. Our experiment yields a ratio of 0.72, suggesting that **visual observations are a key factor not only in VN tasks but also in VLN**, where they play a more decisive role compared to language variations. \n\n---\n### Q5. How does SGR prevent catastrophic forgetting?\n> A : The expected absolute value (EAV) of the gradients quantifies the deviation from the case where neither forgetting nor adaptation occurs, indicating the extent of policy forgetting and adaptation. For brevity, we omit the dimension index $m$ in subsequent derivations.\n>\n>In a standard gradient update, the EAV is given by:\n>$$\\sum \\mathbb{E}[|\\nabla_{\\theta} J(\\theta)|] = \\sum |g_{\\theta}|.$$  \n>\n>For small $\\alpha$ and $p$ such that $|\\alpha| < p$, the EAV for the SGR-modified gradients is:\n>$$\\sum \\mathbb{E}\\left[ \\left\\|\\nabla_{\\theta} J(\\theta)^{\\prime} \\right\\| \\right] = \\sum \\left[ p \\left\\|\\alpha g_{\\theta} \\right\\| + (1 - p) \\left\\| \\frac{g_{\\theta}}{\\alpha p + (1 - p)} \\right\\| \\right].$$  \n>\n>Using a first-order approximation:\n>$$\\sum \\mathbb{E}\\left[ \\left\\|\\nabla_{\\theta} J(\\theta)^{\\prime} \\right\\| \\right] \\approx \\sum \\left[ p \\left\\|\\alpha g_{\\theta} \\right\\| + (1 - p) \\left\\| (1 + p) g_{\\theta} \\right\\| \\right] = \\sum (1 - p^2 - \\alpha p) \\left\\| g_{\\theta} \\right\\|.$$  \n>\n>Applying SGR scales the EAV by a factor of $(1 - p^2 - \\alpha p) \\leq 1$, reducing the gradient magnitude compared to the standard gradient update. \n>- **$\\alpha = 0$ (Gradient Dropout):**  \n> The scaling factor is fixed at $1 - p^2$.\n>- **$\\alpha > 0$ (Gradient Scaling):**  \n> The scaling factor is controlled by $\\alpha$, but remains bounded above:  $1 - p^2 - \\alpha p < 1 - p^2.$\n>- **$\\alpha < 0$ (Gradient Reversion):**  \n> The scaling factor is controlled by $\\alpha$, bounded both above and below:   $1 - p^2 < 1 - p^2 - \\alpha p \\leq 1.$  \n>\n>This result demonstrates that **reversing a subset of gradients as proposed in SGR provides a strategic way to balance plasticity and stability in adapting to unseen environments.**", "claims_and_evidence": "1、I notice that the paper deploys FeedTTA on an A100 GPU, which is a high-end server-grade GPU rather than typical mobile hardware. This raises concerns about its feasibility in real-world navigation scenarios with limited computational resources. \nIt would be helpful to discuss whether FeedTTA increases the computational burden on the base navigation algorithm and compare its hardware requirements and inference speed with standard VLN models to enhance practicality.", "methods_and_evaluation_criteria": "1、Can we leverage large language models (LLMs) to provide more detailed scene-aware feedback or sub-goal feedback instead of simple binary feedback? This might help mitigate the issue of extreme binary feedback.\n\n2、You mention the issue of extreme binary feedback—does this directly correspond to the sparse reward problem in reinforcement learning?\n\n3、I notice that you propose using SGR to generate counterfactual gradient information to address the extremity of binary feedback. Could LLM-generated counterfactual evaluations replace this approach? If so, would slightly less accurate LLM-based navigation evaluations actually improve performance?\n\n4、The paper suggests that TTA should flexibly handle different navigation outcomes, yet it still relies on an LLM-generated binary feedback mechanism, considering only success or failure. Would a more diversified navigation evaluation be more effective? For example, a failed navigation attempt that still gets close to the goal, or a successful navigation where the agent deviates from the ideal final position.", "theoretical_claims": "SGR (Stochastic Gradient Regularization) mitigates the extremity of binary feedback signals by introducing counterfactual gradient updates, which help smooth the learning process. However, its role in preventing catastrophic forgetting needs further clarification.", "experimental_designs_or_analyses": "Can the current method be applied to VN tasks, where there are no complex textual instructions, and test failures stem solely from visual observations and scene layout differences?", "relation_to_broader_scientific_literature": "The paper proposes a novel feedback mechanism utilizing LLM for TTA training and introduces the SGR module to address the binary extremity issue in feedback. Experimental results in the VLN setting outperform previous works.", "essential_references_not_discussed": "Without", "other_strengths_and_weaknesses": "Strengths\nThe paper proposes a novel approach using LLM for TTA training, introducing an innovative feedback mechanism to enhance adaptation.\nWeaknesses\nWould incorporating LLM for feedback provide greater flexibility and lead to better performance? Does using LLM introduce a higher computational burden for fine-tuning? Could modifying the feedback mechanism eliminate the need for the SGR module? These questions require further discussion.", "other_comments_or_suggestions": "See the above comments.", "questions_for_authors": "See the above comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Kg47V53ZW3", "reviewer": "Reviewer_P7CW", "summary": "This paper investigates the problem of Vision-Language Navigation (VLN) adaptation during deployment and introduces FEEDTTA, which uses feedback-based reinforcement learning. The main idea is to give the agent simple binary feedback after each navigation attempt (+1 for success or -1 for failure). To enhance learning from this binary feedback, they present Stochastic Gradient Reversion (SGR), a gradient regularization technique that helps maintain a balance between plasticity and stability.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your **positive evaluation** of our work. We are committed to enhancing the quality of the figures in the final version upon acceptance. If there are any further questions or suggestions during the discussion period, we would greatly appreciate the opportunity to address them and further refine our work **towards clear acceptance**.", "claims_and_evidence": "The claims in the paper are backed up by evidence from experiments. The authors show that FEEDTTA works better than Test-time Adaptation baselines, while being comparable to Offline-training based methods. The paper provides sufficient evidence.", "methods_and_evaluation_criteria": "The evaluation focuses on flexibility and interactivity, with standard evaluation protocol and a new metric called Adaptive Success Rate (ASR) to measure how well the agent adapts. This method is tested on three well-known benchmarks, which cover different navigation scenarios and instruction types.", "theoretical_claims": "was not reviewed in depth", "experimental_designs_or_analyses": "This paper tests on three benchmarks (REVERIE, R2R, and R2R-CE) and provides clear explanations of their results and why FEEDTTA performs as it does. The experimental design is sound. The comparison with both offline training methods and other test-time adaptation approaches provides a comprehensive evaluation of FEEDTTA's performance.", "relation_to_broader_scientific_literature": "I am not familiar with the literature in this area", "essential_references_not_discussed": "I am not familiar with the literature in this area", "other_strengths_and_weaknesses": "Strengths:\nThe problem of test-time adaptation in VLN is interesting.\nThe Feedback-based Test-Time Adaptation is simple and aligns well with how humans might interact with navigation agents in real-world scenarios.\n\nWeaknesses:\nFigures 2, 3, 5, and 7 are not vector graphics and appear blurry.", "other_comments_or_suggestions": "Improve the quality of figures to enhance readability", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a method for adapting to unseen environments during the test phase in Vision-Language Navigation (VLN) using Large Language Models (LLMs). The reviewers noted the strengths of this work as its practical approach (vESv), novel methodology (gT5s
), and the interesting nature of the problem (P7CW), and the overall evaluation leaned toward acceptance. The authors’ rebuttal effectively addressed the reviewers’ concerns, leading one reviewer to raise their score, and ultimately, all reviewers recommended a weak accept. Therefore, the AC recommend accepting this paper. Incorporating the feedback received during the rebuttal phase into the final version is strongly encouraged.
Decision: Accept (poster)