Uncertainty-Aware Gaussian Map for Vision-Language Navigation
Authors: Jianzhe Gao
Rui Liu
Yuxuan Xu
Tongtong Cao
Yingxue Zhang
Zhanguang Zhang
Sida Peng
Yi Yang
Wenguan Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Vision-Language Navigation
Unvertainty Estimation
3D Value Map
Gaussian Splatting
Primary area: applications to robotics, autonomy, planning

Abstract: Vision-Language Navigation (VLN) requires an agent to navigate 3D environments following natural language instructions. During navigation, existing agents commonly encounter perceptual uncertainty, such as insufficient evidence for reliable grounding or ambiguity in interpreting spatial cues, yet they typically ignore such information when predicting actions. In this work, we explicitly model three forms of perceptual uncertainty (i.e., geometric, semantic, and appearance uncertainty) and integrate them into the agent’s observation space to enable informed decision-making. Concretely, our agent first constructs a Semantic Gaussian Map (SGM), composed of differentiable 3D Gaussian primitives initialized from panoramic observations, that encodes both the geometric structure and semantic content of the environment. On top of SGM, geometric uncertainty is estimated through variational perturbations of Gaussian position and scale to assess structural reliability; semantic uncertainty is captured by perturbing Gaussian semantic attributes to reveal ambiguous interpretations; and appearance uncertainty is characterized by Fisher Information, which measures the sensitivity of rendered observations to Gaussian-level variations. These uncertainties are incorporated into SGM, extending it into a unified 3D Value Map, which grounds them as affordances and constraints that support reliable navigation. Comprehensive evaluations across multiple VLN benchmarks show the effectiveness of our agent.

Published as a conference paper at ICLR 2026
UNCERTAINTY-AWARE GAUSSIAN MAP FOR VISION-LANGUAG NAVIGATION
Jianzhe Gao'
Rui Liu'
Yuxuan Xu'
Tongtong Cao
Yingxue Zhang?
Zhanguang Zhang" Sida Peng*
Yi Yang'
Wenguan Wang 1+
The State Key Lab Brain-Machine Intelligence, Zhejiang University
2 *Department of Foundation model. 2012 Labs, Huawei NNoah's Ark Lab, 2012 Labs, Huawei
"School of Software Technology. Zhejiang University
https: / //gtttb...../ http://aaaeaae//ttttttaattaaaaee
ABSTRACT
Vision-Language Navigation (VLN) requires an agent to navigate 3D environ- ments following natural language instructions. During navigation, existing agents commonly encounter perceptual uncertainty, such insufficient evidence for re- liable grounding or ambiguity in interpreting spatial cues, yet they typically ig- nore such information when predicting actions. In this work, we explicitly model three forms of perceptual uncertainty (i.e., geometric, semantic, and appearance uncertainty) and integrate them into the agent's observation space enable in- formed decision decision-making. Concretely, our agent first constructs a Semantic Gaus- sian Map (SGM), composed of differentiable 3D Gaussian primitives initialized from panoramic observations, that encodes both the geometric structure and se- mantic content of the environment. On top of SGM, geometric uncertainty is estimated through variational perturbations of Gaussian position and scale to as- sess structural reliability; semantic uncertainty is captured by perturbing Gaussian semantic attributes to reveal ambiguous interpretations: and appearance uncer- tainty characterized by Fisher Information, which measures the sensitivity of rendered observations to Gaussian-level variations. These uncertainties incor- porated into SGM, extending it into a unified 3D Value Map, which grounds them affordances and constraints that support reliable navigation. Comprehensive evaluations across multiple VLN benchmarks show the effectiveness of our agent.
INTRODUCTION
Vision-Language Navigation (VLN) requires embodied agents to navigate diverse 3D environments following natural language instructions [1-5]. To achieve robust performance, agents must combine accurate spatial perception with reliable decision-making strategies [6-9].
Early agents adopted sequence-to-seqquence frameworks [1, 10], directly mapping language and vi- sual observations into actions. Later works introduced map-based paradigms that explicitly encoded spatial connectivity via topological graphs [9, 11], incorporated semantic information for object- level reasoning [12-14], and leveraged grid-based [15, 16] or volumetric voxel-based representa- tions [17] to capture 3D structure. For policy learning, agents evolved from pure imitation [18, 19] to hybrid approaches that combine imitation and reinforcement with tailored rewards [10, 20-23]. More recently, several agents have employed world models [24-26] to perform look-ahead planning- Despite these advances, existing agents typically ignore uncertainty in perception when making de- cisions. Their training recipes discourage expressing uncertainty or recognizing unreliable situa- tions, instcad incentivizing them to predict actions regardless of confidence [17]. For instance, as illustrated in Fig. 9, agents may confuse visually similar doors, especially when the interior cues behind them provide insufficient evidence, leading unreliable grounding of the correct target. Be- sides, Fig. O illustrates how occlusions mask critical spatial information, introducing ambiguity in assessing path traversability. Previous agents often fail under such conditions, whereas uncertainty offers valuable cues about the reliability of perception and the feasibility of actions [11, 15, 17].
"Corresponding author

---

Published as a conference paper at ICLR 2026
egents
which target
Preeess opats
Fere the 313tc to treverse
Whit 01-DEnc Fiaaficcln
Koners
ageet
sifer dedaru right
OUT OPART
Figure I: Motivation. Previous VLN agents typically ignore perceptual uncertainty when making decisions. As result, they often confuse visually similar structures (e-g., multiple doors) due to lim- ited interior evidence (Q) and struggle when occlusions obscure spatial cues, leaving traversability ambiguous and causing unsafe or suboptimal paths (9). In contrast, our agent explicitly models and leverages such uncertainty for more reliable navigation. Brighter colors indicate higher uncertainty.
In light of the foregoing discussions, this work explicitly models geometric, semantic, and appear- ance uncertainty in perception and consolidates them into a unified 3D Value Map for reliable navi- gation. First, our agent constructs Semantic Gaussian Map (SGM) that represents the environment as a collection of differentiable 3D Gaussian primitives. Each primitive is initialized from sparse pseudo-lidar point clouds obtained from multi-view RGB-D observations and further enriched with semantic properties based on their object instance or stuff membership in the 3D scene. Second, building on SGM, the agent estimates three forms of perceptual uncertainty in navigation. Geomet- ric uncertainty is modeled through variational inference, which approximates the posterior distribu- tion over position and scale perturbations of Gaussians, thereby assessing structural reliability and enabling the pruning of unreliable primitives. In the same manner, semantic uncertainty is estimated by perturbing the semantic attributes of Gaussians, which reveals ambiguous interpretations and allows the agent down-weight unreliable semantic cues during decision-making Appearance un- certainty reflects the sensitivity of rendered observations to Gaussian-level perturbations, quantified by Fisher Information the reconstruction loss surface curvature around each Gaussian. Third, our agent composes a unified 3D Value Map by transforming these uncertainties into affordances and constraints within its perceptual space, thereby guiding informed and more reliable trajectories.
Our agent is evaluated on three VLN benchmarks, i.e., R2R [1], RxR [27], and REVERIE [28] ($4.2, $4.3). improves SR by 2% and SPL by 1% on R2R, yields 1.1% and 1.7% increases SR and nDTW on RxR with comparable SDTW, and achieves 2.94% and 2.57% higher RGS and RGSPL. scores on REVERTE. Extensive ablation studies confirm the contribution of each component ($4.4).
2 RELATED WORK
Vision-Language Navigation (VLN). Early VLN agents adopted sequence-to--sequnee frame- works that directly map instructions and multi-view observations to actions [1, 10, 29]. Yet such models struggle with long-horizon reasoning and robustness in unseen environments, which has spurred a variety of extensions. As a primary step, subsequent agents introduced explicit memory mechanisms, such as topological graphs [11, 30] or episodic memory buffers [31-33], better re- tain and recall spatial and semantic cues over extended trajectories. Later works further advanced the agent with transformer-basel architectures that jointly encode instruction and observation [25, 34]. Moreover, extensive efforts are devoted to mitigating data limitations through instruction gener- ation [35-39] and synthetic data creation [26]. Several works enable the agent to explore steps forward by anticipating future observations before decision-makingg [24, 25, 40]. To improve pol- icy robustness, the combination of imitation learning [41] and reinforcement learning [42] has been widely adopted in VLN agents [20]. Others aim to reduce computational costs while maintain- ing VLN performance by designing lightweight cross-modal or selective memorization architec- tures [43]. Benefiting from the quality and speed of 3D Gaussian Splatting [44-46], growing line of work has adopted it as the agent's scene representation, showing strong performance [5, 47-49].
In parallel, few recent studies begin to explore uncertainty signals in VLN. For instance, VLN-Copilot [50] estimates decision-level uncertainty from the action distribution to decide when request external large language model assistance. In contrast, our work focuses on perception-
2

---

Published as a conference paper at ICLR 2026
Semontic Gaussion Map
3D Value $3.3
Goussion Representation F
MLT
Page 6, 7,9
(T, DJ
Emioadding X
fegg 11,
Instruction: Turn left and go through the door in front of the bed to enter the bathreom.
Geometric Uncertcinty U1
Semontic Uncertaiity 2
Appearance Uncertointy
Figure 2: Pipeline overview. At each step, our agent constructs a Semantic Gaussian Map ($3.1) from its panoramic observation O = (I, D). On top of this map, it estimates geometric U9, seman- tie U and appearance U uncertainties ($3.2) and embeds them back to obtain a unified 3D Value Map ($3.3) that grounds affordances and constraints. Finally, Gaussian representations F9 derived from the value map are concatenated with the instruction embedding X and fed into multi-layer transformer JELL to predict the next action over candidate waypoints ($3.3).
level uncertainty that arises from the agent's observations and models geometric, semantic, and appearance reliability to support informed decision-making.
Uncertainty Estimation in Deep Learning. Uncertainty estimation has long been recognized as a central challenge in deep learning, with variety of approaches proposed across vision and robotics [51]. A prominent line of work follows the Bayesian paradigm, which characterizes un- certainty through predictive distributions over model parameters, often approximated via variational inference, Laplace approximation, or sampling methods [52-54]. Another common strategy relies ensembling, where multiple models trained with different initializations, data subsets, or hyperpa- rameters are aggregated to approximate appearance uncertainty [55-59]. In parallel, sampling-basee techniques like Hamiltonian Monte Carlo provide asymptotic guarantees but incur prohibitive costs for high-dimensional models [60]. To alleviate this computational burden, several works leverage regularization-batt based approximations, such as Monte Carlo Dropout [61] and its variants [62]. which approximate Bayesian inference with minimal changes to standard training. Recent efforts exploit second-order information, where the Hessian or Fisher Information of the loss surface approxi- mated to assess how sensitive predictions are to parameter variations [63-65].
However, most existing approaches rely on implicit latent representations, where globally entangled features obscure uncertainty estimation and hinder region-specific reasoning. By contrast, the ex- plicit structure of 3D Gaussian Splatting [44] provides a natural and interpretable way to associate physically meaningful attributes (i.e., position, scale, semantics) with each primitive. While re- cent studies have explored this explicit structure for estimating uncertainty, they largely concentrate on novel view synthesis and image reconstruction [66, 67]. In contrast, agent leverages these physically grounded primitives to construct a unified 3D Value Map, which explicitly quantifies uncertainty and encodes it as affordances and constraints to guide navigation.
3 METHOD
Problem Formulation. In VLN, an agent placed 3D scene and required to reach target location [1] (or identify target object (28]) following instructions X. At each step f, the agent receives a panoramic observation composed
multiple RGB views I, = [I,, E B x3 K jkmi
and associated depth maps D.- = (Dr.k E RHH ) Based on these observations, the agent learns navigation policy (a, X', I', Dr) that predicts actions a E A,, which includes navigable neighbor nodes, previously observed nodes accessible via backtracking, and a [STOP] action.
Overview (Fig, 2). At each step, our agent constructs Semantic Gaussian Map (SGM) from multi-view RGB-D observations, where primitives are enriched with semantic properties ($3.1). Building on SGM, the agent models geometric, semantic, and appearance uncertainty to capture the perceptual unreliability in VLN ($3.2). These uncertainties are then integrated into a 3D Value Map, encoding affordances and constraints in the agent's perceptual space for decision-making ($3.3).
3

---

Published as a conference paper at ICLR 2026
3.1 SEMANTIC GAUSSIAN MAP
At each waypoint, the agent transforms multi-view RGB-D observations into a collection of differ- entiable 3D Gaussian primitives, each encoding both geometric and semantic properties. Through differentiable rendering, these primitives are jointly optimized to form a Semantic Gaussian Map (SCM), which serves the foundational substrate for subsequent uncertainty modeling.
Initialization. Given multi-view RGB-D observations O = {I,, Dr} at step t. the agent first generates sparse pseudo-lidar point cloud via camera-to-world transformation. Each pixel (u, v) in Ie,k is back-pne nack-projected into the 3D coordinates V, using its depth De, (u, v) and camera intrinsics (c" c',, f", f where (c",,") are the principal point and (f",, f") are the focal lengths: (v c")=
(r,(u, 2 (u-c")² fr
(I)
fv
These 3D points are then transformed to world coordinates using the camera pose, yielding a sparse point cloud P.. Each point initializes a Gaussian primitive g.. parameterized by position (mean)
μ4 E R3, covariance matrix Σ; E 3x33 opacity a; [0, 1]. spherical harmonics coefficients for color C E RR, and semantic property s; E R (t is omitted for simplicity). In detail, Σ is factorized into scale matrix E and a rotation matrix R as E = REE R where E diag(c", ,e" ,ee))) and R is constructed from a unit quaternion r E R.1. For 8, we apply SAM2 [68] to segment the panoramic observation I into spatially coherent regions {m&) ) and extract their CLIP [69] embeddings, which are then attached to the corresponding Gaussians as new semantic attributes.
Construction. SGM is progressively constructed by optimizing Gaussian primitives through dif- ferentiable rendering, which enforces consistency with the current observation. Specifically, the rendered color î at pixel (u, v) is obtained by a-blending depth- poder-d Gaussians:
i(,, v) = E, c.
d()) RR, a = az exp { n4) " - 14) E RR, (2)
where x' = ((u; v) and μut is the Gaussian center in the image plane, and Σ is the 2D covariance.
Following the same principle, the rendered depth D(u, v) and semantic S(u, 2) are computed as:
(1-03) D(u, = =E 2:0 II -1) ER. S(u, >. 8:05 (1- E E
(3)
Furthermore, Gaussians with small scale often capture irrelevant surface noise, while low-opacity primitives represent negligible background clutter. These Gaussians contribute minimally to the agent's 's spatial comprehension and can potentially introduce misleading cues for its decision decision-making. Therefore, after several rounds of differentiable rendering optimization, retaining only Gaussians subject to the constraints ||e; > Te A a; >> Ta- Consequently, the refined SGM serves foundational substrate for subsequent uncertainty modeling and estimation.
further refine SGM by
3.2 UNCERTAINTY ESTIMATION
SGM provides an explicit 3D map enriched with spatial geometry and semantic context. On top of this map. three forms of perceptual uncertainty are modeled. Geometric uncertainty assesses struc- tural reliability through perturbations of Gaussian position and scale. Semantic uncertainty exposes ambiguous interpretations at the object and region levels by perturbing semantic attributes. Appear- ance uncertainty characterizes inherent visual ambiguity in observations, arising from occlusions, texture inconsistencies, and other uncontrollable factors.
Geometric Uncertainty. To quantify spatial reliability of SGM, we model position and scale pa- rameters of each Gaussian as random variables with learnable perturbations x!" E R33 and xi E R3:
n. p4 x
= + x
(4)
where μ1 and e denote perturbed spatial parameters that encode alternative structural hypotheses of gt. The posterior distribution p(x O) over perturbations x = {xx" x conditioned on observa- nigh-dimensional continuous space.
tions O is generally intractable, as it involves integration To approximate it, like [66], we introduce variational distributions q (x) = {Tor (xf"), gør (xi));
and optimize them by minimizing the Kullback-Leibler (KL) divergence to true posterior p(x
min KL. (gx(g) ll p(x(O)) = log p(O) (Ban (x) ((loog((OO))) dkL (x)
(5)
Evidence Bound (BLBO)
4

---

Published as a conference paper at ICLR 2026
Since log p(O) is constant with respect to X. minimizing KL divergence is equivalent to maximizing ELBO, which serves as the training objective in learning (s(X), In this process, the prior p(x) is defined as zero-mean Gaussian (0, 8 for position perturbations X", and a scale-dependent uniform distribution ((11je, 1je) for scale perturbations x where ó controls the standard deviation of position variances and 1 determines the perturbation range relative to the original scale e.
Based on the learned variational distribution g (x), the geometric uncertainty U of each Gaussian gi is estimated by the variability of its perturbations. In particular, we extract the standard deviations of position and scale perturbations from 96 and aggregate them into scalar score
U. = || 90 (x") + || (aoz (xi))la E R²,
(6)
where (-) (-) is the operation that extracts the standard deviation of the variational distribution.
Semantic Uncertainty. In addition to geometric unreliability, agents also face semantic ambigu- ity, where object- and region-level understanding may be unstable. To capture this, we perturb the semantic attribute 8, of each Gaussian with a learnable offset X E R$, while keeping geometric pa- rameters fixed to preserve spatial consistency. Following the same variational inference framework, we learn a posterior 9s- (x") by maximizing the corresponding ELBO, regularized by a zero-mean Gaussian prior p(x") (0 (T), where e controls the perturbation magnitude. The semantic uncertainty U/ of Gaussian g; is defined as the variability of X, under the posterior q6- (x*):
U = اامر (go- (xi)) E IR.
Appearance Uncertainty. To further capture visual instability, we define appearance uncertainty as the sensitivity of the reconstruction loss L = 2 Ş11Ž- Ill2 to variations in SGM. In principle, such sensitivity is characterized by the Hessian matrix V3.4r [64, 65]. Because computing this matrix directly is infeasible, like [63, 67]. we adopt the Fisher Information as a tractable approximation:
VOLh VoiVei II- Vgi ((())_1)))(()))))))
(8)
Fisher Information
Residual Term
where Vg denotes the gradient of the rendered observations with respect to all Gaussian parameters in 9. VgI represents their second-order derivatives, [9] the number of Gaussians in SGM, and d9 is the feature dimension of each Gaussian. In a refined SGM, where (2 - I) in the Residual Term approaches zero, the Hessian reduces to the Fisher Information, which serves as a tractable proxy of the sensitivity. High Fisher Information reveals that even minor Gaussian shifts induce large variations in the perceptual space, destabilizing both scene understanding and action predictions.
While Fisher Information avoids computing costly second-order derivatives, it still has the same di- mension as the Hessian (i.e., (19) dd) x (19) -d))), which remains computationally expensive. To reduce this cost, we group parameters associated with each Gaussian g, E R yielding a diagonal block of size R.d x within the Fisher Information matrix. Each block isolates the sensitivity of gi, quantifying the impact of its perturbations on the reconstruction loss. Based on this, the appearance uncertainty U/ is defined as the log degemmmtrmin of the corresponding Fisher Information block:
U = log | i V T E RR,
where | denotes the matrix determinant. The log-determinant quantifies the volume of the uncer- tainty ellipsoid in parameter space, yielding a scalar measure of the sensitivity for each Gaussian.
3.3 3D VALUE MAP
To operationalize the estimated uncertainties for navigation, we integrate them into a 3D Value Map. traditional 3D scene reasoning and robotics, a value map represents a spatial field in which each element encodes task-relevant signals that guide downstream decisions, such as affordance fields [70], cost maps [71], and traversability [72] or reliability maps [73]. Following this notion, our 3D Value Map instantiates value field on top of SGM, where each Gaussian is augmented with geometric, semantic, and appearance uncertainty estimates. These uncertainties provide unified reliability eues, which can be naturally interpreted affordances and constraints for navigation.
5

---

Published as a conference paper at ICLR 2026
Construction. By attaching U,, Ug. and U. each Gaussian 9t, extend SGM into 3D Value Map. For ease of notation, we reuse gi to denote the Gaussian representation of this value map:
g: [4: Cs Th og Ci 8,, UY, U. UP = R
(10)
This augmented representation preserves geometric and semantic information while further incor- porating complementary uncertainty measures. Consequently, the 3D Value Map characterizes both the structural and semantic reliability of the environment, grounding affordances and constraints into the agent's observation space to support reliable decision-making.
Action Prediction. Given the instruction embedding X R776, each 9i is nonlinearly projected into feature vector F E 12788, This projection embeds all Gaussian attributes, including ge- ometry, semantics, and uncertainty, into a unified feature space. In this space, the agent maintains a direct correspondence between local geometric structure and its associated uncertainty. F9: are then aggregated into global representation. The aggregated representation F9 preserves the fine- grained coupling between geometry and uncertainty, enabling the agent to make decisions that are directly informed by such structure-aware uncertainty. Consequently, F9 concatenated with X and processed by multi-layer transformer FMI.T [11] to produce candidate node probabilities p:
Softmax( ( ([F9, x()) E [0, 1]M,
(11)
where | is the number of candidate waypoints and [ denotes concatenation. These scores are then aligned with the action space A through nearest-neighbor mapping.VV .o:
p= (p, V) [0, []
(12)
where N denotes the mapping of node-level scores to executable actions via nearest-neighbor search. This fusion enables the agent to jointly reason about geometric structure and perceptual confidence, thereby promoting reliable and uncertainty-aware decision-makinn making.
3.4 LOss FUNCTION
SGM Loss. To supervise SGM construction, we apply pixel-wise rendering loss between the rendered outputs and ground-truth observations. Specifically, we combine L loss and Structural Similarity [74] loss [SSI for color consistency, and apply L for depth and semantic alignment:
C = | I|| + SSIM (I,Z), doopt = | D/1,- _ = ||Š - 5|1,
(13)
where I, D,S denote the ground-truth color, depth, and semantic features, respectively, while 1, D.S S are the corresponding rendered outputs from current SGM.
Navigation Loss. Following the conventional procedure [11, 17, 30], our agent is optimized with two-stage training scheme: pretraining with auxiliary objectives such as masked language model- ing and single-step action prediction to strengthen multimodal representations, and finetuning with behavior cloning and pscudo-expert guidance to refine policy learning. (See details in Appendix.)
3.5 IMPLEMENTATIO DETAILS
To support long-horizon reasoning, similar to [11, 17, 30], our agent main- tains dynamic topological memory that records both visited and navigable nodes as exploration unfolds. Each node is associated with multimodal features, including the 2D panoramic embed- dings and the 3D Value Map representations, while edges encode traversability between locations. This memory forms a graph structure that evolves with the trajectory, enabling the agent to revisit prior viewpoints or evaluate alternative routes when needed. Regions that are ambiguous at the previous node may become more certain when viewed from more informative location, while con- sistently uncertain areas remain marked as unreliable. By jointly storing 2D and 3D information in spatially coherent manner, the memory provides global context that strengthens consistency and stabilizes decision-making in diverse environments, (See more details in Appendix.)
Topological Memory.
Network Pretraining. For R2R [1] and RxR [27], we adopt Masked Language Modeling (MLM) [75, 76] and Single-step Action Prediction (SAP) [21, 76] as auxiliary objectives. For REVERIE [28], we additionally employ Object Grounding (OG) [77] to enhance object-level rea- soning. Pretraining is conducted for 100k iterations with a batch size of 64, optimized by Adam [78] with learning rate of le-4. At each mini-hatch, only one task is sampled with equal probability.

---

Published as a conference paper at ICLR 2026
Table I: Quantitative results on REVERIE [28]. '-": unavailable statistics. See $4.2 for more details.
REVERIC [3]
Method
SR
RGSPL
OSR
RGS
RGSPL
ROMTE
11.35
14:23
9.20
6.37
4.30
3.30
10.60
11.36
7334
6.07
3.67
3.14
FAST-M [2S]
45.28
26:20
14,40
7.19
7.84
4.67
39.05
10.83
19:98
11.61
11.26
6.05
ReCHERT [21]
16.75
35.02
30.67
24.50
15.77
15:27
15.86
$3.91
29.61
23:00
15.50
19.51
Aithert [81]
18.71
34.51
21.89
21.88
15.23
14.18
17.91
34,20
30.28
23.61
15.83
13.24
HAMT [76]
14.08
36.84
$2.95
30.20
15.92
17.28
13.62
33.41
30.40
26.67
14.38
13.00
HOP [52]
16.45
36.24
31.78
26.11
18.85
15.33
16.35
33.06
30.17
24,34
17.69
14.54
DUET
22,11
51.07
46.95
35.23
12.15
23.00
21.30
56.91
52.51
36,06
31,88
20.05
DU AURT-Imagiee [40)
48.25
35.76
30.97
23.25
COSMO [03]
56.09
35.93
59.33
52.53
36.12
GrdMM [166
23.20
57.48
51.39
36.47
34.57
24.56
19.97
59.55
53.15
36,00
34.87
23.65
LANA
52.97
BEVBer
36.41
BSG
05
20.34
VER
58
37.67
36.73
20.05
56.90
16.90
22.351114
36.3
37.6
37.05111
27.01 m031
20.141001
00112113
35,000031
2007
Network Finetuning, Following standard protocol [11], we finetune the pretrained model using DAgger [79]. For REVERIE [28], additional Object Grounding (OG) loss incorporated with weight of 0.20. Finetuning is performed for 25k iterations with a batch size of 8 and a learning rate of 1e-5, The best checkpoint is chosen based on performance of val unseen split.
Testing, At each navigable viewpoint, our agent constructs SGM from panoramic observations and extends into a 3D Value Map for reliable action prediction. This process terminates once the agent reaches the target location or decides to execute the [STOP] action. (See details in Appendix.)
Runtime Analysis. The main overhead arises from constructing the 3D Value Map, particularly semantic attribute extraction in SGM and uncertainty estimation. For training, we mitigate this cost through offline pretraining. During inference, RGB-D observations are resized to 224 x 224, and SAM2 [68] can be flexibly replaced by lightweight variants to trade off segmentation quality against runtime efficiency. Once the 3D Value Map is established, action prediction incurs negligible additional cost compared to existing VLN agents [1]]. (See more details in Appendix.)
EXPERIMENT
4.1 EXPERIMENTAL SETUP
Datasets. We evaluate our agent three benchmarks, each posing distinct challenges for VLN. All datasets are built upon the Matterport33 simulator [80], and are split into train, val -seen, val val wal-unseen, and test sets according scenes. REVERIE [28] provides 21,702 high-level instructions paired with 4, 140 remote target objects. The agent must navigate to the described region and precisely ground the referred object. R2R [1] contains 7,189 shortest-path trajectories from 90 indoor scenes with 22K instructions, where the agent is required to follow detailed step-by-ster directions. RxR [27] offers 126K multilingual instructions (i.e., English, Hindi, Telugu) over 16,522 trajectories, requiring the agent to cope with long- horizon navigation across diverse languages.
Evaluation Metries. We comprehensively evaluate agents using standard metrics [11] across dif- ferent benchmarks. For R2R [1], we report Success Rate (SR), Trajectory Length (TL.), Navigation Error (NE), Oracle Success Rate (OSR), and Success weighted by Path Length (SPL.. For RxR [27], we additionally adopt Normalized Dynamic Time Warping (nDTW) and Success weighted nDTW (SDTW) to assess trajectory fidelity and path alignment. For REVERIE [28], evaluation further considers Remote Grounding Success (RGS) and RGS weighted by Path Length (RGSPL), which measure whether the agent successfully localizes the target object at the correct location.
4.2 QUANTITATIVE COMPARISON RESULT
Our results are averaged over five runs on three datasets, with standard deviations reported.
Performance on REVERIE [28]. Table reports the results REVERIE, which evaluates the agent's 's ability to ground remote target objects given high-level instructions. On the val unseen split, our agent outperforms the best reported results (i.e, BEVBert [15]) by a significant margin in terms of RGS (37.65% vs 34.71%) and RGSPL (27.01% vs 24.44%). These improvements of 3.94% in RGS and 3.31% % in RGSPL clearly demonstrate the effectiveness of our 3D Value Map for accurate navigation and precise object grounding.

---

Published as a conference paper at ICLR 2026
Walk the stairs. tarn left and walt straight towerds the windons right after
Exit the kitchen heading the dining nook, Go aroune the table, and enter from froms. the the and conches.
windows
navigable mode
ours
Finish
VER
Start
(a)
(b)
Figure 3: Qualitative results on R2R [1]. (a) Under the instruction "straight towards the windows" VER [17] misinterprets the layout and stops early, whereas our agent correctly follows the path and reaches the landmark. (b) Our agent bypasses the obstacle and enters the designated region, while VER halts at the "table" without completing the task. See $4.3 for more details.
Walk the red-carputed sinirestee, continur straight deven Bulliway, and stop room with large table.
Figure 4: Representative visual results on R2R [1]. At each step. we show the constructed SGM, the rendered observations, and the aggregated uncertainty map. While SGM captures the geometry and semantic layout, the uncertainty emphasizes ambiguous regions such as reflective surfaces and repetitive structures, offering complementary cues for reliable grounding. See 54.3 for more details.
Performance R2R [1]. As shown in
Table Quantitative results on R2R [1] val unseen.
Table 2, our agent consistently surpasses recent state-of-the-art methods R2R. On the val unseen split, achieves
unavailable statistics. See $4.2 for more details.
RIR[]
stuvee
TL.
SPL.
TL...
an SR of 78% compared to 76% from
HAMT (7))
16
3.10
66
61
13.23
3.95
655
60
DUET [1]]
13.94
3.31
72
60
14.73
3.65
55
VER [17] and improves SPL from 65% to 66%, corresponding
BURT-Imagre [4]]
gains of 2% SR and 1% in SPL. These results clearly high- light the ability of our agent follow de- tailed instructions in unseen environments.
GrdMM
2.53
3335
16.55
BSQ [X]
14.00
2.59
74
62
14.85
3.19
73
62
(ER
16.83
2.40
76
69
19.21
2.76
76
66
3DGS-VLNN VLN
14.83
2.43
66
14.38
3.17
Ops
14 14.00001
2.2
1000
14
76.02)
66:000
Performance on RxR [27]. Table 3 presents the results on RxR, which features longer paths and multilingual instructions. Our agent attains higher SR and nDTW (65.2% 64.1%, 65.6 % vs 63.9%) and comparable SDTW (53.5% vs 52.6%) on val spseen Such improve- ments further demonstrate the benefit of the un- certainty information in long-horizon navigation.
Table 3: Quantitative results on RxR [27] va.
unseen. "-": unavailable statistics. See $4.2.
Method
NF.
SR
nDTW
SDTW
LSTM [27]
10.9
38.9
EavDrop [84]
42.6
55.7
HAMT [76]
56.5
63.1
48.3
EnvEdit [85]
62.8
68.5
54.6
BEVBet [15]
4.6
64.1
63.9
$2.6
Ours
4.2
65.2s022
63.6:1.19
53.5
4.3 QUALITATIVE COMPARISON RESULT
Case Studies. We compare our agent with VER [17] on the R2R val unseen split. In Fig. 3(a), with multiple visually similar "windowss". VER misgrounds the target and deviates early, while our agent resolves the ambiguity and follows the correct landmarks. requires bypassing the "table" and reaching "the corner near the couches" ". VER collides with the table and stops, whereas our agent detours safely and completes the instruction. These cases show how uncertainty helps disambiguate confounding structures and encode traversability constraints.
Fig. 3(b), the instruction
Moreover, Fig. 4 presents a step-wise visualization the agent's 's trajectory. At each step, we show the constructed SGM, the rendered observations, and the aggregated uncertainty map (summing ge-
8

---

Published as a conference paper at ICLR 2026
Observation
56M
Rendered Obs.
Geometric Unc.
Semantic Unc.
Appearance Unc.
Figure 5: Visualization of diverse perceptual forms. From left to right: current observation, SGM, rendered observation, geometric uncertainty map, semantic uncertainty map, appearance uncertainty map. Brighter colors indicate higher uncertainty. See $4.3 for more details.
ometric, semantic, and appearance components). SGM captures the geometry and semantic layout, while the uncertainty highlights visually ambiguous regions such as reflective surfaces, chandeliers, and repetitive structures. These complementary views demonstrate how our agent grounds the in- struction throughout navigation and show that uncertainty provides additional information.
Visualization. Fig. 5 illustrates our diverse perceptual forms. i) SGM preserves detailed geometric structures while maintaining high-fidelity rendering of the scene. ii) Geometric uncertainty reveals structural reliability, particularly highlighting uncertain boundaries and irregular surfaces. iil Se- mantic uncertainty exposes ambiguity in object- and region-level interpretations, reflecting unsta- ble semantic cues. iv) Appearance uncertainty highlights regions where rendered observations are highly sensitive to visual variations, e.g., texture complexity, occlusions, or lighting variations.
4.4 DIAGNOSTIC EXPERIMENT
For thorough examination, we conduct a series of ablative studies on the val unseen split of R2R [1] and REVERIE [28].
Key Component Analysis. We first study the efficacy of the core components our framework, i.e. SGM ($3.1) and 3D Value Map (3DVM, $3.3). In Table 4, row #1 gives the performance of our base agent DUET [L]]. For row #2, the scores are obtained by using SGM as the 3D scene representation without uncertainty values. In contrast, row #3 leverages only the uncertainty information (i.e.,
U9, Uo. U as the 3D scene representa- tion, without the raw Gaussian parameters. Row #4 reports the scores of our full frame- work. i) Row #1 #2: SGM leads to table performance improvements against the baseline (e.g.. 32.15% + 35.48% RGS on REVERIE). This demonstrates that the agent benefits from the geometric structure and semantic cues within SGM, achieving stronger navigation performance. ii) Row #1 vs #3: the uncertainty information boosts the performance of the baseline (e.g., 72.22% + 74.20% SR on R2R), which indicates that perceptual uncertainty inherent nav- igation encodes informative cues that assist navigation decisions. iii) Row #2 vy #3: Explicit 3D structure with contextual awareness provides a stronger foundation for navigation than uncertainty alone (e.g., 35.48% 34.02% RGS on REVERIE). iv) Row #1 vs #4: Combining all contributions results in the largest gain over baseline, which confirms the effectiveness of our overall design.
Table 4: Ablation studies on val unseen split of
R2R [1] and REVERIE [28]. See $4.4 for more details.
Components
R2R [1]
REVERIE [2]]
SGM 3DVM
SR
SPL.
SR
RG8
RGSPL
60.41
46.98
32.15
2
76.21
64.57
50.20
35.48
25.64
74.20
62.89
49.12
34,02
78.32
66.47
53.37
37.65
27.01
Analysis on SGM ($3.1). We in- vestigate how the scale of SGM (i.e., the number of Gaussians) af-
Table 5: Effectiveness of Te and To on val unseen splits of
R2R [1] and REVERIE [28]. Nial are Gaussian count within SGM.
fects navigation performance. To control SGM scale, we apply prun- ing thresholds Te and To to fil- ter out Gaussians with small scale
Te
Pruning
N'l 4
FPS
R2R [|]
SPL
SR
REVERIE [28]
RGS
RGSPL
0.00
50,000
11.2
77.30
63.26
52,00
35.00
0.01
0.002
45,000
13.1
77.87
64,80
52.70
35.30
27.00
0.015
42,000
15.5
66.47
53.37
37.65
27.01
0.02
0.010
35,000
18.7
74.80
61.68
46.50
32.30
24.80
([ee)ll < Te) or low opacity (a;; < <.). as these typically represent noise or irrelevant background
clutter. Table shows that, i) Slightly removing low-contribution Gaussians improves action accu- racy (Row #2). ii) Moderate additional pruning yields clear rendering speedups while maintaining competitive accuracy (Row #3). tii) Aggressive removal markedly degrades performance (Row #4).

---

Published as a conference paper at ICLR 2026
Analysis on 3D Value Map ($3.3). In Ta- ble 6, we investigate the contribution of dif- ferent uncertainty types in our 3D Value Map. Row #1 utilizes SGM as the 3D scene repre- sentation. i) Row #1 (#2 or #3): Consistent performance gains appear when incorporat- ing any form of perceptual uncertainty, con- firming that such signals provide useful guidance for navigation. iî) Row #2 vs #3: Geometric and semantic uncertainty contribute richer navigational cues than appearance uncertainty, as the agent benefits more from recognizing uncertain spatial structure or semantic interpretation than from sen- sitivity visual rendering. üi) Configuration with uncertainties achieves the best performance, highlighting their complementary roles.
Table 6: Effectiveness US, U U on val unseen
of R2R [1] and REVERIE [28]. See $4.4 for more details.
Uncertainty UP
R2R
REVERIE
0"
SR T
SPL
SR 1
RGS
RGSPL ?
7621
61.37
50.20
35.48
25.64
4
77.05
65.12
51.82
36.96
26.11
65.31
78.32
66.47
53.37
37.65
27.01
CONCLUSION
This work presents framework for Vision-Language Navigation that explicitly models geometric, semantic, and appearance uncertainty on top of a Semantic Gaussian Map. By integrating these uncertainties into unified 3D Value Map, our agent grounds affordances and constraints into its perceptual space and achieves more reliable decision-making REVERIE demonstrate consistent improvements over strong baselines, while qualitative analyses further validate the effectiveness of our secertainty-wwar design.
Experiments across R2R, RxR, and
ACKNOWLEDGEMENT
This work supported by Zhejiang Provincial Natural Science Foundation of China (No. LR26F020002), Fundamental Research Funds for the Central Universities (226-2025-00057,, and the State Key Laboratory of Brain Cognition and Braininspired Intelligence Technology (No. SKLBI-K202504)..
REFERENCES
[1] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce Mark Johnson, Niko Sünderhauf, Ian D. Reid, Stephen Gould, and Anton van den Hengel. Vision-and -nna-aua- navigation: Interpreting visually- grounded navigation instructions in real environments. In CVPR. 2018. 1. 2, 3. 6, 7, 8, 9, 10, 18, 19, 22, 24
[2] Chen Gao, Xingyu Peng, Mi Yan, He Wang, Lirong Yang, Haibing Ren, Hongsheng Li, and Si Liu. Adaptive zone-aware hicrarchical planner for vision-language navigation. In CVPR, 2023.
[3] Dong An, Hanqing Wang, Wenguan Wang, Zun Wang. Yan Huang. Keji He, and Liang Wang. Etpnav: Evolving topological planning for vision-language navigation in continuous environments, IEEE TPAMI, 2024.
[4] Rui Liu, Wenguan Wang, and Yi Yang. Vision- laiiunla navigation with energy- based policy. In NeurlPS, 2024.
[5] Jianzhe Gao, Rui Liu, and Wenguan Wang. 3d gaussian map with open-set semantic grouping for vision- language navigation. In ICCV, 2025. 1, 2, 7,
[6] Danny Driess, Ingmar Schubert, Pete Florence, Yunzhu Li, and Mare Toussaint. Reinforcement learning with neural radiance fields. NeurlPS, 2022.
Hanqing Wang. Wei Liang, Lue Gool, and Wenguan Wang. Towards versatile embodied navigation. In NeurlPS, 2022.
[8] Hanqing Wang, Wei Liang, Jianbing Shen, Lue Van Gool, and Wenguan Wang. Counterfactual cycle- consistent learning for instruction following and generation in vision-language navigation. In CVPR, 2022.
[9] Xichen Ding, Jianzhe Gao, Cong Pan, Wenguan Wang, and Jie Qin. History-enhanced two-stage trans- former for acrial vision-and-language navigation. In AAAI, 2026.
10

---

Published as a conference paper at ICLR 2026
[10] Hao Tan, Licheng Yu, and Mohit Bansal. Learning to navigate unseen environments: Back translation with environmental dropout. In NAACL, 2019. 1,2
[11] Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Think global, act local: Dual-scale graph transformer for vision-and-language navigation. CVPR, 2022. 1, 2, 6, 7, 8, 9, 18, 19, 20, 22
[12] Yicong Hong, Yang Zhou, Ruiyi Zhang, Franck Demonncuur, Trung Bui, Stephen Gould, and Hao Tan. Learning navigational visual representations with semantic map supervision. In CVPR, 2023. 1
[13] Hanqing Wang. Wenguan Wang, Wei Liang. Steven CH Hoi, Jianbing Shen, and Luc Van Gool. Active perception for visual-language navigation. IJCV, 131(3):607-66 2023.
[14] Chen Gao, Si Liu, Jinyu Chen, Luting Wang, Qi Wu, Bo Li, and Qi Tian. Room-object entity prompting and reasoning for embodied referring expression. IEEE TPAMI, 46(2):994- 2023.
[15] Dong An, Yuankai Qi, Yangguang Li, Yan Huang, Liang Wang. Tieniu Tan, and Jing Shao. Bevbert: Multimodal map pre-training for anguagg-gguide navigation. ICCV, 2023. 1, 7, 8, 19, 20, 22
[16] Zihan Wang, Xiangyang Jiahao Yang. Yeqi Liu, and Shuqiang Jiang. Gridmm: Grid memory map for vision-and-languue navigation. In ICCV, 2023. 7,
[17] Rui Liu, Wenguan Wang. and Yi Yang. Volumetric environment representation for vision-language navi- gation. In CVPR, 2024. 1, 6, 7, 8, 18, 20, 22
[18] Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural computation, 3(1), 1991.
[19] Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In ICMI, 2004. 1
[20] Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang. William Yang Wang, and Lei Zhang. Reinforeed cross-modal matching and self supervveed imitation learning for Vision-language navigation. In CVPR, 2019. 1,2,7
[21] Vicong Hong, Qi Wu, Yuankai Qi, Cristian Modriguez-Opaza and Stephen Gould. VIn bert: A recurrent vision-and-language hert for navigation. In CVPR, 2021. 6,7
[22] Hanqing Wang. Wenguan Wang, Tianmin Shu, Wei Liang, and Jianbing Shen. Active visual information gathering for vision-language navigation. In ECCV, 2020.
[23] Yusheng Zhao, Jinyu Chen, Chen Gao, Wenguan Wang. Lirong Yang. Haibing Ren, Huaxia Xia, and Si Liu. Target-driven structured transformer planner for vision-language navigation. In ACM MM, 2022.
[24] Jing Yu Koh, Honglak Lee, Yinfei Yang, Jason Baldridge, and Peter Anderson. Pathdreamer: world model for indoor navigation. ICCV, 2021. 1, 24
[25] Hanqing Wang, Wei Liang, Lue Van Gool, and Wenguan Wang. Dreamwalker: Mental planning for continuous lision-language navigation. In ICCV, 2023. 2
[26] Amir Bar, Gaoyue Zhou, Danny Tran, Trevor Darrell, and Yann LeCun. Navigation world models. In CVPR, 2025. 1.2
[27] Alexander Ku, Peter Anderson, Roma Patel, Eugene Ic, and Jason Baldridge. Room-across-room: Multi- lingual vision-and-le and-language navigation with dense spatiotemporal grounding. In EMNLP, 2020. 2, 6, 7, 8, 18,
[28] Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang. William Yang Wang, Chunhua Shen, and Anton den Hengel, Reverie: Remote embodied visual referring expression real indoor environments. CVPR, 2020. 2, 3, 6, 7,9, 9, 10, 18, 24
[29] Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Korg-Kirrpatrick, Kate Sacnko, Dan Klein, and Trevor Darrell, Speaker-follower models for aaioo-and-language navigation. In NeurIPS, 2018. 2
[30] Rui Liu, Xiaohan Wang, Wenguan Wang, and Yi Yang. Bird's-eye-view scene graph for lision-language navigation. In ICCV, 2023. 2, 7,8
11

---

Published as a conference paper at ICLR 2026
[31] Zhiwei Deng. Karthik Narasimhan, and Olga Russakovsky. Evolving graphical planner: Contextual global planning for vision-and lang-ang navigation. In NeurlPS, 2020. 2
[32] Yujing Chen, Jinmin Zhang, and Yunjiang Lou. Topological and semantic map generation for mobile robot indoor navigation. In ICIRA, 2021.
[33] Hanqing Wang, Wenguan Wang. Wei Liang, Caiming Xiong, and Jianbing Shen. Structured scene mem- ory for vision-language navigation. In CVPR, 2021. 2
[34] Yicong Hong, Cristian Rodriguez, Yuankai Qi, Qi Wu, and Stephen Gould. Language and visual entity relationship graph for agent navigation. In NeurIPS, 2020.
[35] Dewei Zhou, You Li, Fan Ma. Xiaoting Zhang, and Yi Yang. Mige: Multi-instance generation controller for text-to-image synthesis. In CVPR, 2024.
[36] Xianghao Kong. Jinyu Chen, Wenguan Wang. Hang Su, Xiaolin Hu, Yi Yang, and Liu. Controllable navigation instruction generation with chain of thought prompting. In ECCV, 2024.
[37] Sheng Fan, Rui Liu, Wenguan Wang, and Yang. Navigation instruction generation with bev perception and large language models. In ECCV, 2024.
[38] Sheng Fan, Rui Liu, Wenguan Wang. and Yi Yang. Scene map-based prompt tuning for navigation instruction generation. In CVPR, 2025.
[39] Zun Wang, Jialu Li, Yicong Hong, Songze Li, Kunchang Li, Shoubin Yu, Yi Wang. Yu Qiao, Yali Wang. Mohit Bansal, et al. Bootstrapping language-guided navigation learning with self-refining data flywheel. In ICLR, 2025. 2
[40] Akhil Perincherry, Jacob Krantz, and Stefan Lee. Do visual imaginations improve vision and languae navigation agents? In CVPR, 2025. 7,8
[41] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and-languaae navigation: Interpreting visually-gronnded navigation instructions in real environments. In CVPR, 2018.
[42] Xin Wang, Wenhan Xiong, Hongmin Wang, and William Yang Wang. Look before you leap: Bridging model-free and model-hased reinforcement learning for planned-ahead asiionaand-language navigation. In ECCV, 2018. 2
[43] Siqi Zhang, Yanyuan Qiao, Qunbo Wang. Zike Yan, Qi Wu, Zhihua Wei, and Jing Liu. Cosmo: Com- bination of selective memorization for low-cost
navigation. In ICCV, 2025. 2, 7,
[44] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM TOG, 42(4), 2023. 2, 3
[45] Rui Liu, Zhibo Duan, Jianzhe Gao, Yi Yang, and Wenguan Wang. Uncertainty-aware 3d reconstruction for dynamic underwater scenes. In ICLR, 2026.
[46] Yucheng Yan, Chen Liang, Wenguan Wang, and Yi Yang. Open-set semantic gaussian splatting slam with expandable representation. In ICLR, 2026. 2
[47] Xiaohan Lei, Min Wang, Wengang Zhou, and Houqiang Li. Gaussnav: Gaussian splatting for visual navigation. IEEE TPAMI, 2025. 2
[48] Tuo Feng, Wenguan Wang, and Yi Yang. Caussian-basee world model: Gaussian priors for voxel-based occupancy prediction and future motion prediction. In ICCV, 2025.
[49] Rui Liu, Sheng Fan, Wenguan Wang, and Yi Yang. Underwater visual slam with depth uncertainty and medium modeling. In ICCV, 2025. 2
[50] Yanyuan Qiao, Qianyi Liu, Jiajun Liu, Jing Liu, and Qi Wu. Llm as copilot for coarse-grained vision- and-language navigation. In ECCV, 2024.
[S1] Bálint Mucsányi, Michael Kirchhof, and Seong Joon Oh. Benchmarking uncertainty disentanglement: Specialized uncertainties for specialized tasks. In NeurlPS, 2024.
[52] Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012. 3
12

---

Published as a conference paper at ICLR 2026
[53] Jianxiong Shen, Adria Ruiz, Antonio Agudo, and Francesc Moreno-Noguer. Stochastic neural radiance fields: Quantifying uncertainty implicit representations. In 3DV, 2021.
[54] Jianxiong Shen, Antonio Agudo, Francese Moreno-Noguee and Adria Ruiz. Conditional flow nerf: Ac- curate 3d modelling with reliable uncertainty quantification. In ECCV, 2022.
[55] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive un- certainty estimation using deep ensembles. In NewrIPS, 2017. 3
[56] Jeremiah Liu, John Paisley, Marianthi-Anna Kioumourtzaglou, and Brent Coull. Accurate uncertainty estimation and decomposition in ensemble learning. In NewrlPS, 2019.
[57] Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov. and Dmitry Vetrov. Pitfalls of in-domain un- certainty estimation and ensembling in deep learning. In ICLR, 2020.
[58] Niko Sünderhauf, Jad predictive uncertainty in neural radiance fields. ICRA, 2023.
and Dimity Miller. Density- aware nerf ensembles: Quantifying
[59] Anshuk Uppal, Kristoffer Stensbo-Smidt, Wouter Boomsma, and Jes Frellsen. Implicit variational infer- ence for high-dimensional posteriors. In NeurlPS, 2023. 3
[60] Yunhum Kook, Yin-Tat Lee, Ruoqi Shen, and Santosh Vempala. Sampling with riemannian hamiltonian monte carlo in a constrained space. In NewrlPS, 2022.
[61] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncer- tainty in deep learning. In ICML, 2016. 3
[62] Durk P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparamcterization trick. In NeurlPS, 2015. 3
[63] Lily Goli, Cody Reading, Silvia Sellán, Alec Jacobson, and Andrea Tagliasacchi. Bayes' rays: Uncer- tainty quantification for neural radiance fields. In CVPR, 2024. 3,5
[64] Marco Miani, Lorenzo Beretta, and Søren Hauberg. Sketched lanczos uncertainty score: a low-memory summary of the fisher information. In NewrlPS, 2024. 5
[65] Kiwan Maeng, Chuan Guo, Sanjay Kariyappa, and Edward Suh. Bounding the invertibility of privacy- preserving instance encoding using fisher information. In NeurlPS, 2023. 3,5
[66] Ruiqi Li and Yiu-ming Cheung. Variational multi-scale representation for estimating uncertainty in 3d gaussian splatting. In NewrIPS, 2024. 3,4
[67] Alex Hanson, Allen Tu, Vasu Singla, Mayuka Jayawardhana. Matthias Zwicker, and Tom Goldstein. Pup 3d-gs: Principled uncertainty pruning for 3d gaussian splatting. In CVPR, 2025. 3,5
[68] Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024. 4, 7,
[69] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transfcrable visual models from natural language supervision. In ICML, 2021. 4
[70] Xiaohan Wang, Yuehu Liu, Xinhang Song. Yuyi Liu, Sixian Zhang, and Shuqiang Jiang. An interactive navigation method with effect-oriented affordance. In CVPR, 2024. 5
[71] Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Compos- able 3d value maps for robotic manipulation with language models. In CoRL, 2023.
[72] Yuxiang Yang, Xiangyun Meng, Wenhao Yu, Tingnan Zhang, Jie Tan, and Byron Boots. Learning semantics-aware locomotion skills from human demonstration. In CoRL, 2023.
[73] Kashu Yamazaki, Taisci Hanyu, Khoa Vo, Thang Pham, Minh Tran, Gianfranco Doretto, Anh Nguyen, and Ngan Le. Open-fusion Real-time open- vocabulayyul 3d mapping and queryable scene representation. In ICRA, 2024. 5
[74] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE TIP, 13(4):600-612, 2004.
13

---

Published as a conference paper at ICLR 2026
[75] Jacob Devlin Ming-Wei Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. In NAACL, 2019.
[76] Shizhe Chen, Pierre-Lour Guhur, Cordelia Schmid, and Ivan Laptev. History aware multimodal trans- former for vision-and-lanuuee navigation. In NeurlPS, 2021. 6, 7,8
[77] Xiangru Lin, Guanbin Li, and Yizhou Yu. icene-intuitive agent for remote embodied visual grounding. In CVPR, 2021. 6
[78] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, ICLR, 2015. 6
[79] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. reduction imitation learning and structured prediction to no-regret online learning. In AISTATS, 2011. 1. 7, 18
[80] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niebner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. In 3DV. 2017. 24
[81]
Pierre-Louis Guhur, Makarand Tapaswi, Shizhe Chen, Ivan Laptev, and Cordelia Schmid. Airbert: In- domain pretraining for vision-and-language navigation. In ICCV, 2021. 7
[82] Yanyuan Qiao, Yuankai Qi, Yicong Hong, Zheng Yu, Peng Wang, and Qi Wu. Hop: hiddor aware pre-training for vision-and- hainn-and-language navigation. CVPR, 2022. 7
[83] Xiaohan Wang, Wenguan Wang, Jiayi Shao, and Yi Yang. Lana: language-capable navigator for instruction following and generation. In CVPR, 2023. 7,8
[84] Sheng Shen, Liunian Harold Li. Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can clip henefit assion-and-languaee tasks? In ICLR, 2021. 8
[85] Jialu Li, Hao Tan, and Mohit Bansal. Envedit: Environment editing for vision-and-language navigation. In CVPR, 2022. 8
[86] Chaoning Zhang, Dongshen Han, Yu Qiao, Jung Uk Kim, Sung-Ho Ho Bac, Seungkyu Lee, and Choong Scon Hong. Faster segment anything: Towards lightweight sam for mobile applications. arXiv preprint arXiv:2306. 14289, 2023. 19
[87] Stefan Depeweg, Jose-Migue) Hernandez-Lobate Finale Doshi Velez. and Steffen Udluft. Decomposi- tion of uncertainty in bayesian deep learning for efficient and risk-sensitive learning. In ICML, 2018 21
[88] Hao Li, Jingkuan Song, Lianli Gao, Xiaosu Zhu, and Hengtao Shen. Prototype-based aleatoric uncertainty guantification for cross cross-modal retrieval. In NeurlPS, 2023.
[89] Matthew Chan, Maria Molina, and Chris Metzler. Estimating epistemic and alcatoric uncertainty with a single model. In NeurlPS, 2024. 21
[90] Xiaoyi Cai, Siddharth Ancha, Lakshay Sharma, Philip R Osteen, Bernadette Bucher, Stephen Phillips, Ji- uguang Wang, Michael Everett, Nicholas Roy. and Jonathan P How. Evora: Deep evidential traversability learning for risk-aware off-road autonomy. IEEE Transactions Robotics, 2024. 21
[91] Xiaoyi Cai, Michael Everett, Jonathan Fink, and Jonathan P How. Risk-aware off-road navigation via a learned speed distribution map. In IEFF/RSI IROS, 2022. 21
[92] Parker Ewen, Adam Li, Yuxin Chen, Steven Hong. and Ram Vasudevan. These maps are made for walking: -ime- terrain property estimation for mobile robots. IEEE Robotics and Automation Letters, 7(3)70083-7000, 2022. 21
[93] Nikhil Keetha, Jay Karhade, Krishna Murthy Jatavallabhula, Gengshan Yang. Sebastian Scherer, Deva Ramanan, and Jonathon Luiten. Splatam: Splat track & map 3d gaussians for dense rgb-d slam. In CVPR, 2024. 22, 24
[94] Shubo Liu, Hongsheng Zhang, Yuankai Qi, Peng Wang, Yanning Zhang, and Qi Wu. Acrialvin: Vision- and-language navigation for uavs. In ICCV, 2023. 24
[95] Karmesh Yadav, Jacob Krantz, Ram Ramrakhya, Santhosh Kumar Ramakrishnan, Jimmy Yang. Austin Wang, John Turner, Aaron Gokaslan, Vincent-Pierre Berges, Roozbeh Mootaghi, Oleksandr Maksymets, Angel X Chang, Manolis Savva, Alexander Clegg, Devendra Singh Chaplot, and Dhruv Batra. Habitat challenge 2023. https:/ / / habi tat..rgrrrr. cha Tenge/2023/, 2023. 24
14

---

Published as a conference paper at ICLR 2026
[96] Ziyu Zhu, Xilin Wang, Yixuan Li, Zhuofan Zhang, Xiaojian Ma. Yixin Chen, Baoxiong Jia, Wei Liang, Qian Yu, Zhidong Deng, Siyuan Huang, and Qing Li. Move understand scene: Bridging visual grounding and exploration for efficient and versatile embodied navigation. ICCV, 2025. 24
[97] Ziyi Yang. Xinyu Gao, Zhou, Shaohui Jiao, Yuqing Zhang, Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. CVPR, 2024, 24
[98] Guanjun Wu, Taoran Yi, Jiemin Fang. Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In CVPR, 2024. 24
[99] Nico Messikommer, Giovanni Cioffi, Mathias Gehrig, and Davide Scaramuzza. Reinforcement learning meets visual odometry. In ECCV, 2024,
15

---

Published as a conference paper at ICLR 2026
SUMMARY OF THE APPENDIX
This appendix contains additional details for the ICLR 2026 submission, titled -nwer
Gaussian Map for Vision-Language Navigation. The appendix is organized follows:
A summarizes the notations used throughout the framework.
B reports additional model details.
$C gives more runtime analysis.
$D covers additional experiments.
SE offers discussion of our uncertainties and failure cases.
SF provides discussion of the limitations and future, societal impact, terms of use, privacy, and license, and use of large language models.
16

---

Published as a conference paper at ICLR 2026
A
LIST OF SYMBOLS.
Table 7 concisely lists the symbols, excluding unnecessary subseripts for clarity.
Description
X
Natural language instructions
83
RGB images step
$3.1; Eq. (
D.
Depth images step
$3.1; Eq. (2)N())
O
Multi-view RGB-D observations step f
$3.1
P
Sparse point cloud from observations
$3.1;
A
Predicted action at step t
$3
M;
(mean)
primitive 1
E
Covariance matrix of Gaussian primitive
$3.1
a,
Opucity of Gaussian primitive
$3.1: Eq. (200))
Color (spherical harmonics) Gaussian primitive
1 2
Semantic property of Gaussian primitive
$3.1: Eq (&&())
Scale of Gaussian primitive
R.
Rotation mattix Gaussian primitive
$3.1
Unit quaternion for rotation of Gaussian primitive
$3.1 Eq ()ON&
g
Gaussian primitive representation
I
Rendered RGB image
$3.1; Eq. (1)3())&(13)
Rendered depth map
83.1; 83.1; (3)((13)
F
Rendered semantic feature
83.1; Eg. (3)8(13)
Perturbation for position of Coussian
$3.2; Eq., (4)
X
Perturbation for scale of Gaussian i
$3.2; Eq. (4)
XC
Perturbation semantic
$3.2; Eq.
Variational distribution for perturbations
$3.2; Eq. (5)
Ber" (xr""""
Variational distribution for position perturhation Gaussian
$3.2; Eq. (6)
9e: (c.)
Variational distribution scale perturbation of Gaussian
$3.2: Eq. (6)
Geometric uncertainty Gaussian
$3.2; Eq,
Semantic uncertainty of Gaussian
$3.2; Eq. (7)
Appearance uncertuinty Gaussian
$3.2; Eq
X
Instruction embedding
$3.3; Eq. (11)2(12)
Fo-
Projected feature of Gaussian
$3.3; Eq. (11)
Fw
Aggregated Gaussian representation
Eq:
p
Candidate node probabilities
$3.3: Eq. (11)
p
Action probabilities after mapping
$3.3; Eq. (12)
RGB rendering Joss
$3.4; Eq. (13)
Clkph
Depth sendering loss
Eq
geen
Semantic rendering loss
$3.4; Eq. (13)
Subscript 1 denotes the navigation step.
Table 7: Notation and Description of Key Symbols.
17

---

Published as a conference paper at ICLR 2026
B MODEL DETAILS
Our approach built upon the 2D observation-basee baseline DUET [11], and the proposed 3D) Value Map serves as an additional decision branch on top of it. In this part, complement the details by introducing: i) 2D Action Score and ii) Navigation Losses.
B.1 2D ACTION SCORE
Besides the 3D Value Map branch, we retain the 2D perception pathway inherited from DUET [11], which leverages 2D panoramic observations to guide navigation. The panoramic views and detected objects are first encoded by a multi-layer transformer (MLT) into 2D visual embeddings FED E 7768 These features then concatenated with the instruction embedding X E R 168 through another transformer head JMLT to yield 2D action scores:
and passed
2D Softmax ([F2), x()) E []
(14)
where [.] denotes concatenation and V is the number of candidate viewpoints. Next, we apply a nearest-neighhor function N aggregate p2D across neighboring nodes V in topological memory:
DdD VN 2D v) E [0.1]
(15)
This operation merges scores from spatially adjacent nodes and outputs unified value for each candidate, thereby aligning the predictions with the action space A.
B.2 NAVIGATION LOSSES
Following standard protocol [11, 17], our training follows a two-stage paradigm: pretraining with auxiliary objectives to enhance multimodal representations, and fine-tuning with behavior cloning and pseudo-expert guidance to refine navigation policy.
In the pretraining stage, three objectives used depending on the benchmark. For R2R [1] and RxR [27], we include Masked Language Modeling (MLM) and Single-step Action Prediction (SAP) tasks. For REVERIE [28], further incorporate Object Grounding (OG) support precise local- ization of target objects. The corresponding losses are formulated as:
C MLM log (,,,,,
(16)
SAP
log p(a] K. Ree),
(17)
0G - log |X R)),
(18)
where X' is the instruction sequence, W/ randomly masked token, and X,; its remaining context. R denotes the trajectory, with Ret indicating the path prefix. The expert action at step is az, and o" represents the target object.
During fine-tuning, we adopt DAgger [11, 79], which alternates between agent rollouts and pseudo- expert corrections. The pseudo-expert leverages the partially constructed topological memory to generate shortest-paie -aath guidance, allowing the agent to recover from suboptimal actions and gradu- ally improve its policy in unseen environments.
18

---

Published as a conference paper at ICLR 2026
C
RUNTIME ANALYSIS
Table 8: Runtime Analysis on R2R [1] val unseen. Our feature time is decomposed into SGM, variational inference (VD,, and Fisher Information (FI) estimation. Wall-clock Time measures per- step latency, Mem. is peak GPU memory usage, and FLOPs measure computational complexity
Method
Wall-clock Time (5)
Mem. (GB)
FLOPs (G)
R2R
Action
Action
BEVBen [[S]
2,41
15.71
25.0
41.37
DUET [1]]
0.42
0.89
1.31
3.13
2.71
18.68
21.39
72
Ours (MobileAAM)
0.62
0.11
0.19
0.92
1.71
2.63
3.45
3.82
20.53
24.35
Ours
1.47
0.11
0.19
1.77
1.71
3.48
3.84
4.36
20.53
24.89
78
C.I RUNTIME ANALYSIS ON THE OVERALL DESIGN
Table reports the runtime decomposition terms of wall-clock time, memory usage, and FLOPs across different components R2R [1] val unseen split in inference. Compared to DUET [11], our agent achieves substantial performance gains (+6% SR) with only modest increases in runtime (+ (+1s,5s), memory (+0.71GB) and FLOPs (+2.96G), highlighting the efficiency of our design.
C.2 RUNTIME ANALYSIS SGM
As shown in Table. 8, the majority of overhead arises from constructing SGM, dominated by se- mantic attribute extraction with SAM2 [68] (1.47s). Owing to the flexibility our framework, SAM2 can be seamlessly replaced with lightweight variants (e.g., MobileSAM [86]), enabling flex- ible trade-offs between segmentation quality and runtime efficiency (e.g.. 78% 76% SR with 1.47s+0.62s) This flexibility allows agent adapt different deployment scenarios.
C.3 RUNTIME ANALYSIS ON VARIATIONAL INFERENCE
Table 9: Runtime Analysis of Variational Inference (VI) on R2R [1] val unseen.
Perturbed Parameters
Inference Time (8) +
SR
SPL. f
None (w/o VI)
60
Position + Scale
0.08
77
65
Semantic Only
0.07
All SPos..)cale+Sem.)
0.11
78
66
As illustrated Table. 8, VI introduces only minimal cost (0.11s), since it perturbs Gaussian param- eters with lightweight noise and updates variational distributions. In addition, to further assess the efficiency of VI, we measure per-step inference time and performance on R2R [1] val unseen split in Table 9. We can observe that perturbing only spatial parameters (i.e., position and scale) or only semantic attributes incurs negligible overhead (0.08s and 0.07s, respectively). When applied jointly, VI maintains similarly low cost (0.11s) while yielding the best navigation performance.
C.4 RUNTIME ANALYSIS ON FISHER INFORMATION ESTIMATION
Fisher Information (FI) estimation emerges as a lightweight component in our framework, requiring only 0.19s per step (Table 8). This efficiency stems from approximating as the outer product of first-order gradients, which circumvents the costly computation of the full Hessian. Furthermore, we adopt block-diagonal approximation at the Gaussian level, isolating sensitivity within each primitive and further reducing complexity.
19

---

Published as a conference paper at ICLR 2026
D ADDITIONAL EXPERIMENTS
This section presents supplementary experiments, including hyperparameter sensitivity analysis and statistical significance tests.
D.1 HYPERPARAMETER EXPERIMENTS
Table 10: Sensitivity Analysis of uncertainty-related hyperparameters R2R va unseen split. (a) 6 and (b) regulate geometric uncertainty, while (c) governs semantic uncertainty.
(a) Sensitivity to
(b) Sensitivity 7
(c) Sensitivity to
SR 1
SPL
SR
SPL
SR 1
SPL
0.0015
78.27
66.41
0.05
78.30
66.45
0.0015
78.28
66.42
0.002
78.29
66.44
0.1
78.32
66.47
0.002
78.30
66.44
78.32
66.47
0.15
78.30
66.44
78.32
66.47
0.003
78.30
66.46
0.003
78.29
0.2
78.28
66.41
0.0035
78.28
66.48
0.0035
78.27
66.43
0.005
78.25
66.40
0.005
78.25
66.39
We evaluate the sensitivity of the three uncertainty-related hyperparameters on R2R val unseen split: and 17, which regulate geometric uncertainty, and E, which governs semantic uncertainty. The default settings used in our agent are 8= 0.0025, 0.1, and == 0.0025. As shown in Table 10, the agent maintains similar performance when varying within 0.0015-0.005, y) within 0.05-0.2, and within 0.0015-0.000 demonstrating that our uncertainty estimation is stable over broad range of paramcter values.
D.2 STATISTICAL SIGNIFICANCE TESTS
Table 11: Statistical Significance Tests on R2R val unseen split. We report the mean + std, confidence intervals (CI), and paired t-test p-values over runs.
SR
SPL. CI
Agent
mean:istd
CI
p-value
mean=std
p-value
DUET [1]]
72.22 + 0.21
[72.08, 72.36]
3.42
12
60.41 0.27
[60.27, 60.56]
6.60
16
BEVBert [15]
75.82 0.25
[75.70, 75.95]
8.91
10-11
64.14 0.22
[64.01, 64.28]
7.30
10-10
VER [17]
76.37 + 0.18
[76.27, 76.47]
1.58 10-10
65.07 = 0.21
[64.91, 65.23)
2.90 x 10-15
Ours
78.32 = 0.13
(78.25, 78.39)
66.47 = 0.17
66..39, 66.54)
To assess whether the performance improvements are statistically meaningful beyond random vari- ation, we conduct significance tests on R2R val unseen split over 5 runs. For each agent, we re- port the mean and standard deviation, the confidence interval (CI), and paired t-test p-values against ours. As shown Table 11, the improvements our agent over DUET [11], BEVBert [15], and VER [17] are statistically significant (all 0.05 for both SR and SPL), confirming that the gains are not attributable to stochastic variance.
20

---

Published as a conference paper at ICLR 2026
E UNCERTAINTY DISCUSSION AND FAILURE CASE
In this section, we provide an extended discussion of our uncertainty formulation, present additional empirical analyses, and summarize representative failure cases.
E. DISCUSSION OF OUR UNCERTAINTY
Robustness to observation noise R2R val unseer. split. We evaluate an epistemic only variant (geometric + semantic), aleatoric only variant (appearance), and our agent under increasing levels of Gaussian noise in RGB observations.
Table 12:
Noise Level
Epistemic Only SK
Alcatoric Only SRT
Ours SR
77.05
76. 86
78.32
10%
76.78
76.80
78.09
20%
76.53
76.77
78.12
30%
75.93
76.81
77.98
In deep learning, uncertainty is typically categorized into two types: epistemic and aleatoric [87- 89]. Epistemic uncertainty arises from a lack of knowledge or limited evidence in the model, whereas aleatoric uncertainty denotes irreducible randomness or inherent variability in the data that cannot be reduced by collecting more samples. In embodied navigation [90], epistemic uncertainty typically associated with insufficient or unreliable perceptual evidence (e.g.. missing views or out- of of-distribution observations), often leading to ambiguous target grounding around visually similar landmarks [91]. Aleatoric uncertainty captures irreducible ambiguity caused by partial observability, occlusions, clutter, or sensor noise, which makes traversability inherently uncertain [91, 92].
Under this taxonomy, our design is as follows. i) We interpret geometric and semantic uncertainty as epistemic uncertainty. These two arise from missing or ambiguous perceptual evidence, such as sparsely observed regions or visually similar landmarks. Because they can, in principle, be reduced by acquiring more views, they align with the notion of epistemic uncertainty. ii) We interpret appear- ance uncertainty as aleatoric uncertainty. It reflects the sensitivity of rendered observations to small local perturbations. This variability is intrinsic the rendering measurement process and cannot be eliminated even additional scene cues are available, which aligns with aleatoric uncertainty.
In addition, examine whether these components behave consistently with the above interpreta- tions. We compare three variants on R2R val unseen split: an epistemic only variant that uses geometric and semantic uncertainty, an aleatoric only variant that uses appearance uncertainty, and our agent. We gradually inject 10%, 20%, and 30% Gaussian noise into RGB observations while keeping all other settings fixed. As shown in Table 12, two trends align with the intended distinction. i) The epistemic only variant degrades as noise increases. reflecting its dependence on the sufficiency and reliability of perceptual evidence. ii) The aleatoric only variant remains stable across noise lev- els, consistent with uncertainty that models inherent observation variability. Moreover, our agent remains robust under all noise levels and achieves the best overall performance.
E.2 ANALYSIS OF APPEARANCE UNCERTAINTY
Ground
Bendared obs
Figure 6: Ground Truth Rendered Observations. The renderings closely match the ground truth, supporting the Fisher-based appearance uncertainty proxy.
We provide visual comparisons between the rendered observations and the ground truth. Fig. 6 illustrates that the renderings closely match the ground truth, indicating that the residual term in the Hessian decomposition is negligible. Consequently, the Fisher Information serves as reliable
21

---

Published as a conference paper at ICLR 2026
proxy for appearance uncertainty and yields faithful sensitivity signal, thereby corroborating the soundness of our design.
E.3 EFFECTIVENESS OF UNCERTAINTY INFORMATION
Table 13: Effectiveness of Uncertainty Information on R2R val unseen split. The uncertainty infor- mation encoded in our 3D Value Map is rendered into 2D panoramic uncertainty map and provided an additional observation to support decision-making existing agents.
Agent
SR f
SPL |
DUET [1]]
72.22
60.41
DUET + 2D uncertainty map
73,52
61.43
BEVBer [15]
75.82
64.14
BEV Bert 2D uncertainty map
76.91
65.77
VER [17]
76.37
65.07
VER 2D uncertainty map
77.45
65.94
To further verify that the estimated uncertainty information is beneficial for VLN, we apply our uncertainty cues to other agents. Since these agents adopt other forms of scene representation rather than 3DGS [93], we render the uncertainty encoded in our 3D Value Map into 2D panoramic uncertainty map and provide it as an additional observation to DUET [11]. BEVBert [15], and VER [17].
As shown in Table 13, this 2D uncertainty map consistently improves navigation performance across all three agents on R2R [1] val unseen split. For example, DUET improves from 72.22% to 73.52% SR, while BEVBert and VER obtain similar gains of more than 1% in SR together with corresponding improvements SPL. These results demonstrate that our uncertainty cues provide useful guidance for improving navigation decisions.
E.4 FAILURE CASES
Start in the Aedresme and walk chrough the dhornay into the hall. Continue snaight past the kitchen area, then turn liftt into the loving room. Stap in front the refi.
Exit the hallsay and walk noward the Androsm at the for end Pars the painingg, fhie skrouph the diaragy, Susp neur the goreok hoard on the ruble baside the bed
cavigible
tanget
(b)
Figure 7: Failure Cases. (a) Our agent stops "the sofa" comes into view, the current observation already provides sufficient evidence of the target, creating confusion about whether further steps are required. (b) Our agent halts at the doorway instead of reaching "the gomoku board" near the bed, since the board lies inside the room and cannot be observed from the entrance, leaving the agent uncertain and leading to premature termination.
To illustrate the challenges our agent may still face, we present two representative failure cases. As shown in Fig. 7(a), although the instruction requires stopping at the sofa, the agent terminates as soon as "the sofa" enters its observation. This because the current view already provides sufficient evidence of the target, leaving the agent confused about whether perceiving "the sofa" is equivalent to reaching the intended stopping point. In addition, in Fig. 7(b), although the instruction requires "stopping near the gomoku board by the bed", the agent halts at the doorway without entering the room. This because "the board" is not visible from its current viewpoint, leaving the agent uncertain about whether further exploration is necessary.
22

---

Published as a conference paper at ICLR 2026
Table 14: Quantitative Analysis uncertainty scaling
unseen split. scale estimated
uncertainty values by factors in {1.0, 0.8, 0.6, 0.4, 0.2, 0} and evaluate the agent in simple and complex scenes to analyze the influence of uncertainty on navigation performance.
Uncertainty Scale
Simple Scenes SR
Complex Scenes SR
1.0 (ours)
10/10
10/10
0.8
10/10
9/10
0.6
10/10
7/10
0.4
10/10
7/10
0.2
10/10
5/10
(No uncertainty)
10/10
2/10
In addition, we further provide quantitative analysis to examine how uncertainty affects navigation performance under different levels of perceptual ambiguity. Specifically, we select 20 representa- tive scenes from the R2R val unseen split: 10 simple scenes (e.g.. open spaces, few obstacles, clear landmarks) and 10 complex scenes (e.g., narrow spaces, occlusions, visually similar struc- tures). For each episode, {1.0, 0.8, 0.6, 0.4, 0.2, 0}.
the agent while scaling the three uncertainty values by factor in
As shown in Table 14, performance in simple scenes remains consistently high across all scaling factors, indicating that these environments contain minimal perceptual ambiguity and rely little on uncertainty
In contrast, navigation performance in complex scenes drops progressively as the uncertainty values are suppressed. Removing the uncertainty information entirely reduces the success from 10/10 2/10, demonstrating that uncertainty cues play critical role in guiding reliable navigation under challenging visual conditions.
23

---

Published as a conference paper at ICLR 2026
F DISCUSSION
F.I LIMITATION AND FUTURE WORK
This work has several limitations that also highlight directions for future exploration. i) Simula- for Constraints. Our framework is trained and evaluated in the static Matterport3D simulator [80], which omits real-world challenges such as moving objects, sensor noise, or actuation errors. Ex- tending to dynamic and noisy environments will be crucial for deployment. ti) Task Scope. We focus on indoor VLN tasks (i.e., R2R [I], RxR [27], REVERIE [28]). Applications to broader navigation domains, such as aerial VLN [94] outdoor scenarios [95], remain unexplored. iti) Environmental Coverage. Our approach is primarily validated in structured indoor layouts. Future studies should examine its robustness in more cluttered, unstructured, cross-domain environments. iv) Predic- five or Active Perception. Our framework currently estimates perceptual uncertainty based solely on the available viewpoint, without actively acquiring additional evidence. Incorporating predictive view synthesis, such as world-model based future observation forecasting [24], or integrating active perception mechanisms [96] may allow the agent select more informative viewpoints and thereby mitigate perceptual ambiguity. Exploring such predictive and action- -uudddd perception strategies represents a promising direction for future research.
F.2 TOWARD REAL- WORLD DEPLOYMENT
Although experiments are conducted in simulation, transferring the proposed framework real robots is an important direction. We outline several practical considerations and discuss how our design can be exlended address them.
Sensor noise and uncertainty degradation. To assess robustness to imperfect sensing, we inject 10-30% Gaussian noise into RGB observations and re-evaluate the agent on R2R val unseen split in Table. 12. The performance remains stable under these perturbations, suggesting that the uncertainty estimates remain stable under moderate sensor noise.
Dynamic objects and time-varying geometry. Real environments often contain moving objects and non-static geometry. Our Semantic Gaussian Map can be coupled with dynamic 3D Gaussian Splatting pipelines [97, 98], which continuously update Gaussians as the scene changes. In such a setup, both the scene representation and its associated uncertainties are updated online, enabling the agent to react to newly introduced ambiguity.
Actuation errors and control dynamics. Actuation errors affect the executed robot pose rather than the uncertainty estimation itself. For real-world deployment, our 3D Value Map can be in- tegrated with standard closed-loop control and localization modules (e.g., visual odometry [99] or SLAM [93]), so that pose uncertainty and perceptual uncertainty are jointly considered when plan- ning reliable trajectories.
F.3 SOCIAL IMPACT
This work introduces an uncertainty-aware framework for Vision Language Navigation. By ex- plicitly modeling geometric, semantic, and appearance uncertainties, the agent learns to interpret environments not only terms of structure and semantics but also terms of reliability. This design strengthens decision-making and improves navigation performance across multiple bench- marks. Beyond quantitative gains, the framework highlights the importance of uncertainty modeling for embodied AI, suggesting that safer, more interpretable, and reliability can be developed for broader real-world applications. We hope that this perspective will inspire future research on integrating uncertainty into embodied reasoning and planning.
navigation systems
F.4 TERMS OF USE, PRIVACY, AND LICENSE
Matterport3D [80], R2R [1]. RxR [27]. and REVERIE [28] are available for non-commercial re- search purposes.
Use of Large Language Models. We did not use any large language models in this work.
24


Reviews:
[{"review_id": "3ShYkkkYDb", "reviewer": "Reviewer_BefP", "summary": "To address the perceptual uncertainty in robot navigation, this paper formalizes three forms of uncertainty, including geometric, semantic, and appearance uncertainty, enabling more informed decision-making. Specifically, a Semantic Gaussian Map (SGM) based on 3D Gaussian primitives is constructed to encode both the geometric structure and semantic content of environments. Then, three uncertainties are estimated: 1) geometric uncertainty is modeled through variational perturbations of Gaussian position and scale to assess structural reliability; 2) semantic uncertainty is estimated by perturbing the semantic attributes of Gaussians to reveal ambiguous interpretations; 3) appearance uncertainty is quantified by Fisher Information to measure the sensitivity of rendered observations to Gaussian-level variations. Then, a unified 3d Value Map is composed to ground these uncertainties as affordances and constraints, thereby guiding informed and more reliable trajectory planning. Extensive experiments on VLN benchmarks demonstrate the effectiveness of the proposed method.", "questions": "Please try to address the weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1)\tThis paper focuses on a valuable but usually ignored research topic, i.e., the perceptual uncertainty in robot navigation, and systematically formalizes three forms of perceptual uncertainties, including geometric, semantic, and appearance uncertainty, which reveals a new improvement direction for robot navigation.\n2)\tExtensive experiments are constructed to demonstrate the effectiveness of the proposed uncertainties.\n3)\tThis paper is well-written and easy to follow.", "weaknesses": "1）\tCould the recognized perceptual uncertainty help to further lower the uncertainty? How do you improve the confidence/reliability? Will it help with robot navigation?\n2）\tIn the action prediction of the 3D Value Map, all the representations of each Gaussian are projected into a feature vector, and then the aggregated representation is utilized to predict candidate nodes. So the uncertainties indirectly and aggregately impact action prediction, does it need to compute the uncertainty of each Gaussian? \n3）\tAccording to the results in Tables 1 and 2, the improvement of the proposed method is marginal over the state-of-the-art, especially on the SR, SPL. \n4）\tAccording to the results in Table 6, it seems the three uncertainties do improve the baseline method, but the baseline is obviously outperformed by the SOTA methods. Could the proposed method still work on a better baseline?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jgxSF87TdX", "reviewer": "Reviewer_XCWv", "summary": "The paper proposes an Uncertainty-Aware Gaussian Map (UAGM) framework for Vision-Language Navigation (VLN). The method represents the environment as a 3D Gaussian field and explicitly models three types of perceptual uncertainty—geometric, semantic, and appearance—via variational inference and Fisher-information-based estimation. These uncertainty estimates are aggregated into a 3D Value Map used by the navigation policy. The approach aims to improve robustness and interpretability by allowing the agent to reason about uncertain visual observations. Experiments on R2R, RxR, and REVERIE show consistent performance gains over prior VLN baselines, and qualitative visualizations demonstrate uncertainty-aware perception.", "questions": "Have the authors analyzed whether high predicted uncertainty can be used to predict or prevent navigation failure?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces an explicit Gaussian-based 3D uncertainty representation, offering a principled way to quantify perception uncertainty in VLN.\n2. The theoretical formulation (variational inference + Fisher information) is well motivated and mathematically sound.\n3. Experiments are comprehensive, spanning three benchmarks with ablations and qualitative visualizations.\n4. The paper is generally clear and technically solid, with detailed appendices.", "weaknesses": "1. The claim of being the first to introduce uncertainty modeling in VLN is somewhat overstated. Prior work such as LLM as Copilot for Coarse-grained Vision-Language Navigation (ECCV 2024)[1] already explores confidence or uncertainty-related reasoning. The main novelty here lies in adopting a Gaussian-based uncertainty representation, rather than introducing the uncertainty concept itself. Moreover, the estimated uncertainty is not explicitly incorporated into decision-making—it serves more as an auxiliary feature than a truly uncertainty-guided policy.\n2. The paper employs ambitious terminology (e.g., 3D Value Map, Unified Uncertainty Grounding), which may slightly overstate conceptual novelty.\n3. Although the appendix includes qualitative failure cases, the analysis remains superficial. There is no quantitative study demonstrating how uncertainty correlates with navigation errors or ambiguous observations.\n4. The latest compared method is VER (CVPR 2024), which is relatively dated. It would strengthen the paper to include comparisons with more recent works such as [2][3][4].\n\nReferences\n\n[1]  LLM as Copilot for Coarse-grained Vision-Language Navigation, ECCV 2024.\n[2]  Do Visual Imaginations Improve Vision-and-Language Navigation Agents?, CVPR 2025.\n[3]  COSMO: Combination of Selective Memorization for Low-cost Vision-Language Navigation, ICCV 2025.\n[4]  Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheel, ICLR 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YPSmd8BFvT", "reviewer": "Reviewer_ZYEA", "summary": "This paper presents an uncertainty-aware Vision-Language Navigation (VLN) framework that explicitly models and leverages perceptual uncertainty to improve the navigation decision making. The key contribution is constructing a Semantic Gaussian Map (SGM) using differentiable 3D Gaussian primitives from panoramic observations, then estimating three forms of uncertainty: geometric, semantic, and appearance uncertainty, which are then integrated into a unified 3D Value Map that encodes affordances and constraints, guiding more reliable navigation decisions. Empirically, the proposed method improves the performance across benchmarks including R2R, RxR, and REVERIE, demonstrating the method's effectiveness.", "questions": "1. Could the authors provide a comprehensive hyperparameter sensitivity analysis?\n2. Could the authors provide a stronger justification regarding the uncertainty categorization (mapping geometric, semantic, appearance to epistemic and aleatoric)?\n3. Could the authors conduct significance tests to prove their improvements are statistically meaningful?\n4. As mentioned in the Limitation and Future Work, since the major contributions of the work are made in simulation environments. Any thoughts on addressing expected transfer problems to the real world? Like potential uncertainty estimate degradation with real sensor noise, dynamic objects, or actuation errors.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The motivation and problem identification is clear. The paper motivates why uncertainty matters in VLN through concrete examples (e.g., confusing similar doors, occlusion ambiguity) that existing agents struggle to handle.\n2. The proposed framework is technically sound. The combination of 3D Gaussian Splatting with variational inference for uncertainty estimation is well-grounded. The mathematical formulation is rigorous and clearly presented. The uncertainty estimation addresses three types of uncertainty, providing a complete picture.\n3. Experiments are well-conducted, validating the effectiveness of the proposed method. Comprehensive ablation study demonstrates each component's contribution. Runtime analysis shows acceptable overhead, eliminating the concern about  excessive computational resource consumption.", "weaknesses": "1. Limited hyperparameter sensitivity analysis. Only the pruning threshold $\\tau_e$ and $\\tau_{\\alpha}$ are analyzed (in Table 5). Sensitivity of uncertainty-related hyperparameters like $\\delta$ and $\\eta$ for geometric uncertainty and $\\epsilon$ for semantic uncertainty.\n2. Theoretical grounding of uncertainty identification is weak. While Appendix E.1 briefly discusses epistemic vs. aleatoric uncertainty, the mapping of geometric/semantic to epistemic and appearance to aleatoric lacks rigorous justification. This categorization seems somewhat arbitrary.\n3. While standard deviations are reported, no significance tests confirm whether improvements are statistically meaningful beyond variance.\n4. It would be more persuasive if the authors could upload video demos.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)