Distilling LLM Prior to Flow Model for Generalizable Agent’s Imagination in Object Goal Navigation
Authors: Badi Li
Ren-Jie Lu
Yu Zhou
Jingke Meng
Wei-Shi Zheng
Venue: NeurIPS 2025 poster ()
Keywords: Visual Navigation; Object Goal Navigation
Primary area: applications
TL;DR: We distill commonsense spatial priors from large language models into a flow-based generative model to help agents imagine unseen parts of indoor scenes for Object Goal Navigation task.

Abstract: The Object Goal Navigation (ObjectNav) task challenges agents to locate a specified object in an unseen environment by imagining unobserved regions of the scene. Prior approaches rely on deterministic and discriminative models to complete semantic maps, overlooking the inherent uncertainty in indoor layouts and limiting their ability to generalize to unseen environments. In this work, we propose GOAL, a generative flow-based framework that models the semantic distribution of indoor environments by bridging observed regions with LLM-enriched full-scene semantic maps. During training, spatial priors inferred from large language models (LLMs) are encoded as two-dimensional Gaussian fields and injected into target maps, distilling rich contextual knowledge into the flow model and enabling more generalizable completions. Extensive experiments demonstrate that GOAL achieves state-of-the-art performance on MP3D and Gibson, and shows strong generalization in transfer settings to HM3D.

Distilling LLM Prior to Flow Model for Generalizable Agent's Imagination in Object Goal Navigation
Badi Li Ren-Jie Lu Yu Zhou', Jingke Meng': Wei-Shi Zheng 1.3
Sun Yat-sen University The University of Hong Kong
3 Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education badi. cs@connect. hku.hk, mengjke@gmail.com wszhengdieee.org
Abstract
The Object Goal Navigation (ObjectNav) task challenges agents locate a spec- ified object in an unseen environment by imagining unobserved regions of the scene. Prior approaches rely on deterministic and discriminative models com- plete semantic maps, overlooking the inherent uncertainty in indoor layouts and limiting their ability to generalize to unseen environments. In this work, we propose GOAL,, generative flow-based framework that models the semantic distribution of indoor environments by bridging observed regions with L.MM-enrchhed full- scene semantic maps. During training, spatial priors inferred from large language models (LLMs) are encoded as two-dimensional Gaussian fields and injected into target maps, distilling rich contextual knowledge into the flow model and enabling more generalizable completions. Extensive experiments demonstrate that GOAL achieves state-of-the-art performance on MP3D and Gibson, and shows strong generalization in transfer settings to HM3D, Codes and pretrained models are available htttt:t///// httptgggtttttttteeeeg com/Badi Badi Li/GOAL
Introduction
which enables agents to move purposefully through complex, realistic environments, is fundamental challenge in embodied intelligence. Within this domain, Object Goal Navigation (ObjectNav) tasks an agent with locating an instance of a user- specified object category (e.g., "find a chair") in unseen environment, relying solely visual observations.
Embodied navigation 2
30
43
56
70
To succeed ObjectNav, the agent must not only recognize the goal object when it becomes visible but also infer its likely location before it is seen. This imagination step is particularly challenging, it requires reasoning about contextual and co-oceurrence relationships between objects (e.g.. chairs often appear near tables). Recent approaches [17
25 75 address this by incrementally constructing top-down semantic maps and predicting full-scene semantic maps through discriminative and deterministic models. However, their deterministic nature, which directly maps inputs to fixed outputs with strict one-to-one mapping, inherently limits generalization unseen data.
In contrast, we argue that semantic map completion inherently uncertain: Multiple plausible full scenes can correspond to the same partial map, and multimodal outputs could benefit generalization capabilities [10]. We therefore formulate this task as a probabilistic generation problem, leverag- ing recent advances in bow-based generative modeling (21 26 28 53 54 to learn the semantic distribution of indoor scenes (illustrated in Fig. []. However, while generative models offer better generalizability, find that applying generative models to ObjectNav poses three core challenges, which we address in this work.
Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

I've similar scenes 3 and was distilled objects contescuull information lhink the semantic
from LLM @8 distrbutioon
Goal: find a table
Figure 1:
We incorporate flow model to generate the semantic distribution of unobserved regions (in dark), based on dataset-internal patterns and external knowledge from language models.
First, generative models typically require large and diverse datasets effectively learn the latent distribution. However, existing indoor scene datasets remain limited in both scale and variety. To overcome this, we incorporate external knowledge from large language models (LLMs) by firstly modeling it as natural Gaussian distribution in the latent space of the flow model, which enriches contextual signals during training. particular, we prompt LLMs to infer spatial contextual priors, including common distances between object pairs and associated confidence scores. These distance- confidence pairs are transformed into two-dimensional Gaussian priors and injected into semantic maps during training. This process distills the rich contextual information from LLM to generative flow model, enhancing its understanding of object co-occurrence and spatial context. Crucially, this external supervision is applied only during training. avoiding inference- serreecce-ime costs such API latency and memory overhead, and enabling the flow model to operate as a plug-and-plaa semantic reasoner.
Second, we identify the problem of inefficient conditioning. Traditional diffusion models assume the source distribution be standard Gaussian and develop their theory based on this assumption. result, conditional generation in the diffusion literature typically relies on additional mechanisms, such as cross-altention, concatenation, or FiLM (38], to incorporate conditioning information. While these mechanisms are acceptable for image generation and restoration, they introduce additional computation, which may prohibitive visual navigation tasks that require multiple inferences per episode and real-time interaction. In contrast, the Flow Matching algorithm does not strictly assume a Gaussian source distribution, allowing us to design more efficient conditioning mechanism: directly modeling the dependent couplings between noise injected partial semantci maps and full LLM-enriched semantic maps.
Third, semantic maps constructed during navigation are prone to accumulating errors from upstream segmentation models, which can degrade the performance of generative models. To mitigate this, aggregate past RGB-D observations into unified point clouds representations and perform joint segmentation using 3D perception models, inspired by how humans implicitly integrate multi-frame observations. This method captures both spatial geometry and temporal consistency more effectively than traditional 2D semantic segmentation methods used in ObjectNav 20 As a result, we achieve more accurate and consistent scene understanding.
In conclusion, we propose GOAL (Guiding Agent's imaginatiOn with generAive iLow), a generative framework that incorporates externl knowledge as training supervision, models direct couplings to better leverage semantic map priors, and integrates multi-view observations for enhanced scene-level understanding. These components together lead to strong and generalizable performance on the ObjectNav task.
Evaluations on large-scale datasets Gibson 63 and MP3D g show that our approach significantly outperforms baselines. Additionally, transfer experiments, training MP3D and testing HM31) μТ], demonstrate the strong generalization capabilities of our approach to unseen environments.
2
Related Work
ObjectGoal navigation. ObjectNav approaches fall into two main categories: end-to-end and modular. End-to-end methods map visual inputs to actions using reinforcement imitation learning,

---

focusing on improving visual representations 7
64 65
66 or tackling policy learning challenges
like sparse rewards and overfitting 442 50,
59
67
68 Modular methods decompose the task into
components such as mapping, planning, and policy learning. Given a semantic map, these methods ex- plore waypoint or frontier selection [9 32 40, distance estimation [79], target probability prediction
(73], and semantic map completion 17 25 75]. T-Diff (72) firstly introduced generative modeling
to ObjectNav via a DDPM conditioned on semantic maps for trajectory generation. Alternatively we propose generative flow-based model that imagines full-scene semantics, using LLM-derived priors to improve generalization to unseen environments.
Diffuusion and flow-based generative models. Generative models such as diffusion (21] and flow-based methods [26] [28] generate data via iterative denoising or learned velocity fields. They
have shown strong performance in image generation and restoration tasks [3
45
48
78
We
draw an analogy between agent's 's imagination via completing a partial semantic map and image restoration, where the goal is to reconstruct missing content from degraded input. Most approaches begin from Gaussian noise and condition on the input via concatenation, FiLM [38] or cross-attention.
Recent alternatives have explored directly bridging degraded images and targets via Schrödinger Bridges [27] and stochastic interpolants [I] but often trade off quality for interpretability. In contrast, we show that for sparse semantic maps, direct coupling via flow matching can be more efficient without sacrificing the performance. We adopt flow matching framework for its faster sampling and mathematically simple yet expressive formulation.
3 Flow Matching Preliminaries
The generative task is typically defined as mapping vht, / E [0, 1], which transports samples Xo from a source data distribution to samples X₃ in a target distribution q. In general, the source and target samples may come from joint distribution (1) (X.
To address this transformation from Xo to X1, Flow Matching 226 interpolates a probability path pe between the source and target samples:
X, = 04 Xo + 1,X1 ~ Pt:
with boundary conditions ag 31 = 1 and 0n BB = 0. To learn this path, we solve the following Ordinary Differential Equation (ODE):
d
X, ue (Xt).
(2)
dt
where ut is time-dependent velocity field, also referred to the drift, typically parameterized by neural network uf which trained by minimizing:
LFM (0) = Exxx,rp D (x,,,,(
where X; dr_x, X1 denotes dime-derivative of X, and D denotes general Bregman Divergences measuring the dissimilarity. Once trained, this velocity field can be used to generate samples by integrating the ODE (Eq. numerically. In our work, we solve the ODE by simplest method Euler integration:
Xeth = Xx + huu (X+,,
where h = and n a hyperparameter representing the number of forward steps.
Method
4.1 ObjectNav Definition
We consider the ObjectGoal Navigation (ObjectNav) task, where an embodied agent is initialized at random location in an unseen indoor environment and is instructed to navigate to an instance of a user- specified object category (e.g.. a chair). At each timestep t, the agent receives egocentric observations I (i.e., RGB-D images) and its pose wt, which includes its location and orientation. Based on this sensory and positional input, the agent selects an action 0, E AA where A includes move_forde forward turn turn_left, turn_rii wrigh, and stop. The navigation episode terminates either when the agent issues the stop action or after a maximum of 7 steps. An episode is deemed successful the agent issues stop with the target object visible and within threshold distance.
3

---

4.2 Navigation Overview
In line with previous works 40 72 73
751, we incrementally build local semantic map, but via scene segmentation instead of single-fram frame understanding. Our trained generative flow model then completes the partial map by generating the full semantic distribution, especially for unobserved areas. This distribution guides the agent to likely goal object locations, followed by deterministic local navigation. An overview of the navigation pipeline is illustrated in E (a).
Navigation (inference)
Trining
Flow guided agent's imagination
(1(3))
GOA
Edtiniildd Vilocty
A
iromnsllatt X.
Grums-tndh videcity Up
update
3d segment & project
GOAL
navigate
X
(c)
X
X1
Obsurved point clouds
Observed map
Target: toilet
Generated map
prompt
Rale
eppetio adboe mone byven.....
Distance Confidence
Observed objects: cushioo, bed...
Fil ogen plornart
chai:
Hacardaca df extrctions
iilrnco prineally
LLM
mettyndhe
distane 2.50 cont drean buBs
Edtpt fasaul
Contextual prior
Gaussian map
Figure 2: Overview of GOAL framework. with navigation (inference) in blue and training in green. (a) shows the navigation pipeline where the agent imagines future maps using flow-guides model. (b) illustrates how we prompt LLM with hierarchical instructions to generate contextual priors (for full prompt and response Appendix [E). (c) visualizes how we use LLM priors to construet data-dependent couplings. (d) demonstrates how the flow model is trained using these couplings through interpolated velocity supervision.
4.3 Generative Flow Models For Agent's Imagination
For training GOAL, we first prompt an LLM to obtain contextual priors between objects (F)))))). use these priors to construct dependent couplings as training samples (Fig (C)) and then train the model using standard interpolation scheme (Fig. d)).. We detail each step below.
Prompting LLM for contextual prior. Due to the scarcity of large-scale, densely annotated indoor scene datasets, models often struggle generalize beyond environments. This is largely attributable to the limited diversity of contextual object arrangements in existing data. Here, we describe how we solve the problem by enriching training signals with commonsense knowledge about object co-occurrence, extracted from LLMs.
Specifically, prompt LLM using various modern prompting techniques such Chain-of- ofoThoughttt (CoT) [58], Role Conditioning [44], Few-shot prompting [5]. to generate the common distances between different objects D {diz J and its confidences on each response C = {c:g) ) 2 Then, given the partially observed semantic maps, cluster the connected grids with the same semantic labels into objects O = {or | where N is the number of observed objects. For each observed object o,, refer to the LLM responses for its likely co-occurring objects using preset distance threshold Ta and confidence threshold тe- Other objects 0j that either has a distance to the observed object diş within тa confidence score eij larger than тe will be considered as co-occurring candidates. Formally, co-occurring candidates set for object o; is defined as:
(o,,:) foz (d:j S Ta) V (cij 2 re)).
During training, every clustered observed object will randomly choose co-occurring object among their candidates set (o.)) enrich semantic context and avoid model collapse (In practice, rather

---

than strictly random selection, we actually prioritize the objects in the intersection of candidates set of multiple observed objects, take account for a functional cluster with more than objects.). Intuitively, if an object is expected to co-occur near an observed object, but has not yet heen observed, is likely located the unobserved region of the scene. Thus we connect the clustered objects' centers with their nearest frontiers (the spatial boundaries between known free space and unexplored regions, by definition), and the centroid of the co-occurring objects will be on this connected line, offset by distance dij. Specifically:
M ド d,
(6)
where fot and μi are the centroid of observed object o and predicted centroid of co-occurring object 0g, and v is unit vector pointing from o; to its nearest frontier grid. Then the confidence score Cij is converted to a standard deviation via linear transformation:
jjj min + 0 (1 Cij),
where min and max are hyperparameters representing the lower and upper bounds of standard deviation, respectively. With the computed deviation and centroid, we model the co-occurring objects as two-dimensional Gaussian distributions, which are added across all observed objects:
(2) J
PLLM
Σ 0.E0
2To
(1) (2) 1j
1
exp
1 2
x
(2)
Of ~ ((N(o))), (8)
ij
where we take 0i (1) - o G.j. resulting in an isotropic Gaussian over the semantic map. Notation (5) U(X) refers uniform distribution, which is used for random sampling. PLLM is then the LLM prior distribution reflecting the commonsense spatial expectations of object 0, derived from LLMs. The final LLM prior is stacked across all channels:
(1) (2) PLLM PLLM PLLM
(N.)
PLLM
which has the same shape as the input semantic maps and is used to guide the flow model toward plausible object arrangements in unobserved regions of the scene during training.
Building data-dependent couplings with LLM-derived supervision. Following standard practice in diffusion and score-based models 21 53 the source distribution typically set to standard Gaussian N I), resulting in an independent coupling To.1 (Xo, X2) p(Xo)9(( (())))(X)) However, we find this strategy will complicate the model architecture with additional conditioning mechanism. Instead, we directly couple the partial semantic map with the LLM-enhanced target, eliminating the need for conditioning mechanisms like cross-attention and yielding more consistent and effective generation for navigation. Below, we describe how this dependent coupling is constructed.
During training, we have access to the full-scene semantic map X'. Following [40], we employ the Fast Marching Method (FMM) [4 to simulate a realistic navigation trajectory by planning a path between two randomly sampled points on the map. The visible region along this path serves binary mask 7, representing the observed area. To incorporate stochasticity and avoid model collapse, we still inject Gaussian noise with relatively small deviation Ao into the unobserved regions of the scene. Specifically, the source semantic map is defined as:
X. = 7 @ X' + + 3 (0 Ao?).
(10)
where denotes the Hadamard product, and mask. The target semantic map X, is constructed by adding LLM--vvived priors to the unobserved regions of the full ground-truth map:
denotes the complement of the visibility
X1 = = P
(11)
where PLLM derived from Eq. and Eq. 9 and À a hyperparameter to control the prior strength. Notably, this formulation results in data-dependent coupling between Xg and X1, since both maps are conditioned on the same ground-truth map X' and share the same visibility mask 7. As a result, the joint distribution To.1 vo.1(X, (Xo, Xi) described in Sec. 3 is no longer factorizable into independent marginals p( target data.
but instead captures a strong, structured relationship between source and
5

---

Training, Given the constructed data deteadeenndent couplings, we adopt the Optimal Transport (OT)
displacement interpolant as our interpolation scheme (see Eq. [.. defined as:
X (1 t) X) + t.
(12)
For the Bregman Divergence term in Eq. 3 we use the Euclidean distance, resulting in the training objective being a Mean Squared Error (MSE) loss:
LFM = X [| uo (X,, 1)13
(13)
where the ground-truth velocity field is given by X, = X1 - Xo. derived from the linear interpolation For details of the training pipeline, refer the Appendix B for pseudocode.
in Eq. 12
4.4 ObjectNay with GOAL
In this subsection, we detail the process of building and preprocessing the semantic map serve as input to GOAL. followed by how the agent uses the output of it to guide exploration and take effective actions.
Semantic map construction via 3D scene understanding. We construct a semantic map by transforming RGB-D observations into 3D point clouds, segmenting them, and projecting the results to a top-down view, as detailed below. As described in Sec. [c.] the agent receives RGB-D observations I and its pose wi at each timestep t, along with the camera intrinsic matrix P inherently available. The observations 4 are first back projecero to point clouds by aligning the RGB values and depth values of each pixel. Then egocentric-to -e-eccectct transformation is conducted using we and P. The registered point clouds will be segmented at once to take account of historical context, geometric structure, and achieve scene-level perception. Though modern architectures such as the Point Transformer series [61 62 76 have demonstrated strong performance in 3D scene understanding, we adopt a Sparse Convolutional network (18), the 3D counterpart of standard 2D CNN-based segmentation models, to reduce potential confounding effects from using highly expressive architectures. After segmentation, points are then projected to bird-eye view to form a local map M, E (2) where No represents the number of semantic categories, and the additional 2 dimensions correspond to obstacles and free space, respectively (we denotes M' E R Nexxw to 'semantic map') )
To guide navigation toward the goal, the agent uses a generative model to imagine unobserved regions of the scene and select the grid with the highest probability to be the target as long-term way-point. Given the semantic map M/ at time t. we first compute the minimum bounding box of the observed area and crop surrounding region scaled by a factor c. This factor controls the imagination's 's spatial extent: too small may limit foresight, while too large may reduce reliability. The cropped region is resized to a fixed shape m E >x (with L 256), where unobserved areas are injected with Gaussian noise similar to Eq 10 This map serves as the source sample Xo in Eq 12 and is passed to the generative flow model, which applies iterative Euler steps (Eq. 4 to generate a complete semantic map. The output is then resized and merged back into the original map, resulting in the imagined full map M,. Finally, the grid cell with the highest value in the target object channel Cg is selected as the long-term waypoint g::
Agent's 's imagination With GOAL.
(eg)
9t. = arg max Me(h.w)- (h,w)
(14)
Navigation policy. After determining the waypoint 9t = (Ir. yt). we follow prior works 9 75
by implementing a local planner based on the Fast Marching Method (FMM) [49]. which computes the shortest path from the agent's current position to the waypoint using the occupancy channels of the semantic map. Unlike previous approaches that select waypoints at fixed intervals, we adaptively sample new waypoint only when the agent is either too close to or too far from the previous one. This strategy reduces computational overhead while maintaining effective path planning.
5 Experiments
5.1 Experimental Setup
Datasets. We evaluate our method, GOAL, on the validation sets of two standard ObjectNav benchmarks: Gibson [63), Matterport3D (MP3D) [8] Additionally, we perform transfer experiments

---

Targtt chair
beca
Larget (false positive)
targe
Figure 3: Visualization of navigation with GOAL on MP3D (val). The top row shows RGB obser- vations and agent trajectories; the bottom row displays the observed semantic maps and generated full-scene maps.
Table Transfer experiments results on HM3D. We compare the SR, SPL,, and DTS of state-of-the- art methods in different settings and training set. denotes our implementation using model weights and codes from official repositories, with improvement using scene segmentation.
Method
Train set
LLM usage
HM3D
Training
Inference
SR t
SPL t DTS &
ZSON
34
HM3D HM3D
X
x
25.5
12.6
PixNav 6
X
/
37.9
20.5
ESC 77
/
39.2
22.3
VoroNav 160
/
42.0
26.0
PONI 40
MP3D
X
X
41.8
20.1
4.63
GOAL w/o LLM
MP3D
X
X
47.6
22.5
4.14
GOAL
MP3D
/
X
48.8
23.1
4.11
by training on MP3D and evaluating on HM3D. For Gibson, we follow the tiny-split protocol from 99, 40), using 25 training and 5 validation scenes, with 1,000 validation episodes covering target object categories. For MP3D, we use the standard Habitat simulator setting [35] 39 [55], which includes 56 training and 11 validation scenes, 2,195 validation episodes, and 21 target categories. For HM3D, we use only the 20 validation scenes, with 6 goal categories and 2,000 validation episodes for transfer evaluation. For details of target object categories, please refer to Appendix C.2
Evaluation metrics. We adopt three standard metrics for evaluating the navigation performance. SR (Success Rate) indicates the proportion of success episodes. SPL (Success weighted by Path Length) represents the success rate of episodes weighted by path length, measuring the efficiency of navigation. DTS (Distance To Goal) is the distance to the goal at the end of the episode. For mathematical expression of these metrics, please refer to Appendix C.I
Implementation details. For training of GOAL, we sample 400K sub-maps for training on each dataset. We implement GOAL based on DiT 37 models. We use AdamW (24] optimizer with base learning rate of 1.5e-4, warmed up for 2 epochs, and applied cosine decay after that. The model is trained for 25 epochs, and exponential moving average (EMA) is used with decay of 0.999 during training. We trained the GOAL model on NVIDIA RTX 4090 GPUs with a batch size of 64 per GPU. For details of training scene segmentation module, please refer to Appendix C.5
5.2 Evaluation Results
Visualization navigation with GOAL. Figure 3
illustrates an example episode where the agent is tasked with finding a chair. Initially, the agent is misled by false positive prediction due to limited observations. As the agent explores and uncovers more of environment, model refines its prediction, effectively guiding exploration and ultimately leading agent correctly identify and navigate to the target (rightmost column). Additional visualizations demonstrating generation quality and diversity provided in Appendix
7

---

Table Comparison between using dependent and independent couplings (Gaussian source). Navi-
gation performance is reported in MP3D and Inference time is tested using an NVIDIA RTX 4090
GPU. The external knowledge distilled from ChatGL.M.
Base PDF
Number of Memory
GFLOPs
Inference time
Navigation (MP3D)
Parameters
Usage
SR
SPL
DTS
5o No0, I) To p(X', .7)
149.92M 138.76M
693.86 MB
29.34
17.13 ms
39.0
14.0
5.16
562.57 MB
24.12
12.38 ms
41.5
15.5
4.85
Table 4: Effectiveness of model variants. Navi-
gation performance is tested in MP3D.
Table Ablation study of individual compo- nents on MP3D. 'LP" refers to distillation of LLM priors; 'SS' indicates scene segmentation.
Model
LLM
Navigation (MP3D)
Variants
SR t
SPL
DTS
ID
Modules
Navigation (MP3D)
DīT-B
ChatGLM
41.5
15.5
4.85
SS
LP
SR
SPL
DTS 4
DÎT-L
ChatGLMM
40.4
14.9
5.01
32.4
11.7
5.25
DiT-B
Deepseek
40.9
15.0
4.89
2
/
38.8
14.6
5.08
DIT-L
Deepseek
40.3
14.7
4.97
/
/
41.7
15.5
4.84
DiT-B
ChatGPT
41.7
15.5
4.84
DiT-L
ChatGPT
40.5
15.0
5.09
Generalizability of GOAL. We assess GOAL's generalization ability by training it on MP3D and directly evaluating on the HM3D dataset. We compare its performance against state-of-the-ar methods across various training settings. As shown in Tab. [ GOAL significantly outperforms prior methods, including those that heavily rely on LLMs or are trained directly on HM3D, demonstrating strong generalization capabilities. Notably, the generative flow model without LLM supervision achieves competitive transfer performance, highlighting the benefits of generative modeling and diverse generation.
Effectiveness of data-dependent couplings. We compare bridging data-dependent couplings de- scribed in this paper and the traditional cross-attention method DiT As shown Tab. [2] building dependent couplings yields better navigation performance while simplifying model architec- ture and reducing inference time. Note that to condition the flow model on the partially observed semantic map, we introduce extra convolutional encoder to downsample the partial maps and then feed the patches to cross-attention layers of each DiT blocks, as an expressive mechanism to process and fuse the complex semantic map.
Ablation study on LLM prior and scene segmentation components. As shown in Tab. scene segmentation alone significantly improves performance by enabling more consistent and complete scene understanding, already establishing a strong baseline (row 2). Notably, further integrating the L.M prior on top this strong foundation yields substantial additional gain (row 3), despite the typical difficulty of improving over high-performing baselines. This highlights the effectiveness of the LLM prior, and suggests that our bridging scheme between partial maps and full semantic distributions requires reliable understanding of the observed scene.
Effectiveness of Flow Matching. Since GOAL shares the objective of inferring unob- served scene semantics with prior approaches it is essential to compare the Flow Matching algorithm we adopt with these base- line prediction methods. most recent approach in this line of work, we compare Flow Matching (26) with masked
Table 5:
Comparison between Flow Matching (FM) and Masked Autoencoder (MAE) for seman- tics imagination. Models are trained on MP3D and evaluated on different datasets.
17 25 75),
SGM 775 is the
Alg. MAE
Eval. Dataset
SR
SPL
DTS
MP3D
31.9
11.71
5.31
FM
MP3D
32.4
11.67
5.25
autoencoder (MAE) [99 used in SGM, replac- ing our proposed scene segmentation module
MAE
HM3D
32.1
14.7
4.85
FM
HM3D
35.9
14.69
4.65
with the widely adopted RedNet 23 for fair

---

Table 6: Object-gore Poject-goal navigation results on Gibson and MP3D. We compare the SR, SPL and DTS of state-of-ther tf--hhe-rt art methods in different settings. For SemExp g L2M 17 and Stubborn [32], we
report results from (74). For SSCNav [25, we report results from 75 under Training indicates
zero-shot methods that do not require any training. means the second best results.
Method
Venues
LLM usage
Gibson
MP3D
Training Inference
SR t SPL DTS & SR
SPL
DTS
Semexp
9
NeurIPS 20
X
71.1
39.6
1.39
28.3
10.9
6.06
SSC-Nav [25
ICRA
X
27.1
11.2
5.71
PONI 40
CVPR 22
X
X
73.6
41.0
1.25
31.8
12.1
5.10
L2M
ICLR 22
X
32.1
11.0
5.12
Stubborn
32
IROS 22
31.2
13.5
5.01
CoW 16
CVPR 23
7.4
3.7
3DAware
14
CVPR 23
X
74.5
42.1
1.16
34.0
14.6
4.78
T-Diff
72
NeurIPS 24
X
X
79.6
44.9
1.00
39.6
15.2
5.16
L3MVN
IROS 23
76.1
37.7
1.10
34.9
14.5
SG-Nav
69
NeurlPS 24
40.2
16.0
UniGoal 70
CVPR 25
41.0
16.4
SGM 75
CVPR 24
/
78.0
44.0
1.11
37.7
14.7
4.93
GOAL
Proposed
83.5
44.2
0.83
41.7
15.5
4.84
comparison. As shown in Table 5 while MAE and Flow Matching achieve comparable performance on MP3D, Flow Matching demonstrates stronger generalization in the transfer setting, where models are trained on MP3D and evaluated on HM3D.
Effectiveness of model variants and LLMs. As shown in Tab increasing model complexity (e.g., using DiT-L) yields little to no performance gain, and may even degrade performance compared to the base model (DiT-B). We hypothesize this is due to overfitting, as complex models tend to overfit when trained on limited and less diverse data, even generative setting. assess the impact of LLMs in GOAL, compare three models: ChatGLM-4-plus [4], DeepSeck-RI [12). and GPT-4 [36]. As shown in Tab. 4 the overall navigation performance remains similar across LLMs (though observe significant per-scene variance). Since number of prior works [69] 75 77 also report minimal differences between LLM variants, we hypothesize that current evaluation datasets are too small and biased to reliably reflect the effects of LLM choice.
Comparison with related works. We evaluate the performance of our method, GOAL, on the ObjectNav task by comparing with relevant baselines, categorized by their use of LLMs during training and inference. SemExp [9] first introduced semantic reasoning into object-goal navigation. PONI 40) improves using supervised learning to predict two potential functions, avoiding the ineffi- ciencies of reinforcement learning. L2M [17] and SSC-Nav [25] improve navigation by predicting the full top-down semantic map. SGM 75 further advances these approaches using the MAE algorithm [9 to train VIT [13 model in a self-supervised manner for full-scene semantic imagination. To enhance generation capabilities, SGM also prompts an LLM for contextual object information, but requires it during both training and inference, introducing additional complexity via cross-attention mechanism. contrast, methods like L3MVN 77,, SG-Nav [69), and UniGoal 770 rely solely on LLMSs to achieve zero-shot performance. While these approaches reduce training requirements, they often suffer from drawbacks such as API latency and excessive memory consumption.
As shown in Tab. 6 GOAL consistently outperforms the current state-of-the-an approach 77).. across all metrics and datasets. Although the improvement in SPL is relatively small, this can be attributed to the modified local policy we employ, described in Sec. 4.4 Specifically, the agent only updates its long-term goal when it becomes significantly closer to or farther from the previous one. As result, the agent's ' ability to correct false positive predictions is limited, even when new observations provide better guidance, leading to relatively long path length. And we stress that this limitation can be solved by applying modern techniques [5]
to reach faster inference for generative flow and adopt the general policy of changing long-term goal at fixed interval. We leave this for future work as these techniques will introduce complex formulation.
29 33
51
52

---

Discussions and Limitations
There are several technical limitations and potential further improvements to consider.
First, aim to introduce the generative flow match- ing algorithm into ObjectNav task, so our implemen- tations adhers the core theory in initial paper. How- ever, recent advances in FM for natural image genera- tion suggests some powerful training techniques be used, such as time discretizations and time shift, as well as techniques that reduce the Number of Evalua- tions (NFEs) of flow matching 29 33 S1 [52]. In-
Figure 4: Comparison between the simulated visible area and actual visible area given the agent position (red dot). The left shows the simulated mask adopted by [40] and our work, while the right shows the actual mask, reveal- ing substantial gap.
corporating these techniques could enhance training stability and sampling efficiency, potentially yielding more robust navigation policies.
Second, our method operates on a fxed-diimeniionl semantic map, with channels corresponding pre- defined set of object categories. This design inher-
ently restricts generalization to open-vocabulary or zero-shot settings, where novel object classes may appear at test time.
Finally, we follow the setting of [40] by computing path between two randomly selected points on the ground-truth semantic map, where a rectangular region centered at points along the path is treated the visible to generate the partially observed maps for training GOAL. However, found significant gap between these simulated partial observations and those encountered during real navigation. In practice, the agent's visible is fan-shaped rather than rectangular (See Fig. 4 Designing training samples that better simulate actual agent observations could substantially improve model performance.
7 Conclusion
In this work, we propose GOAL, a generative flow model that distills rich contextual priors from LLM into training supervision. We show that data-dependent couplings hetween partially observed maps and full semantic distributions significantly improve generation quality, outperforming traditional independent couplings commonly used in natural image domains. Additionally, we introduce a scene segmentation module to enhance holistic, geometry-aware, and temporally consistent scene under- standing. Experimental results on large-scale datasets Gibson and MP3D validate the effectiveness of GOAL, while cross-dataset transfer experiments on HM3D further highlight its strong generalization capabilities.
Acknowledgments
This work supported partially by NSFFC(U21A20411,224777 62206315), by the National Key Research and Development Program of China (2023YFA 1008503), Guangdong NSF Project (No. 00231151504002 No. 2024A1515010101, Guangzhou Basic and Applied Basic Research Scheme
(No. 2024A0414067).
10

---

References
[1] Michael S. Alhergo, Mark Goldstein, Nicholas Matthew Boffi, Rajesh Ranganath, and Fric Vanden- Rijnen- Stochastic interpolants with data-dependent couplings. Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27. 2024, 2024.
[2] Amir Bar, Gaoyue Zhou, Danny Tran, Trevor Darrell, and Yann LeCun. Navigation world models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11-15, 2025, 2025.
[3] Maxim Berman, Amal Rannen Triki, and Matthew B. Blaschko. The lovász-softmaa loss: A tractable surrogate for the optimization of the intersection over -viiuiooooo measure in neural networks. In 2018 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT. USA, June 18-22, 2018, 2018.
[4] Léon Bottou. Stochastic gradient descent tricks. In Neural Networks: Tricks of the Trade Second Edition. 2012.
[5] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Fric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever. and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[6] Wenzhe Cai, Siyuan Huang, Guangran Cheng, Yuxing Long, Peng Gao, Changyin Sun, and Hao Dong. Bridging zero-shot object navigation and foundation models through pixcl-guided navigation skill. In IEEE International Conference Roboticy and Automation, ICRA 2024, Yokohama, Japan, May 13-17, 2024, 2024.
[7] Tommaso Campari, Paolo Eecher, Luciano Serafini, and Lamberto Ballan. Exploiting scene-specific features for object goal navigation. In Computer Vision ECCV 2020 Workshops, 2020.
[8] Angel X. Chang, Angela Dai. Thomas A. Funkhouser. Maciej Halber, Matthias Nießner. Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from RGB-D data indoor environments. In 2017 International Conference on 3D Vision, 3DV 2017, Qingdao, China, October 10-12. 2017, 2017.
[9] Devendra Singh Chaplot, Dhiraj Gandhi, Abhinav Gupta, and Ruslan Salakhutdinov. Object goal navigation using goal- oriented semantic exploration. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6- 2020, virtual, 2020.
[10] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Robotics: Science and Systems XIX, Daegu, Republic of Korea, July 10-14, 2023, 2023.
[11]
Pointcept Contributors. Pointcept: A codebase for point cloud perception research. httpa:/ / /github com/ httpPPPointcet//oincceeptp 2023.
[12] DeepScek-Al, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang. Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi. Xiaokang Zhang, Xingkai Yu. Yu Wu, Z. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu. Bing Xue, Bingxuan Wang. Bochao Wu. Bei Feng. Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo,, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjic Qiu, Junlong Li, L. Cai, Jiaqi Ni, Jian Liang. Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li. Miaojun Wang, Mingming Li, Ning Tian, Pampan Huang, Peng Zhang. Qiancheng Wang. Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang. R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, and S. S. Li. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. CoRR, 2025.
11

---

[13] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly. Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. 9th International Conference on Learning Representations, ICLR 2021, Viruall Event, Austria, May 3-7, 2021, 2021.
[14] Zhengxiao Du Yujie Qian. Xiao Liu, Ming Ding, Jiezhong Qiu. Zhilin Yang. and Jie Tang. GLM: general language model pretraining with autoregressive blank infilling. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, 2022.
[15] Kevin Frans, Danijar Hafner, Sergey Levine, and Pieter Abbeel. One step diffusion via shortcut models. In The Thirteenth International Conference on Learning Representations, 2025.
[16] Samir Yitzhak Gadre, Mitchell Wortsman, Gabriel Itharco, Ludwig Schmidt, and Shuran Song. Cows pasture: Baselines and benchmarks for language-driven zero-shot object navigation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, 2023.
[17] Georgios Georgakis, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, and Kostas Danillidis. Learning to map for active semantic goal navigation. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, 2022.
[18] Benjamin Graham, Martin Engelcke, and Laurens van der Maaten. 3d semantic segmentation with submanifold sparse convolutional networks. In 2018 IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2018, Salt Lake City, UT. USA. June 18-22, 2018, 2018
[19] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, 2022.
[20] Kaiming He. Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, 2017.
[21] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems 33; Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020. December 6-12, 2020, virtual, 2020.
[22] Steeven Janny, Hervé Poirier, Lconid Antsfeld, Guillaume Bono, Gianluca Monaci, Boris Chidlovskii. Francesco Giuliari, Alessio Del Bue, and Christian Wolf. Reasoning in visual navigation of end-to-end trained agents: A dynamical systems approach. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11-15, 2025, 2025.
[23] Jindong Jiang, Lunan Zheng, Fei Luo, and Zhijun Zhang. Rednet: Residual encoder-decoder network for indoor rgb-d semantic segmentation. arXiv preprint arXiv: 08064, 2018
[24] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[25] Yiqing Liang. Boyuan Chen, and Shuran Song. Ssenav: Confidence- aware semantic completion for visual semantic navigation. In IEFE International Conference on Robotics and Automation, ICRA 2021, an,, China, May 30 June 5, 2021, 2021.
[26] Yaron Lipman, Ricky T. Q Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Lc. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, 2023.
[27] Guan-Homg Liu, Arash Vahdat, De-An Huang, Fvangelos A. Theodorou, Weili Nie, and Anima Anandku- mar. 12sb: Image-to-inet to-inage schrödinger bridge. In International Conference on Machine Learning, 2023, 23-29 July 2023, Honolulu, Hawaii, USA, 2023.
[28] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, 2023.
[29] Cheng Lu and Yang Song. Simplifying, stabilizing and scaling continuous-ting consistency models. In The Thirteenth International Conference on Learning Representations, 2025.
12

---

[30] Renjie Lu, Jingke Meng, and Wei-Shi Zheng. PRET: planning with directed fidelity trajectory for vision and language navigation. In Computer Vision ECCV 2024 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part LXVI, 2024.
[31] Andreas Lugmayr, Martin Danelljan, Andrés Romero, Fisher Yu, Radu Timofte, and Lue Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, 2022.
[32] Haokuan Luo, Albert Yue, Zhang- Wei Hong, and Pulkit Agrawal. Stubborn: A strong baseline for indoor object navigation. In JEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2022, Kyoto, Japan, October 23-27, 2022, 2022.
[33]
Weijian Luo. A comprehensive survey on knowledge distillation of diffusion models. CoRR, 2023.
[34] Arjun Majumdar, Gunjan Aggarwal, Bhavika Devnani, Judy Hoffman, and Dhruv Batra. ZSON: zero-shot object-goal navigation using multimodal goal embeddings. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
[35] Manolis Savva*, Abhishek Kadian*, Oleksandr Maksymets*, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform Embodied Al Research, In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
[36] OpenAI. GPT-4 technical report. CoRR, 2023.
[37] William Peebles and Saining Xie. Scalable diffusion models with transformers. In JEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, 2023.
[38] Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. Film: Visual reasoning with general conditioning layer. In Proceedings of the Shirty-Sccond AAAI Conference on Artificial Intelligence, (AAAI-18) 18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposiun on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, 2018
[39] Xavi Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Ruslan Partsey, Jimmy Yang, Ruta Desai, Alexander William Clegg, Michal Hlavac, Tiffany Min, Thco Gervet, Vladimír Vondruš, Vineent- Pierre Berges, John Turner, Olcksandr Maksymets, Zsolt Kira, Mrinal Kalakrishnan, Jitendra Malik, Devendra Singh Chaplot, Unnat Jain, Dhruv Batra, Akshara Rai, and Roozbeh Mottaghi. Habitat 3.0: co-habitat for humans, avatars and robots, 2023.
[40] Santhosh Kumar Ramakrishnan, Devendra Singh Chaplot, Ziad AI-Halah, Jitendra Malik, and Kristen Grauman. PONI: potential functions for objectgoal navigation with interaction-fie frre learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition. CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, 2022.
[41] Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexander Clegg, John M. Turner, Eric Undersander, Wojeiech Galuba, Andrew Westbury, Angel X. Chang, Manolis Savva, Yili Zhao, and Dhruv Batra. Habitat- Habitat-matterport 3d dataset (HM3D): 1000 large-scale 3d environments for embodied AI. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021.
[42] Ram Ramrakhya, Eric Undersander, Dhruv Batra, and Abhishek Das. Habitat-web: Learning embodied object-search strategics from human demonstrations at scale. IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA. USA, June 18-24, 2022, 2022.
[43] Hao Ren, Yiming Zeng, Zetong Bi, Zhaoliang Wan, Junlong Huang, and Hui Cheng. Prior does matter: Visual navigation via denoising diffusion bridge models. In IEEF/CVF Conference Computer Vision and Pattern Recognition. CVPR 2025, Nashville, TN, USA, June 11-15, 2025, 2025.
[44] Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few- shot paradigm. In CHI 21: CHI Conference on Human Factors in Computing Systems, Virtual Event Yokohama Japan, May 8-13, 2021, Extended Abstracts, 2021.
[45] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffuuson models. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, 2022.
13

---

[46] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assiste Intervention MICCAI 2015- 18th International Conference Munich, Germany, October 5- 9, 2015, Proceedings, Part III, 2015.
[47] Chitwan Saharia, William Chan, Huiwen Chang, Chris A. Lee, Jonathan Ho, Tim Salimans, David J. Fleet, and Mohammad Norouzi, Palette: Image-to-image diffusion models. In SIGGRAPH '22: Special Interest Group on Computer Gruphics and Inteructive Techniques Conference, Vancouver, BC, Canada, August 7-
11, 2022, 2022.
[48] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEFE Trans. Pattern Anal. Mach. Intell. 2023.
[49] James A. Sethian. Fast marching methods. SIAM Rev. 1999.
[50] Kunal Pratap Singh, Luca Weihs, Alvaro Herrasti, Jonghyun Choi, Aniruddha Kembhavi, and Roozbeh Mottaghi. Ask4help: Learning to leverage expert for embodicd tasks. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
[51] Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. In The Twelfth International Conference on Learning Representations, 2024.
[52] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, 2023.
[53] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, 2019.
[54] Yang Song, Jaseha Sohl-Dickstein Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
[55] Andrew Szot, Alex Clegg, Fric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot. Oleksandr Maksymets. Aaron Gokaslan. Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojciech Galuba, Angel Chang, Zsolt Kira. Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra. Habitat 2.0: Training home assistants to rearrange their habitat. In Advances in Neurul Information Processing Systems (NeurlPS), 2021.
[56] Huilin Tian, Jingke Meng, Wei-Shi Zheng, Yuan-Ming Li, Junkai Yan, and Yunong Zhang. Loc4plan: Locating before planning for outdoor vision and language navigation. In Proceedings of the 32nd ACM International Conference on Multimedia, MM 2024, Melbourne. VIC, Australia, 28 October 2024 1 November 2024, 2024.
[57] Alexander Tong, Kilian Fatras, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector-Brooks, Guy Wolf, and Yoshua Bengio. Improving and generalizing flow-hased generative models with minibatch optimal transport. Trans. Mach. Learn. Res., 2024.
[58] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian lehter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain thouuut prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
[59] Mitchell Wortsman, Kiana Ehsani, Mohammad Rastegari, Ali Farhadi, and Roozbeh Motagghi. Learning learn how to learn: Self-adaptive visual navigation using meta-learning, In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, 2019.
[60] Pengying Wu, Yao Mu, Bingxian Wu, Yi Hou, Ji Ma, Shanghang Zhang, and Chang Liu. Voronaw: Voronoi- based zero-shot object navigation with large language model. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, 2024.
[61] Xiaoyang Wu, Jiang, Peng-Shinn Shuai Wang, Zhijian Liu. Xihui Liu. Yu Qiao, Wanli Ouyang. Tong He, and Hengshuang Zhao. Point transformer V3: simpler, faster, stronger. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, 2024.
[62] Xiaoyang Wo, Yixing Lao, Li Jiang, Xihui Liu, and Hengshuang Zhao. Point transformer V2: grouped vector attention and partition- based pooling. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA. USA, November 28- December 9, 2022, 2022.
14

---

[63] Fei Xia, Amir R. Zamir, Zhi- Yang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: Real-world perception for embodied agents. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT. USA, June 18-22, 2018, 2018.
[64] Karmesh Yadav, Ram Ramrakhya, Arjun Majumdar, Vincent- Pierre Berges, Sachit Kuhar, Dhruv Batra, Alexei Baevski, and Oleksandr Maksymets. Offline visual representation learning for embodied navigation. In Workshop on Reincarnating Reinforcement Learning al ICLR 2023, 2023.
[65] Jiaoje Yan, Qieshi Zhang, Jun Cheng, Ziliang Ren, Tian Li, and Zhuo Yang. Indoor target-driven visual navigation based on spatial semantic information. In 2022 IEEE International Conference on Image Processing, ICIP 2022, Bordeaux, France, 16-19 October 2022, 2022.
[66]
Wei Yang, Xiaolong Wang, Ali Farhadi, Abhinav Gupta, and Roozbeh Mottaghi. Visual semantic navigation using seene priors. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
[67] Joel Ye, Dhruv Batra, Abhishek Das, and Erik Wijmans. Auxiliary tasks and exploration enable objectgoal navigation. In 202/ IEEF/CVF International Conference Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, 2021.
[68] Xin Ye and Yezhou Yang. Efficient robotic object search via HIEM: hierarchical policy learning with extrinsccexttrinscc modeling. IEEE Robotics Autom. Lett., 2021.
[69] Hang Yin, Xiuwei Xu, Zhenyu Wu andJic Zhou, and Jiwen Lu. Sg-nav: Online 3d scene graph prompting for Iim-based zero-shot object navigation. Advances in Neural Information Processing Systems 38: Annual Conferenceon Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver,BC BC, Canada. December 10 15, 2024, 2024.
[70] Hang Yin, Xiuwei Xu, Linqing Zhao, Ziwei Wang, Jie Zhou, and Jiwen Lu. Unigoal: Towards universal zcro-shot goal-oriented navigation. arXiv preprint arXiv:2503. 10630, 2025.
[71] Bangguo Yu, Hamidreza Kasaci, and Ming Cao. L3MVN: leveraging large language models for visual target navigation. In IROS, 2023.
[72] Xinyao Yu, Sixian Zhang, Xinhang Song, Xiaorong Qin, and Shuqiang Jiang. Trajectory diffusion for objectgoal navigation. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15. 2024, 2024.
[73] Alhert J. Zhai and Shenlong Wang. PEANUT: predicting and navigating to unseen targets. In IEEF/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, 2023.
[74] Jiazhao Zhang, Liu Dai, Fanpeng Meng. Qingnan Fan, Xuclin Chen, Kai Xu, and He Wang. 3d-aware object goal navigation via simultaneous exploration and identification. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, 2023.
[75] Sixian Zhang, Xinyao Yu, Xinhang Song, Xiaohan Wang, and Shuqiang Jiang. Imagine before go: Self- supervised generative map for object goal navigation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, 2024.
[76] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip H. S. Torr, and Vladien Koltun. Point transformer. In 2021 JEEF/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, 2021.
[77] Kaiwen Zhou, Kaizhi Zheng, Connor Pryor, Yilin Shen, Hongxia Jin, Lise Getoor, and Xin Eric Wang. ESSC exploration with soft commonsense constraints for zero-shot object navigation. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii USA, 2023.
[78] Yu Zhou, Dian Zheng, Qijie Mo. Renjie Lu, Kun-Yu Lin, and Wei-Shi Zheng. Decoupled distillation to erase: A general unlearning method for any class-centric tasks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN USA, June 11-15, 2025, 2025.
[79] Minzhao Zhu, Binglei Zhao, and Tao Kong. Navigating to objects in unseen environments by distance prediction. In IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2022, Kyoto, Japan, October 23-27, 2022, 2022.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We clearly demonstrate our claims in both abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discussed the limitations in Sec
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
This paper do not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We illustrated all experimental setup and details for reproduction in Sec. and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification:
We will release the code upon the decision the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
sttpiiisssooooiocyyy. for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We detailed all essential information for experimental setting in Sec. 5.1 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We report results across five random seeds in Appendix D.2 to capture vari-
ability.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
18

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We reported the compute resources in Sec. 5.1 and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[NA]
Justification:
Our proposed method only used academic research currently.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All the datasets used in this paper are publicly available, and cite them all.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
21

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: We incorporate LLMs into supervision of our model, which is one of the main claim of this paper. We describe the specific use in Sec. 4.3
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
22

---

Appendix
We provide additional information about our method and experiments in the appendix. Below is summary of the sections:
Appendix A presents the strict mathematical formulation of Conditional Flow Matching (CFM), with additional notation omitted the main text for simplicity.
Appendix describes the training algorithm with pseudocodes.
Appendix outlines the experimental setup and further implementation details.
Appendix
D
provides more experimental results and analysis.
Appendix
includes the prompts used for querying the LLM and its responses.
Appendix
provides additional visualizations of our results.
Appendix A
Strict Formulation of CFM
The Flow Matching loss defined Equation is, practice, not directly solvable (since the target velocity X; is not tractable). Throughout the paper, we refer to the Conditional Flow Matching (CFM) algorithm without explicitly including the conditioning variables in the notation, for simplicity. In this section, we present the strict and complete formulation of the Conditional Flow Matching algorithm.
Conditioning design in CFM vary, examples include conditioning on source sample Xos the target sample X1, or joint coupling (X,,X) X1), and they are essentially equivalent. We exhibit the general formulation (conditioned on any random variable Z).
Following [57], suppose that marginal probability path pr()) is mixture of probability paths pr (x]]=) that vary with some conditioning variable z:
pr(x) - f pe(a =)d(z)dz.
(15)
The marginal velocity field, which generates this marginal probability path, is given by averaging the conditional velocity field (x 2) across the condition
ue ue(x) = f ux(xx) [x] 2))2)((2 x))2 E (ux( X, x]
(16)
Conditional Flow Matching Loss is then defined to be:
LCFM (0) = Ex, 2) [D (ue(Xe 2), ui (X+))]
(17)
In practice, we actually apply data-coupling conditioning (namely Z = (Xo, X1)) shown in Eq. 12 X, ~ Ps is given by a linear combination of X, and X1, and velocity field is hence given by X, Then the loss in Eq 13 can indeed solve the Flow matching problem introduced in Sec.
Xo-
Appendix B
Algorithm
Algorithm Doutlines the procedure for constructing data-dependent couplings and training the GOAL model. The corresponding sampling process is detailed in Algorithm 2 We stress that the sampling requires no additional pre- processing, making it alug-and-pla module.
Appendix C Experimental setup and implementation details
C.1 Metries
As mentioned in the main text, we select SR, SPL,, DTS the evaluation metrics for ObjectNav performance, Following give their mathematical formulation and explaination.
23

---

Algorithm Training algorithm for GOAL
1: Input: dataset X' initial model parameter 0, learning rate distance matrix 'D and confidence matrix C from LLM responses.
2: repeat
3: sample X' ~ X
4: Randomly sample two points 91 92 on the grid
5: Compute visible mask 7 by planning path from 91 to 92
6: Xo < + X"
7: Cluster Xg into observed objects {o:}
8:
Initialize PLEM
zero vector with the same shape X"
9:
for each o; in {or]:== do
10:
PLLM PLLM ComputeLLMPrior((,, D,C)
(5)
(3)
See Eq.
x + x1 + * o PLLM
11:
12:
end for
13:
No + Xo + +7@N(O, Ao?)
14:
to [0
15:
X, + (1- t)Xo + 1
d. uo (1, 1)
17:
C MSE út, X1 - Xo)
18:
0 0+0-
19:
until convergence
Algorithm 2 Sampling algorithm for GOAL
1: Input: trained GOAL model ug. partially observed semantic map M, number of steps n, standard deviation Åo.
2: y + mask of empty area of M
3: M + + (0, A
4: for + to n do
5:
is <- k
6:
ÁM uo tk)
7: M t M + AM
8: end for
9:
return M
Success Rate (SR) represents the agent's accuracy reaching user-specified object goal, where higher values indicates better performance:
N Σ
SR
S
(18)
N
where N is the number of validation episodes and S; indicates whether the i-th episode is successful.
Success weighted by Path Length (SPL) evaluates success relative the shortest path, normalized by the actual path length agent takes:
N SPL Σ S.
(19)
max (4, 47)*
where 1ú denotes the shortest path length and I, is the actual path length agent takes.
Distance To Goal (DTS) measure the distance of agent towards the target object when the episode ends:
DTS max(Tnig E, 0):
(20)
where Ligg is the distance between agent and goal, and is the success threshold (0.1 meter in MP3D).
24

---

Table 7: Chosen object categories in Gibson
63 MP3D
and HM3D 41
Dataset
Training
Evaluating
Gibson
chair, couch, potted plant, bed, toilet, dining-table, oven, sink, refrigerator, book, clock, vase, cup, bottle chair, table, picture, cabinet, cushion, sofa, bed, chest of drawers, plant, sink, wilet, stool, towel, tv monitor, shower, bathiub, counter, fireplace, gym equip- ment, seating, clothes
chair; couch, Iv, bed, toilet, potted plant
MP3D
chair, table, picture, cabinet, cushion, sofa, bed, chest of drawers, plant, sink, toilet, stool, towel, tv monitor, shower, buthtub, counter, fireplace, gym equip-
ment, seating, clothes
HM3D
chair, couch, potttd plant, bed, toilet,
C.2
Object categories
Following the setup of previous works 40 categories for validation in the Gibson dataset, and 21 categories for both training and validation in the MP3D dataset. Additionally, validation episodes the HM3D dataset contain 6 categories. The adopted categories are detailed in 7
72
75 we adopt 15 categories for training and
Hyper-Parameterr
Values
C.3 Trivial hyper-parametens
Ty (Eq.
2.5
Tc (Eq.
0.85
There are number of trivial hyper-parameters which are not tuned, we detail the choices we adopt intuitively in Tab. ffor better reproduction.
miin (Eq.
20
max (Eq.
50
A (Fq.
8
1500
Ar (Eq.
0.01
Table 8: Values for trivial bypee-parameters
C.4 Memory and Time Cost Analysis
Our method involves maintaining point cloud representations and performing scene segmenta-
tion, along with generative modeling for exploration. These components naturally raise concerns about memory and computation overhead.
Unlike semantic maps, whose memory usage fixed by tensor grid dimensions, point cloud memory consumption varies significantly across scenes and episodes, depending on how much the environ- ment has been explored. As result, scene-independent comparison difficult. In practice, observed that running 6 parallel threads on a 24GB NVIDIA RTX 3090 consumes approximately 22GB of memory. For comparison, PONI 440 reports around 20GB under similar settings, indicating our approach adds roughly 350MB per thread.
Time cost also varies across scenes and depends on the agent's 's waypoint update frequency. Since GOAL is only invoked when the agent is either close to or far from the previous target, inference is sparse and input-dependent. Empirically, running threads in parallel yields average FPS between 1.2 and 1.8, while PONI ranges from 1.5 to 3.5. Despite this, we consider the trade-off worthwhile, method achieves over 30% improvement in success rate (SR), the primary metric of interest.
C.5 Training of scene segmentation model
We train Sparse UNet 18 46 with the assistance of Pointcept [T]] codebase, using sum of weighted Cross-Entropy loss Lcg and Lovász-Softmax loss Liz training objectives:
= LCF + LLz.
(21)
For optimizer, we simply select Stochastic gradient descent (SGD) 4 with base learning rate of 0.05, momentum 0.9 and weight decay of 0.0001. Additionally, the first 5% of the training steps are used for warm-up, and the learning rate smoothly decays using cosine annealing over the remaining 95% of the training steps. We train the scene segmentation model on 2 NVIDIA RTX 4090 GPUs with total batch size of 64.
25

---

Appendix D
More experimental results
D.1 Hyper-parameters tuning
545
41.25
DIV
147
We tune two key hyperparameters: the expansion ra- tio of the observed map e, and the number Euler steps used during generation. The effect of varying number of Euler steps n is shown Fig. 5 Nav- igation performance generally improves with more steps, saturating around n 96. As discussed in Sec while the overall performance across dif- ferent LLMs is comparable, each exhibits distinct preference for the expansion ratio Therefore, tune individually for each LLM, as shown in 6 We observe that the flow model distilled with exter- nal knowledge from ChatGPT produces more reliable semantic distributions with larger expansion ratio.
41.00
146
145
435
40.25
14.3
40.00
142
6.50
64
Figure 5: Effect of the number of Euler steps Ti on navigation performance.
5628
SPC
SEC
525
159
45
4779
10.3
4225
4.25
480
647
6776
0.71
(a) ChatGLM
(b) Deepseek
(c) ChatGPT
Figure 6: Tuning curve for hyper- parameter across different LLMs.
D.2 Evaluation Variability and Error Analysis
While probabilistic generative schemes offer improved generalization, they naturally introduce concerns about evaluation stability due inherent stochasticity. To assess the robustness of our model, we conduct additional evaluations on the MP3D dataset using different random seeds (42, 75, 100, 123, 3407). The resulting success rates 41.0%, 41.6%, and 41.7%, 41.6%, 42.2% respectively, yielding an average success rate of 41. 41.6% + 0.4%. These results indicate that our flow model can reliably capture the semantic distribution, despite the stochasticity introduced by the generative process. We report the result with seed 100 in the main text, as all experiments and hyperparameter tuning were conducted under this setting.
Appendix E Prompts and LLM responses
E.1
Prompts
We provide an example of prompts that query LLMs for objects contextual information. Specifically, first condition the L.M with its role in the system prompt. Next, provide contextual infor- mation, followed by a chain-of-thought (58] style of hierarchical prompting, which involves scenes, rooms, and objects. Moreover, for each step, also provide some positive and negative examples serve as few-shot learning samples. In addition to the distance-confided pair, we further require the LLMs output brief reasoning for their responses, An example prompt is follows:
26

---

Prompt
You are an expert in indoor scene layouts, with strong reasoning skills regarding object co-occurrence. Your task to infer the typical distances between different object types in indoor environments, considering both object placement patterns and functional relationships. Output your answers clear, structured format with a confidence level that reflects the uncertainty of each estimate.
System Prompt:
In indoor scenes, object layouts typically follow certain patterns; for example, chairs are usually placed around table but are unlikely be near toilet. Suppose you are analyzing scale-scale indoor scene (c.g., house with multiple rooms such as living rooms, bedrooms, bathrooms, etc.). Given the following list of objects: [chair, table, ...] and specific central object in it, your tasks are as follows, step-by-step:
User Prompt:
Inferring Object Placement: Determine where each object is typically placed in the scene based on common indoor layouts. For example:
Toilet + bathroom
Room Proximity: Identify the typical rooms surrounding the room where central object is placed. For instance:
Chairs (living room) - nearby: kitchen, hallway
Bed (bedroom) > nearby: bathroom, closet
3. Functional Relationships: Consider the functional relationships between the central object and other objects:
Chair & table: strong (dining/study areas)
Chair & toilet: weak (different functional zones)
4. Distance Inference: Based on the inferred scene layout and functional relationships,
estimate the typical distance between central object and other objects. For example, the central object is 'chair', then:
table to chair: close (functional pair)
Chair to closet: relatively farther away (possibly the same room but with weak func- tional relationship)
Chair to toilet: large distance (different functional zones)
Provide Distance and Confidence: Given central object = =... for each object provide:
Distance: continuous float value in meters
Confidence: score between 0 (uncertain) and (certain)
Reasoning: brief explanation how the distance and confidence were determined, considering the room layout and common object placements
E.2 Responses
Few examples for LLMs response are as follows:
Response
central object: table
quried object: chair
distance: 0.5
confidence: 0.95
reasoning:
Chairs are commonly placed directly around a table, especially dining areas or meeting rooms, making the distance very short and the placement highly predictable.
27

---

quried object: picuture
distance: 1.5
confidence: 0.8
reasoning:
Pictures are often hung on walls near tables, especially in living or dining rooms, but not as close chairs, leading to moderate distance with good confidence.
quried object: counter
distance: 4.5
confidence: 0.65
reasoning:
Counters often found in kitchens, which may be adjacent to dining areas but not directly next tables, leading to moderate distance with moderate confidence.
quried object: bathtub
distance: 8.0
confidence: 0.5
reasoning:
Bathtubs are also found bathrooms, separate from areas where tables are commonly placed, resulting in a large distance with low confidence.
10-
chai
0:03
1:50
3.55
2.60
2:55
1:00
1.37
2:13
0006
012
0.9
2.50
2.80
oudkicn
0.22
2.22
0:22
020
2:40
1.49
3.
4:2
5.09
1.20
3:55
1.59
6.00
1:32
5.59
1:70
435
2.25
2.59
0.55
2.54
0.50
0/22
D:72
10:20
LAK
3ed
0.76
2.50
0:70
6:4
5:57
3:00
4.72
07-
thett
0.22
8.00
Confidence
0.6
0.37
250
9:59
Distance
hilee
0:17
529
6m20
830
674
4:2
6.22
1.30
2:0
3.20
3:20
S/4:
2.05
0.5
0.32
3.0
0.65
0.77
0.63
7:22
0.60
0.50
0:00
375
032
4.9
2.19
9%
2007
0.34
223
0.64
9:23
0.29
291
0.10
4.30
9.72
0.36
220
2.23
0.26
0.54
0.40
KKK
1000
9.10
countar
0.85
2.00
0.50
0.23
0.43
0:33
0.40
0.32
0:72
297
0.30
0.24
0.70
3.40
0.42
0.32
6.23
5.00
2.25
4.25
Frop ace
0:02
3.89
10.0
044
0.20
015
020
02-
0.57
0.33
0.25
0.20
0.24
0.1-
0.22
230
0.28
0.12
0.23
382
5214
Womorbor
Figure 7: Structured distance matrix D and confidence matrix C obtained from GPT-4 for MP3D. We visualize the upper triangle the distance matrix and the lower triangle of the confidence matrix within the same figure for compactness and clarity
28

---

Since the distance between objects should be bidirectional, we take the average of the symmetric distances and confidences, resulting two symmetric matrices. Structured matrix representation of I.M response from GPT-4 is shown in Fig.
Obsrnedrrop
map
Smondd-trutime map
Figure 8: Example for semantic map generated by generative flow model. It successfully generates objects GT semantic map like cabinet (highlighted in red circle) and cushion (highlighted in blue circle). Moreover, it also generate objects not in the ground-truth map but indeed reasonable, such as seating and chair (highlighted purple circle) behind the table, which can improve the capability of generalization.
out-of bound
foor
wall
chair
table
bed
chest www.orrooooooooooooooooooom
plant
slink
toilet
<tool
counter
fireplace
sym, equipme
seating
clother
out-of-boound
floor
wall
chair
twhle
sofe
bed
chest at dra
plant
sinik
tollet
wy_monitor
bathitub
counter
firepiace
seating
Observed maps
Generated maps
Ground-truth -ruth maps
Figure More visualizations for outputs of generative flow model.
29

---

Appendix F More visualizations and analysis
F1
visualizations for generative flow
In this subsection, we present visualizations and analyses of the outputs produced by our generative flow model. Generating complete semantic distribution of a scene is an inherently challenging task and unlikely to be perfectly accurate. While the visualizations may not appear flawless, they signifi- cantly contribute to navigation performance. We begin with an illustrative example accompanied by detailed analysis (see B) followed by additional qualitative results (Fig In this paper, we stress that indoor scene semantics can vary greatly, and multiple plausible distributions may exist given the same partial semantic map. To capture this diversity, GOAL adopts probabilistic generation scheme, which enhances the model's ability to generalize to unseen environments. We showcase sample of this generative diversity in Fig. TO While we highlight examples where objects are generated near observed ones to provide intuitive ovaluation, the diversity applics to the full semantic distribution and is not limited to individual object placements.
F.2 visualizations for scene segmentation
In Fig. II
we additionally present few visualizations for comparison between scene segmentation proposed in this paper and image segmentation traditionally adopted in ObjectNav.
Iowrd noor (on) courter
Fimplacee noar table (living area)
chair neer talée
sitk near fonl counter
Seeting& coer table
Doumter
lable
showver
battiuub
table near bed
(balhroom)
and toller, (lathrom)
Seating
Figure 10: Visualization of generation diversity. Given a single partial map (left most), GOAL can generate multiple plausible full semantic distribution (right three).
30

---

chest of drawers
gym equipment
toilet
plant
Image segmentation
Scene segmentation
Ground-tritt tuut
Figure 11: Comparison between built semantic maps using image segmentation models and scene segmentation models.
31


Reviews:
[{"review_id": "cXNqPPEAjb", "reviewer": "Reviewer_Hp6p", "summary": "This paper introduces a framework for ObjectNav based on semantic map completion with a flow-matching-based generative model. It uses an LLM to perform a form of data-augmentation, injecting objects into semantic maps during training based on a pre-trained LLM’s prediction about likely co-occurring objects. At test time, the flow-matching generative model completes a plausible semantic map of the scene given the history of observations, which is then used to select navigation targets for ObjectNav. The method is compared against several baselines for navigation via envrionment imagination, showing SOTA and transfer to environments that the generative model is not trained on.", "questions": "Based on equations 6,7,8 it seems as if there is no protection for augmenting objects being placed across a wall from it’s co-occurring object, making it’s geodesic distance much farther than the predicted distance. \n\nIt’s not clear to me exactly how segmentation is performed. As is it seems as if this work trained a sparse convolutional network from scratch on RGB pointclouds. If this is is the case there should be more details included about training.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "*Strengths*\n- This paper proposes a strong system for object navigation with several good ideas and engineering choices. Their LLM based map-augmentation scheme incorporates priors from LLMs effectively while avoiding the need for running an LLM at inference time, and the use of flow matching for map inpainting is a natural upgrade in methodology for this line of work based on more current inpainting research. \n- Paper is well written and the language is clear. Many related works are compared against and many relevant ablations are included. \n- The results are strong and clearly give SOTA performance.\n\n\n*Weaknesses*\n- Missing details on the “Effectiveness of data-dependent couplings” ablation: Inpainting/outpainting with diffusion models is already a well studied problem. While the inpainting formulation used in this paper is compelling and could make sense, there are very few details about the baseline used to justify the contribution of this non-standard inpainting scheme. UNet style models typically concatenate the conditioning to noise [1] while several different conditioning schemes are used for DiT based models [2,3]. This paper would be stronger if these methods were explicitly compared against, or one of these inpainting schemes were adopted directly. As is, there is not enough to justify the proposed inpainting scheme as a contribution.\n\n- Incomplete ablations against SGM. Some of the major differences between this work and SGM that need more experimentation:\n1) Scene segmentation vs per-frame segmentation\n2) The use of flow matching vs MAE\n3) LLM conditioning techniques (conditioning on features vs directly augmenting maps)\nSGM claims to do segmentation using single frame models (Mask-RCNN, RedNet) while this work uses scene segmentation. In Table 3 this work reports a 5 point SR increase from using scene segmentation. If the same increase were found from simply using scene segmentation with SGM, then SGM would actually be out-performing this method on MP3D, implying that either MAE or the SGM language conditioning technique is superior to the techniques from this paper. This paper would benefit from a comparison against SGM using scene segmentation, and SGM using this method’s language augmented maps as part of training.\n\n[1] Rombach, Robin, et al. \"High-resolution image synthesis with latent diffusion models.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.\n\n[2] Nitzan, Yotam, et al. \"Lazy diffusion transformer for interactive image editing.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.\n[3] Wu, Xian, and Chang Liu. \"DiTPainter: Efficient Video Inpainting with Diffusion Transformers.\" arXiv preprint arXiv:2504.15661 (2025).", "was_revised": true, "final_justification": "As authors have addressed my questions and concerns, I change my score to accept.", "rebuttal": "We sincerely thank you for your comprehensive review and valuable feedback on our submission. We are pleased that you recognized the strengths of our work, including the novel LLM-based map augmentation, the effective use of flow matching for map inpainting, the well-written paper, and the strong, state-of-the-art results. We address your concerns below.\n\n# Weakness1: Missing details and potential baselines for 'Data-dependet coupling'\n## Details of the 'Independent Couplings' Baseline\nWe apologize for inclarity brought by our incomplete details for the baseline for 'independent couplings', only give a brief introduction in line 265-267. Here we first dive more into the conditional mechanism for baseline: We adopt **two convolution blocks to downsample the partial semantic map to avoid computational bottleneck**, and adopt a Patch embedder to project it into higher dimension. Then we interpolate Gaussian noise with ground truth map and also send into a patch embedder. We then use **DiT blocks with cross attention layers** [1], since we suppose the partial semantic map condition is highly expressive and simple layer norm may not suitable. The interpolated noisy patches are send into sequential DiT blocks as input, and partial semantic map patches are **sent to the cross attention layer of each DiT block to serve as condition**. \n\n## Comparison with DiTPainter\n\nSince reviewer also mentioned some of other conditioning mechanism, we choosed to compare with DiTPainter [2] mentioned as it also adopt a flow matching training pipeline and reveal similar idea. It is important to note that **DiTPainter was preprinted on Arxiv only three weeks before the NeurIPS 2025 submission deadline, making both works concurrent in their exploration of direct coupling mechanisms**.\n\nAs DiTPainter does not open-source their codes, we first reimplement it according to the main idea and adapt to our objectgoal task. It adopts a somehow similar mechanism with ours: They also adopt a dependent couplings mechanism but with more inputs. They project interpolated noise, corrupted video, and binary masks to higher dimensions individually, and add them to serve as the input to model, namely $x = \\text{embedder}_1(x_0) + \\text{embedder}_2((1-t)\\epsilon + tx_1) + \\text{embedder}_3(M)$ where $x_0$ is the corrupted data, $x_1$ is the full videos, and $M$ is the mask representing corrupted regions. However, we merge the information of multiple inputs before projection, namely $x = \\text{embedder}(f(x_0, x_1, \\epsilon, M))$ which eliminate the cost of extra embedders. \n|model| SR$\\uparrow$| SPL$\\uparrow$ | DTS$\\downarrow$\n|-------| ------|--------|-------|\n|DiTPainter|41.4| **15.6**| **4.84**|\n|GOAL| **41.7** | 15.5 |**4.84**|\n\nAs shown, GOAL achieves comparable navigation performance with slightly higher SR, while incurring less model complexity due to our streamlined input processing. Given the comparable performance and reduced complexity, we will retain our conditioning mechanism. While our work is concurrent with DiTPainter, we acknowledge that this specific conditioning mechanism should be presented as a technical improvement for building a strong flow-matching system for object-goal navigation rather than a core conceptual contribution in the revised version of the paper.\n\n# Weakness2: Incomplete comparison with SGM\nWe conduct multiple comparison with SGM. The experimental results shows that **SGM benefit less from scene segmentation**, inheriting from the random masking scheme of MAE. Furthermore, **flow matching exhibits stronger capabilities of generalization in transfer setting**.\n\nSince SGM does not open-source their models and evaluation scirpts on MP3D, we reproduce it in reference to their codes for training and evluating SGM on Gibson. However, as they also did not open source the exact prompts and details on how they get language guidance, we reproduce only the naive version without language guided, and refer it to MAE. To reach a fair comparison, we compare MAE with our method wihout LLM, referring to FM. \n## Scene segmentation vs per-frame segmention\nwe refer scene segmentation to 'SG'\n\n|Algorithm| SR$\\uparrow$| SPL$\\uparrow$|DTS$\\downarrow$\n|---------|-----| ------| -----\n| **w/o SG** ||||\n|MAE| 31.9 |**11.7**|5.31\n|FM|**32.4**|**11.7**|**5.25**\n| **w/ SG** ||||\n|MAE+SG|36.7|14.3|5.16\n|FM + SG|**38.8**|**14.6**|**5.08**\n\nOur method delivers stronger performance in both settings, with more performance gains in the w/ SG configuration. This is largely because our direct coupling method in FM relies more heavily on the reliability and consistency of the entire observed semantic map, which scene segmentation enhances. For MAE, which typically selects a fixed number of patches as input, the inherent consistency of the input semantic map can be disrupted by the patch selection process, thus limiting the benefit of consistent scene segmentation.\n\n## Flow matching vs MAE\nWe further test the capabilities of generalization in transfer setting of Flow matching and MAE (trained on MP3D and directly test in HM3D), the results are as follows:\n\n|Algorithm| SR$\\uparrow$| SPL$\\uparrow$| DTS$\\downarrow$|\n|----|-----|-----|-----|\n|MAE|32.1|**14.70**|4.85|\n|FM|**35.9**|14.69|**4.65**|\n\nFM achieve a 3.8 SR gains, and the reason of minor decrease of SPL (0.01) has already discussed in the main paper. **Flow matching exhibits a significant better generalizability probably due to its multimodal output**,  which better capture the underlying distribution of possible semantic maps in unseen environments.\n\n## LLM conditioning techniques.\nWe acknowledge that directly transferring our LLM augmentation mechanism to MAE is challenging, as MAE's training paradigm (masking random patches) does not inherently involve the concept of partial and complete maps in the same way our flow-based model does. Our augmentation mechanism is specifically designed to simulate real navigation paths and augment only at the boundaries of observed regions.\n\nMoreover, our primary goal with LLM conditioning was **to completely eliminate the need for LLM usage during inference while still benefiting from its commonsense knowledg**e. We did not aim to surpass the language guidance in SGM, which operates on a different principle by directly conditioning on language features during inference. Our contribution lies in distilling this knowledge offline into a generative model that can then operate independently.\n\n# Question1: Protection for placing objects across a wall\n\nWe appreciate you raising this important practical consideration. While our LLM-distilled prior provides commonsense spatial relationships based on **Euclidean distances** (instead of geodesic), our modified local planner (initially designed for efficient inference) planning **according to FMM distance (geodesic)** can mitigate the problem during actual navigation.\n\nAs stated in line 218, our modified local planner **selects a new long-term goal when the FMM distance of current goal is too far away**. If GOAL selects a goal across the wall, **the agent's planner would immediately detect that the geodesic distance is excessively large and prompt agent to plan again**. This makes the issue not critical for the overall navigation performance.\n\n# Question2: Details for segmentation.\n\nYes, we confirm that we trained a sparse convolutional network from scratch on aggregated RGB point clouds for 3D semantic segmentation. We apologize for the oversight in not directly referencing the detailed training specifications in the main paper.\n\n**These crucial details regarding the segmentation model's training can be found in Appendix C.5**. We will ensure that a clear and prominent reference to Appendix C.5 is added to the main paper in the revised version to enhance reproducibility and clarity for the readers.\n\nWe hope these clarifications and additional results address your concerns. We are confident that with these improvements, the paper will be even stronger.\n\n[1] Scalable Diffusion Models with Transformers. ICCV 2023.\n\n[2] DiTPainter: Efficient Video Inpainting with Diffusion Transformers. arixv preprint 2025."}, {"review_id": "ymPmeppaC4", "reviewer": "Reviewer_wb79", "summary": "This work presents a generative flow-based framework that models the semantic distribution of indoor environments by bridging observed regions with LLM-enriched full-scene semantic maps.", "questions": "1. Does the LLM give the same numbers for the distance and confidence, even when the same prompt is used as the input of the LLM? \n2. In Figure 8, the gym equipment and fireplace seem to be in multiple, distant places in the same environment. How accurate are the generated maps? Is there a numerical method to measure this?", "limitations": "I am not sure how reliable are the numerical values of distance and confidence that are an output of LLM. As the map is generated using this information from the LLM output, I am unsure how reliable is the quality of generated maps.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The authors show state-of-the-art performance on Gibson and MP3D datasets. Moreover, they show good generalisation on HM3D dataset.\n\nThe authors have not compared the performance of their model with RIM [1].\n\nThe generated maps rely on an LLM prompt to predict numerical values of distance and confidence, alongwith reasoning. I think there would be some randomness in the output of values for distance and confidence. As LLMs are autoregressive models, I am not sure how reliable they would be at predicting numerical values, also given that they are prone to hallucination. Perhaps, you could train a transformer encoder to predict the values of distance and confidence, instead of prompting an LLM.\nMoreover, the qualitative results in the appendix in Figure 8 show gym equipment and fireplace in multiple, distant places in the same environment. Usually in a house, all the gym equipment is present in one area and not scattered in different places. So, I am not sure how accurate and reliable are generated maps.\n\n[1] Chen, Shizhe, Thomas Chabal, Ivan Laptev, and Cordelia Schmid. \"Object goal navigation with recursive implicit maps.\" In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 7089-7096. IEEE, 2023.", "was_revised": true, "final_justification": "The authors have addressed all the concerns that I raised.", "rebuttal": "We appreciate your insightful questions and concerns, which we address comprehensively below.\n\n# Weakness1: Comparison with RIM\nWe appreciate you raising the question of comparison with RIM [1]. While RIM is an interesting and important work in object goal navigation, we did not include a direct empirical comparison for the following reasons:\n* RIM involves **extensive data collection through robot interaction in simulators and incorporates auxiliary training tasks**, as also noted by T-Diff [2]. This makes a direct and fair comparison challenging.\n* RIM **exhibits a fundamentally different paradigm from our work**. It does not incrementally build a map to map on, nor incorporating LLM knowledge. We mainly compared with methods more aligned with our explicit map-centric and LLM-enhanced approach (e.g., UniGoal, GAMap, SG-Nav, T-Diff, SGM).\n\nHowever, we agree on RIM's relevance within the broader object goal navigation landscape. We will ensure it is properly cited and discussed in the \"Related Work\" section of the revised manuscript to contextualize our contributions more comprehensively within the field.\n\n# Weakness2&Question1: The randomness of LLM response.\n\nWe acknowledge the reviewer's valid concern regarding the inherent variability in LLM responses, even with identical prompts. **We provide quantitative results to demonstrate that the differences are minor.** To quantitatively assess the consistency of our distilled LLM prior, we measured the similarity between multiple distance and confidence matrices generated by the same LLM (ChatGPT) using the same prompt.\n\nWe employed two key metrics:\n\n* **Cosine Similarity**: Measures the angular similarity between two matrices, indicating if they share similar patterns of higher/lower distances/confidences for object pairs:\n$\n\\text{Cosine}(A,B) = \\frac{A\\cdot B}{||A|| ||B||}.$\n* **Average Frobenius Norm**: Quantifies the average absolute element-wise difference between two matrices, providing a direct measure of numerical divergence:\n$\n\\text{AverageFrobenius}(A,B) = \\frac{1}{N_C^2}\\sqrt{\\sum_{i=1}^{N_c}\\sum_{j=1}^{N_c}|A_{ij} - B_{ij}|^2}.$\n\n**Methodology**: We prompted ChatGPT five times with the same input. Treating the first result as a reference (A), we computed these two metrics between the reference and the subsequent four results (B1-B4).\n\n**Results**: The consistency analysis is summarized below (values for Distance Matrices / Confidence Matrices):\n\n||Cosine$\\uparrow$|Frobenius$\\downarrow$|\n|----|----|-----|\n|B1|0.976/0.992|0.048/0.0040|\n|B2|0.968/0.988|0.050/0.0047|\n|B3|0.981/0.991|0.039/0.0041|\n|B4|0.974/0.990|0.047/0.0045|\n|Average|0.975/0.990|0.046/0.00433|\n\n\nThis analysis clearly demonstrates that responses from the same LLM, with identical prompts, **exhibit highly similar patterns** (average Cosine Similarity of 0.975 for distance and 0.990 for confidence). Furthermore, the **absolute differences are very small** (average Frobenius Norm indicating differences typically less than 5cm for distances and 0.5% for confidence values). Given that our semantic maps are represented as relatively smooth Gaussian fields, these small numerical differences are not expected to cause significant performance degradation. The generative model effectively learns to interpret this consistent, albeit slightly variable, prior for robust semantic scene completion.\n\n# Weakness3: Hallucination of LLMs.\n\nWe acknowledge the reviewer's concern regarding LLM hallucination, a known characteristic. Our approach **specifically mitigates this through a comprehensive, hierarchical prompting strategy designed to guide structured reasoning and constrain the output space**, thereby significantly reducing instances of hallucination in the distilled prior.\n\nWe strongly contend that **despite their limitations, the commonsense reasoning capabilities of LLMs are immensely valuable for ObjectNav tasks**, particularly for generalizing to unseen environments and novel object categories. This perspective is increasingly shared within the community, with a growing number of recent works demonstrating effective LLM applications in ObjectNav, including L3MVN [2], SGM [6], VoroNav [5], GAMap [4], SG-Nav [7], and UniGoal [3] etc. These works, alongside ours, highlight that judicious application of LLMs can significantly enhance embodied agents' capabilities, and the concern of hallucination does not warrant discarding their utility entirely for this domain.\n\n# Weakness4: Potential method for a transformer encoder instead of LLMs.\n\nWe thank the reviewer for this constructive suggestion regarding a transformer encoder. However, we stress that **a transformer encoder trained on existing datasets will not be comparable to the LLM which is trained with web-scale data.** An encoder trained on existing scene datasets would inherently be limited to the in-domain knowledge present within those specific datasets. Such an approach would essentially regress to patterns already observable in current scene data, offering **no fundamental difference from directly predicting semantic maps** trained with solely existing scene datasets.\n\nOur core motivation for augmenting semantic maps with LLMs is precisely to **leverage their vast commonsense knowledge, derived from web-scale data, which is crucial for out-of-domain generalization**. This allows our model to \"imagine\" plausible semantic layouts for unseen environments and novel object categories – a capability that a transformer encoder trained purely on existing scene datasets, without equivalent pre-training on diverse world knowledge, would not possess. Therefore, the LLM serves as a unique and invaluable source of generalized world knowledge for our task.\n\n# Weakness5&Question2: Misunderstanding on qualitative results.\n\nWe appreciate the reviewer's close inspection of our qualitative results in Appendix Figure 8.\n\nFirst, we wish to clarify a misunderstanding: **the objects identified by the reviewer as \"gym equipment\" are actually \"seating\"**. We ask the reviewer to further check and identify the figure carefully. \n\nBased on this clarification, we justify the qualitative results as follows:\n\n* **\"Seating\" Placement**: The scattering of seating across a scene, as observed, is quite common in diverse indoor environments (e.g., multiple living areas, dining nooks, individual chairs in hallways). In Figure 8, the generated seating instances are consistently placed in reasonable contexts, such as near walls with pictures or alongside generated tables, reflecting plausible arrangements.\n* **\"Fireplace\" Placement**: The reviewer noted \"multiple, distant\" fireplaces. Upon careful re-inspection of Figure 8, only the second row depicts two fireplaces, each placed behind a wall with pictures and accompanied by generated seating. This configuration is a common and reasonable arrangement for distinct living areas within a larger residence.\n\n* **Purpose of GOAL**: The purpose of GOAL is not to reconstruct exactly the whole scene semantic, but **focus more on plausible objects placement to guide object navigation**. This is the reason why we try to distill the object co-occurence information into flow model, for our object-level task.\n# Question2: Numerical method to measure the quality of generated map\nWe want to emphasize that **measuring the quality of the generated map quantitatively via numerical metrics can be misleading because our focus is on the downstream object navigation task**, and the performance on this task (SR, SPL, DTS) is the most fair and ultimate judgment. Our generative model produces plausible semantic completions, not exact reconstructions of a hidden ground truth, and these plausible completions are what enable robust navigation.\n\nHowever, since the reviewer is curious about relevant information, we have conducted an evaluation using metrics commonly used in segmentation tasks: mIoU, mAcc (on all 21 objects and free space, totaling 22 categories), and mean squared error (MSE). It is important to note that we generate validation data samples similarly to training samples, but without adding the LLM prior, as the LLM prior is introduced with stochasticity, making direct ground truth comparison for the augmented maps less suitable for validation. We also test the performance of the flow model without LLM knowledge distilled for a fairer comparison, as the supervision for the flow model with LLMs is no longer identical to the expected whole scene semantic map, but instead incorporates strong object co-occurrence information.\n\nWe compare the validation results with a recent work, SGM (also without LLM knowledge for fair comparison), in the same setting:\n\n|Model|mIoU$\\uparrow$|mAcc$\\uparrow$|MSE$\\downarrow$|\n|------|-----|------|------|\n|SGM|11.05| 12.70| 16.18|\n|GOAL| **35.20**|**35.63**|0.013|\n\nNote that SGM adopt a MAE-like algorithm, which will normalize the input and the MSE may not comparable, we refer the reviewer only to our absolute MSE instead of comparing with SGM.\n\nWe hope these comprehensive responses and additional analyses address all your concerns.\n\n[1] Object goal navigation with recursive implicit maps. IROS 2023.\n\n[2] L3MVN: Leveraging Large Language Models for Visual Target Navigation. ICRA 2023.\n\n[3] UniGoal: Navigate to Any Goal in Zero-shot! CVPR 2025.\n\n[4] GAMap: Zero-Shot Object Goal Navigation with Multi-Scale Geometric-Affordance Guidance. NeurIPS 2024.\n\n[5] VoroNav: Voronoi-based Zero-shot Object Navigation with Large Language Model. ICML 2024\n\n[6] Imagine Before Go: Self-Supervised Generative Map for Object Goal Navigation. CVPR 2024.\n\n[7] SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigation. NeurIPS 2024."}, {"review_id": "dFrebeRjhy", "reviewer": "Reviewer_ZGge", "summary": "The submission proposes GOAL, a generative flow-based framework that incorporates external knowledge from large language models to enhance the agent's ability to imagine unobserved regions in object goal navigation tasks. The method achieves state-of-the-art performance on MP3D and Gibson datasets and demonstrates strong generalization capabilities in transfer settings to HM3D.", "questions": "- When using models that can run on consumer-grade GPU, such as LLaMA 7B, for training, what is the performance of this method?\n\n- Since the generated content of LLM only includes distance and confidence, is there a significant gap in the content generated by different LLMs? Please provide quantitative analysis or examples for illustration.\n\n- For hyper-parameters tuning, is there a more reasonable interpretation of the current performance curve? Is it possible to propose a more systematic optimization approach?\n\n- Table 2 takes the model's real-time performance into account, and dependent couplings achieve a significant reduction in inference time (from 17.13ms to 12.38ms). However, for an episode, it seems that map prediction for waypoint selection does not need to be performed at every step. So, how much delay will be caused in an entire episode?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strength\n- The method outperforms existing approaches on multiple benchmarks, including MP3D and Gibson.\n\n- The submission provides experiments and ablation studies, demonstrating the effectiveness of each component of the framework and its overall robustness.\n\n- The writing and illustration of the submission are convenient for understanding.\n\n## Weakness\n- The primary weakness of the submission is the lack of novelty in methods/architecture. Similar to SGM, it predicts unobserved maps to guide navigation, differing mainly in specific technical approaches. But I also question the necessity of the proposed methods.\n\n- I question the necessity of the proposed methods:\n  - Regarding the \"Distilling LLM Prior\", the data shown in Table 1 also indicate that the improvement brought by LLM is very limited (SR:47.6→48.8). And when using LLMs, the prompts in this submission only provide central objects without specific information about other actual objects surrounding them, lacking environmental distribution information. This approach is insufficient for handling complex environments.\n  - Regarding the \"Flow Model\", Table 2 shows that the dependent couplings method(SR:41.5) outperforms independent couplings (SR:39.0), but the improvement is marginal. Given that dependent couplings rely on extensive hyper-parameter tuning (D.1 Hyper-parameters tuning), I find it difficult to be convinced of the effectiveness of the proposed method.\n\n- Demonstrated in \"D.1 Hyper-parameters tuning\" of the submission, the hype-parameters mainly relies on experimental testing and optimization. The performance fluctuates significantly with several hyper-parameters, and there is a lack of reasonable explanation, which deepens the concerns about the reliability of the method.\n\n- Although the experiment tested LLMs from different manufacturers, they are all LLMs with a large number of parameters (for example, DeepSeekR1 has 671B parameters), lacking the ability test of small-parameter models. However, works such as SG-Nav, often achieve good results on models with relatively small parameters (such as LLaMA 7B in SG-Nav).", "was_revised": true, "final_justification": "My concerns have been partially addressed. I vote for borderline accept.", "rebuttal": "We appreciate your critical questions and concerns, which we address comprehensively below.\n# Weakness1: Novelty of the paper.\n\nMap prediction is a fundamental direction of ObjectNav, and a series of works (L2M [1], SSC-Nav [2], SGM[3]) have endeavoured to improve this scheme, we follow this direction with following novelties:\n\n* **Novel Flow-Based Generative Inpainting**:We introduce a robust system of **flow-based generative model for semantic inpainting**, providing superior fidelity and handling uncertainty. \n* **New Scheme for Distilling Task-Specific LLM Prior**:Our method uniquely distills **structured, task-relevant commonsense knowledge from LLMs offline**. This specialized prior is directly integrated into our generative model, enabling it to **operate efficiently during inference without requiring any LLM calls**.\n\n# Weakness2: Effectiveness of \"Distilling LLM Prior\".\nWe direct reviewer's attention to Table 3, where the improvement from incorporating the LLM prior is much higher in the in-domain setting, **increasing SR from 38.8 to 41.7**. This represents a substantial 2.9 absolute gain, clearly demonstrating the effectiveness of this module in enhancing performance. While the benefits might be slightly attenuated in the highly challenging cross-dataset transfer setting (Table 1), the in-domain results strongly validate the module's contribution.\n\nWe appreciate you highlighting the challenge of prompting LLMs for complex co-occurrence patterns. While prompting with only a central object might indeed fall short in capturing the richer context of clustered objects, we **address this in complex scenarios by employing a practical data augmentation trick**: during training data generation, **we prioritize augmentating  objects that are part of co-occurring sets identified by more than one observed objects (namely the intersection of co-occuring sets of observed objects).** We initially omit this detail as it's simplicity, but we will include it in appendix in the final version for better reproductivity. We hope this can to some degree solve the problem in complex scenario without extremely more prompts. (Say if we consider the probability of cluster with 3 objects, there will be $\\binom{21}{3} = 1330$ possible pairs, making it exhausting)\n\n# Weakness3: Effectiveness of flow model with \"Dependent couplings\".\n\nWe clarify as follows:\n\n* First, in challenging real-world benchmarks like MP3D, **a 2.5 absolute gain in SR is substantial, not marginal**, especially from an already strong baseline (SR: 39.0). For context, prior works such as SG-Nav [5] report a 2.1 SR gain by incorporating complex scene graph relationships, and similar conclusions can be drawn from PONI's [4] ablations on MP3D. This demonstrates that our observed improvement is significant within this domain.\n\n* Second, our primary motivation for introducing dependent couplings was to **reduce computational complexity and significantly improve inference time** (as the reviewer also notes, from 17.13ms to 12.38ms). The observed performance gain was **an unexpected and welcome side-product**, highlighting the dual benefits of this design choice in achieving both efficiency and enhanced navigation performance.\n\n* Third, the hyper-parameters tuned in Appendix D.1 (number of steps $n$ and expansion ratio $\\epsilon$) are fundamental to the operation of our generative flow model itself, irrespective of whether independent or dependent couplings are used. For fair comparison, all the hyper-parameters are kept the same when testing the effectiveness of dependent couplings.\n\n# Weakness4&Question3: Interpretation for hyper-parameters tuning curve.\n\nWe appreciate the reviewer's careful review of our hyper-parameter analysis.\n\nFirst, we clarify that performance **does not fluctuate significantly with hyper-parameter settings**. We apologize if Figures 4 and 5 in the Appendix convey a misleading impression of drastic changes due to their scaling. In absolute terms, the maximum SR difference is less than 2.0 for the number of steps ($n$) and less than 1.5 for the expansion ratio ($\\epsilon$). We will update these figures to better illustrate the absolute performance changes. Moreover, we will analyze that **$n$ is not about extensive tuning and generally increase it makes thing good.**\n\nNext, we provide the following interpretations for effective tuning:\n\n* Number of Steps ($n$): As n controls the discretization of the Ordinary Differential Equation (ODE) solver, **performance generally improves with increasing $n$ due to enhanced precision**. However, this gain rapidly saturates, becoming negligible against inherent navigation randomness. Thus, **tuning $n$ is primarily a trade-off to find the saturation point that balances performance with computational cost.**\n\n* Expansion Ratio ($\\epsilon$): As detailed in lines 206-208, $\\epsilon$ scales the influence of the LLM-derived prior on the generative process. **A moderate value (e.g., 1.2-1.3) typically yields strong results**. For LLMs that demonstrate higher reliability in inferring relationships for more distant objects (e.g., a TV being opposite a sofa may exhibit a large distance but also with large confidence), increasing $\\epsilon$ can provide additional performance benefits, as it allows the model to leverage this \"distant but strong\" relational knowledge more effectively.\n\nFinally, we emphasize that GOAL is **generally not highly sensitive to hyper-parameters**. As shown in Table 7, many parameters were selected intuitively without extensive tuning, yet the model performs robustly. Our focus on tuning $n$ and $\\epsilon$  was partly due to their computational efficiency, as they only require re-evaluation, not full model retraining.\n# Weakness5&Question1: Performance with models on consumer-grade GPU.\n\nWe first stress that, in our setting, whether the LLM can be run on consumer-grade GPU does not count, as we only need to prompt API at the very beginning, which **cost less than 0.3$ for ChatGPT4**, and **without any memory comsumption brought by local deployment**.\n\nHowever, we understand your curiosity regarding the performance of GOAL with a smaller LLM. We are currently conducting experiments with Llama3:8b (we noted that SG-Nav, while stating LLaMA 7B in their paper, uses Llama3.2-vision:11b according to their official GitHub repository). We will provide these results to you within the discussion period as soon as the experiments are completed.\n# Question2: Gap of content generated by different LLMs\n**We evaluate quantatively of the gap of content from different LLMs, the results show there is difference, but in small scale.**\nTo assess the variability in distilled knowledge across different Large Language Models, we measured the difference in content between the generated distance and confidence matrices using the following metrics:\n\n1. **Consine Similarity** measures if two matrices have similar patterns, that is, if they always agree a spcific object pair to have a higher/lower distance/confidence: \n$\n\\text{Cosine}(A,B) = \\frac{A\\cdot B}{||A|| ||B||}.\n$\n\n1. **Average Frobenius Norm** measures the average absolute differences of different results: \n$\n\\text{AverageFrobenius}(A,B) = \\frac{1}{N_C^2}\\sqrt{\\sum_{i=1}^{N_c}\\sum_{j=1}^{N_c}|A_{ij} - B_{ij}|^2}.\n$\n\nThe computed similarity scores are as follows (each element scheduled as distances / confidences)\n\n|LLMs pairs|Consine|Frobenius|\n|---|---|---|\n|ChatGPT-ChatGLM|0.947/0.964|0.062/0.0092|\n|ChatGPT-DeepSeek|0.956/0.960|0.057/0.0085|\n|ChatGLM-DeepSeek|0.933/0.954|0.071/0.0103|\n|Llama-ChatGLM|0.952/0.952|0.059/0.0115|\n|Llama-ChatGPT|0.921/0.952|0.074/0.0095|\n|Llama-Deepseek|0.892/0.938|0.088/0.0104|\n\nFrom these results, we observe that:\n\n* Among the industry-level models (ChatGPT, ChatGLM, DeepSeek), ChatGLM exhibits slightly more distinct content compared to the other two.\n* The consumer-grade model, Llama, shows comparatively larger differences when paired with other models.\n* Despite these variations, all LLM pairs **demonstrate high cosine similarities** (ranging from 0.892 to 0.956 for distances and 0.938 to 0.964 for confidences). This indicates that all models consistently **output fundamentally similar spatial relationship patterns**.\n* The absolute differences, as measured by the Average Frobenius Norm, **typically range from 5cm to 10cm for distances and around 1% for confidences**.\n\nSince we formulate the LLMs' responses as smooth Gaussian fields, they are expected to be robust to slight differences.\n\n# Question4: Per-episode inference time reduction for dependent couplings.\n\nWe thank the reviewer for suggesting this more practical metric. We analyzed the average per-episode inference time, accounting for varied scene and goal object types.\n\nSince inference time of an episode varies according to various factors, we take the first 20 episodes of all 11 scenes (totally 220 episodes) and compute the average inference time:\n\n|Data Couplings|Inference time per episodes|\n|---|--------------|\n|Independent Couplings (w cross-attention)|3m 43s|\n|Dependet Couplings|2 m 30s|\n\nThis represents **a significant 32.7% reduction in average episode inference time (from 223s to 150s)**, demonstrating the practical efficiency benefits of dependent couplings beyond individual forward passes.\n\nWe hope these comprehensive responses and additional analyses address all your concerns.\n\n[1] Learning to Map for Active Semantic Goal Navigation. ICLR 2022.\n\n[2] SSCNav: Confidence-Aware Semantic Scene Completion for Visual Semantic Navigation. ICRA 2021.\n\n[3] Imagine Before Go: Self-Supervised Generative Map for Object Goal Navigation. CVPR 2024.\n\n[4] PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learning.CVPR 2022.\n\n[5] SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigation. NeurIPS 2024."}, {"review_id": "4EA9Sp1ZBL", "reviewer": "Reviewer_eNUZ", "summary": "This paper proposes GOAL, a generative method for object-goal navigation. Previous methods build incremental BEV semantic map and apply discriminative method to infer the position to navigate. On the contrary, this paper model objectnav as a generative problem. It proposes a novel framework to distill LLM's common knowledge into a flow-based model, which guesses unseen region to facilitate navigation planning. Experimental results on several benchmarks demonstrates the effectiveness of GOAL.", "questions": "1. The choice of generation. Why generate semantic map rather than other representations like 3D scene graph? State-of-the-art methods like UniGoal leverages 3D scene graph rather than BEV semantic map.\n2. Can this method be extended to more tasks rather than object goal?", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "# Strength:\n1. Model map-based ObjNav as a generative problem is interesting and makes sense.\n2. The technical contribution is sufficient.\n3. During inference, LLM is not required, which may be friendly for on-device deployment.\n\n# Weakness:\n1. For 3D perception, the paper claims \"To mitigate this, we aggregate past RGB-D observations into unified point clouds representations and perform joint segmentation using 3D perception models, inspired by how humans implicitly integrate multi-frame observations.\" However, it seems the authors only leverage per-frame 3D segmentation using sparseconv, without modeling the inter-frame temporal information. Applying online 3D perception may achieve better performance [1][2][3].\n2. One advantage of GOAL is that it does not require LLM during inference. I'd like to know its inference speed compared with other methods. This should be demonstrated in experimental results.\n\n\n[1] Fusion-aware point convolution for online semantic 3d scene segmentation. CVPR 2020.\n\n[2] Memory-based adapters for online 3d scene perception. CVPR 2024.\n\n[3] Embodiedsam: Online segment any 3d thing in real time. ICLR 2025.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your very positive and insightful review. We are delighted that you found our approach of modeling map-based ObjectNav as a generative problem interesting, our technical contributions sufficient, and our paper clear and well-illustrated. We also appreciate your recognition of the significant advantage of not requiring an LLM during inference for on-device deployment. We address your points below.\n\n# Weakness1: Suggestion on using online 3D perception method\n\nWe thank the reviewer for this constructive advice and for highlighting the valuable advancements in online 3D perception methods ([1], [2], [3]). We completely agree that explicitly leveraging inter-frame temporal information, as these methods do, is highly suitable for tasks involving streaming RGB-D observations and holds significant potential to enhance the fidelity of our observed semantic maps.\n\nWhile our current approach implicitly integrates multi-frame data by aggregating past RGB-D observations into unified point clouds, integrating advanced online perception modules like those you referenced presents a **significant engineering challenge and requires substantial architectural changes** to our existing framework. **Thoroughly implementing and evaluating such a complex integration is unfortunately beyond the scope and feasibility of this rebuttal period**.\n\nHowever, we fully recognize its potential to further enhance our observed semantic map fidelity. We will ensure to discuss these relevant methods more extensively in the our revised paper, highlighting them as a valuable avenue for continued research.\n\n# Weakness2: Inference speed comparison \n\n**There is overwhelming difference in magnitude between methods prompting LLM online and ours.** Since UniGoal [4] has not open-sourced its code for object-goal navigation evaluation, we reimplemented SG-Nav [5], which represents a similar paradigm of real-time LLM interaction.\n|Method| average time per episode|\n|-----|-------|\n|SG-Nav|1h 54m|\n|GOAL | 2m 30s |\n\n**Note on Comparison Fairness**: It is important to acknowledge that this comparison is not perfectly balanced. We were only able to test SG-Nav on 6 episodes (which already took approximately 11.5 hours on our setup), whereas GOAL was tested on 220 episodes in parallel on two RTX 3090 GPUs, with the reported time being the average. Despite this slight unfairness in the number of episodes, the overwhelming difference in magnitude demonstrates the profound superiority of our method's inference speed. Our framework's design, which distills LLM knowledge offline, eliminates the latency of real-time LLM calls, resulting in an order-of-magnitude faster execution for entire navigation episodes. Reviewer can further refer to the issues of SG-Nav repo, where relevant questions on inference time can validate our results.\n\nWe also want to emphasize that while SG-Nav incurs significant inference time, it offers valuable benefits from its training-free and open-world setting, which is a different and equally valuable research direction. Our work presents an alternative trade-off: leveraging offline LLM distillation for high efficiency during deployment.\n\n# Question1: The choice of generation.\n\nWe thank the reviewer for their valuable question regarding the choice of semantic map representation over alternatives like 3D scene graphs. Our perspective is that while any representation can be adopted with specific technical designs (For example, leveraging discrete flow matching algorithms with graph neural network backbones and suitable LLM augmentation can also generate a 3D scene graph), our core contribution lies in the generative modeling methodology and LLM prior distillation.\n\nOur decision to generate a BEV (Bird's-Eye View) semantic map is **primarily for convenience and direct applicability to navigation**. Even state-of-the-art methods that represent entire scenes with 3D scene graphs, such as SG-Nav and UniGoal, still **fundamentally rely on a basic BEV map representing occupancy and free space for navigation planning (FMM)**. Directly building the semantic map upon this necessary BEV foundation is therefore a natural and efficient choice for our framework.\n\n\n\n# Question2: Extension of the method to more tasks\n\nYes, our method is extensible beyond object-goal navigation. The core strength of our approach lies in capturing inter-object relationships and co-occurrence patterns to generate plausible semantic layouts for unobserved regions. This fundamental capability makes it applicable to various other scene-level tasks such as 3D Indoor Scene Generation: Our method's ability to \"imagine\" coherent object placements and semantic distributions can provide valuable guidance for 3D indoor scene generation, where multiple objects need to be generated in a contextually appropriate manner. If integrated with techniques for generating 3D scene graphs (as discussed in Question 1), this could further benefit complex scene generation tasks based on scene graph[6].\n\nHowever, it is important to note that these extensions are not trivial and would require specific design choices and adaptations. Our current model primarily focuses on generating object presence and coarse spatial relationships crucial for navigation, without explicitly modeling fine-grained object sizes or other necessary detailed assets for tasks like direct scene rendering. Therefore, direct application without modification would be impractical, necessitating further research and development tailored to each specific task.\n\nWe would discuss more about the broader impact and application in the appendix in the final version of paper. \n\n\nWe hope these comprehensive responses and additional analyses address all your concerns. We are confident that these clarifications further strengthen the paper.\n\n[1] Fusion-aware point convolution for online semantic 3d scene segmentation. CVPR 2020.\n\n[2] Memory-based adapters for online 3d scene perception. CVPR 2024.\n\n[3] Embodiedsam: Online segment any 3d thing in real time. ICLR 2025.\n\n[4] UniGoal: Navigate to Any Goal in Zero-shot! CVPR 2025.\n\n[5] SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigation. NeurIPS 2024.\n\n[6] Controllable 3D Outdoor Scene Generation via Scene Graphs. ICCV 2025."}]

Author rebuttal:

Meta-review: The paper proposes to use a generative model leveraging the prior encoded in LLMs to complete the map of the scenes. It received two Accept and two Borderline Accept ratings. The reviewers mentioned various strengths of the paper, such as:
- Achieving strong SoTA performance on the object navigation task.
- Clear writing and illustrations.
- Notable features, such as eliminating the need to run an LLM during inference.
- Useful ablation experiments

The rebuttal addressed the concerns well (e.g., incomplete ablations wrt SGM and the randomness of LLM outputs). The AC checked the paper, the reviews, and the rebuttal. The AC concurred with the recommendation of the reviewers and recommended acceptance.
Decision: Accept (poster)