Trajectory Diffusion for ObjectGoal Navigation
Authors: Xinyao Yu
Sixian Zhang
Xinhang Song
Xiaorong Qin
Shuqiang Jiang
Venue: NeurIPS 2024 poster (2024)
Keywords: Embodied AI
visual navigation
Primary area: robotics

Abstract: Object goal navigation requires an agent to navigate to a specified object in an unseen environment based on visual observations and user-specified goals. 
Human decision-making in navigation is sequential, planning a most likely sequence of actions toward the goal. 
However, existing ObjectNav methods, both end-to-end learning methods and modular methods, rely on single-step planning. They output the next action based on the current model input, which easily overlooks temporal consistency and leads to myopic planning.
To this end, we aim to learn sequence planning for ObjectNav. Specifically, we propose trajectory diffusion to learn the distribution of trajectory sequences conditioned on the current observation and the goal. 
We utilize DDPM and automatically collected optimal trajectory segments to train the trajectory diffusion.
Once the trajectory diffusion model is trained, it can generate a temporally coherent sequence of future trajectory for agent based on its current observations.
Experimental results on the Gibson and MP3D datasets demonstrate that the generated trajectories effectively guide the agent, resulting in more accurate and efficient navigation.

Trajectory Diffusion for ObjectGoal Navigation
Xinyao Yu Yu!.2- Sixian Zhang' 1.2- Xinhang Song" .2, Xiaorong Qin Shuqiang Jiang 1.2.3 Keey Lab of Intelligent Information Processing Laboratory the Chinese Academy Sciences (CAS), Institute of Computing Technology. Beijing, UUniversity Chinese Academy of Sciences, Beijing Institute of Intelligent Computing Technology, Suzhou, CAS {xinyao. yu, sixian. zrain, xinhang. sonna, xiaorong... segrronggiinevippl.... httprippp.it.a......m sqjiang@ict.ac. cn
Abstract
Object goal navigation requires an agent to navigate to specified object in an unseen environment based on visual observations and user-specified goals. Human decision-making in navigation is sequential, planning a most likely sequence of actions toward the goal. However, existing ObjectNav methods, both end-to-end learning methods and modular methods, rely single-step planning. They output the next action based on the current model input, which easily overlooks temporal consistency and leads to myopic planning. To this end, we aim learn sequence planning for ObjectNav. Specifically, we propose trajectory diffusion to learn the distribution of trajectory sequences conditioned and the goal. We utilize DDPM and automatically collected optimal trajectory segments train the trajectory diffusion. Once the trajectory diffusion model is trained, it can generate a temporally coherent sequence of future trajectory for agent based on its current observations. Experimental results the Gibson and MP3D datasets demonstrate that the generated trajectories effectively guide the agent, resulting in accurate and efficient navigation. The code is available https:///ithur: com/sx sx zhang/T-dnnia git
the current observation
Introduction
Embodied AI aims to develop agents with comprehensive understanding of their environment, capable of interacting with humans, other agents and entities in real physical environments. As fundamental task of embodied AI, visual object goal navigation (ObjectNav) task involves placing agent in unseen environment and tasking it to navigate to user-specified object (c.g., 'find toilet') based on visual sensory input. To efficiently complete the ObjectNav task, the agent needs to construct information-rich memory store past experiences (i.e., what it has previously seen) to avoid redundant searching. Additionally, needs to learn a planner to plan the future (i.e., determine the most efficient sequence of actions to navigate to the target) to avoid unnecessary exploration.
Humans are innately smart navigators, as they encode both short-term memories from current navigation and long-term memories from daily life into cognitive map. This map records detailed information about the semantics, positions, and relationships within spatial environments [40]. In practice, the human decision decision-making making process is understood finding the most possible sequence of releases based on cognitive information [41]. Human planning is scquential process that ensures temporal consistency and global optimality. However, prior methods of ObjectNav employ single-step planning rather than sequential planning for navigation. As illustrated in Fig. prior end-to-end
learning methods [66] 45
33 34] formulate their planners end-to-end networks and train them using reinforcement learning (RL)
62 28 [8]
66
45
62
28
or imitation learning (IL)
'These authors contributed equally to this work.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Maxtiizzing marginal Ikelhood
RL/II traning
Location-related supervised
repression
Cond
Wlyyscitt
Observation
Gerrattic map
pred
Chair
Goa!
'Caal'
Goal
Single-step implicit memory
Single step planner with geametric memory
Sequencee planner geometricc memory
(a)
(b)
(c)
Figure Different planners (denoted as green modules) existing methods of ObjectNav. (a) represents end-to-end learning methods, and (b) refers to modular methods. (c) denotes the proposed
trajectory diffusion, which plans future sequence trajectory based on geometric semantic map.
These planners perform single-step planning at each moment, outputting an action based on the current egocentric view, which leads to a lack of temporal consistency and interpretability. Furthermore, these methods implicitly encode all past observations, leading loss of geometric
33 34
spatial information, thus limiting their generalization in complex environments [8 63 Modular
59 64) attempt to address this by constructing semantic map during navigation to geometrically store historical observations, helping the agent avoid redundant searching. However, their decision -ariiiiinnmmainngggg process is still single-step. They train waypoint predictor via supervised learning to plan the next subgoal based on the current semantic map and target. Motivated by learning sequential planning, we propose model the conditional distribution of the trajectory sequence, i.e., learning to plan a sequence of future trajectory conditioned on the current semantic map and target.
methods 4 [3 65
Since both the semantic map and trajectory sequence are nigh-dimensional, the feature space and computational complexity are significantly high. Thus, directly learning this conditional distribution is challenging []. Recently, diffusion models have achieved great success in expressing complex Diffusion models gradually add noise to the data during the diffusion process, transforming the data distribution into a simple distribution (c.g., Gaussian distribution). In the reserve process (i.c., denoising process), noise is iteratively predicted and removed, generating complex data distributions from the simple random distribution. This iterative refinement learning manner enhances the stability and controllability of learning dighenimensional data distributions. Therefore, utilize diffusion models to learn this conditional distribution and propose the trajectory diffusion (T-diff) model as a navigation planner, which performs sequence planning and generates the future trajectory sequence for the agent.
distributions
16 54 and generating high-quality images 37 35 32
In this paper, we propose the trajectory diffusion for the ObjectNav task. The trajectory diffusion is a sequence planner designed to generate optimal trajectory sequence for an agent based on its historical observations (i.e., semantic maps) and target object. Specifically, we collect optimal trajectories by using precise maps in training rooms. Then, agent equipped with semantic map module is driven to follow these trajectories, gathering semantic maps and poses at each timestep along the way. The collected data are further divided into data pairs consisting of semantic map at a given moment and the corresponding future trajectory segments. Based on the collected data pairs, we employ DDPM [6] to train our trajectory diffusion. Our trajectory diffusion is implemented by modifying Transformer based diffusion, which takes noised latent as the starting input and iteratively refines them to produce trajectory sequence. Once the trajectory is generated, we drive the agent to move along the predicted trajectory sequence until the target is found. Evaluations on the Gibson [47] and MP3D (3] simulators show our trajectory diffusion model significantly outperforms baselines. Visualization results confirm effective guidance from generated trajectories. Additionally, we further showcase the scalability of our trajectory diffusion model across different simulators.
Related Work
is a fundamental task in embodied AI, and this paper focuses on a specific variant of this task, namely the ObjectNav task, where the goal is defined by object semanties. Current works for ObjectNav task can be categorized into two types: end-to-end learning methods and modular methods. End-to-end learning methods develop a navigation policy by interacting with the environment trained by reinforcement learning (RL) [45 66 61 imitation learning (IL) 34 33). These approaches typically take inputs such as
ObjectGoal Navigation. Goal-driven navigation

---

target object categories and extra information, including visual representations 20, 29 18 and object relationship graphs [5] 62 IT 55]. Then they predict single-step planning each timestep. The end- to-end method implicitly encodes all past observations, which results in the loss of geographic spatial information. Consequently, its generalization ability in environments with complex layouts limited. Modular methods 4 5 6 typically preserve top-down geometric map enriched with semantic details. They are also single-step planners that predict waypoints as sub-goals at each time step based on the constructed semantic map. Through supervised learning, PONI learns to predict the nearest frontier with the current semantic map to infer where to explore, while PEANUT directly predicts the target location in the form of probabilistic goal map. Current end-to-end learning methods are single-step planners with implicit memory, while modular methods are single-step planners with geometric memory. Alternatively, our trajectory diffusion approach is a sequence planner with geometric memory, predicting future sequential trajectories based on the current semantic map. Sequence planning ensures temporal consistency and interpretability of decisions. The geometric memory prevents the agent from redundant exploration.
Diffusion Model. Diffusion models (DDPMs [T6] are generative models that learn complex data distributions by iteratively predicting and removing randomly sampled noise to obtain target samples. Diffusion models have been successfully applied in image-related fields, including image generation 32, 37 30), super-resolution 36 46). image inpainting (6,, 48), and image editing 2
[3]. Diffusion models predominantly use UNet-based [36 (37 and Transformer-batt basee [30 32 46
architectures. UNet excels in preserving spatial details for high-resolution images, while Transformers capture global context, suited for sequential data. Therefore,
adopt a Transformer -badd diffusion
model to implement our trajectory diffusion. Recently, diffusion models are gradually employed [3] leverage Data collection from real-world often suffers from scarcity or lacks diversity. As natural data synthesizers, diffusion models are [3] for data augmentation purposes. Furthermore, methods like Diffuser (7), Crossway Diffusion 21 and Diffusion policy (9) utilize diffusion models to fit multi-modal behavioral data of agents. In line with our work, the proposed Diffusion Trajectory aims to learn the distribution of trajectory sequence conditioned on semantic maps and the goal, primarily to address the ObjectNav task.
in the field of roboties. several works g 58
25
leveraged by several works [8 58 25
3 Preliminaries of ObjectNav
The task of ObjectNav involves navigating an agent to a specific type of object (e.g., 'chair') in unseen environments. At the beginning of each episode, the agent is initialized at random position. During navigation, at every timestep f, the agent receives egocentric RGB-D images It, the target object o, and the senor pose pa. which includes the spatial coordinates and the direction the agent is facing. The agent performs one of several discrete actions, includeing move looe_forwarr's turn_left, turn_right and stop. The action stop is autonomously initiated by the agent once it determines to complete the task. An episode is considered successful if the agent stops within preset number of steps at spot where the target object is within specified distance (e.g., less than 1m) and is visible in the agent's 's field of view.
Existing works for the ObjectNav task can be categorized into end-to-end learning methods and modular methods, as shown in Fig. | The end-to-end learning methods [66] 18 33
34 generate
single-step plans at each time step, formulated at timestep t. They typically utilize RL or IL to train policy functions π T(ar 14, o). The training objective of RL is to maximize the expected sum of discounted rewards. Let x denote a sequence of object, action, reward tuples sampled based on æ within an episode. The training objective is argmaxEx~> [Rx], Rx E where y is a discount factor, and r' denotes the reward
policy [Ta,)4,, o), where a denotes the action
function, typically implemented as a sparse success reward. This is because dense rewards can inhibit agent's 'ss exploration (28), thus impairing its generalization in unseen environments [33] Sparse rewards are desirable, but they result in most sample trajectories having difficulty obtaining positive rewards, making the learning process challenging. Moreover, as the policy parameters are updated, previously sampled trajectories become obsolete, necessitating the collection of new data. Consequently, the low sample efficiency of end-to-end learning methods results in high computational costs for training. As for II.-based methods [33 34), the training objective is to minimize the difference between the policy's 's output and human demonstrations (i.e., behavior cloning), summarized as (rg. log(T(af (af) (4, o]]], where D is a dataset of human
3

---

navigation state
Pyet
Puri
concitoo
Noised Latent
Inpul
outpui
T-Diff (Eg)
Injeetory sogmont
Decader
<hap
data pair (Im). 6,) data pair (mmp o)
Transformer blocks N
(c)
Horalivaly dancising
El Bal
Vallitedd Cress Aiterit
Semanticc Macping
Nolood Intont
T-Difl (eg)
Condition
Mo
MLP
Oiservation XJ (RGB DI
Nevigation stade
Sence pick (P)
Semantic map mi
Embooding
parkcy
Erieeciing
navigation
schedule
atale
Target object (/1
Action (x)
Noised Latent
Figure 2: Frameworks of the trajectory diffusion (T-Diff). (a) refers to the process of dividing the collected data into data pairs. (b) shows the implementation structure of T-Diff. (c) illustrates the navigation process guided by trajectories generated by T-Diff.
demonstrations. However, collecting human demonstrations is highly expensive, making IL training costly. Additionally, end-to-end learning methods implicitly encode historical observations, which results lack of geometric memory. This also limits the generalization capability of these methods.
Modular methods [ 31 65 59
typically construct a local semantic map (the map module will be detailed in Sec. 4.2 during navigation. The semantic map geometrically stores historical observations, helping the agent avoid redundant exploration. For the navigation planner, they formulate it as f52mm, m,, o), where 52 is
of points in m,, e.g., Ω can be defined frontiers of m, [31] unknown regions outside my [59]. The module m(m) o) is trained as supervised regression task with the training loss nn C (V (S) J (S)me o)), where V (S2) represents the ground truth, and its values are calculated based on the actual location of the target (i.e. the closer Ω is to the target, the higher value of V(2)). Based on the predictions of f, the agent selects the point with the optimal value as sub-goal to guide the agent. The planner re-predicts at each timestep, which is also form of single-step planning. Supervised learning is efficient for training, however, due the limited room diversity (i.e., current simulators 47
mostly contain fewer than 1000 unique rooms), these lecation-related supervision constraints lead the learned planner f to overfit the layouts of
3 49
the training rooms 63
In summary, end-to-end learning methods learn single-step planner with implicit memory, and their learning process suffers from sample inefficiency and high training costs. On the other hand, modular methods learn single-step planner with geometric memory, but the generalization of their planner is constrained by location-related supervision. In our work, our trajectory diffusion model sequence planner with geometric memory, which aims learn the conditional distribution P(T frrmme,0 o). where т, represents the planned future sequential trajectory. The sequential plan ensures temporal consistency and interpretability of decisions, while the geometric memory ensures efficient exploration. Furthermore, in terms of model training, we leverage DDPM with automatically collected trajectories learn this conditional distribution, effectively avoiding sample inefficiency, the necessity for expensive human demonstration collection, and the risk of overfitting recation-related information.
Trajectory Diffusion
4.1 Diffusion Model
Diffusion models are probabilistic generative models trained to learn data distributions by iteratively denoising variables sampled from Gaussian distributions. The forward process of diffusion models (DDPMs (16) is defined the diffusion process, which is implemented via Markov chain that

---

gradually applies Gaussian noise to the real data which can be formulated as follows:
(x1r)/0)) II a (x-| q (x = N -N
Var, (zr: ū+)
(I)
where constants ar are hyper-parameteer. zo is the real data, while I.,..., IT are noised latent data. The dimension both latent and real data are the same, i.e. Torr E Rd. By applying the reparameterization trick, the noised data can be sampled by 2, vā vI C, where 6, ~ ~ I). Diffusion models learn the real data distribution by reversing the diffusion Markov chain, denoted p(zr-1(27)) which is referred to denoising process. Theoretically, this process reduces to predict the noise added to 2y. The noise prediction network 60 is trained using the mean-squared error between the predicted noise and the ground truth sampled Gaussian noise C,. Additionally, diffusion models can be conditioned on other inputs (e.g., text 2 30 or image ( 52 then the training objective is formulated by
Lo Eg e, [lle, co (x,, c, r) 112)
(2)
where e is the embeddings of input condition. Once the model co is trained, real data are iteratively generated starting from random noise.
4.2 Navigating with Trajectory Diffusion
Our trajectory diffusion aims to generate the optimal future trajectory for the ObjectNav agent based on its current state, thereby assisting the agent in efficiently navigating to the target object.
Trajectory collection. To train the trajectory diffusion, we collect a set of data pairs ((m,, o), Ex). where m, is the semantic map at time t, o denotes target object, and & E 2 [Pt+1.... Pc++]] represents the trajectory segment, defined the concatenation of the senor poses from time t 1 to t + k. Each sensor pose Pt is recorded in 2 dimensions representing the 2D coordinates. To obtain efficient trajectories and corresponding semantic maps, we first randomly initialize start position in the training room. Then, the Fast Marching Method (FMM) (38] is utilized to compute an optimal path from the start position to specific target based the precise collision maps of the current training room. Note that such precise maps are unavailable during testing since the test rooms are unseen. Subsequently, we drive agent equipped with semantic mapping module along this optimal path, recording the semantic map m, at each timestep. Based on the collected data, we sample series of data pairs for training our trajectory diffusion. Furthermore, the collected trajectories are segmented into sub-trajectories & of length k. Together with the corresponding semantic maps m, and target object o, these data pairs ((m,, o), Ec) ultimately constitute the training data for our trajectory diffusion, as illustrated in Fig. (a).
Trajectory diffusion model. We utilize DDPM train our trajectory diffusion to estimate the conditional distribution p(Eelme, In the diffusion process, we sample noised data by &T = Vā, + VI- C, where c, -NO0, represents Gaussian noise and & is the real data &,, i.e. the trajectory segments in the collected data pairs ((m,, E:). âr and T are hyper-parameters that control the variance schedule. Note that two timesteps (1 and r) are involved here, where / represents a specific moment in navigation, and 7 denotes the noise schedule in diffusion or denoising processes. We train the model to predict the added noise, and the training objective is modified from Eq. 2
L E.g....,,elllc.- co (E7, St, T) (1))
where 0 represents the parameter for trajectory diffusion. The navigation state of the current agent acts as the condition for trajectory diffusion. The state 8; is the concatenation of the embedding of the semantic map m; and the target object o. The semantic map m, is encoded by ResNet18 (without pretrained), while the target object is encoded using linear projection.
Regarding the implementation of trajectory diffusion, since the navigation trajectory & is sequence of i tokens, each with dimension of m, our trajectory diffusion follows DiT 30) which is a diffusion model based on the Transformer architecture. As illustrated in Fig. 2 (b), the noised latent EI encoded via a linear projection with added positional embeddings, and then processed through a series of transformer blocks. In addition to the noised latent input, trajectory diffusion also conditions on the diffusion timestep 7 and the navigation state 8t. The condition information interacts with the encoded noised latent through a multi-heas head cross-attention layer. After passing through N
5

---

8 8 (960
(R()%)
5009
SR
SP UULO
MSE
C850
0.08
300
(a) length of trajectory &
(b) sechadule Tmuz
(c) selected kg waypoint
Figure 3: The impact of three syper-parameters. (a) represents the impact of the length of generated trajectories & during training. (b) reflects the impact of maximum noise schedule Tmax in diffusion and denoising process of T-diff. (c) shows the impact selected proportion of generated trajectory length for navigation performance.
Transformer blocks, we utilize a standard linear decoder to output the noise prediction. Note that the output has shape equal to the original noised latent.
Navigating with generated trajectory. During navigation, at each timestep t, the agent receives the current egocentric RGB-D view 4, the current sensor pose Pr, and the target object shown in Fig. 2 (c). The semantic mapping module aids the agent in constructing an incrementally growing semantic map m, At each timestep during navigation, the received RGB-D image is segmented into semantic categories using a pre-trained segmentation model [14]. Subsequently. the depth image is employed to project each pixel, along with its semantic label, into 3D space. Points within specific height range (relative to the agent's height) are designated as obstacles. Then, This point cloud is transformed into voxel occupancy grid, which is integrated across the height dimension to create an egocentric map. The egocentric map is then transformed into an allocentric coordinate system based on the agent's 's pose and is aggregated with the pre-existing global map. Consequently, the semantic map m, (+++)))xxx comprises (n + 4) x h X w elements, where denotes the number of semantic categories, and h, w are the map size. Channels I and 2 depict the obstacle map and the explored regions, respectively. Channels 3 and 4 represent the agent's current position and all previous positions.
The embeddings of the semantic map m, and target object are concatenated to form the navigation state 8t, serving as generation condition. The trained trajectory diffusion model eœ begins with an initialized Gaussian noise E.mar as the initial input, and predicts the noise Eq (ET St, T) contained within the noised latent 47. Then, the noised latent trajectory is denoised by 4--1 E -eg (ET 8c, T). This denoising process is repeated for Tmon steps, iteratively generated the final trajectory E
Once the generated trajectory 57 (i.e.. ft) is obtained, local policy 4 6 is employed to drive the agent along this trajectory. prevent the agent from encountering unreachable points (i.e.. obstacles) in the generated trajectory, we select the ky-th waypoint of & as the navigation goal, where ky is hyper-parameter discussed in Sec. 5.2 The local policy converts the navigation goal into low-level actions by computing a collision-fres free path using the FFM method based on the obstacle channel from the semantic map my. then determines deterministic actions according to the agent's step distance to navigate agent towards the navigation goal (i.e., -th waypoint of &.). The trajectory diffusion generates a new trajectory every LT-diff step, while the local policy replans deterministic action at each step of navigation.
Experiments
5.1 Experimental Setup
Dataset. We evaluate the performance of model standard ObjectNav datasets, including Gibson 47 and Matterport3D (MP3D) [3] in the Habitat simulator. For Gibson, we use 25 train 5 val scenes from the tiny-split, following the settings of [31], with 1000 validation episodes containing 6 target object categories. For MP3D, we utilize 56 train / 11 val scenes, with 2195 validation episodes containing 21 target object categories. The detailed goal categories mentioned in Appendix.
Implementation Details. For the training of trajectory diffusion model, we sample 84k and 465k data pairs from the training scenes in Gibson and MP3D (85% train / 15% val), respectively. The semantic maps are resized to 224 x 224. We implement the trajectory diffusion model based on the DiT [30] structure. Additionally, the semantic map in condition information is encoded by

---

denoise process
Ground-truth map
Goal: Potted plant
99 25
Goal: toilet
99
25
Figure 4: Visualization of trajectory generation. Each row in the figure overlays the results of five repeated experiments, where the same semantic map and target are used as conditions but different random initialization noises are used as inputs. The color intensity of trajectory points represents temporal order, with lighter colors indicating points closer in time to the current navigation timestep.
a ResNet-18 18 [15] with the first convolutional layer's input channel adjusted to accommodate the dimension of the semantic map. The diffusion process contains N 8 Transformer blocks. Training is performed using AdamW optimizer 19, 24 with a base learning rate of le-4, warmed up for 1000 steps using linear warmup and cosine schedule. After the warmup steps, the learning rate for the diffusion model is decayed by a factor of 1e-3, and the learning rate of the semantic map encoder decayed by factor of le-6. Each model is trained for 200 epochs. Following some common generative methods, exponential moving average(EMA. is used with a maximum decay of 0.9999 during training. All reported results obtained using EMA model. The maximum noise schedule Tmaz is set to 100. The length of the predicted trajectory & = 32 and the selected k--th point is set 28. The experiments on these three paaeerrrraeeerrs are detailed in Sec. [5.2] The agent's 's turn angle is fixed at 30 degrees and each F Forward step distance is 25 cm. The maximum timestep limit set to 500 during navigation and f is set 5. Note that, since the navigation performance of multiple repeated experiments does not show significant differences, error bars are not reported.
Evaluation metrics. Three standard metrics are utilized to quantify the performance of the model in ObjectNav task, following 53 4] SR (Success Rate) indicates the proportion of success episodes. SPL (Success weighted by Path Length) represents the success rate of episodes weighted by path length, thereby reflecting the efficiency of the agent's 's path relative to the shortest path. DTS (Distane To Goal) denotes the distance of the agent from the goal when the episode ends. In addition, use MSE (Mean Squared Error). which reflects the distance between the denoised generated trajectory and the real trajectory, to assess generation accuracy of our T-Diff.
5.2 Evaluation Results
determines the length of generated trajectory. We evaluate the impact of parameter i on the generated trajectory (on MSE metric) and navigation performance (on SR and SPL metrics), as shown in Fig. 3 (a). The results indicate that both overly short and overly long generated trajectories are suboptimal. We infer that when the trajectory length is too short, original sequence-plannin
Hyper-parameter tuning of T-Diff.
gradually turns into step-planning, which undermines the temporal consistency of the planning. Conversely, if the trajectory too long (i.c., greater than 32), the distribution p(E:|m,, o) becomes more complex. making the learning process more difficult. Based on experimental results, we determine that 32 is the optimal value.
The
Timx represents the maximum noise schedule, determining the upper limit of iterations in both the diffusion and denoising processes. As evaluation results shown in Fig. (b), the noise schedule Tmax is empirically set to 100.
Visualization of trajectory generation. We visualize the iterative trajectory generation process, as
shown Fig. g Initially, the trajectories are initialized as random coordinate points. As the denoising process progresses, these points gradually converge, forming the final trajectory at
0. Note that
7

---

Table 1: Ablation study on minimal trajectory length for T-Diff training in Gibson (val). When the trajectory length is (rows 2-6), T-Diff is trained to directly predict waypoint, i-th denotes that the training ground truth is the point at the i-th step from the current position on the optimal trajectory.
Trajectory
Navigation (Gibson)
ID
Method
Length
ith
MSE 4
SR(%)
SPL(%) f
DTS(m) 1
1
Single-step (PONT)
73.6
41
1.25
2
T-Diff
0.2247
71.0
37.6
1.49
3
T-Diff
1
8
0.2253
72.8
40.4
1.44
4
T-Diff
16
0.2308
72.6
40.8
1.45
5
T-Diff
1
24
0.2291
74.2
42.1
1.39
6
T-Diff
32
0.2456
73.8
41.3
1.40
7
T-Diff
1
0.1042
75.9
43.1
1.32
8
T-Diff
32 1
0.0357
79.6
44.9
1.00
the visualization is conducted on the validation set of Gibson, where scene layout is unknown to our trajectory diffusion method. Despite this, the trajectory diffusion still capable of generating the most efficient path the target, as indicated by the goal position marked the ground-truth map.
Ablation study on minimal trajectory length for T-Diff training. We conduct an ablation study using minimal trajectory lengths (e.g.. and 4) for training T-Diff, shown in Tab. [ Note that when the length is set to 1, the prediction of T-Diff is a single waypoint. The results indicate that when the length is set to 1, T-Diff's s performance is influenced by the choice of ground truth point (i.e.. the i-th point from current position on the optimal trajectory). The performance with shorter lengths (1 4) is lower compared to longer lengths (32).
We hypothesize that predicting a sequence of trajectories, as opposed to a single point, allows each predicted point to receive contextual information from neighboring points. This helps correct and smooth out prediction errors of individual points, reducing the sensitivity the results to single-point errors. Consequently, this ensures more stable predictions and enhances overall accuracy of trajectory prediction. This finding further supports our motivation for using sequence planning.
Evaluations of T-Diff vari- ants. We compare differ- ent variants of T-Diff as shown in Tab. where rows 2-4 represent differ- ent variants of T-Diff (i.e., sequence planner with ge- ometric memory). comparison in row I uses an enhanced FBE method (proposed by PONI B1I combined with a local pol- icy (i.e., single-step planner with geometric memory). The 'X' marks in row 1 indicate that T-Diff is not used, but this alternative still employs semantie map and goal for navigation. Row 0 refers the case where navigation process does not use any map or goal. The comparison between row and T-Diff variants (rows 2-4) demonstrates that sequence planning achieves superior navigation performance. Moreover, compared to different T-Diff variants, row 2 represents the variant that utilizes only a single timestep observation 1, for trajectory generation. Rows and represent variants that use semantic map m, encompassing all historical observations as conditions for trajectory generation. Comparing rows 2 and 3, the results indicate that using m, not only improves trajectory generation but also enhances navigation performance. We hypothesize that navigation is a sequential decision decision-making task, and relying solely on current single-step observations can lead to suboptimal decisions due to a lack of temporal consistency, thereby reducing overall navigation efficiency.
Table 2:
Comparison with different variants of T-Diff on Gibson (val). I, means using RGB information condition while my refers to employing semantic map as condition. Here, LP denotes local policy, and FBE corresponds the area potential function proposed by [3T.
Trajectory
Navigation (Gibson)
ID
Method
T-Diff variants
MSE +
SR(%) SPL(%) DTS(m) |
Random policy
0.4
0.4
3,89
The
FBE+LP
x Visual(T:)
72.3
38.5
1.32
T-Diff+LP
0.2058
73.5
41.6
1.23
T-Diff+LP
Visual (mz)
0.0546
76.9
44.1
1.08
T-Dif++LP
Visual (ma))Goal)
0.0357
79.6
44.9
1.00
Furthermore, comparing rows 3 and 4, the inclusion of target object improves navigation accuracy, demonstrating that the agent's 's trajectory is goal-driven rather than aimless. Finally, compared to the baseline (comparing rows and 4), integrating our trajectory diffusion method improves navigation performance by 7.3% in SR, 6.4% in SPL, and reduces DTS by 0.32m. These results validate the effectiveness of the proposed trajectory diffusion.

---

Lipetted
Goll
Osservation
Local Mooa with topeetory
Figure 5: Visualization of navigating with generated trajectory on Gibson (val). The top row shows the agent's first-person RGB observation and the bottom row displays the local semantic map along with generated trajectory by T-Diff. The right figure shows the ground truth map with the actual trajectory of the agent and the target location marked. The generated trajectory effectively guides the agent towards the target, even when the target remains unseen.
Comparisons with simpler model for trajectory generation. We con- sider the following simple decoder to learn the trajectory generation P(T,|me o) as a comparison. adopts similar maunforrerrbaseer ar- chitecture to T-Diff, with comparable parameters and the same conditional inputs. However, unlike T-Diff, which is trained using DDPM, this competitor is trained with MSE loss. The results are shown in the Tab. The results indicate that directly learning P(r|mm,, o) through supervised training yields poor performance. Our analysis suggests that since both my and 7; are high-dimensional,, the target distribution P(Tx [ma, o) is also high-dimensional. Given the limited number of training rooms (less than 100). P(T o) is sparse and difficult to learn directly. In contrast, the diffusion model (DDPM). through its diffusion and denoising process, gradually simplifies complex distribution into multiple simpler distributions. This allows for better learning of P( m, o) distribution. Therefore, our experiments and analysis confirm the necessity of using diffusion models for learning trajectory generation.
Table 3: Comparisons with simpler model for trajectory gen- eration. MSE measures the quality of generated trajectories, while SR, SPL, and DTS indicate navigation performance.
MSE 4
SR(%)
SPI[(%)
DTS(m)) 4
Simple decoder
0.6541
59.2
33.5
2.05
T-diff (Ours)
0.0357
79.6
44.9
1.00
Scalability. We compare the per- formance of our trajectory diffusion method with the modular method (i.e., PONI 31) across different simula- tors, as shown in Tab. 4 When train- ing and testing are performed on the same simulator, despite the testing rooms being unseen, the layouts of the training and testing rooms are still similar. In this scenario, the modular method, which uses location-related information for supervision, achieves relatively good performance, as shown in line of Tab. 4 However, when the training and testing rooms come from different simulators, the performance of the modular method significantly deteriorates (line 2). In contrast, our trajectory diffusion method demonstrates better scalability. Even when the training and testing rooms come from different simulators, it maintains good navigation performance (compare lines 3 and 4).
Table 4: Comparisons of navigation performance across different training and testing simulators.
ID
Method
Train
Test
SR(%)
SPL(%) f DDTS(m) +
1
PONI 31
Gibson Gibson
73.6
41.0
1.25
2
PONT 31
MP3D Gibson 43.9
26.3
2.56
T-Diff (Ours)
Gibson Gibson
79.6
44.9
1.00
T-Diff (Ours)
MP3D Gibson
78.2
45.2
1.07
Navigation with trajectory diffusion. The generated trajectory & represents the future location points of the agent from time 1 to + k. As mentioned in See 4.2, instead of directly using the first points on the generated trajectory as the waypoint to guide the agent, we select the - point as the guidance. The impact of the hyper-pau parer-prreeterr ky is evaluated as shown Fig. (c), where the horizontal axis represents kg/ k. The results indicate that the optimal value is achieved when kg = 0.875, i.e., when = 32, ki 28. We attribute this to the fact that, while the generated trajectory follows the correct overall trend, it still fluctuates within a certain range, as shown in Fig. Therefore, if the selected point is too close to the current coordinate, the planning of near-term actions is more frequently affected by these fluctuations, leading performance decline.

---

Table 5: Comparing ObjectNav performance on Gibson and MP3D of related studies. Note that Red-
Rabbit (53 utilizes auxiliary tasks for training, while THDA 28 and Habitat- Web [4 incorporate
additional training data. Results of SemExp 4. L2M
12 and Stubborn 227
are reported from [60]
Results marked with indicate our implementation.
Gibson
MP3D
ID
Method
SPL(%)
DTS(m)
DTS(m)
Random
0.4
3.89
0.5
8.05
DD: DDO 44
3.24
Red-Rabbit
53
34.6
7.9
THDA 28
28.4
5.62
SSCNav
21
27.
11.2
5.71
EmbCLIP*
68.1
39.5
1:15
29.2
10.1
5.40
Habitat Web
34
35.4
10.2
ENTL 20
17.0
5.0
OVG-Nav
50
35.8
12.3
5.69
50
1.78
22.7
6.70
ANS
1.66
SemExp
73.6
1.25
5.10
II
L22M
12
32.1
11.0
5.12
15
Stubborn
31.2
13.5
5.01
16
3D-sware
00
74.5
42.1
1.16
34.0
14.6
4.78
17
1.3MV
S
76.9
38.8
1.01
18
SGM [4
78.0
44.0
1.11
37.7
14.7
4.93
T-Diff (Ours)
79.6
44.9
1.00
39.6
15.2
5.16
Additionally, we visualize the navigation process of the agent, as shown in Fig. 5 According to the ground truth map target location on the right side, is evident that during navigation, the trajectories generated by our trajectory diffusion are consistently correct and efficient, even when the target object is not visible in the early stages (t<76). Consequently, the agent efficiently finds the target guided by the generated trajectories. More visualizations can be found in the Appendix.
Comparisons with the related works. We evaluate the performance of T-Diff ObjectNav task
by comparing it with related baselines, categorizing into end-to-end 44 53 28
23
18 34 20 [66]
and modular [50 5 31
12
27
60 57] methods. is worth noting that some methods incorporate
additional information (28
10 34 or auxiliary tasks [53] making challenging to achieve fair
comparison. Therefore, we primarily focus on the following baselines: SemExp 4, PONI [3]]
OVG-Nav (56). L.MVN (57, 1.2M [2], and SSCNav (23]. PONI enhances SemExp by introducing supervised learning to predict goal-related information. OVG-Nav uses semantic topological maps to plan sub-goal nodes at high level for the agent. L3MVN leverages LLM to infer unknown regions based on semantic information of the current boundary. L2M and SSCNav improve navigation by learning build top-down map that is egocentric in single timestep. Since L2M and SSCNav only report results on their custom datasets, we use either re-implementation results from other work [60] with similar experimental settings our own implementation results for comparison.
We compare our T-Diff with these methods on the validation sets of Gibson and MP3D, as shown in Tab. 5 On the Gibson, our T-Diff outperforms the current state-of-the art-art method [57] by 2.7%, % and -0.01 in SR, SPL,, and DTS metrics, respectively. On the MP3D, T-Diff improves by 3.8%, 2.9%, and -0.53m in the same metrics compared to the current sate-of-theeart method [56].
Conclusion
We propose trajectory diffusion model (T-Diff) as sequence planner for ObjectNav task. Our method leverages agent's historical observations and target object as conditions iteratively generate the future sequential trajectory. Experimental results on standard datasets, including Gibson and MP3D, demonstrate that our T-Diff effectively improves navigation performance compared to baselines. Furthermore, additional visualizations and experiments, detailed in the Appendix and supplementary materials, show that the future trajectory generated by T-Diff provides effective guidance for the agent and exhibits scalability and generalizability across different simulators.
10

---

Acknowledgements
This work was supported by the National Natural Science Foundation of China under Grant 62125207, 62032022, 62272443 and U23B2012, in part by Beijing Natural Science Foundation under Grant JQ22012 and L242020.
References
[1] Richard Bellman. Adaptive Control Processes A Guided Tour (Reprint from 1961), volume 2045 of Princeton Legucy Library. Princetoon University Press, 2015.
[2] Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver BC, Canada, June 17-24, 2023, pages 11992-1402.. IEEE, 2023.
[3] Angel X. Chang, Angela Dai, Thomas A. Funkhouser, Maciej Halber, Matthias Nießner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from RGB-D data in indoor environments. In 2017 International Conference on 3D Vision, 3DV 2017, Qingdao, China, October 10-12, 12, 2017, pages 667-676. IEEE Computer Society, 2017.
[4] Devendra Singh Chaplot, Dhiraj Gandhi, Abhinav Gupta, and Russ R. Salakhutdinov. Object goal navigation using goal-oriented semantic exploration. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[5] Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhutdinov. Learn- ing explore using active neural SLAM. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net net, 2020.
[6] Devendra Singh Chaplot, Ruslan Salakhutdinov, Abhinav Gupta, and Saurabh Gupta. Neural topological SLAM for visual navigation. In 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle WA, USA, June 13-19, 2020, pages 12872-12881, 2020.
[7] Shizhe Chen, Thomas Chabal, Ivan Laptev, and Cordelia Schmid. Object goal navigation with recursive implicit maps. CoRR, Abs/2308.05002 2023.
[8] Zoey Qiuyu Chen, Shosuke C. Kiami, Abhishek Gupta, and Vikash Kumar. Genaug: Retargeting behaviors to unseen situations via generative augmentation. In Kostas E. Bekris, Kris Hauser, Sylvia L. Herbert, and Jingjin Yu, editors, Robotics: Science and Systems XIX, Daegu, Republic of Korea, July 10-14, 2023, 2023.
[9] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Erie Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Kostas E. Bekris, Kris Hauser, Sylvia L. Herbert, and Jingjin Yu, editors, Robotics: Science and Syxtems XIX, Daegu, Republic of Korea, July 10-14, 2023, 2023.
[10] Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Jordi Salvador, Kiana Ehsani, Winson Han, Eric Kolve, Ali Farhadi, Aniruddha Kembhavi, and Roozbeh Mottaghi. ProcTHOR: Large-Scale Embodied Al Using Procedural Generation. In NeurlPS, 2022. Outstanding Paper Award.
[11] Heming Du, Xin Yu, and Liang Zheng. Learning object relation graph and tentative policy for visual navigation. In Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VII, pages 19-34, 2020.
[12] Georgios Georgakis, Bernadttte Bucher, Karl Schmeckpeper, Siddharth Singh, and Kostas Daniilidis. Learning to map active semantic goal navigation. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[13] Haoran Hc, Chenjia Bai, Kang Xu, Zhuoran Yang, Weinan Zhang, Dong Wang, Bin Zhao, and Xuclong Li. Diffusion model is an effective planner and data synthesizer for multi-task reinforcement learning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors. Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
[14] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. In JEEE International Conference Computer Vision, ICCV 2017, Venice Italy, October 22-29, 2017, pages
2017.
11

---

[15] Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778, 2016.
[16] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6- 12, 2020, virtual, 2020.
(17] Michael Janner, Yilun Du, Joshua B. Tenenbaum, and Sergey Levine. Planning with diffusion for ficxible behavior synthesis. Kamalika Chaudhuri, Stefanie Jegelka, Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 9902-9915. PMLR, 2022.
[18] Apoorv Khandelwal, Luca Weihs, Roozbeh Mottaghi, and Aniruddha Kembhavi. Simple but effective: CLIP embeddings for embodied AL In IEEE/CVF Conference on Computer Vision and Puttern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 14809-14818. IEEE, 2022.
[19] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[20] Klemen Kotar, Aaron Walsman, and Roozbeh Mottaghi. Entl: Embodied navigation trajectory learner. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10863-10872, October 2023.
[21] Xiang Li. Varun Belagali, Jinghuan Shang, and Michael S. Ryoo. Crossway diffusion: Improving diffusion- visuomotor policy via self-supervised learning. CoRR, abs/2307.018499 2023.
[22] Xiangyang Li, Zihan Wang, Jiahao Yang. Yaowei Wang, and Shuqiang Jiang. Kerm: Knowledge enhanced reasoning for vision-and-languss navigation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2583-2592, 2023.
[23] Yiqing Liang, Boyuan Chen, and Shuran Song. Ssenav: Confidence-aware semantic scene completion for visual semantic navigation. In IEFF International Conference on Robotics and Automation, ICRA 102 Xi' a'a,, China, May 30 June 2021, pages 13194-13200. IEEF, 2021.
[24] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[25] Cong Lu, Philip J. Ball, Yee Whye Teh, and Jack Polker-Holder. Synthetic experience replay. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023. New Orleans, LA, USA, December 16, 2023, 2023.
[26] Andreas Lugmayr, Martin Danelljan, Andrés Romero, Fisher Yu, Radu Timofte, and Lue Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 1 11451-11461. IEEE, 2022.
[27] Haokuan Luo, Albert Yue, Zhang- Wei Hong, and Pulkit Agrawal. Stubborn: A strong baseline for indoor object navigation. In IEFE/RSJ International Conference on Intelligent Robots and Systems, IROS 2022, Kyoto, Japan, October 23-27, 2022, pages 3287-3293. IEEE, 2022.
[28] Oleksandr Maksymets, Vincent Cartillier, Aaron Gokaslan, Erik Wijmans, Wojeiech Galuba, Stefan Lee, and Dhruv Batra. THDA: treasure hunt data augmentation for semantic navigation. In 2021 IEEF/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 15354 15363, 2021.
[29] Bar Mayo, Tamir Hazan, and Ayellet Tal. Visual navigation with spatial attention. In IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 16898-16907, 2021.
[30] William Peebles and Saining Xie. Scalable diffusion models with transformers. In /EEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 1172-4183 IEEE, IEEE 2023.
12

---

[31] Santhosh Kumar Ramakrishnan, Devendra Singh Chaplot, Ziad AI-Halah, Jitendra Malik, and Kristen Grauman. PONE: potential functions for objectgoal navigation with interaction- learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 18868-18878, IEEE, 2022.
[32] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with CLIP latents. CoRR, abs/2204.06 .0122,,,.222.. 2022.
[33] Ram Ramrakhya, Dhruv Batra, Erik Wijmans, and Abhishek Das. Pirlnav: Pretraining with imitation and RL finetuning for OBJECTNAV. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 17896-17900 IEFE, 2023.
[34] Ram Ramrakhya, Eric Undersander, Dhruv Batra, and Abhishek Das. Habitat-web: Learning embodied object-searcd strategies from human demonstrations at scale. IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 3163-5173.
IEEE, 2022.
[35] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolttion image synthesis with latent diffusion models. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 10674-10685. IEEE, 2022.
[36] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 10674-10685. IEEE, 2022.
[37] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kam- yar Seyed Ghasemipour, Raphael Gontijo Lopes. Burcu Karagol Ayan, Tim Salimans. Jonathan Ho. David J. Flect, and Mohammad Norouzi. Photorcalistic text-to-image diffusion models with deep language understanding. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
[38] James A. Sethian. Fast marching methods. SIAM Rev. 41(2):199-23) 235, 1999.
[39] Shelly Sheynin, Adam Polyak. Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman. Emu edit: Precise image editing via recognition and generation tasks. CoRR, abs/2311,1000$$ 2023.
[40] Edward C Tolman. Cognitive maps in rats and men. Psychological review, 55(4):189, 1948.
[41] Yingxu Wang and Günther Ruhe. The cognitive process of decision making. Int. J. Cogn. Informatics Nat. Intell., 75,8,7, 2007.
[42] Zihan Wang, Xiangyang Li, Jiahao Yang. Yeqi Liu, Junjie Hu, Ming Jiang, and Shuqiang Jiang. Lookahead exploration with neural radiance representation for continuous vision- language navigation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 13753-13762, 2024.
[43] Zihan Wang, Xiangyang Li, Jiahao Yang. Yeqi Liu, and Shugiang Jiang. Gridmm: Grid memory map for vision-and and langugg navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15625 15636, 2023.
[44] Erik Wijmans, Abhishck Kadian, Ari Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. DD-PPO: learning near-perfect pointgoal navigators from 2.5 billion frames. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30. 2020. OpenReview.net, 2020.
[45] Mitchell Wortsman, Kiana Ehsani, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Learning to learn how to learn: Self-adaptive visual navigation using meta-lcarning. In IEFE Conference on Computer Vision and Pattern Recognition, CVPR 2019. Long Beach, CA, USA, June 16-20, 2019, pages 6750-6759, 2019.
[46] Chanyue Wu, Dong Wang. Yunpeng Bai, Hanyu Mao, Ying Li, and Qiang Shen. Hir-diff: Hyperspectral image super-resolution via conditional diffusion models. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 7060-7070. IEFF, 2023.
[47] Fei Xia, Amir Roshan Zamir, Zhi- Yang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: Real-world perception for embodied agents. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages Computer Society, 2018
IEEE
13

---

[48] Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz. and Kun Zhang. Smartbrush: Text and shape guided object inpainting with diffusion model. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 22428-22437. IEEE, 2023.
[49] Karmesh Yadav, Ram Ramrakhya, Santhosh Kumar Ramakrishnan, Théophile Gervet, John M. Turner, Aaron Gokaslan, Noah Maestre, Angel Xuan Chang, Dhruv Batra, Manolis Savva, Alexander William Clegg, and Devendra Singh Chaplot. Habital-matterport 3d semantics dataset. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 4927-4936, IEEE, 2023.
[50] Brian Yamauchi. A frontier-based approach for autonomous exploration. Proceedings 1997 IEFE International Symposium on Computational Intelligence in Robotics and Automation CIRA '97 - Towards New Computational Principles for Robotics and Automation, July 10-11, 1997, Monterey, California, USA, pages 146-151. IEEE Computer Society, 1997.
[51] Wei Yang, Xiaolong Wang, Ali Farhadi, Abhinav Gupta, and Roozbeh Mottaghi. Visual semantic navigation using scene priors. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
[52] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. CoRR, 065/2308.06621,, 2023.
[53] Joel Yc, Dhruv Batra, Abhishek Das, and Erik Wijmans. Auxiliary tasks and exploration enable objectgoal navigation. In 2021 IEEE/CVF International Conference Computer Vision, ICCV 2021, Montreal, QC. Canada, October 10-17, 2021, pages 16097-16106. IECE, 2021.
[54] Mao Ye. Lemeng Wu, and Qiang Liu, First hitting diffusion models for generating manifold, graph and categorical data. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
[55] Xin Ye and Yezhou Yang. Hierarchical and partially observable goal-driven policy learning with goals relational graph. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021. pages 14101-14110 2021.
[56] Hwiyeon Yoo, Yunho Choi, Jeongho Park, and Songhwai Oh. Commonsense-awan object value graph for object goal navigation. IEEE Robotics Autom. Lett., 95))4423-44300 2024.
[57] Bangguo Yu, Hamidreza Kasaci, and Ming Cao. L3MVN: leveraging large language models for visual target navigation. In IROS, pages 3554-3560 2023.
[58] Tianhe Yu, Ted Xiao, Jonathan Tompson. Austin Stone. Su Wang, Anthony Brohan, Jaspiar Singh, Clayton Tan, Dee M. Jodilyn Peralta, Karol Hausman, Brian Ichter, and Fei Xia. Scaling robot learning with semantically imagined experience. In Kostas E. Bekris, Kris Hauser, Sylvia Herbert, and Jingjin Yu. editors, Robotics: Science and Systems XIX, Daegu, Republic of Korea, July 10-14, 2023, 2023.
[59] Albert J. Zhai and Shenlong Wang. Peanut: Predicting and navigating unseen targets. In Proceedings of the IEFF/CVF International Conference on Computer Vision (ICCV), pages 10926-10935, Octoher 2023.
[60] Jiazhao Zhang, Liu Dai, Fanpeng Meng, Qingnan Fan, Xuelin Chen, Kai Xu, and He Wang. 3d-aware object goal navigation via simultaneous exploration and identification. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6672-6682, June 2023.
[61] Sixian Zhang, Weijie Li, Xinhang Song, Yubing Bai, and Shuqiang Jiang. Generative meta-adversarial network for unseen object navigation. Computer Vision ECCV 2022 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIX, volume 13699 of Lecture Notes in Computer Science, pages 301 120-
[62] Sixian Zhang, Xinhang Song, Yubing Bai, Weijie Li. Yakui Chu, and Shuqiang Jiang. Hierarchical object-to-zone graph for object navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 15130-15140, October 2021.
[63] Sixian Zhang, Xinhang Song, Weijie Li, Yubing Bai, Xinyao Yu, and Shuqiang Jiang. Layeut-based causal inference for object navigation. In IEFF/CVÉ Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 10792-10802. IEEF, 2023.
[64] Sixian Zhang. Xinyao Yu, Xinhang Song. Xiaohan Wang, and Shuqiang Jiang. Imagine before go: Self- supervised generative map for object goal navigation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages
IEEE, 2024.
14

---

[65] Minzhao Zhu, Binglei Zhao, and Tao Kong. Navigating to objects in unseen environments by distance prediction. In IEEE/RSJ International Conference Intelligent Robots and Systems, IROS 2022, Kyoto, Japan, October 23-27, 2022, pages 10571-10578. IEEE, 2022.
[66] Yuke Zhu, Roozbeh Mottaghi Eric Kolve, Joseph J. Lim, Abhinav Cupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation indoor scenes using deep reinforcement learning. 2017 IEEE International Conference on Robotics and Automation, ICRA 2017, Singapore, Singapore, May 29 June 3, 2017, pages 3357-3364, 2017.
15

---

Table 6: Chosen object categories in Gibson 47 and MP3D
3
Dataset
Training
Evaluating
chair, couch, potted plant, bed, toilet, dining-table, chair, couch, tv, bed, toilet, potted plant oven, sink, refrigerator, book, clock, vase, cup, bottle char, table, picture, cabinet, cushion, sofa, bed, chest of drawers. plant, sink, toilet, stool, towel, tv monitor, shower, bathtub, counter, fireplace, gym equipment, seating, clothes
Gibson
MP3D
Table 7: FLOPssFloating Point Operations) of SemExp T-Diff.
PONI
31
3D-aware
660 and our
Method
SemExp
PONI
3D-Aware
T-Diff (Ours)
FLOPs (M)
3080.41
46621.90
11617.01
16078.40
A Appendix/ / supplemental material
A.1 Experiments Setup
Evaluation metrics. We employ SR, SPL and DTS metrics to evaluate the ObjectNav performance and assess the trajectory generation accuracy by MSE metric.
SR (Success Rate). SR measures the success rate of the agent in successfully finding the target object. is defined as SR = N Σi S₁, where N the total number of validation episodes and S, is an indicator that representing whether the i-th episode is successful or not.
SPL (Success weighted by Path Length). SPL considers whether the path length of navigation is efficient based on success rate which formulated as SPL = t Σ S !) where i means the shortest path length calculated by the simulator and I, refers the actual path length of i-th episode.
DTS (Distance to Goal). DTS evaluates the distance Li.g of the agent towards the target object when the episode ends. It can be calculated as DTS = + Em i-1 mar(Lig O), where success threshold ę 1m. DTS is 0 when an episode is successful.
MSE (Mean Squared Error). MSE assesses the accuracy of the generated trajectory which defined MSE- 1 D 2. where n is the number of validation set of our collected
trajectory data pairs. 2; refers the ground truth trajectory and f is the generated trajectory.
Object Categories. Our experimental setup follows previous settings [3 60
4] and the adopted
object categories are detailed Tab. 6 For Gibson, we choose 15 categories for training and 6 for evaluating. For MP3D, there are 21 categories for both training and evaluating.
More Visualizations. Fig. illustraaes more visualizations of the generated trajectory by T-Diff during navigation process. As shown in Fig. 6 T-Diff precisely generates the agent's future trajectory conditioned the current state, thus effectively steering the agent towards the target object. The accuracy of the generated future trajectory reduces unnecessary exploration and guides the agent along an almost optimal path. ulimately navigating to stop directly front of the target object, demostrating that our T-Diff significantly improves the efficiency of navigation.
Furthermore, we provide video demo that presents more intuitive view of the process of trajectory denoising and navigation. Please refer to the MP4 file in the supplements zip archive.
A.2 Computation Complexity
To compare the computational complexity of our T-Diff with the modular baselines(SemExp [] PONI (3I) and 3D-aware (60]), we utilize FLOPs(Floating Point Operations) metric assess the computational complexity. shown in Tab. 7 where higher value of FLOPs refers greater computational complexity. Note that, T-Diff iterates Tmaz = 100 steps for each time step to generate the denoised future trajectory, and this trajectory is generated every L- navigation. Thus, we compute the computation complexity of 100 iterations and average over every 5 steps. The results in Tab. 7 indicate that the computation complexity of our T-Diff is higher than
5 steps during
16

---

T Bell
Salt pe
L+S
Clsnvvation
Figure 6: More visualizations. At each timestep, we visualize RGB observation, local semantic map along with the previous path and the generated future trajectory. Note that the target object location marked with a circle.
that of SemExp, comparable that of 3D-aware, but significantly lower than that PONI. Although T-Diff requires multiple iteration, its operating latents is relatively small (when k = 32, the input dimension 2 32) compared PONT's input (the input dimension B (4+ + n) 480 480, where n refers to the number of semantic categories). Therefore, the compuation complexity of T-Diff is considered acceptable.
A.3
Limitations
(I) The method of generating future trajectories based on diffusion has prediction biases. The proposed trajectory diffuusion model primarily considers the current semantic map, historical trajectory information, and target category. However, other information such as the egocentric view and the relationships between object semantics, may help generate better trajectories for the ObjectNav task. Therefore, in future work, we plan to apply more conditions into the trajectory diffusion model.
(2) The training data for the trajectory diffusion model mainly comes from the collection of optimal paths, which are relatively straight and short. The types of training data pairs are not yet diverse enough, and the collection of complex trajectories requiring multiple turns and detours is still
17

---

insufficient. In future work, we plan to introduce data pairs of complex paths to enhance the model's applicabilitty more complex scenarios.
A.4 Broader Impacts
Our trajectory diffusion model is a general method applied to the ObjectNav task. Although our training and testing are currently limited the simulator stage, the trajectory diffusion model can be deployed on real robots. However, prediction errors in the model may lead to incorrect actions by the robot, potentially causing damage personal or social property. Therefore, must be used cautiously to ensure safety in real-world applications.
A.5
Data License
We use two datasets (Gibson 47 and MP3D 3 and employ Habitat simulator. All of them published on the official papers with no licenses stated on their papers and websites. Thus, we just cite all corresponding papers without licenses.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have clearly delineated the main claims of our proposed method in both the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the proposed method in Sec. A.3
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
We provide all the essential details for replicating the main experimental results, with thorough descriptions of the experimental procedures See. 5. In addition, anonymous code repository is provided in Sec. ??.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
20

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: provide an anonymous code repository in Sec. ??, containing comprehen- sive data, code, and relication instructions.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ https//ccppbbccc/uuudddeeoo Coaee//ddeeeSbmmmmma ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify all the training and test details in Sec. 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We mention error bars in Sec. 5. and explain that due to the relatively small variance in the experiments conducted in paper, error bars not reported.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
21

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the related information Sec.
A.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't It make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
The research conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss both potential positive societal impacts and negative impacts in Sec. A.4
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
22

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
See in Sec. A.5
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
23

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.


Reviews:
[{"review_id": "fVl7C4HteX", "reviewer": "Reviewer_ria9", "summary": "This paper presents a novel, diffusion-based modular sequential planning algorithm for image-based goal-conditioned ObjectNav tasks. Concretely, the method performs supervised training to solve the following task: given a partial map constructed from past image observations and a user-specified object, predict a finite sequence of future 2D waypoints that would lead the object.\n\nThe method's main novelty is a diffusion-based conditional sequence generator. Specifically, the condition denoted $s_t$ is a learned embedding of the incomplete map and the target object. The whole training pipeline starts with data preparation, where ground truth paths are used to generate the incomplete map incrementally, serving as the diffusion model conditions.\n\nThe authors perform extensive empirical studies in the Gibson and MP3D simulated environment. The results show that the proposed method is more generalizable across domains than the existing approach. Moreover, it outperforms a collection of end-to-end and modular algorithms, some of which are trained on more data.", "questions": "- For the hyper-parameter search experiment in Figure 3, how are the other variables chosen when experimenting with one specific parameter?\n- The method seems able to plan correct sequences with a very limited initial map. Does this indicate the model learns some inductive bias on where objects are in the rooms? Are there examples where the agent makes a wrong prediction and has to back out?", "limitations": "The authors cover the limitations and broader impacts on pages 14 and 15.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The overall presentation of the work is clean and easy to follow.\n- The specific learning problem, source of data, and training method are covered in great detail.\n- The framework figure is precise and helps the understanding of the approach.\n- Using a conditional diffusion model to predict the waypoint sequences makes sense.\n- I'm unfamiliar with relevant literature, but the presented empirical results are strong in terms of both in-domain validation results and cross-domain generalization.", "weaknesses": "- I find myself scrolling back and forth to read the figures. Maybe their placements could be adjusted.\n- The method uses a heuristic $k_g$ to select the one point within the sequence as guidance. While this design seems to work as is, it would be nice if some post-processing on the sequence could be used to pick the point more smartly.\n- Additional studies could make the design choices more convincing:\n    * If only one point is eventually used as guidance, what would a model perform if trained to directly predict the $k_g$-th point?\n    * How would simpler conditional generation models work, for example, directly predicting the sequence or learning a latent variable model?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewer for the appreciation and suggestions for our work. We address the concerns in the following lines.\n### (W1) Figure placement.\nThanks for the valuable suggestion. We will adjust the figure placement in the final version to enhance readability.\n\n---\n### (W2) Post-processing on the sequence for picking the waypoint more smartly.\nThanks for the valuable suggestion. The parameter $k_g$ is important for selecting the appropriate waypoint on the generated trajectory. \nPreviously, point selection was determined through parameter tuning experiments on the validation set.\nAs the reviewer suggests, we add post-processing on the generated trajectory to autonomously select the optimal waypoint.\n\nWe build an additional trajectory scoring model that takes the generated trajectory, the embedding of semantic map and target as inputs, and outputs a score vector of the same length as the input trajectory.\nBased on collected trajectory segments from training rooms, we calculate the distance of each points on the trajectory from the target location, and construct a one-hot vector with the point closest to the target set to 1 as the training ground truth. The scoring model is trained with Cross-Entropy loss.\n\nBy incorporating the score model, the performance of the enhanced T-Diff is shown in the table below. Without hyper-parameter tuning, it achieves comparable or slightly better performance compared to the original version.\nWe will include this enhanced version in our final version to further improve the proposed T-Diff.\n|                         | SR(%) ↑  | SPL(%) ↑ | DTS(m) ↓ |\n|-------------------------|----------|----------|----------|\n| T-diff                  | 79.6     | 44.9     | 1.00     |\n| T-diff + s(τ,m_t,o)     | 79.8     | 45.1     | 1.00     |\n\n---\n### (W3-1) What would a model perform if trained to directly predict the $k_{g}$-th waypoint?\nWe appreciate your concern. This topic is addressed in the Overall Response above, under Common Q1.\n\n### (W3-2) How would simpler conditional generation models work?\nWe discuss this concern in the Overall Response, please refer to Common Q2.\n\n---\n### (Q1) How are the other variables chosen when experimenting with one specific parameter in Fig. 3?\nWhen experimenting with one specific parameter, the values of other variables are held constant. For example, in Fig. 3(c) of the main text, which tests the impact of different $k_g$ values on navigation performance, the length of the trajectory is fixed at $k=32$ and the max de-noising step is set to $\\tau=100$.\n\n---\n### (Q2-1) Does this indicate the model learns some inductive bias on where objects are in the rooms?\nYes, T-Diff learns some inductive bias regarding the object layout, which is an expected outcome.\nIn the ObjectNav task, agents need prior knowledge about object layout to infer the likely location of the target for more efficient navigation.\nFor example, if the target is a sofa, the agent should learn to first plan a trajectory to the living room to explore if the target is there.\nThis inductive bias provides a form of prior knowledge to improve navigation efficiency.\n\nHowever, as the reviewer concerns, this inductive bias can hinder navigation efficiency if it differs from the current room layout.\nTherefore, we control the length of sequence planning (i.e., trajectory length).\nAs shown in Fig. 3(a) of the main text, excessively long sequence planning harms navigation performance, while controlling it within a reasonable range ensures that the planning results can be updated in time based on new observations. Within this reasonable range, such inductive bias can improve navigation performance.\n### (Q2-2) Are there examples where the agent makes a wrong prediction and has to back out?\nWe provide visualizations of instances where T-Diff makes incorrect predictions, leading to exploration in the wrong direction, as shown in RFig.2 in the supplementary PDF in the rebuttal.\nThe results show that in the initial steps of navigation, due to limited observations of the environment, T-Diff plans an incorrect direction.\nHowever, since the generated trajectory is controlled to be of a reasonable length and the frequency of new trajectory generation is appropriately set, the agent can quickly change direction (see the orange circle), mitigating the impact of a single-step incorrect prediction by T-Diff.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "epfpE2xfUW", "reviewer": "Reviewer_JCTG", "summary": "The paper \"Trajectory Diffusion for ObjectGoal Navigation\" introduces a novel method called \"trajectory diffusion\" for the task of ObjectGoal Navigation (ObjectNav), where an agent is required to navigate to a specified object in an unseen environment based on visual observations. The existing methods for ObjectNav often rely on single-step planning, leading to a lack of temporal consistency. The proposed approach leverages diffusion models to learn the distribution of trajectory sequences conditioned on the current observation and the goal. By training with Diffusion Denoising Probabilistic Models (DDPMs) and using optimal trajectory segments, the model can generate a coherent sequence of future trajectories for the agent. The paper demonstrates significant improvements in navigation accuracy and efficiency using the Gibson and MP3D datasets, showcasing the effectiveness of trajectory diffusion in guiding agents in real-world navigation tasks.", "questions": "1. Are there other papers in the literature that use a sequence of waypoints for ObjectGoal Navigation, and how does this method compare?\n2. How necessary are diffusion models for predicting waypoints? Could simpler methods like a standard decoder suffice?\n3. How would the performance change if the trajectory length were reduced further, possibly to the point of a single-step planner?", "limitations": "n/a", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and clearly positions itself in the literature, highlighting the use of diffusion models to generate a sequence of waypoints for ObjectGoal Navigation.\n2. The method is innovative, leveraging diffusion models to enhance temporal consistency in navigation, which addresses a common issue in existing approaches.\n3. The evaluations are thorough, with the use of datasets like Gibson and MP3D demonstrating the effectiveness of the approach. The visualizations provided are also helpful in understanding the results.", "weaknesses": "1. The paper does not compare its approach with other methods that might use a sequence of waypoints for navigation, leaving a gap in understanding the uniqueness or superiority of using diffusion models for this purpose.\n2. The necessity of using diffusion models to predict waypoints is not well justified. There is no comparison with simpler models, such as a standard decoder that outputs a sequence of waypoints, to establish the added value of the diffusion approach.\n3. The study might be better suited for a robotics-focused conference rather than NeurIPS, as it leans more towards robot learning than core machine learning innovations.\n4. The differences in performance across various hyperparameter settings in the ablation study are relatively small, suggesting that the problem may not be as challenging as presented. For example, varying the length of generated trajectories from 8 to 32 only changes performance by about 3%.\n5. There is no exploration of the performance with minimal trajectory lengths, such as 4 or even 1, which could provide insights into the importance of multi-step planning versus single-step planning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the detailed questions and address them in the following lines.\n### Comparison with other methods of sequence planning for ObjectNav.\nWe choose Habitat-web [1] and ENTL [2] for comparison, as they also output sequence predictions for ObjectNav task. The comparison is based on the following aspects:\n\n- **Training Data.** Both Habitat-web and ENTL rely on human demonstration trajectories for training. In contrast, T-Diff uses automatically collected trajectories, which incur lower collection costs.\n- **Model Structure.** T-Diff's predictions are based on a semantic map. However, both Habitat-web and ENTL rely solely on the implicit encoding of the egocentric view from a few adjacent steps for sequence planning. The semantic map geometrically preserves all past observations, whereas the implicit encoding of a few egocentric views leads to a loss of geometric spatial information, thus limiting their performance in complex environments.\n- **Navigation Performance.** The comparison of navigation performance on MP3D is shown in Table 3 of the main text (for better readability, we have copied the results below). The results indicate that T-Diff achieves higher performance compared to other sequence planning methods, especially with a more significant improvement over ENTL.\n\nThese comparisons demonstrate the superiority of using T-Diff for sequence planning.\n\n|              | SR(%) ↑ | SPL(%) ↑ |\n|--------------|---------|----------|\n| ENTL         | 17.0    | 5.0      |\n| Habitat-Web  | 35.4    | 10.2     |\n| T-diff (Ours)| 39.6    | 15.2     |\n\n[1] Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scale. CVPR 2022\n[2] ENTL: Embodied Navigation Trajectory Learner. ICCV 2023\n\n---\n### Comparison with simpler model for trajectory generation.\nPlease refer to Common Q2 in the Overall Response, where we discuss this concern in detail.\n\n---\n### The study emphasizes robot learning over machine learning, possibly unsuitable for NeurIPS.\nWe focus on the ObjectNav task, which is related to robotics.\nTo efficiently complete this task, we learn a conditional distribution $p(\\tau|m_t,o)$, i.e., inferring the trajectory $\\tau$ from the current position to the likely location of the target $o$ based on partial observations $m_t$ of the current scene.\nThus, the foundation of our work lies in learning this conditional distribution, which is inherently a machine learning problem.\n\nMoreover, NeurIPS has established a Robotics area in the main track, which is the track we submitted to.\nRecently, several ObjectNav-related works have been accepted by NeurIPS.\nA subset is listed below.\nTherefore, we believe our work is suitable for NeurIPS.\n\n[1] CaMP: Causal Multi-policy Planning for Interactive Navigation in Multi-room Scenes. NeurIPS2023\n[2] ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddings. NeurIPS2022\n[3] ProcTHOR: Large-Scale Embodied AI Using Procedural Generation. NeurlPS2022\n[4] Object Goal Navigation using Goal-Oriented Semantic Exploration NeurIPS2020\n\n---\n### The difference in performance across trajectory length from 8 to 32 is relatively small (about 3%).\nWe analyze why the navigation performance in Fig. 3 is not significantly affected by hyperparameter variations within a certain range (e.g., trajectory length from 8 to 32). Two possible reasons are:\n\n- **Simulator difficulty.** Experiments in Fig. 3 of the main text are conducted on Gibson, which has a smaller average area (221.33 $m^2$) compared to other datasets (e.g., MP3D, 682.68 $m^2$), making navigation easier and resulting in smaller performance fluctuations. After repeating the experiments on MP3D, we observe that different hyperparameters have a greater impact on performance, causing differences of 8.1% in SR and 3.9% in SPL.\n- **Model robustness.** Another factor is that T-Diff essentially achieves its full potential when the trajectory length is set within a reasonable range (neither excessively large nor small). Consequently, the model's performance exhibits relatively low sensitivity to hyperparameter variations within a certain range.\n\n---\n### Performance with minimal trajectory lengths (such as 4 or even 1) of T-Diff.\nWe address this concern in the Overall Response. Please refer to Common Q1 for more details.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q4ej5VNFM2", "reviewer": "Reviewer_DH2d", "summary": "The authors propose a diffusion trajectory planner in the context of indoor object navigation that takes current semantic maps (could be partial) and the target object as input to produce a planned future sequential trajectory.\nEvaluation is done in simulation using the habitat simulator on two datasets.", "questions": "-  Why does it make sense to use optimal trajectories from an omniscient planner for training when at deployment partial semantic maps do not know where the goal is ? How does using the Fast Marching Method (FMM) to compute optimal paths to a specific targets with knowledge of precise collision maps translate to efficient exploration in unseen rooms ?\n- How is this expected to generalise across objects and scenes? What modifications would it require to become generalisable?\n- When does the size of the map become an obstacle in itself as rescaling to 224x224 reduces the granularity of trajectories.", "limitations": "- No real world experiments to validate the approach’s practicality (goes hand in hand with the lack of semantic map+planning)\n- Limited technical novelty that is not convincingly motivated.\n- Poor manuscript quality\n\nOn the basis of these limitations, in my appreciation the paper does not meet the conference's quality level for acceptance.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- Comparison to an exhaustive list of other methods with improvement in performance.", "weaknesses": "- The main weakness of the paper is the lack of novelty in the methods/architecture; using DDPM for trajectory generation as well as goal and observation conditioning have all been presented in the literature as cited by the authors in the related works.\n- Given the previous point, although the authors propose a sequential implementation with the capability of predicting trajectories based on partial semantic maps, the work lacks demonstrations of a complete solution, including semantic map building and diffusion planning in parallel.\n- An agent can either see the target or not, if it does not see the target and is in a new room it has to explore, if it does see it then the task comes down to reaching a goal. The authors claim based on results in table 1 that the sequential nature of the task is sufficient motivation for the approach chosen. However, in that same table the results seem to suggest that even without any conditioning, let alone image or goal conditioning the success rate is above 70%, while with the use of semantic maps the score does not exceed 80%. This is very odd and seems to go against the authors’ claim or suggests that the task is ill specified.\n- The manuscript’s quality is below average, with multiple typos (the use of “senor” instead of “sensor” in multiple instances and grammatical errors (“They are is also single-step planners that predict waypoints”, “For the navigation planner, they formulate it as” amongst many other instances). It need a lot of polishing to make it enjoyable to read.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewer for the constructive and insightful feedback.\nWe address your concerns below.\n\n### (W1 & L2) Lack of novelty.\nWe discuss the novelty of our work from two perspectives:\n\n(1) Comparison with existing diffusion-based planning methods\n\nConditional diffusion models are widely adopted generative frameworks.\nSeveral works leverage diffusion models for planning, as shown in table below. \n|Method|Task|State|Model Structure|Target State|\n|-|-|-|-|-|\n|Diffuser[1]|Point-to-point path planning|Grid matrix|-|Specific state (goal location)|\n|Diffusion Policy[2]|Robot motion planning|Fully observed RGB |CNN-based + Transformer-based|Specific state (image)|\n|CrossWay[3]|Robot motion planning|Fully observed RGB|CNN-based|Specific state (image)|\n|T-Diff|ObjectNav|Partially observed semantic map|DiT with cross-attention|Abstract state (User-specified object)|\n\nThese studies consistently utilize agent states as conditions for diffusion models to generate planning paths.\nThe trend is to extend diffusion models to more complex tasks (e.g., more complex condition and state). Our work follows this trend by applying conditional diffusion models to the ObjectNav task, where the current state is only partially observable, and the goal condition is more abstract. The improvements are as follows:\n\n- Higher capacity model architecture: We adopt an improved DiT structure to build the model, enabling it to describe more complex distributions and accommodate more intricate conditions.\n- Advanced condition representation: We replace previously used few single-step visual observations with a continuously updating semantic map as a condition. The semantic map integrates all historical observations into a unified geometric space, facilitating the accumulation of partial observations for planning.\n\n(2) Comparison with existing ObjectNav works\n\nThis aspect has been thoroughly discussed in the main text.\nIn summary, our work is the first to utilize a diffusion model for sequence planning based on geometric memory in the context of ObjectNav task.\n\n---\n### (W2) Lack demonstrations of semantic map building and diffusion planning in parallel.\nWe draw attention to Sec. 4.2 (Page 5, lines 211-239). Specifically, lines 211-224: building the semantic map; lines 225-230: generating trajectory based on the semantic map and target; lines 231-239: waypoint selection and planning.\n\nTo further clarify, we re-summarize these two modules in parallel. As shown in Fig. 2 (c) of the main text, at each navigation timestamp $t$, the semantic map $m_t$ is continuously updated. Every $t_{T-diff}$ steps, our T-Diff is activated and iteratively generates trajectories over $\\tau_{max}$ steps. \n\nThen, the local policy selects waypoints at each timestamp $t$: \n- When the goal is observed in $m_t$, the local policy simply selects the goal's position as the waypoint and drives the agent towards it.\n- When the goal is not observed, the local policy adopts T-Diff guidance (i.e., selects points on the T-Diff generated trajectory as waypoints) for more efficient exploration. Note that the generated trajectory remains until a new trajectory is generated.\n\nWe will add these descriptions in our final version to enhance clarity.\n\n---\n### (W3-1) An agent can either see the target or not?\nExcept for simple cases (e.g., the agent starts close to the target), the target is invisible from the agent’s initial location. \nTherefore, to effectively complete the ObjectNav task, agent needs to: 1) efficiently explore to quickly locate the target, and then 2) move to the target once it becomes visible. \nOur T-diff is designed to enhance exploration efficiency by deducing paths to potential locations of the target.\n\n### (W3-2) The result in Tab. 1 (without any conditioning) is odd.\nWe discuss this concern in detail in the Overall Response, please refer to Common Q5.\n\n---\n### (Q1) Why does it make sense to use optimal trajectories for training, and why can these trajectories from training rooms translate to unseen rooms to improve exploration efficiency?\nThe ObjectNav task typically focuses on indoor environments, where the key to efficiency is improving exploration stage.\nEven though test environments are unseen during deployment, there are regularities in object layouts, e.g., sofas are often found in living rooms, surrounded by cushions and blankets.\nTherefore, to improve exploration efficiency, recent works[4,6,7] have focused on learning such prior knowledge $P(p_o|m_t, o)$, i.e., inferring the target's position $p_o$ based on the map $m_t$ that records historical information and the target $o$.\n\nSimilarly, we aim to learn the prior knowledge about $P(\\tau|m_t, o)$, i.e., deducing a path $\\tau$ from the current position to the likely location of the goal. For example, if the agent observes a sofa, a coffee table, and a microwave, and the goal is set as a toaster, the agent should plan a path towards the microwave to locate the target faster.\nDue to the regularity in the contextual layout of objects, trajectories from training rooms can be transferred to unseen rooms.\nAdditionally, recent studies [5] have shown that learning trajectories from human demonstrations can improve ObjectNav efficiency, further evidencing that navigation trajectories are transferable to unseen rooms.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ospttrq2MF", "reviewer": "Reviewer_k5Ci", "summary": "This paper argues that the previous object navigation algorithms generally only consider one-step decision-making, which can lead to temporal inconsistency and shortsightedness. Therefore, the authors propose using diffusion models to learn sequential decision-making. By collecting expert trajectories and then using the DiT model to learn trajectory diffusion, experimental results have demonstrated that the trajectory diffusion model significantly improves object navigation performance across multiple benchmarks.", "questions": "1. In Line 6 and Line 35, what does the \"temporal consistency\" mean in the paper? Can the authors give an example?\n\n2. Is the generated trajectory a coordinate or an action? In context, it appears to be an xy coordinate, so is this equivalent to some waypoints?\n\n3. Is predicting a sequence really better than predicting a step? The information used to make decisions is all the same, and a well-trained policy can theoretically accomplish the task by simply predicting the optimal next step.\n\n\n\n# After Rebuttal\nThe authors' paper and rebuttal demonstrated to me very well, the motivation and necessity of diffusion models applied in the field of navigation, and more importantly, that diffusion models significantly improve navigation performance. What the authors supplement is so convincing. \n\nWhile the novelty level is still limited, the paper's analysis on diffusion models and navigation makes enough of a contribution to boost my score.", "limitations": "the authors discussed the limitations.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 4, "strengths": "1. The authors attempt to apply diffusion models to the object navigation task, and experimental results show that this approach improve navigation performance and generalization in unseen scenarios.\n2. The figures in the paper are very refined, effectively explaining the differences between their sequential prediction and one-step prediction.\n3. In Section 3, the authors provided a good summary of end-to-end and modular methods and their drawbacks, despite some redundancy.", "weaknesses": "1. I would cautiously suggest that authors might appropriately streamline some of the discussion in Section 3 or place too much of it in supplemental material. Sec 3 also has some overlap with what's in Sec 1, and I would cautiously suggest some adjustments.\n   \n2. I'm not very convinced about the motivation of the paper. In Line 35, The authors claim that the previous end-to-end approach suffered from temporal inconsistencies, and I suggest that the authors provide some examples to demonstrate this. And I think there seems to be no difference between the waypoint prediction among modular methods and the method in this paper: because after the waypoint prediction, a trajectory is also formed. And let's consider a sample case like this: when the policy predicts only the next step, the robot executing this step may receive new information that makes the further next step better; whereas predicting a sequence, if the robot finds a better direction before it has finished executing the sequence, then the robot will continue in the wrong direction. In such an example, it seems that only one step is predicted to perform better.\n\n3. In Line 147， the authors claim that \"end-to-end learning methods .... suffers from sample inefficiency and high training costs\". Can the authors provide the number of samples and training consumption required for the trajectory diffusion model? How effective would these samples be if they were used to train methods of imitation learning?\n   \n4. In Line 148, \"the generalization of their planner is constrained by location-related supervision\". I think that the current modular algorithms based on LLM and VLM segmentation models in the field of zero-shot object navigation do not have the problems described by the authors and outperform many trained algorithms. Could the authors have some discussion or comparison of this like Table 2?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewer for the insightful and valuable feedback. We address your concerns below.\n### (W1) Streamlining Sec. 3 of the main text.\nWe appreciate the reviewer's suggestion and will adjust the content arrangement to ensure conciseness.\n\n---\n### (W2-1 & Q1) Temporal inconsistencies of end-to-end RL methods.\nTemporal consistency in planning should ensure:\n- Consistency in past trajectories: Planning should avoid revisiting past trajectories to prevent redundant exploration.\n- Consistency in future planning: Planning for future exploration at short time intervals should maintain spatial consistency, avoiding frequent large-scale goal switching, which can hinder exploration efficiency.\n\nHowever, end-to-end methods implicitly encode past information (e.g., past trajectories and plans) and predominantly rely on single-step egocentric observations for planning.\nTherefore, their planning cannot avoid redundant exploration or prevent frequent goal switching, thus failing to ensure temporal consistency.\n### (Q2) Is the generated trajectory a coordinate or an action?\nThe generated trajectory consists of a series of coordinates. The local policy selects a point on the trajectory as a waypoint.\n### (W2-2 & Q2) Difference between waypoint prediction in modular methods and T-Diff.\nBoth methods provide a waypoint to the local policy, but they differ in several key aspects:\n- **Direct goal vs. Gradual goal.** Modular methods use the absolute position of the target as supervision for training, meaning the predicted waypoint represents the target's final position, i.e., Direct goal. In contrast, T-Diff is trained with segments of the optimal trajectory to the goal, with each segment representing a progressive sub-goal towards the target position. Therefore, the waypoint predicted by T-Diff is a Gradual goal. Compared to Direct goals, Gradual goals are more reachable for the agent. Splitting one absolute position into several sub-goals enriches the state space and prevents sparse supervision, improving generalization, particularly when the target position changes significantly.\n- **Temporal consistency.** The modular method predicts waypoints based on observed objects and obstacles, while T-Diff also considers historical trajectories and current agent pose. This ensures the predicted waypoints are consistent over short time intervals, preventing frequent waypoint switching.\n- **Interpretability**: T-Diff outputs a series of coordinates with explicit trajectories, making the predictions interpretable.\nAdditionally, the similarity in usage (predicting a waypoint for the local policy) ensures T-Diff's compatibility with existing modular navigation frameworks, allowing it to benefit from modular methods like improved mapping and local policy.\n### (W-3 & Q3) Is predicting a sequence really better than predicting a step?\nThe advantage of sequence planning lies in ensuring consistency, preventing redundant exploration, and avoiding frequent spatial jumps of predicted waypoints over short intervals.\nHowever, as the reviewer concerns, its disadvantage is that if the sequence is too long and an error occurs, the agent cannot be corrected promptly.\nThus, balancing the pros and cons of sequence planning is crucial, which involves controlling the sequence length.\nAs shown in Fig. 3 (a) and (c) of the main text, the ablation study on sequence length and waypoint selection indicates that when the sequence is too long or waypoints are too far apart, the performance of sequence planning decreases due to the lack of timely correction. However, when an appropriate scale is set, performance gradually increases with the sequence length and surpasses that of single-step planning.\n\n---\n### (W3-1) Comparisons of samples and training consumption.\nPlease refer to the RFig.4 in the supplementary PDF in the rebuttal, where we compare end-to-end learning methods (DD-PPO, Habitat-Web), modular methods (PONI), and our T-Diff in terms of training samples and training consumption. The results show that T-Diff's training samples and consumption are similar to modular methods and significantly lower than end-to-end methods.\n### (W3-2) Using collected trajectories for imitation learning.\nWe use collected trajectories to fine-tune SemExp[1] through imitation learning.\nResults are shown below.\nBoth SemExp and T-Diff utilize the same inputs, i.e., semantic map and target object. \nThe results indicate that collected trajectories help improve performance further, but imitation learning is less effective than using DDPM for training. \nWe hypothesize that DDPM, through the noise-adding process, enriches the training state space, allowing the diffusion model to learn the target distribution better.\n| Method           | SR(%) ↑ | SPL(%) ↑ | DTS(m) ↓ |\n|------------------|---------|----------|----------|\n| SemExp           | 71.1    | 39.6     | 1.39     |\n| SemExp (finetune)| 73.3    | 41.2     | 1.18     |\n| T-Diff           | 79.6    | 44.9     | 1.00     |\n\n---\n### (W-4) Comparing T-Diff with zero-shot navigation methods.\nWe choose ESC [2] for comparison, which achieves zero-shot navigation using the VLM model (i.e., GLIP model) and LLMs.\nAs shown in RTab.4 in the supplementary PDF in the rebuttal, results indicate that zero-shot method performs better in cross-domain generalization than previous modular methods relying on location-related supervision, i.e., the performance difference is minimal when test data comes from different simulators.\nHowever, due to the lack of optimal trajectory training, their success rate, especially in the SPL metric reflecting navigation efficiency, is lower than our method.\n\n---\n### Reference \n[1] Object Goal Navigation using Goal-Oriented Semantic Exploration, NeurIPS 2020\n\n[2] Esc: Exploration with soft commonsense constraints for zero-shot object navigation, ICML 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1geyeX7k7g", "reviewer": "Reviewer_sqUP", "summary": "This paper tackles the object goal navigation problem, where given visual observations and a target goal object, the task is to plan a navigation path to find said object. The paper proposes a trajectory diffusion model, where the future navigation trajectory is predicted starting from random points. The diffusion model is conditioned on an a semantic map and past trajectories, and these choices are validated in an ablation study. Finally, the proposed approach is compared with previous methods on two different simulators, demonstrating its effectiveness.", "questions": "How is the goal object represented? By text, one-hot vector, or something else?\n\nHow can the method work at all without goal information (rows 1-3 in Table 1) or without environment knowledge (i.e. no map or image in row 1)? Or does this only refer to the diffusion process and the model always takes the object and map as input? Does row 1 correspond to predicting trajectory points directly?\n\nWhat happens if the model explores the wrong part of the room, i.e. when two directions are equally good at first glance? Does the model stop prematurely or does it manage to backtrack, overlapping its previous path and find the goal?", "limitations": "Some limitations are discussed in the appendix, such as prediction biases and optimal paths for supervision.\n\nAdditional limitations include:\n* The proposed approach seems to assume known pose, which is a limiting factor if it is not available.\n* The fixed map resolution limits the generalization to large-scale environments.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Overall, the paper is well written and presents relevant prior work as well as its contributions in a clear way that is easy to follow.\n\nThe problem under consideration is relevant for robotics, and is challenging as it considers navigation in unknown environments, utilizing only a camera as its sensor.\n\nThe paper proposes a novel approach based on diffusion models to generate trajectory points for a local policy to follow.\n\nAblations and comparisons with other methods demonstrate strong performance.", "weaknesses": "Since a fixed 224x224 size is used for the semantic map, the effectiveness of the proposed approach is limited on large-scale environments. Either fine details are lost by representing the full map at that resolution, or distant regions are lost by representing a local neighborhood.\n\nExperiments are only conducted in simulation. Experiments on a real robot would further improve the quality of the paper.\n\nNo error bars. Although explained in the paper that the variance is low, reporting quantitative values is more precise and complete.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewer for the appreciation and suggestions for our work. We address the concerns in the following lines.\n\n### (W1 & L2) Concerns about small map size limiting the proposed method's generalization to large-scale environments.\nWe evaluate navigation performance with various map sizes, as illustrated in RFig.3 of the supplementary PDF in the rebuttal. Our model is compatible with higher-resolution semantic maps as input. However, for navigation tests in the simulator, as shown in RFig.3 (for more details, please refer to Common Q3 in the Overall response), a map size of 224×224 optimally balances navigation performance and computational efficiency. Thus, for navigating simulators, we set the map to 224×224 based on performance-to-cost considerations. Nevertheless, due to its compatibility with different map sizes, our T-Diff can transfer to large-scale environments by utilizing maps with richer fine details.\n\n---\n### (W2) Real world experiments.\nWe provide additional evaluation results and navigation visualizations in real world environments. Please refer to the Overall Response, Common Q4.\n\n---\n### (W3) Lack of Error bars.\nCompared to end-to-end learning methods, both modular approaches and our T-Diff demonstrate significantly smaller result variations due to the use of explicit semantic maps and local policies. Results indicate that after multiple experimental runs, the error bars for the end-to-end method (PIRL[1]) are 0.47% in SR and 0.56% in SPL, whereas for T-Diff, they are only 0.03% in SR and 0.09% in SPL.\n\nWe agree with the reviewer's concerns and will incorporate error bars into the results in our final version to enhance the precision and completeness of our experimental results.\n\n---\n### (Q1) Representation of target object.\nThe target object is represented as a one-hot vector. We will emphasize this in the revised version for better understanding.\n\n---\n### (Q2) Detailed explanation of Tab.1 in the main text.\nIn Tab.1 of the main text, row1 indicates that the baseline method does not utilize T-Diff but is still equipped with semantic maps, target object, and local policy for navigation.\nThe **visual** or **goal** conditions in the main text table merely represent different settings in the diffusion process of T-Diff, rather than completely omitting semantic maps and goals in navigation.\n\nFor clarity, we have modified this table, and the revised version is presented as RTab.3 in the supplementary PDF in the rebuttal. RTab.3 offers clearer explanations and includes an additional row (row 0), corresponding to navigation without any visual or goal information.\n\n---\n### (Q3) What happens if the agent explores wrong rooms?\nDuring navigation, when T-Diff plans an incorrect direction, the agent may explore the wrong room, as illustrated in RFig.2 in the supplementary PDF in the rebuttal.\n\nTo mitigate the impact of such erroneous cases, we do not employ predefined, fixed strategies, such as backtracking the previous path upon entering an incorrect room. Fixed strategies could potentially trap the agent in local optima, limiting its ability to explore and adapt to newly acquired observations. Instead, we opt for a dynamic approach by controlling the planning length (i.e., trajectory length) to prevent the agent from being misguided for extended periods, and setting an appropriate update frequency for generating new trajectories based on continuously updated local maps to correct previous erroneous planning.\n\nAs shown in RFig.2, during the initial steps of navigation, T-Diff plans an incorrect direction due to limited environmental observations. However, as more environmental information is observed on the semantic map, new correct trajectories overwrite the previous erroneous ones with the update frequency. Consequently, the agent can quickly adjust its direction (see the orange circle). The visualization results demonstrate the robustness of our T-Diff to single-step prediction errors.\n\n---\n### (L1) Assumption of known pose.\nThe experimental setting with known pose follows previous modular methods, as pose information is necessary for constructing semantic maps.\nWhile pose-known methods are less flexible than those requiring only RGB-D input, they exhibit greater stability in complex, large-scale environments and better generalization to real-world scenarios [2].\nMoreover, in practical deployments, pose can be obtained through sensors such as odometers.\n\n---\n### Reference \n[1] PIRLNav: Pretraining with Imitation and RL Finetuning for OBJECTNAV, CVPR 2023\n\n[2] Object Goal Navigation using Goal-Oriented Semantic Exploration, NeurIPS 2020", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Overall Response
We thank all reviewers for their valuable and insightful feedback. We appreciate the supportive comments regarding our well-written presentation (ria9, JCTG, sqUP), novel approach (JCTG, sqUP), precise figures (ria9, k5Ci), sound motivation (ria9), robust performance (ria9, k5Ci, sqUP), and comprehensive evaluations (JCTG, DH2d).
Several common questions have been raised, which we address below.

### Common Q1 (@ria9 @JCTG) Using minimal trajectory lengths for T-Diff training.
As suggested by the reviewers, we add an ablation study using minimal trajectory lengths (e.g., 1 and 4) for training T-Diff, as shown in RTab.1 in the supplementary PDF in the rebuttal. Note that when the length is set to 1, the prediction of T-Diff is a single waypoint.

The results indicate that when the length is set to 1, T-Diff's performance is influenced by the choice of ground truth point (i.e., the **i-th** point from current position on the optimal trajectory). The performance with shorter lengths (1 or 4) is lower compared to longer lengths (32).

We hypothesize that predicting a sequence of trajectories, as opposed to a single point, allows each predicted point to receive contextual information from neighboring points. This helps correct and smooth out prediction errors of individual points, reducing the sensitivity of the results to single-point errors. Consequently, this ensures more stable predictions and enhances overall accuracy of trajectory prediction.
This finding further supports our motivation for using sequence planning.

---
### Common Q2 (@ria9 @JCTG) Comparison with simpler model for trajectory generation.
We consider the following simple decoder to learn the trajectory generation $P(\tau|m_t, o)$ as a comparison.
It adopts a similar Transformer-based architecture to T-Diff, with comparable parameters and the same conditional inputs.
However, unlike T-Diff, which is trained using DDPM, this competitor is trained with MSE loss. The results are shown in the table below.

The results indicate that directly learning $P(\tau|m_t, o)$ through supervised training yields poor performance. Our analysis suggests that since both $m_t$ and $\tau$ are high-dimensional, the target distribution $P(\tau|m_t, o)$ is also high-dimensional. Given the limited number of training rooms (less than 100), $P(\tau|m_t, o)$ is sparse and difficult to learn directly.
In contrast, the diffusion model (DDPM), through its diffusion and denoising process, gradually simplifies complex distribution into multiple simpler distributions.
This allows for better learning of $P(\tau|m_t, o)$ distribution.
Therefore, our experiments and analysis confirm the necessity of using diffusion models for learning trajectory generation.
||MSE ↓|SR(%) ↑|SPL(%) ↑|DTS(m) ↓|
|-|-|-|-|-|
|Simple decoder|0.6541|59.2|33.5|2.05|
|T-diff|0.0357|79.6|44.9|1.00|

> Note that MSE measures the quality of generated trajectories, while SR, SPL, and DTS indicate navigation performance.

---
##### Common Q3(@DH2d @sqUP) Concerns about map size.
We conduct experiments with T-Diff using various map sizes. Results are presented in RFig.3 of the supplementary PDF. Our findings demonstrate that T-Diff is compatible and performs well with different map sizes (except when excessively small). The results indicate that as map size increases, both information granularity and performance improve, albeit with increased computational complexity. Performance plateaus beyond a size of 300, while complexity continues to rise. To balance performance and computational cost, we opt for a 224×224 map size. However, when T-Diff requires adaptation to larger-scale scenarios, map size can be scaled up and readily integrated (addressing **sqUP**'s concerns about small map sizes limiting T-Diff's performance in large-scale environments). Additionally, we observe that performance degrades significantly when the map size falls below 150 (addressing **DH2d**'s inquiry about the point at which map size constrains performance).

---
### Common Q4 (@DH2d @sqUP) Real world experiments.
To validate T-diff in real-world environments, we provide additional evaluation results and navigation visualizations in real world, as shown in the supplementary PDF in the rebuttal. 
We set up a 140 $m^2$ space, utilizing movable walls to create 3 scenes with different layouts. Each scene is divided into several rooms and furnished with common furniture and objects. The space contains a total of 35 object categories, from which we select 7 object types for our object navigation experiments. The experiments are deployed on a Locobot-wx250s. For each object type, we conduct trials from 10 different starting positions in each of the 3 scenes, and calculate the SR metrics.
As shown in RTab.2, our method achieves a higher success rate compared to the baseline (PONI[1]). Additionally, the visualizations in RFig.1 show that the generated trajectories align well with actual target positions, demonstrating T-diff's robustness in real-world scenarios.


---
### Common Q5 (@DH2d @sqUP) Misunderstanding regarding the result of row 1 in Tab. 1 in the main text.
In Tab. 1 of the main text, rows 2-4 represent different variants of T-Diff (i.e., sequence planner with geometric memory). 
The comparison in row 1 uses an enhanced FBE method (proposed by PONI[1]) combined with a local policy (i.e., single-step planner with geometric memory). 
The 'X' marks under 'visual' and 'goal' condition in row 1 indicate that T-Diff is not used, but this alternative still employs semantic map and goal for navigation.

To improve clarity, we revise this table, as shown in RTab.3 in the supplementary PDF in the rebuttal. RTab.3 provides clearer explanations and includes an additional row (row 0) to address the reviewer's concern about the case where navigation process does not use any image or goal.

---
### Reference 
[1] PONI: potential functions for objectgoal navigation with interaction-free learning, CVPR 2022

Meta-review: The paper "Trajectory Diffusion for ObjectGoal Navigation" presents a novel approach for object goal navigation using a trajectory diffusion model called T-Diff. The proposed method leverages a conditional diffusion model to generate trajectories based on a semantic map and target object, guiding the agent toward potential target locations in unseen environments. The reviewers agree that the paper makes a contribution to the field of embodied AI and recommend its acceptance. The strengths of the paper lie in its innovative use of a diffusion model for trajectory generation in the context of object goal navigation. By conditioning the diffusion process on a semantic map and target object, T-Diff can generate temporally consistent and interpretable trajectories that guide the agent towards potential target locations in unobserved areas. This approach addresses the limitations of existing end-to-end and modular methods, which often suffer from temporal inconsistencies and inefficient exploration.

The reviewers raised several important questions and concerns, which the authors addressed in their rebuttal. They provided additional experiments and analyses to support their claims, including the effectiveness of using optimal trajectories for training, the impact of map size on performance, and the comparison with simpler models for trajectory generation. The authors also clarified the differences between T-Diff and related works, such as modular methods and zero-shot navigation approaches. The authors addressed the concerns regarding the temporal consistency of end-to-end and modular methods, and provided evidence that T-Diff achieves more consistent planning by considering past trajectories and current agent pose. They also explained the advantages of sequence planning over single-step prediction, highlighting the importance of balancing the sequence length to ensure timely corrections. While some reviewers raised concerns about the small map size potentially limiting T-Diff's generalization to large-scale environments, the authors demonstrated that their method is compatible with different map sizes and can be scaled up for larger environments. They also provided additional real-world experiments to validate the effectiveness of T-Diff in practical scenarios.

The AC also considers this work to be both interesting and important, as it introduces a novel approach for object goal navigation that addresses the limitations of existing methods. The use of trajectory diffusion for generating temporally consistent and interpretable trajectories based on semantic maps and target objects is a significant contribution to the field. The experiments and analyses provided by the authors further demonstrate the effectiveness and robustness of their approach. The reviewers' concerns have been adequately addressed in the rebuttal, and the authors have shown a commitment to improving the paper based on the feedback received. 

Based on the above reasons, the AC recommends accepting this paper for publication.
Decision: Accept (poster)