TP-MDDN: Task-Preferenced Multi-Demand-Driven Navigation with Autonomous Decision-Making
Authors: Shanshan Li
Da Huang
Yu He
Yanwei Fu
Yu-Gang Jiang
Xiangyang Xue
Venue: NeurIPS 2025 poster ()
Keywords: Long-Horizon Task Planning
Continuous Vision and Language Navigation
Demand-Driven Navigation
Autonomous Decision-Making
Unseen Environment
Large Vision-Language Models
Primary area: reinforcement_learning

Abstract: In daily life, people often move through spaces to find objects that meet their needs, posing a key challenge in embodied AI. Traditional Demand-Driven Navigation (DDN) handles one need at a time but does not reflect the complexity of real-world tasks involving multiple needs and personal choices. To bridge this gap, we introduce Task-Preferenced Multi-Demand-Driven Navigation (TP-MDDN), a new benchmark for long-horizon navigation involving multiple sub-demands with explicit task preferences. To solve TP-MDDN, we propose AWMSystem, an autonomous decision-making system composed of three key modules: BreakLLM (instruction decomposition), LocateLLM (goal selection), and StatusMLLM (task monitoring). For spatial memory, we design MASMap, which combines 3D point cloud accumulation with 2D semantic mapping for accurate and efficient environmental understanding. Our Dual-Tempo action generation framework integrates zero-shot planning with policy-based fine control, and is further supported by an Adaptive Error Corrector that handles failure cases in real time. Experiments demonstrate that our approach outperforms state-of-the-art baselines in both perception accuracy and navigation robustness.

TP-MDDN: Task-Preferenced Multi-Demand-Driven Navigation with Autonomous Decision-Making
Shanshan Li", , Da Huang Yu Hel Yanwei Fu Yu-Gang Jiang', Xiangyang Xue
'Fudan University Shanghai Jiao Tong University
Shanghai Innovation Institution
Abstract
In daily life, people often move through spaces to find objects that meet their needs, posing a key challenge in embodied AI. Traditional Demand-Driven Navigation (DDN) handles one need at time but does not reflect the complexity of real- world tasks involving multiple needs and personal choices. To bridge this gap, we introduce Prefeeeefeee Salli-Demand-Drivvee Navigation (TP-MDDN), a new benchmark for long-horizon navigation involving multiple sub-demands with explicit task preferences. To solve TP-MDDN, we propose AWMSystem, an autonomous decision becision-making system composed of three key modules: Break- LLM (instruction decomposition), LocateLLM (goal selection). and StatusMLLM (task monitoring). For spatial memory, we design MASMap. which combines 3D point cloud accumulation with 2D semantic mapping for accurate and efficient environmental understanding. Our Dual-Tempo action generation framework inte- grates zero-shot planning with policy- fine control, and further supported by an Adaptive Error Corrector that handles failure cases in real time. Experi- ments demonstrate that our approach outperforms state-of-the-arr baselines in both perception accuracy and navigation robustness.
1 Introduction
In daily life, people often identify need and look for something in their environment to meet it Demand-Driven Navigation (DDN) I is a task where an agent receives a natural language instruction (e.g., "I tired") and must find an object that fulfills that need (e.g., bed chair). This is variation of the ObjectNav task. However, a single need can often be met in different ways, depending on personal preferences. For example, "organize the living space" could mean finding cleaning tools, decorative items, or storage boxes. To guide the agent effectively, it is important to clarify the user's specific preference, like focusing decoration, the instruction becomes actionable. People also tend to have a series of needs, such as cleaning, then resting, then cating. Efficiently handling multiple needs and evaluating the success of these actions is still a major challenge. This paper explores how to enhance long-horizon DDN tasks by making user preferences more explicit.
3
Recent work like MO-DDN 5 addresses navigation instructions through Multi-object Demand- driven Navigation with human preferences. For example, given an instruction like "I need to display my photography collection, preferably with good lighting", the demand may involve multiple objects such as picture frames, bookshelves, and ceiling lamps. However, it still focuses on single- demand navigation. Some works have explored long-range instruction navigation by constructing Landmark Semantic Memory for decision-making planning [6) or adopting Autonomous Evolution mechanisms
g) but they primarily target object-driven navigation.
Corresponding Authors
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a) Overview
(b) Performance
How to choose one of the subtask?
How generate Dual-Tampo actions?
Performance v.s. Efficiancy
Beraadss what
Mow can I quiiclly get there and fonduct miriivalaox inspeeilen?
should de nouf?
your
your
Generalist Palky
pocialist Palicy
+1.%
GOM
Indructiin
Construct LandHark
Semantic Memory
0
cloa
Eag Wicken (i3.4
Adlers Lit Ntrraft
(min))
Experiments under TP-MDDN benchmark
check the completion status subtasks?
How know when suitch the aest subiask?
2000 ceacicen mortrnattt Your bagget seject Soze. Scuiptun.
3671
2U.01
Premph: C wee hiretta scétect chy3ss saben and curmet shrenatarr Fnfer The
11:03
$5.00
sefertunmentt optiere
D0M
MO-CON
Iratrvetherr
Duts
Rabe
Figure I: This figure presents our autonomous decision-making process and its performance benefits. (a) Overview: The pink area shows subtask selection using landmark semantic memory; the yellow area explains Duap-TTmpo action generation via generalist and specialist policies; the green area details dynamic subtask completion checks. (b) Performance: Our method achieves a 16% higher
success rate than DDN (I] and InstructNav [2) under the TP-MDDN benchmark, with superior efficiency and individual success rates, highlighting its effectiveness and reliability.
We introduce a new benchmark called Task-Preferenced MDDN) to handle scenarios with multiple needs, where each need includes a clear preference for a specific task-related object category. In simulated multi-room home environment, we use V3 [9] and GPT-4o [10] to generate long-horizon instructions and object pairs across different scenes. These pairs are manually checked to ensure accuracy.
Navigation (TP-
To efficiently manage memory in long-term navigation, we introduce a novel Multidimensional Accumulated Semantic Map (MASMap) that achieves balance between accuracy and efficiency without requiring additional training. MASMap integrates local 3D point cloud accumulation with a global 2D semantic map to build and maintain spatial memory over time. A core challenge of merging semantically similar objects viewed from different perspectives is addressed using loU-based for accurate segmentation and labeling. To reduce storage overhead,
fusion and Nan-Grrunndd-SS we implement efficient update-and ann-prune strategy that preserves critical small objects. Our global map structure further minimizes redundancy while ensuring consistent and reliable semantic memory across extended navigation trajectories.
To address long-horizon visual navigation tasks, we further propose the Autonomous Decision- Making World Model System (AWMSystem), inspired by WMNav [I]. AWMSystem breaks down complex instructions into sub-demands using BreakLLM, dynamically selects goals through LocateLLM based on object memory and execution status, and tracks task progress via StatusMLLLM using real-time observations. For lightweight deployment, we employ a dual-tempo action generation strategy: zero-shot planning using A algorithm with obstacle maps and affordance estimation (as in InstructNav [2)), and fine- aue-grained policy-based control near targets, following DDN M. A major challenge in simulation environments, i.c.. handling logical loops, boundary violations, and unseen obstacles, is addressed with an Adaptive Error Corrector, which adjusts actions in real time based on environmental feedback, greatly enhancing robustness. This modular design supports efficient, reliable long-term navigation without requiring additional end-to-end retraining.
We summarize our main contributions as follows: (1) We introduce TP-MDDN, new long-horizon navigation benchmark with multi-sub-demand tasks and explicit task preferences, featuring high semantic richness and scene diversity through the use of DeepSeekV3 and GPT-4o. (2) We propose AWMSystem, an autonomous decision-making world model system composed of BreakI.LMM LocateLLM, and StatusMLLM, enabling efficient instruction decomposition, dynamic goal selection,

---

and real-time execution monitoring without requiring end-to-end training. (3) We design lightweight MASMap, which fuses 3D object detection and 2D semantic mapping to achieve both accurate perception and computationally efficient navigation. (4) Extensive experiments demonstrate that our method significantly outperforms state-of-the-art baselines, with ablation studies confirming the effectiveness of each component. Our system achieves strong robustness and environmental adaptability while maintaining low computational overhead, showcasing a practical balance between performance and efficiency that has not been addressed by previous works.
2 Related Works
Vision-Language Navigation. Visual-Language Navigation (VLN) involves guiding an agent to goal based on language instructions and visual observations. Early methods focused on progress
estimation [12
13].
backtracking [14), reinforcement learning [15 16], and policy learning [
17
18 19 20 21 Some works extracted object and action types from instructions 22 23
24 [25] 26), while others leveraged transformers for history encoding (27]
21
22
23
28 built
topological maps [29 20 30], or predicted future events [31] Pre-training 32
33
34
35
36 and
data enhancement [37 38
39
35
40, 41 have also been explored. However, these approaches
struggle with long-horizon continuous navigation. In contrast, our method adapts zero-shot scene layout understanding to achieve strong performance in such challenging tasks.
Continuous VLN with Foundation Models. Foundation models, like LLMs, VLMs, and LVLMs, by enabling strong reasoning, high-level planning, and end-to-end action generation abilities. Recent zero-shot approaches use these models for collab-
have advanced visual navigation 42
43
or instruction tuning [47]. However, most work focuses discrete actions, while continuous navigation, more suited for real-world use, remains difficult [48 M Earlier methods used GRU or LSTM [4] (50), while recent ones address object-
oration [44] memory-based reasoning 45
46
targeted [51 52 53 (54) and instruction-following tasks [55] (6.55. (7). Some predict progress (55), use value maps 158, or plan trajectories [59] 60 61. Yet, long-horizon tasks suffer from the high cost of frequent LLM calls. To solve this, we introduce a Dempo action generator, inspired by
dual-system robotics 62 51, which boosts efficiency without losing performance.
Long- Morgzon Navigation. Long- horizon navigation is essential for building agents that can learn and act over time. While benchmarks like LH-VLN 64 have made progress, success rates remain low due limited memory encoding. Recent work has explored memory-based methods-like WMNav [1] for relation prediction and Mem2Ego 6 for using landmarks to guide decisions but they focus on object goal navigation. Real-world systems like ReMEmbR [65] support long-range navigation but lack continuous control. Minecraft agents show promise through skill reuse [ 77) but they rely on explicit object prompts. In contrast, demand-driven navigation, where agents meet high-level needs like "find an office tool". better reflects real scenarios. Yet, handling multi-step goals without object-specific instructions remains difficult. To tackle this, we combine obstacle avoidance, error correction, status tracking, and large foundation models to boost navigation performance in complex environments.
3 Tasked-Preferenced Multi-Demand-Driven Navigation
In a Pask-Preferenced Multi-Demand-Driveee Navigation task, the agent is given natural language long-horizon instruction (e.g., "Organize the living space by arranging decorative items, set up a cozy entertainment corner with seating and media devices") "), comprising multiple subtasks. Each subtask combines a basic requirement and a task preference; for example, the basic requirement is "set up cozy entertainment corner", and the task preference is "with seating and media devices".
Formally, let S denote the set of environments and D denote the set of long-horizon instructions. For given instruction d = (d1, .g..... dz) E D, where L is the number of subtasks and d; is the i-th subtask, the goal is determine whether the agent successfully finds objects satisfying each subtask. Specifically, in each episode: (1) A random environment s E S is selected. (2) The agent is initialized at a random position and orientation within 8. (3) A random instruction is chosen. (4) Let O be the set of objects in the environment. We define function G: D O - (0, 1)4, mapping instructions and environment objects to a binary vector indicating whether each subtask satisfied. To complete the instruction, the agent must find at least one object for each subtask. Each subtask is
3

---

Multiple
Dual-Tempo Gonerator
Instructio
Policy
Specialist policy
Tampo
Amribate Sindile
Specialist
Generalist policy
BreakLLM
Moeel
Memory Bank
Extra Info
Succasks (4p
Depth 30peints +20
Fint View
Environment
RGB
Slow-Tempe
StatusMILLM
Alfordance Map Calculation
Adaptive Error Corrector
Navigation Exampte
31
Rest Drgun tie
Obstack:
2otup doan batrop sppes wil
diokkl
map
Buretical sastly aos etlng
Waypoim Predictor
Dijpcts Memery
Targst Mon
Sucneeds
34333-36 chartz 43.81
Target
Nammrry
Somandic
Target Postiinn
map
Agannt Posttion
Olpects Mer
famitoo2.11 2.12 22-
"Chal'
Extra
Suttiaik Status
somsecuting oscoutin"
Reat Feaition
(Tabs. True]
Total Alfordince
map
LocateLLM
WorldModel
Generalist policy
Figure 2: Overview. This diagram illustrates the dual-tempo action generation process in our system. The BreakLLM module decomposes the instruction. Then, depth images are converted into 2D semantic points. The fast-tempo branch uses a pretrained policy to generate primitive actions, while the slow-tempo branch employs Locatel.LM for high-level navigation reasoning, determining target objects and positions. StatusMILLM tracks task progress and updates memory. The Navigation Network performs affordance map computation, adaptive error correction, and waypoint prediction.
strictly binary, defined as either success or failure. To reduce the difficulty the TP-MDDN task, the 2D Euclidean distance between the agent and the object is within threshold cais. the object is considered found. This differs from DDN II, which requires finding the target object within the field of vision.
At each step, the agent can generate actions using either the Generalist Policy or the Specialist Policy. The Generalist Policy drives the agent toward a target pose selected by the large model, while the Specialist Policy uses a pretrained network to execute one of six actions: Move RotateRight RotateLeft, LookUp. LookDown, and Done. The system invokes the Generalist Policy at regular intervals to explore potentially relevant regions, after which the pretrained policy M used to search for objects matching the task requirements. The agent terminates the navigation task when the number of steps reaches the maximum length Lenmax, or when all subtasks are completed. From the test set of Proc ProcTHOR (66], we generated 200 long-horizon instructions, each containing three subtasks, spanning 68 rooms.
3.1 Autonomous Decision-Making World Model System
AWMSystem Overview. Figure 2 illustrates our Autonomous Decision-Making World Model System, which models the environment and predicts actions based on observations. It constructs real-time 2D semantic map with efficient memory storage and uses past trajectories and layout information to select the next target. The Dual-Tempo action generator plans actions, while an Adaptive Error Corrector adjusts strategies based on feedback.
3.1.1 Construction of Long-Range Memory Banks
Raw Data Processing. Building the memory bank consists of three components: input data pro- cessing, real-time accumulation, and recording of different types of historical data. For raw data processing, we perform object detection and segmentation on RGB images and extract 3D point clouds of detected objects from depth maps. These 3D points are then fused and compressed into 2D semantic map. Specifically, at regular intervals, agent performs environmental sensing. Suppose the current step is t; the agent captures RGB images If....,, 1 and corresponding depth

---

images Depthi..... Depth", For each image 1;, we use the Ram Hemn-Grounded-See model [67 68 to obtain object labels, bounding boxes, and segmentation masks. Next, we compute the real-world 3D point cloud PC from each depth image using the camera's 'e intrinsic parameters and rotation matrix.
Real-time Accumulation. In the real-time accumulation design, since object point clouds obtained from different viewpoints at the current location may correspond the same physical object, we design a point cloud update strategy. Let be an object point cloud in the recorded set PCR, Teur be an object point cloud in PCeur and let final denote the residual point cloud obtained by removing regions of Teur that overlap with any object in PCR. During this process, for each candidate r E PCR- we compute two overlap metrics based on the intermediate point cloud your at given stage:
score(r (reur)
score(r (u))
08
I Yeur-pedl
rOs
|r.ped]
where the overlap score is computed using element-wise Euclidean distance between point clouds and denotes the number of points. Let Update(reur: PCn) denote the operation that updates the reference set PCR based on Teur- We define it as follows:
PCn u (Feur)
if max,ePCs o8* 0.25
r repedd - Merge(r.ped, reur Fénal pedd
Update(rouss PCR))
rd.caas- +- Teur class
of > 0.8 A ros* 0.8
PCR[r] r
If the maximum os* over all re PCn is less than 0.25, rear has negligible overlap with any existing object, so it is treated as a new object and added to PCR- If both os* > 0.8 and ros* > 0.8 for a specific r, this indicates strong overlap, suggesting that Teur and represent the same object. In this case, their point clouds are merged, and the class label of r is updated to that of Tear.
Fusion of Global Semantic Map. After recording the center coordinates of object point clouds, all 3D point cloud data is cleared to save memory. Let O.,, denote the object memory bank storing infor- mation about previously detected objects up time step t. Each object's 2D information is recorded as {class: : Cobj E C, center Pobj = (z) is the set of possible object classes. We compute the 2D IoU between current and historical objects and apply the Hungarian algorithm to find the most similar historical object. If match exists, we update the corresponding entry the 2D semantic map and object memory bank; otherwise, we add the new object. Under a task, we continuously accumulate the names and locations of explored objects to form a target memory. The data in the target memory bank is formatted as (Target Object, (z, y), Feedback Type), where (I, y) denotes the target location the map, and Feedback Type includes: Success, Obstructed, Out-of of -Bounds, and other failure descriptions.
(eenter you bbox :
min max min
obj x obj Moby yoby"] [) where
Explanation of Memory Bank. As illustrated in Figure the memory bank contains two components. Global Records serve as Long-Term Memory by storing visited information, including a cumulative map of detected objects with their 2D poses and history of planned targets with execution outcomes, enabling continuous progress tracking. In contrast, Local Updates function as Short- Terr- Memory by maintaining transient data for immediate decision-male maaiing.--- This includes local 3D point clouds extracted from the current panoramic view, additional information derived from past execution failures to address recurring issues, and the current status of the ongoing subtask.
3.1.2 Summary of Foundation Model Usage
BreakLLM. We employ a Large Language Model (LLM) to automatically decompose long-horize instructions into a subtask list doub. and initialize corresponding subtask execution status list Substatuer which is maintained throughout the task. The instruction decomposition is formalized as (dgut, Subgtatus) BreakLLM (d), where all entries in Substatus are initially set to False.
LoCateLLM. At time step t. we maintain memory of detected objects and their 2D coordinates, denoted as OM, (Object Memory). The primary inputs to the decision-matee the overall instruction d, the subtask list deut, the current subtask status Substatuss the target memory Tr, and the object memory OM, Large models may struggle to fully comprehend long sequences of historical trajectories, potentially leading to repeated failures on the same object. To mitigate this, introduce auxiliary feedback to help the model detect and avoid execution
module include:
5

---

(5) Statts LM
Quesiion TEar yu decorpase complex
Oeesioo Yourgaa deferniee anstactie
Densitnn: Wi WHIC outrin from plan wa mest Beyy UST attempted completed bood on CJRREMT image contuct
Eejece
Organus apoi and aig.amettaaiis eabrage aciupe buttrooon deaniinas ai with hyglera products
Massage
Taree way
planed soqerice etsuitokk sls 3. Prevics Sultaca Staks The CURRENT imuge new.
Output Done Done
en deverige solion with diisueer: Mariias
Sainnaa Organin spois sygonersstoogee
Suitedi mairii aoodines
"Roloand Diceoo sepatsholl
Eoeiantts Reecoiing
ITarge Fortlooo 2X0
Updilldd budlaauuuuude Sustaal Stalaso
AftT
Adaptive Error Correctrr
Hop
Environment
Real time uspanted sunantc
Figure 3: Foundation Model Usage. BreakLLM decomposes the instruction. The agent uses Soin-Grounded-SSee [67 68 segment panoramic RGB-D images and ultimately map them onto 2D semantic maps to form object memory. LocateLLM receives multiple types of data and outputs the next target object and position. StatusMLLM determines whether a subtask has been completed based on the current observed image. Adaptive error corrector re- re-plans the failed trajectory.
loops. Specifically, we track the number of consecutive failed attempts dICF on the target ob- ject TargetObja-1- If nCFE 3 Pooerancee we generate auxiliary prompt Extronto in the form: "Do not select <object> or <position> in the next step". This prompt is passed to the planner to discourage revisiting failed targets. The next target object is determined by:
T TargetObje LocateLLM (Tc, OM,, Substatues d, doub. I (NCFE Tholerance) Extra
(1)
StatusMLLM. Iterative updating of subtask completion status is crucial for long-horizon horizon task navigation. introduce multimodal LLM, StatusMLLM, to update Substatus when the policy network outputs a Done action. This design is motivated by the observation that prior methods often trigger Done upon detecting an object that matches the instruction 22 M. We leverage this behavior to infer subtask completion. Specifically, subtask status Substaatue the target memory Te, and the current image 1 into StatusMLLM. The model outputs inference on which subtask might have just been completed Subcur, explanation for its judgment Reason, updated subtask completion status list Subt status' This process formalized
input the overall instruction d. the previous
(Subeur; Reason, Statua
StatusMILLM (d.Suug Status: Tr, Ii) I (Apolicy Done)
3.1.3 Dual-Tempo Action Generator
62 and considering that frequent invocation of large models at every step incurs significant computational overhead, we propose the Dual- Duml-Tempo Action Generator, illustrated in Figure This architecture decouples planning into a slow-tempo phase and fast-tempo phase to balance reasoning depth with efficiency,
Inspired by prior work two-stage navigation strategies [52
In the slow-tempo execution phase, we follow these steps: (1) Extract the current object point cloud PCeur from the panoramic observation and fuse it into the object memory O.,, (2) Feed historical context and exccution feedback into LocateLLM to determine the next target object and its 2D location. (3) Compute affordance value maps that encodes navigational feasibility and semantic relevance. (4) Apply the A algorithm on the affordance map to generate globally feasible navigation path. (5) Decompose the path into a sequence of intermediate waypoints, convert each segment into discrete actions, and execute them sequentially. (6) Invoke the Adaptive Error Corrector detect and rectify trajectory failures.
In the fast-tempo execution phase, we directly employ the pretrained policy from prior work D. which outputs low-level actions based on the current RGB image and the high-level instruction. triggered to evaluate whether subtask
When the policy outputs Done, the StatusMLLM module is

---

has been completed. Next, we detail the computation of the affordance value map and the operational mechanism of the Adaptive Error Corrector.
Calculation of Affordance Map. At step I, we sequentially read an RGB image 7 and a depth image Depth from the panoramic views. For each depth image, we obtain its real-world 3D point cloud Parr- Based on the height of the points, we classify them into navigable points N and obstacle points O Then, we project Vidi and O onto 2D grid map form Niri grid and O grid and
pavi
perform dilation on the obstacle regions grid
Next, we construct the 2D affordance map by computing the obstacle avoidance affordance and semantic affordance. Following similar practices 22. we calculate the Euclidean distances from navigable points W to category-specine point sets. For any navigable point n, N if its distance to the 2D obstacle point cloud satisfies (ot)(11)) < Tobs. the obstacle avoidance affordance value Gobs (n:)) is set O; otherwise, is normalized as aohx((z:) = (dlot (n,) dmin)/ (/dma//xmx////// dmin). From the target 2D coordinates obtained in Section 3.1.2 we define the semantic affordance value (ng(n;) as the inverse of the normalized distance to the target point cloud: (rg(( = (dgg(n;)
dminn) )/ (d" - d'uin)> where (g,( represents the distance from n; to the target point. This means that positions closer to the target receive higher semantic affordance values. Here, dmai and dmax denote the minimum and maximum distances to the obstacle point cloud, respectively; d' and d'max are the corresponding minimum and maximum distances to the target point cloud. Finally, we compute the final affordance value Ofima (n:): if Oobs (n.) O, then Gfimal [n,] is set to O; otherwise, it set to the clipped value of Oege (n.) between 0.1 and 1. The formula is as follows:
S0, clip (ang( 0.1, 1);
Goba (n4) =
afiiaal (nt)
=cclpp
otherwise
Adaptive Error Corrector. Our Adaptive Error Corrector uses environmental feedback to correct navigation errors. When the agent detects that a MoveAhead action may lead to a collision (e.g.. with doors, walls, furniture), it re-plans new path from the current position and updates the waypoint sampling strategy. Under normal operation, the agent navigates by sampling a waypoint every Rwaypeint steps, using discrete action space defined by forward translations of 0.25 meters and rotational increments of 30 degrees. When re-planning is triggered due to execution failure, the agent continues operate within the same action space. The affordance map is recomputed based on the current state, allowing the planner to generate a revised trajectory that avoids obstacles and resumnes progress toward the target object. To improve navigation precision, the trajectory is divided into two segments: an initial segment and subsequent segment. In the initial segment, a finer sampling interval rbblock is applied support detailed spatial reasoning near obstacles. In the subsequent segment, the sampling frequency reverts to the standard rate waypoinn consistent with the regular navigation strategy.
Experiments
4.1 Experimental Setups
Experiments. We use A12-THOR 669 as our simulator and ProcThor as our scene dataset [66]. We used each of DeepSeek-Ve V3 (9 and GPT-4o [10 to generate 100 task-preferenced, multi-demand- driven unseen instructions in test scenarios (totaling 200 commands). In experimental settings, the success distance threshold édis is 1.5 meters, the maximum step count Lenmax is 50, and the tolerance for repeated failed attempts on the same object Rholeaanc within Extra is 2. The obstacle avoidance distance Tobs used during affordance map computation is 0.25 meters. When processing input data, we set the camera resolution to 300x 300 and the horizontal field of view (HFoV) to 90% the agent. The agent operates in closed-loop fashion, perceiving environmental feedback (e.g., success, collision, or boundary detection) immediately after executing each discrete action. All experiments can be run single NVIDIA H100 80GB GPU.
Evaluation Metries. In line with prior works 1 64, we adopt the following evaluation metries: (1) Success Rate (SR): The proportion of tasks in which the agent successfully reaches the target object associated with each subtask. (2) Independent Success weighted by Path Length (ISPI.): For each task, the success of each subtask is weighted by the ratio of the shortest path length to the
7

---

Instruction: Create welcaming entryway by arranging decorative pieces ① set a functional kitchen space with esventaal tools 2 relaxing reading comfortable seating (3) and reading materials.
Action: LookDoen
Target Subtzsk:
Target Object: buruiture
Turget Subtask:1
"Trrgst Objjcts barnun
Target Subtask
Target Object: image
Sabtank Completed:
I-0
E+3
Target
Object
Target
Target
Subtask Comploted:
Subtask Completed:3 3
Subtask Completed: 3
Subtask Completed: 3.2,1
fast toupo
Intermediste Turget Point
A MforddanoMMap
WayPoint Predictor
Figure 4: Visualization Results. The intelligent agent receives a Task-Preferencedd Multi-Demand- Driven instruction, autonomously decomposes it into multiple subtasks, and identifies objects in the scene that match the unexecuted subtasks. On the affordance maps, redder values indicate higher affordance scores. The arrow in the waypoint predictor graph represents the agent's location and current field of view. As the step count increases, the three subtasks are gradually completed.
actual path length, then averaged across all subtasks. (3) Successful Trajectory Length (STL): The average number steps taken in successful navigation trials. (4) Independent Success Rate (ISR): The success rate for each subtask evaluated individually. We evaluate the above metrics over 50 tasks for each method, averaging three due the randomness in large model outputs.
Baselines. DDN is an end-to-end, single-demand -drvend navigation method []] MO-DDN is an based exploration modular agent designed for multi-object, demand-drived navigation [5]. InstructNav uses dynamic chain-of-navigation and multi-sourced value maps to generate robot- actionable trajectories (2). These three methods are the most relevant and state-of-the-art in the field of demand-driven navigation.
4.2 Main Results
The experimental results with baselines shown in Table Our AWM-Nav achieves the highest scores across all metrics, demonstrating its superior performance long- navigation tasks. In contrast, all baseline methods achieve significantly lower results than ours, which can be attributed to the fact that long-horizon instruction navigation typically requires stronger active exploration capabilities and the ability execute multiple subtasks, while these baselines are designed for single-task navigation and lack such capabilities. InstructNav [2] combines zero-shot reasoning using large models with explicit memory mechanisms. However, its planned paths often result in collisions within the environment. Although InstructNav employs dynamic chain-of thain-of-thhught reasoning over actions, the LLM struggles to infer the completion status of subtasks from raw action sequences due to limited contextual understanding. MO-DDN [S] adopts a two-stage navigation process consisting of coarse search followed by fine-grained localization. As the full implementation MO-DDN has not been publicly released, we instead integrate its policy network with our own MASMap for coarse exploration stage, achieving the lowest success rate. Despite showing some capability in multimodal alignment, MO-DDN still falls short when coping with the challenges posed by

---

Table 1: Comparison with state-of state-oftthe-art the- methods on the TP-MDDN benchmark. In the Large-model Inference column, / indicates the LLM is used for reasoning. "Explicit History" refers to methods that record object names and positions in the scene.
Method
Zeroshot
Large-model Inforence
Explicir History
STL
ISR
SR1
ISPI,
DDN 0 MO-DDN
15.50
44.67
16.00
40.66
X
12.11
39.78
13.33
36.25
InstrctNNav 2
/
/
9.50
42.44
16.00
39.41
AWM-Nav
X
20.11
62.89
32,00
44.19
Table 2: Ablation results for object segmenters, reasoning large models, Adaptive Error Corrector, and StatusMLLM. means no information in this line, v/ means using the method, and means not using the method.
The effect of different object segmenters
Different reastming large models
Method
STL1
ISK1
SR1
ISPL
Method
SIL1
ISK1
SRT
ISPL
GLEb
21.33
41.05
U-
10.97
47.78
19.33
36.45
YOLO
15.56
58.00
29.33
43.69
4PT-4o
17.51
56.44
28.67
39.95
RAM-Grunded-SAA
67
20.11
62.89
32.00
44.19
Qwen2-5-VI-728
20.11
62.89
32,00
44.19
3. Influence of Adaptive Error Correcter
4. The effect of StatuMLLM
BlockCorret
BeyondComet
STL1
ISRT
SR1
ISPL.
With/Witout
STL1
ISR1
SRT
ISPL1
x
13.49
59.33
27.33
43.25
X
28.00
X
15.94
27.33
42.46
20.11
62.89
32.00
44.19
/
20.11
62.89
32.00
44.19
task peseerrefeeeeeed multi- multi-demand navigation scenarios. Note that STL refers to the average length of successful trajectories, and our higher score is due to solving some long- dong-distance tasks that require crossing rooms, which increases the average trajectory length.
Besides the metrics in this table, we also pay particular attention to the average execution time, shown in Figure] [ In continuous-action navigation, using LLM inference at every step can be very time-consuming, as demonstrated by InstructNav's performance in (b). The average execution time per long-horizon instruction is 6.82 minutes for AWM-Nav. 1.74 minutes for DDN M1. 6.79 minutes for MO-DDN [5], and 88.90 minutes for InstructNav 2 In the detailed time breakdown, slow-paced actions account for approximately 22 times the duration allocated to fast-paced actions in our method. In summary, our method adopts a dual-tempo action generator to save time and uses an automatic decision-making system built with large models to enhance the agent's reasoning capability.
Ablation Studies. As shown in Table 2 regarding the effect of different object segmenters, the Gew.nrdd--nnnaarr
[67 688 model achieves the best performance, while GLEE 70 lacks precision and YOLO (Ultralyties YOLOv11) 1) performs suboptimally. For different reasoning large models, the well-known GPT-4o does not lead to significant improvements, possibly due to the strong context understanding and state switching awareness required in long-horizon navigation tasks. The open-source Swen2-5-VLL-221 771 72 achieves the best metrics. After examining the execution behavior of the agent, was found that the number of parameters in the large model affects the performance of intelligent planning. In evaluating the influence of the Adaptive Error Corrector, replanning affordance maps proves effective, and unexpected situations remain important to monitor and avoid, even with strong large model reasoning capabilities. With regard to the effect of StatusMLLLM task status tracking is crucial for long-horizon instructions, as misjudgment or absence of status reasoning can severely impact the success of the entire trajectory.
5 Conclusion and Discussion
This paper introduces a new benchmark, TP-MDDN, address navigation tasks involving multiple sub-demands and explicit task preferences. Meanwhile, it proposes the AWMSystem decision- making system, MASMap spatial memory scheme, Deal-Tempo action generation framework, and an adaptive error corrector, which effectively tackle the challenges in TP-MDDN. Experiments show that the method achieves higher navigation accuracy than existing baselines and offers faster inference speed. However, the method has issues such as involuntary mode switching in the dual-tempo action generation framework and navigation decision errors caused by instruction misjudgment due to over- reliance on pre-trained large language models. Future work includes optimizing the mode switching

---

of the action generation framework through reinforcement learning and training domain language models to reduce dependence on pre-trained models.
Acknowledgments
This work is supported in part by NSFC Project (No. 62176061) and Joint Laboratory of Intelligent Construction Engineering Technology for Operating Railway Lines, and the Science and Technology Commission of Shanghai Municipality (No. 24511103100). The authors gratefully thank these organizations for their support and resources.
References
[1] Hongcheng Wang, Andy Guan Hong Chen, Xiaoqi Li, et al. Find what you want: Learning demand- conditioned object attribute space for demand-drives navigation. Advances in Neural Information Process- ing Systems, 1636366666666, 2023.
[2] Yuxing Long, Wenzhe Cai, Hongcheng Wang, et al. Instructnav: sero-shot system for generic instruction navigation in unexplored environment. arXiv preprint arXiv:2406.04880 048848 2024.
[3] Robert J Taormina and Jennifer Gao. Maslow and motivation hierarchy: Measuring satisfaction of the needs. The American journal of psychology, 125(2):15--177, 2013.
[4] Abraham Harold Maslow. Motivation And Personality: Motivation And Personality: Unlocking Your Inner Drive and Understanding Human Behavior by AH Maslow. Prabhat Prakashan, 2023.
[5] Hongcheng Wang, Peiqi Liu, Wenzhe Cai, Mo-ddn: coarse-to- fine attribute- exploration agent for multi-object demand driven navigation. Advances in Neural Information Processing Systems, 37:64176-64214, 2024.
[6] Lingfeng Zhang. Yuecheng Liu, Zhanguang Zhang, et al. Mem2ego: Empowering vision-language models with global-to-ego memory for long-horizon embodied navigation. arXiv preprint 142544 2025.
[7] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, et Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305. 16291, 2023.
[8] Tongtong Feng, Xin Wang, Zekai Zhou, et al. Evoagent: Agent autonomous evolution with continual world model for long-horizon tasks, arXiv preprint arX5::2502.05907 2025.
[9] DeepSeck-AT. Deepscek-v3 technical report, 2024.
[10] Aaron Hurst, Adam Lerer, Adam Goucher, et al. Gpt-4o system card. arXiv preprint arXiv:2410.2127. 2024.
[11] Dujun Nie, Xianda Guo, Yiqun Duan, et al. Wmnav: Integrating jision-lanuse models into world models for object goal navigation. arXiv preprint 2000::::00300227,,,. 2025.
[12] Chih- Yao Ma, Jiasen Lu, Zuxuan Wu, et al. Self-monitoring navigation agent via auxiliary progress estimation. arXiv preprint Xwt::1900000033. 2019.
[13] Chih-Yao Ma, Zuxuan Wu, Ghassan AlRegib, et al. The regretful agent: Heuristic-aided navigation through progress estimation. In Proceedings of the IEEE/CVF conference on Computer Vision and Puttern Recognition, pages 6732-6740, 2019.
[14] Liyiming Ke, Xiujun Li, Yonatan Bisk, et Tactical rewind: Self-correction via backtracking in vision- and-language navigation. In Proceedings of the IFFF/CVV conference on computer vision and pattern recognition, pages 6741 6741-6749, 2019.
[15] Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, et al. Reinforced cross-modal matching and self-supervised imitation learning for vasion-language navigation. In Proceedings the IEEE/CVF conference computer vision and puttern recognition, pages 6629-6638, 2019.
[16] Xin Wang. Wenhan Xiong, Hongmin Wang, et al. Look before you leap: Bridging model-free and model based reinforcement learning for planned-ahead vision-and languune navigation. In Proceedings of the European Conference Computer Vision (ECCV), pages 37-53, 2018.
10

---

[17] Peter Anderson, Qi Wu, Damien Teney, et al. Vision-and- lan-uuuag navigation: Interpreting visually- grounded navigation instructions in real environments. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pages 3674-3683, 2018.
[18] Daniel Fried, Ronghang Hu, Volkan Cirik, al. Speaker-follower models for vision-and-languagg navigation. In Advances in Neural Information Processing Systems, volume 31, 2018.
[19] Hao Tan, Licheng Yu, and Mohit Bansal. Learning to navigate unseen environments: Back translation with environmental dropout. In Proceedings of WAACL-HLT, pages 2610 2621, 2019.
[20] Dong An, Yuankai Qi, Yangguang Li, et al. Bevbert: Multimodal map pre-training for language-guided navigation. arXiv preprint arXiv:22/2.04385, 2022.
[21] Zihan Wang, Xiangyang Li, Jiahao Yang, et al. Gridmm: Grid memory map for vision-and-language navigation. In Proceedings of the IEEE/CVF International conference on computer vision, pages 15625- 15636, 2023.
[22] Yuankai Qi. Zizheng Pan, Yicong Hong, al. The road to know-where: An object-and-room informed sequential bert for indoor vision-language navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1655-1664, 2021.
[23] Abhinav Moudgil, Arjun Majumdar, Harsh Agrawal, et al. Soat: scene-and and object-aware transformer for vision-and and- lunguag navigation. Advances in Neural Information Processing Systems, 34:7357-7366 7367, 2021.
[24] Yuankai Qi. Zizheng Pan, Shengping Zhang, et al. Object-and-action aware model for visual language navigation. In European conference computer vision, pages 303-317. Springer, 2020.
[25] Chen Gao. Jinyu Chen, Si Liu, et al. Room-and-object aware knowledge reasoning for remote embodied referring expression. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pages 3064-3073, 2021.
[26] Xiangru Lin, Guanbin Li, and Yizhou Yu. Scene-intuitive agent for remote embodied visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7036-7045, 2021.
[27] Shizhe Chen, Pierre-Loure Guhur, Cordelia Schmid, et al. History aware multimodal transformer for rision-and-languaee navigation. Advances in neural information processing systems, 34:5834-5447, 2021.
[28] Yicong Hong, Qi Wu, Yuankai Qi, et al. VIn bert: A recurrent vision-ante aad-language bert for navigation. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pages 2021.
[29] Dong An, Hanqing Wang, Wenguan Wang, al. Etpnav: Evolving topological planning for vision- language navigation in continuous environments. IEEE Transactions Puttern Analysis and Machine Intelligence, 2024.
[30] Hao- Lewis Chiang, Zhuo Xu, Zipeng Fu, et al. Mobility vla: Multimodal instruction navigation with long-context vlms and topological graphs. arXiv preprint -xXx:22407007775 2024.
[31] Zihan Wang, Xiangyang Li, Jiahao Yang. et al. Lookahead exploration with neural radiance representation for continuous vision-language navigation. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 13753-13762, 2024.
[32] Weituo Hao, Chunyuan Li, Xiujun Li, et al. Towards learning a generic agent for -an---uggg navigation via pre-training. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 13137-13146, 2020.
[33] Arjun Majumdar, Ayush Shrivastava, Stefan Lee, al. Improving vision-and-language navigation with image-text pairs from the web. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings. Part VI 16, pages 259 274. Springer, 2020.
[34] Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, et al. Think global, act local: Dual-scale graph transformer for vision-and lanngag navigation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 16537-16547, 2022.
[35] Zun Wang, Jialu Li, Yicong Hong, et al. Scaling data generation in vision-and vision-and-languagg navigation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 12009-12020 2023.
11

---

[36] Bowen Pan, Rameswar Panda, YouYoung Jin, et al. Langnav: Language perceptual representation for navigation. arXiv preprini, arXiv:2310.0788,, 2023.
[37] Pierre-Louis Guhur, Makarand Tapaswi, Shizhe Chen, al. Airbert: In-domain pretraining for vision-and- language navigation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1634-1643, 2021.
[38] Kunyang Lin, Peihao Chen, Diwei Huang, et al. Learning vision-and-langee -anguuage navigation from youtube videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8317-8326, 2023.
[39] Jialu Li, Hao Tan, and Mohit Bansal, Envedit: Environment editing for vision and language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15407-
1,517, 2022.
[40] Keji Hc, Chenyang Si, Zhihe Lu, et al. Frequency-onhamced data augmentation for dision-and-langunee navigation. Advances in Neural Information Processing Systems, 36:4351-4366 2023.
[41] Jialu Li and Mohit Bansal. Panogen: Text-conditioned panoramic enviromment generation for vision-and- language navigation. Advances in Neural Information Processing Systems, 36:21878 21894, 2023.
[42] Jiazhao Zhang, Kunyu Wang. Shaoan Wang. et al. Uni-navid: vadeo-based vision- aasiion-anguagg-attii model for unifying embodied navigation tasks. arXiv preprint arXiv:2412. 062244 2024.
[43] Kaiwen Zhou, Kaizhi Zheng, Connor Pryor, et al. Esc: Exploration with soft commonsense constraints for zero-shot object navigation. In International Conference Machine Learning, pages 42829-42843 PMLR, 2023.
[44] Yuxing Long, Xiaoqi Li, Wenzhe Cai, et al. Discuss hefore moving: Visual language navigation via multi-expert discussions. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 17380-17387. IEEE, 2024.
[45] Gengze Zhou, Yicong Hong, and Qi Wu. Navgpt: Explicit reasoning in vision-and and---ndddddaa and-languagee navigation with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 7641-7649, 2024.
[46] Jiaqi Chen, Bingqian Lin, Ran Xu, et al. Mapgpt: Map-guided prompting with adaptive path planning for vision-and-language navigation. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024.
[47] Gengze Zhou, Yicong Hong, Zun Wang, et al. Navgpt-2: Unleashing navigational reasoning capability for large languae models. In European Conference on Computer Vision, pages 260-278. Springer, 2024.
[48] Jacob Krantz, Erik Wijmans, Arjun Majumdar, et Beyond the nav-graph: Vision-and- and-language navigation in continuous environments. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVIII 16, pages 104-120. Springer, 2020.
[49] Junyoung Chung, Caglar Gulcchre, KyungHyun Cho, et al. Empirical evaluation of gated recurrent neural networks sequence modeling. arXiv preprint 141iv:1412.3555. 2014.
[50] Alex Graves. Long short-term memory. Supervised sequence labelling with recurrent neural networks. pages 37-45, 2012.
[51] Naoki Yokoyama, Sehoon Ha, Dhruv Batra, al. Vlfm: semantic navigation. In 2024 IEEE International Conference on Robotics and Automation, pages 42-48. IEEE, 2024.
Vision-language frontier maps for zero-shot
[52] Samir Yitzhak Gadre, Mitchell Wortsman, Gabriel Ihharco, et al. Cows on pasture: Baselines and benchmarks for language-driven zero-shot object navigation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 23171-23181, 2023.
[53] Yuxuan Kuang, Hai Lin, and Meng Jiang. Openfmnav: Towards open-set zero-shot object navigation via vision-language foundation models. arXiv preprint arXiv:2402. 10670, 2024.
[54] Hang Yin, Xiuwei Xu, Zhenyu Wu, et al. Sg-nav: Online 3d scene graph prompting for I1m man-aase zero-shot object navigation. Advances in Neural Information Processing Systems, 17:5285-5307 2024.
12

---

[55] Yanyuan Qiao, Wenqi Lyu, Hui Wang. et al. Open-nav: Exploring zero-shot vision-and- vision-and-language navigation in continuous environment with open-source Ilms. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 6710-6717. IEEE, 2025.
[56] Peihao Chen, Xinyu Sun, Hongyan Zhi, et al. A nav action-awar zero-shot robot navigation by exploiting vision-ande iision-and-languee and-lanuugeee ability foundation models. arXiv preprint urXi::2308.09997,20000 2023.
[57] Zhiyuan Li, Yanfeng Lu, Yao Mu, al. Cog-ga: large language models-based generative agent for vision- laiio-la navigation in continuous environments. arXiv preprint arXiv:2409,02522, 2024.
[58] Kehan Chen, Dong An, Yan Huang, et al. Constraint-aware zero-shot vision-language navigation in continuous environments. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2025.
[59] Adarsh Jagan Sathyamorrthy, Kasun Weerakoon, Mohamed Elnoor, al. Convoi: Context-aware naviga- tion using vision language models outdoor and indoor environments. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 13837-13844. IEEE, 2024.
[60] Soroush Nasiriany, Fei Xia, Wenhao et al. Pivot: Iterative visual prompting elicits actionable knowledge for vlms. arXiv preprint arXiv:2402.07877 2024.
[61] Dylan Goetting, Himanshu Gaurav Singh, and Antonio Loquercio. End-to-end navigation with vision guage models: Transforming spatial reasoning into question anestion-answering. arXiv preprint 1rX05:2411.05755, 2024.
[62] Johan Bjorek, Fernando Castañeda, Nikita Cherniadev, et al. GrOOt n1: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503, 14734 2025.
[63] Qingwen Bu, Hongyang Li, Li Chen, et al. Towards synergistic, generalized, and efficient dual-system for robotic manipulation. arXiv preprint arX0002410.08001, 2024.
[64] Xinshuai Song, Weixing Chen, Yang Liu, et al. Towards long-horizon vision-language navigation: Platform, benchmark and method. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 12088, 2025.
[65] Abrar Anwar, John Welsh, Joydeep Biswas, et al. Remembr: Building and reasoning over long-horizon spatio-temporal memory for robot navigation. In 2025 IEEE International Conference Robotics and Automation (ICRA), pages 2838 2845. IEEE, 2025.
[66] Matt Deitke, Eli VanderBilt, Alvaro Herrasti, al. Procthor: Large-scale embodied ai using procedural generation. Advances in Neural Information Processing Syxtems, 35:5982-5999, 2022.
[67] Tianhe Ren, Shilong Liu, Ailing Zeng, al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401. 14159..19.9, 2024.
[68] Shilong Liu, Zhaoyang Zeng. Tianhe Ren, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. European Conference on Computer Vision, pages 38-55. Springer, 2024.
[69] Eric Kolve, Roozbeh Mottaghi. Winson Han, et al. Ai2-thor: An interactive 3d environment for visual ai. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. 2017.
[70] Junfeng WU, Jiang, Qihao Liu, et al. General object foundation model for images and videos at scale, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3783-3795, 2024.
[71] Qwen Team. Qwen2.5: A party of foundation models, September 2024.
[72] An Yang, Baosong Yang, Binyuan Hui, al. Qwen2 technical report. arXiv preprint arXiv:2407. 10671, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction explain the task domain of Task Preferred Multi Demand Driven Navigation, and also clarify the contribution of proposing autonomous decision-making system.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Section discusses the limitations of involuntary mode switching in the dual-tempo action generation framework.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
14

---

Answer:
[NA]
Justification:
There is no theoretical proof in this article, it is an innovation in other aspects.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: This article describes the experimental setup, dataset, and evaluation metrics in the fourth section.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code declaration will open source.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ /nips.cc/public/uuiceee hwwpbblllgggudiseeeeeeee CoodeSumeeieeeeee ssi criioonPoicyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The model to be used, the specific method, and how to test it have all been explained clearly in the article.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
My experiment set the number of seeds and did not involve significance
analysis.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: In the fourth section, it was explained that all experiments can be run on single NVIDIA HI00 80GB.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't ' make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics btctps://neurippe: http/puubicc/thcsciieeeeeeeee
Answer: [Yes]
Justification: This study involves algorithm development and evaluation based on standard optimization benchmarks. It does not involve human subjects applications that are clearly ethically sensitive, and we assume it complies with NeurIPS ethical standards.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: This article focuses on technological contributions and does not specifically discuss broader positive or negative social impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
17

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This article proposes new optimization algorithn/framewwrk,, and the model and standard benchmark data used have not been found have high risk of abuse
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer:[Yes:
Justification:
This article correctly cites the sources of existing assets, such as baseline methods and data sources.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
18

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thcode wwwc datasetss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
The main new asset is the source code of the proposed method, which will be open-source.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer[[NA]
Justification:
This study does not involve crowdsourcing experiments research with humans as subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The study does not involve human subjects, therefore IRB approval is not applicable.
Guidelines:
19

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification:
The usage of the large model has been clearly explained in the third section.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
20


Reviews:
[{"review_id": "iBC1d72FEA", "reviewer": "Reviewer_DyKS", "summary": "This paper introduces Task-Preferenced Multi-Demand-Driven Navigation (TP-MDDN), a new benchmark for long-horizon navigation involving multiple sub-demands with explicit user preferences, addressing a key gap in existing research. To solve this, the authors propose the Autonomous Decision-Making World Model System (AWMSystem). This system uses three core LLM modules for reasoning: BreakLLM for decomposing instructions, LocateLLM for selecting goals, and StatusLLM for monitoring task completion. For spatial memory, it employs MASMap, which efficiently integrates 3D point clouds into a 2D semantic map. A key innovation is the Dual-Tempo action generator, which combines slower, high-level planning with fast, policy-based control to enhance efficiency and navigation robustness. An Adaptive Error Corrector further improves reliability by handling real-time navigation failures. Experimental results show the system significantly outperforms state-of-the-art baselines on the TP-MDDN benchmark", "questions": "1. Did the authors use the same segmentation model for the baselines? Because the performance of the segmentation model like in InstructNav can very much affect the navigation performance. Using the same segmentation model for all baselines is a fair comparison that demonstrates systematic superiority.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n1. The method proposed in the paper is a very well developed modular system with very good scalability. The fast-slow system is a good solution to the problems of slow inference for large models and poor generalization of end-to-end models.\n2. The entire system is designed to be robust and safe with an Adaptive Error Corrector.\n3. Overall, the method designed by the authors is effective and exceeds the baseline in the experimental results.\n\n## Weaknesses\n\n1.  I feel a little bit strange, why merge several unrelated demands into one instruction? For example, in Fig 3, \"Organize sports equipment storage, set up a beverage station with drinkware, and maintain bathroom cleanliness with hygiene products\".  In a real-world environment, it doesn't seem to be a bother for a user to give several individual demand instructions. In addition, it does not seem to be a difficult task for LLM to break down these independently demand instructions nowadays. I think the author needs to explain the motivation for designing this benchmark\n2. There are some errors in the author's description of the MO-DDN near lines 33 to 35. The MO-DDN does not emphasize object names; like the DDN, it essentially satisfies a demand, so as long as an object satisfies a demand it does not require a specific object. \"Organize lunch\" implies that the agent needs to find the food first and then find the cooking tools , which is the meaning of its prefix MO (Multi-Object). That's because \"organize lunch\" can't be accomplished without finding food or without finding cooking tools.\n3. While the authors have designed an approach that exceeds the baseline, I feel there is some lack of novelty. As an example, Construction of long-range memory banks and affordance maps are used in InstructNav, and it is not the first time that the Dual-Tempo action generator has been used in navigation. I suggest that the authors elaborate on the differences and re-explain the novelties in their paper.", "was_revised": true, "final_justification": "The author clarifies the difference with the previous paper (DDN and MODDN) very well.", "rebuttal": "# Response to Reviewer DyKS\n\nWe sincerely thank the reviewer for the insightful and constructive feedback.\n\n## Response to Weakness 1: Motivation for Merging Unrelated Demands.\n\n - Thank you for the valuable feedback. We designed this multi-demand benchmark to evaluate and advance embodied AI in multi-task, long-range navigation. Our design considerations fall into two main aspects. First, we aim to enhance agents' multi-task navigation capabilities, as current single-task navigation accuracy is still limited, and multi-task scenarios tend to accumulate errors. Second, given the limited variety of object types in existing simulation environments—which constrains the creation of multiple related sub-tasks under a single overarching goal—we have instead constructed unrelated sub-tasks. This approach is a pioneering exploration in addressing multi-unrelated-demand-driven navigation challenges.\n\nHere are more specific explanations:\n\n- Addressing Cumulative Error in Sequential Execution: Current single-demand navigation methods often exhibit low success rates. For example, even a strong baseline like InstructNav achieves only 30.0% success on the DDN dataset for single demand. When multiple unrelated demands are executed sequentially, the errors accumulate, leading to a significantly lower overall success rate.\n\n- Bridging the Gap in Multi-Demand Paradigms: We identify a notable gap in existing demand-driven navigation methods for their limited performance in handling multi-demand tasks. While some approaches, such as MO-DDN, suggest potential for multi-step, related requirements (as indicated in its teaser), their datasets and prompts typically focus on fulfilling single preference with high precision. In contrast, our work proposes a novel framework specifically designed to improve success rates on instructions with several high-level demands that do not explicitly name the required objects.\n\n- **Comparison with experiment that executes single tasks multiple times**. Verifiably, we also proved through experiments that the success rate of the DDN method in successively and separately executing each sub-task is much lower than that of our method. Their results are **SR=16\\% and ISR=47.74\\%.** This indicates that the method of splitting instructions into several individual demand instructions for sequential execution with the aid of large models does not lead to a significant increase in the success rate of multi-demand instruction navigation. It is still very necessary to propose innovative methods to enable agents to efficiently solve multi-demand long-term tasks.\n\n## Response to Weakness 2: Clarification on MO-DDN Description.\n\n- We sincerely appreciate the reviewer's valuable clarification regarding the nature of MO-DDN. We fully agree with the reviewer's understanding that MO-DDN is indeed a demand-driven navigation method. To clarify our original statement,  the \"emphasizes object names\" we mentioned in Line 32 of the main text actually refers to the Task Dataset Generation prompt and the example it generates, detailed in Section A.1.1 of the MO-DDN attachment. In that section,  the preference requirements for generating raw tasks in their prompt contain \"**Detail any additional preferences that refine the object selection**\", which may introduce actual object information and simplify the instructions. For instance, just as their examples on page 18 prompts like \"**I hope the bed is comfortable enough**\" explicitly include an object name (\"bed\") alongside a preference. **In contrast**, our evaluation set is designed with a stricter requirement: instructions must not contain the name of the target object, and instead express abstract requirements and include task-preferenced preference. It should be noted that the toys in the subtask \"Organize a cozy play area with toys\" in Figure 2 of our main text are not our target objects. This design ensures the agent to infer the most suitable objects or locations based on abstract demands and highlights one of key difference compared to MO-DDN. \n\n- We will revise lines 33-35 in the manuscript to ensure this distinction is more accurately and clearly communicated, preventing any misinterpretations. Thank you very much for your suggestion.\n\n## Response to Weakness 3: Novelty and Differences from Related Work.\n\nOur novelty lies not merely in introducing entirely new concepts, but in the innovative integration, significant refinement, and novel application of these concepts within a holistic framework.\n\n**- Differences from Related Work**\n\n1. Comparison with InstructNav\n\n**(a) Long-range memory banks**. InstructNav directly aggregates all encountered point clouds. This approach works well for short distances, as shown by its average trajectory length of 4.44, but causes high computational overhead and slows down reasoning in long-range exploration. Our approach innovatively proposes the Multi-dimensional cumulative Semantic Map (MASMap) construction method. This method creates a highly efficient 2D semantic graph by intelligently merging 3D point clouds from a local panoramic perspective, crucially avoiding duplicate perspective records in the global scope. This design significantly enhances the efficiency and scalability of semantic mapping for truly long-range navigation, a critical distinction from InstructNav's more direct aggregation.\n\n**(b) Affordance maps**. Instructnav's affordance graph integrates multiple heterogeneous affordance values (e.g., planning direction, action feasibility, landmarks). This fusion process may distort the spatial relevance, causing the highest-scoring location to deviate from the vicinity of the most semantically relevant object in the current context. Furthermore, since instructnav takes the points with the largest afforsibility value as the target, given the high density of points, this method often results in consecutive points being chosen from the same local region, trapping the agent in repetitive, small-scale movements without effective exploration. Therefore, we propose a trajectory planning method that uses an affordance map for precise pathfinding directly to the target object's location, combined with an adaptive error corrector to enhance planning effectiveness and avoid local oscillations.\n\n2. Explanation of Dual-Tempo Action Generation\n\nThank you very much for your insights. Our Dual-Tempo Action Generator is specifically designed to address the inherent trade-off between the meticulous, but slow, planning capabilities of large models and the rapid execution of expert strategies in the context of long-range embodied navigation. We explicitly highlight this distinction by demonstrating a stark difference in planning times: the large model planning takes 1.81 minutes in one step, whereas the fast-process expert strategy executes in just 2.02 seconds in one step. \n\n**- Core Novelties of Our Paper**\n\n- Multi-dimensional cumulative Semantic Map (MASMap): This is a novel contribution specifically designed to tackle the challenge of efficiently managing spatial semantic memory in long-range navigation. Its core innovation lies in its sophisticated fusion strategies, leveraging Ram-Grounded-SAM for object detection, Euclidean distance for spatial relationships, and IoU for object overlap, to extract 3D point clouds of objects from only local 8-view images. More importantly, our global 2D semantic map explicitly tracks and associates observations across multiple views, effectively preventing redundant mapping and duplicate write operations for the same object, ensuring a compact, consistent, and efficient semantic representation crucial for scalability in vast environments.\n\n- Autonomous Decision World Model System (AWMSystem): This system represents a significant step towards dynamic objective decision-making in multi-demand tasks without requiring additional training, which integrates an effective chain of thought for object/pose prediction, an adaptive error corrector based on dynamic environmental perception for real-time robustness, and a multi-modal large model-aided task state tracker for more accurate progress and goal management.\n\n- An Novel Evaluation Task for Multi-Demand-Driven Navigation. This benchmark introduces subtasks that deliberately omit explicit object names, express abstract requirements, and incorporate task-type preferences, challenging agents to move beyond simple object retrieval and demonstrate genuine understanding of complex, real-world demands. Our extensive experiments on this benchmark show that our method achieves state-of-the-art (SOTA) performance in both overall task success rate and subtask completion rate.\n\n# Response to Question 1\n\n- We are grateful for the reviewer's valuable suggestions. In our baselines comparison, the original DDN method does not require a segmentation model, and MO-DDN uses the same segmentation model as us. As for instructnav, to make a fairer comparison, we conducted a set of experimental results after using the same segmentation model as ours by using RAM-Grounded-SAM, and InstructNav achieved a **14\\% SR (Success Rate), 3.34 STL (Success weighted by Trajectory Length), 39.39\\% ISR (Independent Success Rate), and 20.80\\% ISPL (Independent Success weighted by path Length)**. These results highlight that even with identical segmentation capabilities, InstructNav's performance remains significantly inferior to ours, demonstrating our framework's superior efficacy and robustness in complex navigation tasks."}, {"review_id": "0NvYMI0kdZ", "reviewer": "Reviewer_J2bP", "summary": "This paper introduces a multi-demand object goal navigation benchmark TP-MDDN and an AWMSystem decision-making system, which can achieve SOTA performance on the benchmark. TP-MDDN is constructed based on AI2-THOR simulator on ProcThor scenes, where authors construct 100 task-preferenced multi-demand-driven unseen instructions. The AWMSystem is a complicated system which incorporates MASMap spatial memory scheme, Dual-Tempo action generation framework, and the adaptive error corrector.", "questions": "- Line 119 states that the benchmark uses a distance threshold to determine whether the target is found. This is very unclear. What specific distance metric is used—Euclidean distance or geodesic distance? Additionally, should the target be observable by the agent, as in other navigation benchmarks (e.g., Habitat HM3D ObjectNav)?\n\n- The LLM is tightly coupled with the entire system. Does increasing the map size (and thus the number of prompts) lead to a performance drop in the LLM?\n\n- I note that the method supports looking up and looking down, but the real-world experiments do not include these actions. Is there any discussion about this discrepancy?", "limitations": "Yes (Section 5, more detailed discussion will be appreciated.)", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths: This paper is highly comprehensive, with significant contributions to both benchmarks and methods. I particularly appreciate the illustrations and the video, which effectively demonstrate the methodology and performance. The authors’ ability to make such a complex method work in both simulated and real-world environments is commendable. \n\nWeaknesses: \n\n- Justification for demand-driven vs. object-driven navigation: The pipeline attempts to decode instructions into a series of target objects and then perform object-goal navigation. This raises a key question: If the LLM is already effective at translating demands into target objects, why is a separate demand-driven navigation method necessary? This task could potentially be handled by object-goal navigation methods, which are well-established in the field. \n\n- Heuristic-based pipeline relying on LLMs and maps: While a heuristic-based pipeline that performs well is acceptable, the paper lacks methodological novelty and key insights. Authors could highlight the key technical contribution of this paper.", "was_revised": true, "final_justification": "Thanks for the thoughtful discussion. Most of my concerns have been addressed.\n\nI agree that this paper on demand-driven navigation is interesting and the whole work required large efforts, but my remaining concern is the heuristic design of using LLM, which leads to a lack of sufficient novelty. Overall, I maintain my initial score.", "rebuttal": "# Response to Reviewer J2bP\n\nWe sincerely thank the reviewer for their thorough review and valuable feedback.\n\n## Response to Weakness 1: Justification for Demand-Driven vs. Object-Driven Navigation. \n\nWe'd like to clarify that our approach does not directly decode instructions into fixed target objects at the outset. Instead, our demand-driven framework addresses several critical challenges that go beyond the scope of traditional object-goal navigation:\n\n- Dynamic and Abstract Target Selection: Unlike object-goal navigation, where a specific object is the fixed target, our system first identifies objects and their positions from the scene. Then, a large language model (LLM) dynamically determines which sub-task to execute currently and which object to select as the target based on the abstract instruction and the evolving environment. This process also critically involves ensuring that previously selected objects are not duplicated. This dynamic, context-aware decision-making for abstract instructions is a core challenge of demand-driven navigation that object-goal navigation cannot directly resolve.\n\n- Robustness to Object Recognition Uncertainty: In real-world and simulated environments, object recognition is not 100\\% accurate due to factors like lighting, shadows, and color distortion. Even with advanced segmentation models like RAM-Grounded-SAM, the perceived names of the same object from different perspectives can be inconsistent or inaccurate. Our system accounts for this by requiring the agent to explore and verify whether a recognized object truly meets the abstract demand, rather than blindly navigating to a pre-defined object.\n\n- Real-time Adaptation to Unknown Scenes: We cannot determine the ideal target object for abstract requirements in an unknown scene from the very beginning. Our system continuously records newly detected objects in real-time and feeds them as optional targets to LocateLLM. This means that for the same abstract demand, the LLM dynamically adjusts its target object selection based on newly recognized objects during exploration, enabling adaptive behavior in unseen environments.\n\nTherefore, our demand-driven navigation method is essential, which are fundamentally distinct from and more complex than simple object-goal navigation.\n\n## Response to Weakness 2: Methodological Novelty. \n\nWe thank the reviewer for their detailed assessment. We respectfully clarify that our primary contribution is not merely an engineering achievement. Instead, it lies in identifying specific, unresolved problems within the context of the novel multi-demand-driven navigation task and proposing innovative technical designs to overcome these challenges. This constitutes significant methodological novelty. Our key technical contributions are:\n\n- **Efficient and Scalable Spatial Memory Constructing in long-horizon navigation**. Previous work directly aggregates all encountered point clouds, which performs well over short distances but incurs high computational overhead and slows down reasoning in long-range exploration. To address this, we propose the Multi-dimensional cumulative Semantic Map (MASMap), a novel contribution designed to efficiently manage spatial semantic memory for long-horizon navigation. Its core innovation lies in sophisticated fusion strategies that leverage Ram-Grounded-SAM for object detection, Euclidean distance for spatial relationships, and IoU for object overlap to extract 3D object point clouds from only local 8-view images. More importantly, our global 2D semantic map explicitly tracks and associates observations across multiple views, effectively preventing redundant mapping and duplicate writes for the same object, thereby ensuring a compact, consistent, and efficient representation essential for scalability in large-scale environments. \n\n- **Dynamic Objective Decision-Making for Multiple Needs**: Current large-model-based demand-driven navigation methods primarily rely on map-point-driven action generation, making them prone to local repetitive planning and lacking dynamic planning awareness for multiple, evolving tasks. To address these challenges, we propose the Autonomous Decision World Model System (AWMSystem), which represents a significant advance in enabling dynamic decision-making without requiring additional training. It integrates an effective chain-of-thought mechanism for object and pose prediction, an interaction mechanism based on environmental real-time perception, and a multi-modal large-model-aided task state tracker for more accurate progress monitoring and goal management.\n\n- **High-Precision Motion Generation for Robust Navigation**: Nowadays, demand-driven action generating methods include using large models to predict the the current one action. However, agents are prone to being attracted by multiple objects in the scene and deviating from the previously planned path, causing decision-making confusion. Some methods approach the detected related objects, but they are often difficult to apply in unseen scenes due to urgent problems such as being unable to pass through narrow doors and getting stuck at table corners. To fill this gap and address issues such as door frame obstruction and exceeding boundaries without additional training, we propose a waypoint predictor targeting the center coordinates of objects, with multi-size collaborative correspondence (agent step size, object size, waypoint sampling interval), and then use an adaptive action corrector based on environmental feedback. Ultimately, in the multi-demand navigation task, we achieved a trade-off between success rate and reasoning performance by using the Dual-Tempo action generation method with efficient approximation of the target object and fine expert detection.\n\n## Questions\n\n### Response to Question 1: Distance Metric.\n\n- Thank you for the question. To clarify, we define successful sub-task completion using the 2D Euclidean distance between the agent and the center of the target object, with a success threshold of 1.5 units. While benchmarks such as Habitat's HM3D ObjectNav require visual observation of the target object, we do not explicitly verify such observation during evaluation. This choice was made to reduce complexity in the context of multi-demand driven navigation, as discussed in the main text (lines 118–122). A consistent evaluation protocol was applied across all baseline experiments. Furthermore, extensive visual comparisons demonstrate that our method achieves strong performance in accurate target exploration, effective obstacle avoidance, and the resolution of complex multi-demand tasks.\n\n### Response to Question 2: LLM Coupling with Map Size / Performance Drop\n\n- We confirm that the size of the semantic map does not directly affect the number of LLM prompts or lead to a decline in LLM performance. This is because our prompt generation is object-based rather than map-based. we detect objects within the agent’s current field of view, compute their 3D coordinates in the camera coordinate system, and then transform them into the real-world coordinate system based on the current camera pose. Only the names and center coordinates of these detected objects are included in the prompt, meaning the number of prompts depends solely on the visible objects at any given moment, not on the overall map size. Furthermore, our 2D semantic grid map—constructed by extracting and merging 3D point clouds from panoramic RGB-D images (captured from 8 perspectives), projecting them onto a 2D plane, and compressing them into a pixel grid—serves primarily as lightweight spatial memory, increasing storage by only kilobytes to megabytes. Crucially, this map is not fed directly into the LLM as input, and therefore does not contribute to prompt length or impact LLM performance.\n\n### Response to Question 3: Look Up/Down in Real-World Experiments\n\n- We acknowledge that, although our method supports \"look up\" and \"look down\" actions, these are not explicitly included in the real-world experiments. This limitation arises from two main factors: first, the camera on our real robot is mounted along the arc of the mechanical arm, which imposes mechanical constraints that make precise control of vertical gaze directions challenging. Second, the tasks in our real-world scenarios primarily involve common daily-life objects, which can be effectively observed from the robot’s standard horizontal (forward-facing) camera perspective. As such, the absence of explicit vertical viewpoint changes does not significantly hinder the robot’s ability to perceive relevant objects in the environment."}, {"review_id": "bILaynRNCx", "reviewer": "Reviewer_gPWR", "summary": "The paper introduces TP-MDDN, a benchmark addressing task-preferenced, multi-demand-driven long-horizon navigation scenarios. It proposes an integrated solution, AWMSystem, combining several distinct components: BreakLLM for decomposing complex instructions into subtasks, LocateLLM for goal selection, StatusLLM for monitoring task progress, MASMap for spatial memory management, and a Dual-Tempo action generation framework supported by an Adaptive Error Corrector to handle navigation errors dynamically.", "questions": "1. Could the authors clearly differentiate between operational \"preferences\" and simple subtask criteria or object categories?\n2. Is it possible to partially complete subtasks, or are subtasks strictly binary (fully successful or failed)? Clarifying this operational detail would significantly aid in understanding the task complexity.\n3. Could the authors provide more explicit analysis regarding computation time versus robot traversal time? \n4. Why are traditional temporal or precedence-aware planning algorithms not sufficient or applicable here? For example, after BreakLLM decomposes instructions into a formal specification, could a conventional planner (e.g., PDDL-based, temporal constraint solver) replace LocateLLM? If not, what specific limitations (e.g., reasoning over preferences, handling uncertain subgoals, dynamic error correction) prevent that? E.g. see https://ieeexplore.ieee.org/abstract/document/10611163\n5. Can the authors elaborate on why and how each component was selected for integration? More discussion of failure cases that motivated the components or justification for the chosen components would be beneficial. This could also be accomplished by more discussion on the ablation study.", "limitations": "The authors partially address limitations, acknowledging mode switching instability and dependency on pre-trained models. However, explicit discussion of high failure rates and computation overheads would be beneficial. I recommend a dedicated discussion on these points.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 1, "strengths_and_weaknesses": "Strengths:\n- The paper is well-organized, clearly delineating the motivation, methodology, and results.\n- The motivation for addressing longer-horizon tasks with multiple subtasks is compelling and effectively articulated.\n- The comprehensive engineering effort to integrate multiple modules (LLMs for instruction handling, semantic mapping, policy-based navigation) into a cohesive system is impressive and demonstrates considerable practical innovation.\n\nWeaknesses: \n- The emphasis on \"task preferences\" appears conceptually promising, but operationally resembles detailed task criteria rather than true user preferences, such as choosing among multiple suitable objects based on subjective user criteria.\n- The solution seems heavily reliant on the integration of existing components, making the primary contribution more of an engineering achievement than a novel theoretical advancement or the identification of specific problems that inspire the need for different components.\n- The paper lacks a deeper discussion on computational efficiency and failure cases, which are critical for real-world applicability given the described \"high\" failure rates.\n\nOther comments:\nThere is redundancy in early sections where components are repeatedly introduced. Streamlining these descriptions would allow space for deeper analysis, such as expanded discussion of the ablation studies or detailed insights into the specific challenges addressed by each component.", "was_revised": true, "final_justification": "The authors have addressed my primary concerns regarding the differentiation between task preferences and detailed task criteria, provided a thorough rationale for component selection, expanded on computational efficiency with planned baseline comparisons, and offered a more comprehensive discussion of why traditional planning algorithms are not applicable in this setting, including relevant citations. While I still view the work as more incremental than groundbreaking in novelty, the benchmark contribution, experimental results, and clarifications provided tip the balance toward acceptance for me.", "rebuttal": "# Response to Reviewer gPWR\n \nWe sincerely thank the reviewer for their thorough review and valuable feedback.\n\n## Response to Weakness 1 \\& Question 1: Task Preferences vs. Subtask Criteria\n\n- We strongly agree that a clearer differentiation is crucial. Our \"task preferences\" address a key gap in prior demand-driven navigation (DDN) datasets, moving beyond object recognition to model abstract user intent. \n\n- Motivation for Task Preferences: Early DDN tasks often featured very simple commands like \"I want to sleep\". Such instructions are highly ambiguous, as \"sleeping\" could involve a sofa, a bed, or even resting on a desk, which makes inferring intent and target objects difficult. We propose that each subtask should have an implied or explicit preferred task preference to guide agents.\n\n- Differentiating \"Object Preferences\" from \"Task Preferences\":\n\n(1) For a macroscopic instruction like \"organize the living space\", user preferences can be categorized. \"Object preferences\" would involve specifying particular objects (e.g., \"use a broom\", \"find a storage box\"). However, introducing explicit object information can easily simplify the task to object navigation. \n\n(2) In contrast, our \"task preferences\" specify the preference of task type, and we ensure that the name of the target object will not be included in the preference. Examples include \"achieve cleaning tools\", \"prepare decorative items,\" or \"find storage tools,\" as highlighted in lines 23-25 of our main text. This design ensures that the agent must infer the most suitable objects based on the abstract task preference, rather than relying on explicit object names. \n\n(3) Therefore, our \"task preferences\" are not merely detailed task criteria. The clarity of this requirement is important for correctly executing an abstract basic requirement\n\n## Response to Weakness 2: Reliance on Existing Components\n\n- We appreciate the reviewer for the thoughtful feedback. We clarify that our core contribution goes beyond combining existing components. It lies in: (1) identifying previously overlooked limitations of current methods in multi-demand navigation, and (2) introducing novel technical designs to effectively address these challenges, enabling robust and efficient performance.\n\n- Challenges and our contributions:\n\n(1) **Map construction**: Previous work directly aggregates all encountered point clouds, which performs well over short distances but incurs high computational overhead and slows down reasoning in long-range exploration. To address this, we propose the Multi-dimensional cumulative Semantic Map (MASMap), a novel contribution designed to efficiently manage spatial semantic memory for long-horizon navigation. Its core innovation lies in sophisticated fusion strategies that leverage Ram-Grounded-SAM for object detection, Euclidean distance for spatial relationships, and IoU for object overlap to extract 3D object point clouds from only local 8-view images. More importantly, our global 2D semantic map explicitly tracks and associates observations across multiple views, effectively preventing redundant mapping and duplicate writes for the same object, thereby ensuring a compact, consistent, and efficient representation essential for scalability in large-scale environments. \n\n(2) **Dynamic Objective Decision-Making**: Current large-model-based demand-driven navigation methods primarily rely on map-point-driven action generation, making them prone to local repetitive planning and lacking dynamic planning awareness for multiple, evolving tasks. To address these challenges, we propose the Autonomous Decision World Model System (AWMSystem), which represents a significant advance in enabling dynamic decision-making without requiring additional training. It integrates an effective chain-of-thought mechanism for object and pose prediction, an adaptive error corrector based on real-time environmental perception to ensure robustness, and a multi-modal large-model-aided task state tracker for more accurate progress monitoring and goal management.\n\n(3) **An Novel Evaluation Task for Multi-Demand-Driven Navigation**, as important as explained in Response to Weakness 1 & Question 1. Our extensive experiments on this benchmark show that our method achieves state-of-the-art (SOTA) performance in both overall task success rate and subtask completion rate.\n\n(4) **Balancing Zero-Shot Transfer Ability and Efficient Reasoning**. As shown in Table 2 (our ablation study), the design choices and individual contributions of the components have a significant impact on navigation performance, demonstrating the essentiality of each component.\n\n## Response to Weakness 3 \\& Question 3: Computational Efficiency and Failure Cases. \n\n### **Computational Efficiency**\n\n- We agree that a deeper discussion on computational efficiency and failure cases is critical for real-world applicability. The average execution time for a complete task is approximately 17.41 minutes(nearly 39.75 steps). \n- The average time consumption of each module for one task is as follows. BreakLLM :2.89 seconds. Fast-tempo Execution: 67.49 seconds (nearly 32 steps, averaging 2.02 seconds per step). Slow-tempo Execution: 870.74 seconds (nearly steps, averaging 1.81 minutes per step). StatusLLM: 93.93 seconds (averaging 4.68 seconds per call). Adaptive Error Corrector: 12.64 seconds. \n- Smaller modules in slow-tempo execution is as follows. Real-time Mapping: 93.68 seconds. LocateLLM: 5.06 seconds. Affordance: 4.13 seconds. Waypoint Predictor: 0.75 seconds. Trajectory Execution: 5.24 seconds. \n- It takes about 0.39 seconds for a robot to move or rotate once. This detailed breakdown demonstrates how our dual-tempo design balances robustness with efficiency.\n\n### **Analysis of Failure Cases**\n\nWe observed recurring failures related to accuracy and object placement, yet our method still remains viable for real-world use, as shown in our real-robot experiment (see attachment). (1) In cluttered scenes, limited spatial reasoning may cause the LLM to treat nearby objects as separate, leading to redundant visits. However, this occurs significantly less often than with prior map-point-driven methods. We can attempt to enhance the coordinate perception awareness of large models. (2) Small Object Misses: Very small items such as clips or pencils may be missed despite RAM-Grounded-SAM's robustness. Future work could explore zoom-in detection. (3) Hidden Targets: Hidden Targets: If an occluded target is deemed absent, the agent may skip the area, causing failure. This suggests a need for active reasoning, such as checking cabinets, through knowledge bases or uncertainty-driven exploration.\n\n## Response to Other Comments \\& Question 5: Component Selection.\n\nWe acknowledge the reviewer's comment regarding adding expanded discussion. \n\n### **Justification for Component Selection and Integration:**\n\nOur system is modularly designed, comprising map construction, command decomposition, and dual-tempo action generating modules. Each component was carefully selected and designed to address specific limitations of prior work in the context of multi-demand long-range navigation: \n\n**- component :**\n\n1. Map Construction: Problem: Methods like InstructNav store all point clouds, causing slow reasoning (~1 hour/task). Our Fix: We merge 3D points using panoramic views and anti-repetition compression, enabling scalable, fast mapping.\n\n2. Target Object Prediction (LocateLLM): Previous work relies on map-point-driven target prediction, often leading to redundant exploration within the same area. In contrast, our LocateLLM performs object-level prediction, significantly reducing the likelihood of repeated visits. Moreover, we innovatively predicts the target coordinates directly, providing precise spatial input that meets the requirements for subsequent trajectory planning.\n\n3. Adaptive Error Corrector: Despite effective path planning, collisions can still occur in tight spaces. We attempted to use multimodal LLMs for correction, but the action sequences they generated failed to smoothly navigate around obstacles due to a lack of simulation-based planning experience. In contrast, our adaptive corrector leverages real-time environmental feedback to autonomously resolve collisions and boundary issues, significantly improving navigation robustness.\n\n**- Ablation Study Insights (Table 2):**\n\n1. Segmentation: Grounded-SAM outperformed GLEE (used in DDN/InstructNav), especially in detecting fine details (e.g., cup on toilet)\n\n2. State Monitoring: Using LocateLLM to simultaneously perform status prediction failed, likely because the multitasking prompt diluted its focus and impaired judgment accuracy. In contrast, our hybrid approach, where the expert module triggers \"Done\" and the LVLM independently confirms completion, significantly improved success rates.\n\n3. LLM Comparison: Qwen2.5-VL-7B exhibited repeated targets, while GPT-4o switched subtasks erratically. In contrast Qwen3-32B demonstrated a more reasonable rhythm in subtask switching.\n\n## Response to Question 4: Traditional Planning Algorithms\n\nTraditional planning algorithms (e.g., PDDL) are unsuitable for replacing LocateLLM because they: (1) focus on temporal or priority constraints, while our method emphasizes interpreting multimodal inputs to fulfill abstract instructions; (2) require discrete, predefined state spaces that conflict with our unseen, continuous 3D environments; (3) struggle with dynamic, abstract sub-goals and adaptive task prioritization based on evolving context; and (4) lack robust real-time error correction from continuous sensory feedback. Unlike approaches such as AutoTAMP that use LLMs for output correction, our method employs LLMs for core chain-of-thought reasoning and multimodal understanding.\n\n## Details: \n\nResponse to Question 2. Subtasks are evaluated as binary outcomes, either successful or failed."}, {"review_id": "xt4mNTaBWy", "reviewer": "Reviewer_7Quy", "summary": "This paper introduces TP-MDDN, a new benchmark for long-horizon embodied AI navigation that handles multiple sub-demands with explicit task preferences, reflecting real-world complexity where people have multiple related needs. The proposed AWMSystem, an autonomous decision-making system, is composed of BreakLLM (instruction decomposition), LocateLLM (goal selection), and StatusLLM (task monitoring). The framework is combined with a MASMap for efficient 3D-to-2D spatial memory and a Dual-Tempo action generation framework that integrates zero-shot planning with policy-based fine control. The approach also includes an Adaptive Error Corrector for real-time failure handling. Empirical results demonstrate superior performance of TP-MDDN across all metrics while maintaining computational efficiency compared to methods using LLMs at every step. The system successfully balances performance and efficiency for practical long-horizon navigation tasks.", "questions": "Please consider answering the following technical questions for clarification.\n- The authors mentioned in L206 that the output of slow-tempo is a sequence of continuous actions, but the output for fast-tempo/specialist policy are six discrete actions (as stated in L125 - L126), is that correct? If so, why is there such a discrepancy? Moreover, if the six actions for specialist policy are discrete, then the details about distance for MoveAhead, degrees for rotation and look actions are missing,\n- In Figure 2, the Adaptive Error Corrector is placed inside the module of slow-tempo, where the actions should be sequence of continuous ones. However, in L235, you mentioned the agent senses feedback of some discrete action (MoveAhead), which shows contradiction.\n- Still in L206, according to the authors, the sequence of actions are executed after decomposed from the navigation path. Does this mean the actions are executed in an open-loop manner? Does the agent gets feedback from the environment during this execution? If not, then there might be the situation where a collision or out-of-bound has occurred before the action sequence is completely executed.\n- Could the authors provide more details regarding re-planning, how it exactly \"starts with a finer interval  then gradually returns to the\nusual\"?\n- The questions regarding clarity in Weaknesses.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The introduced TP-MDDN benchmark advances beyond existing multi-object navigation by explicitly incorporating task preferences alongside multiple sub-demands, which addresses a real limitation where previous work, e.g., MO-DDN, would introduce ambiguous specification. Instead, the formalization of explicit task preferences better reflects real-world navigation complexity and provides clearer guidance for agent behavior.\n- The systematic decomposition into three specialized LLM modules represents a reasonable and novel approach to handling complex long-horizon tasks. Each module addresses a distinct aspect of the navigation problem, which is both technically sound and aligns well with how humans might approach such tasks. The modular design allows for targeted evaluation and optimization.\n- The Dual-Tempo action generation framework addresses a fundamental computational efficiency problem in LLM-based navigation by separating slow-tempo LLM reasoning for high-level planning and target selection from fast-tempo policy-based control for fine-grained actions near targets. Such design avoids the computational bottleneck of calling expensive LLMs at every step while maintaining the benefits of LLM reasoning for complex decision-making.\n\nWeaknesses\n- Regarding long-horizon instructions generation and split, according to the example illustrated in Fig. 3, the instructions are well-organized with straightforward sentence segmentation, where the BreakLLM simply breaks it into three parts split by comas. This is no more than directly feeding subtasks into the subsequent module. In the real-world scenario, a human instruction can be less clear and structured, although it may still clearly state the preference. Why not make the instructions more similar to human orders and take the advantage of LLM’s ability of language understanding? Moreover, the subtask completion status are uniformly set to False in BreakLLM. All these factors make the BreakLLM module trivial.\n- [L161] It is unclear what global records and local updates are, respectively.\n- There is no explicit reference to Eq. (1) and Eq. (2). Despite relevant, with out references, the two equations look independent of the text, especially for Eq. (1), where $H_t$, $M_t$, $State_t$ are never defined and explained in the text.\n- [L181] The design of including extra info prompt after failing consecutive times is not that elegant. Instead of blaming large model’s capability of understanding long historical sequences and using an enforced way to circumvent it, it would be better to dig into the failed cases and analyze the culprit more carefully, providing a more neat solution to avoiding the failure beforehand.", "was_revised": true, "final_justification": "The submission is generally good but with some clarity issues. The authors clarified multiple technical details and addressed concerns regarding this matter. Their detailed elaboration on the replanning mechanism is particularly appreciated. Therefore, I am inclined to accept this paper.", "rebuttal": "# Response to Reviewer 7Quy\n\nWe sincerely thank the reviewer for their thorough review and insightful feedback.\n\n## Response to Weakness 1:  Instruction Generation and BreakLLM's Role\n\n- We appreciate the reviewer's feedback regarding our instruction set design. The core purpose of BreakLLM is to decompose abstract human instructions into executable subtasks. Our current instructions are structured to ensure that each subtask's requirements are clear and unambiguous, which is essential for accurately evaluating the agent's understanding and execution, and helps prevent failures caused by ambiguous descriptions. We strongly agree that enhancing realism is important and are considering adopting a more natural instruction format. However, these instructions will still only convey basic needs and preferences without explicitly naming objects, in order to preserve the core challenge of the task—consistent with the instruction set used in our main article. Thank you very much for your suggestion.\n- An example of more colloquial instructions is as follows: \"Before the guests arrive, please organize the sports equipment using some storage tools. Also, I'd like to set up a small beverage station, which requires preparing some drinkware, and please do a quick clean of the bathrooms with hygiene products to ensure everything looks good.\" We believe BreakLLM remains effective under such instructions. \n- The uniform initialization of all subtask completion statuses to False in BreakLLM is intentional, as it accurately reflects the logical starting point: no subtasks have been completed immediately after parsing a long-horizon instruction. This procedure provides a clear initial value for subsequent modules (e.g. StatusLLM) to accurately monitor and update progress.\n\n## Response to Weakness 4:  Design for Handling Consecutive Failures\n\nWe thank the reviewer for the insightful suggestion. While our current approach effectively mitigates immediate issues, we acknowledge that it can be further improved and extended with a more in-depth analysis.\n\n1. LLM Base Impact on Planning: The choice of large language model (LLM) significantly influences planning quality. In our experiments, Qwen3-32B generated fewer repeated object predictions than Qwen2.5-VL-7B under identical conditions—using the same segmentation module and prompts. This suggests that the base model’s reasoning capabilities directly affect planning reliability, particularly in processing complex multimodal interaction histories. Our findings underscore the importance of advancing LLM backbone performance for instruction-following navigation tasks.\n\n2. Cause of Consecutive Failures and Our Design Inspiration: A typical cause of repeated failures arises when targeting large objects (e.g., a sofa). The LocateLLM predicts the object's center, and the Waypoint Predictor generates a trajectory ending within it. If the agent fails to reach this point, the LLM may re-select the same object as the next target, creating a failure loop. Since precise 3D object sizing remains challenging, we instead leverage auxiliary information to break this cycle—demonstrating it to be an effective and concise solution.\n\n3. Future Work Toward a More Elegant Solution: We agree that a more elegant approach is desirable. Our future work will explore adaptive trajectory planning using object size estimates derived from the 2D map, generating approach points outside the object's boundary based on its dimensions. This aims to enable robust navigation without relying on ad hoc fixes.\n\n## Response to Questions\n\n### **Response to Questions 1**: \n\n- Thank you for the feedback. We apologize for any confusion caused by the wording in L206 and will correct it. To clarify, both the slow-tempo and fast-tempo policies use discrete actions in our simulation environment, where fundamental actions like MoveAhead, RotateRight, and Lookup are inherently discrete. The difference lies in their execution: (1) the fast-tempo policy (Lines 125–126) executes individual discrete actions—e.g., a single 0.25m MoveAhead or 30° rotation step; (2) the slow-tempo policy (L206) refers to the outcome of a sequence of discrete actions, where a planned trajectory is broken into waypoints, each reached via multiple discrete steps. Slow-tempo movements use 0.25m steps and 45° rotations, chosen to balance precision and efficiency.\n\n### **Response to Questions 2**: \n\n- Thank you for pointing out the potential source of confusion regarding Figure 2 and line 235. To clarify: (1) The adaptive error corrector is triggered when the fast-tempo process fails to execute a specific discrete action toward a given waypoint. Therefore, the environmental feedback it receives corresponds to discrete actions such as MoveAhead. (2) Furthermore, when the adaptive error corrector encounters issues like collisions, it invokes the waypoint predictor to replan the affordance map and fine-tunes the action interval design. Since this mechanism is closely tied to the planning strategy used in the fast-tempo process, it is logically situated within the fast-tempo framework.\n\n### **Response to Questions 3:** \n\n- We appreciate this important question. Our agent operates in a closed-loop manner: after each discrete action, it receives immediate environmental feedback (e.g., success or collision or boundary detection). If an action fails, the system quickly adjusts the strategy, enabling robust navigation—this is key to the effectiveness of components like the Adaptive Error Corrector. The same feedback is available to other researchers using the environment and the difference lies in whether they choose to leverage it effectively.\n\n### **Response to Questions 4: Details on Re-planning Mechanism**\n\n We appreciate the opportunity to elaborate on our re-planning mechanism:\n\n1. Re-planning Triggers: Failures typically arise from collisions during forward movement (e.g., in narrow passages, with table corners, or partially open doors) or boundary violations. When detected, re-planning is initiated to navigate around the obstacle.\n\n2. Need for Refined Re-planning: We have attempted to input current observation images and context to directly prompt the LLM to generate a sequence of actions to resolve the current obstruction situation. However, experiments showed that renowned large models such as DeepSeek, GPT-4o, and Qwen2.5VL-72B were unable to generate effective action sequences, possibly due to a lack of experience in navigation trajectory planning within simulated environments. This motivated our geometric re-planning strategy.\n\n3. Refined Re-planning Approach. The approach consists of three key components: First, a new affordance graph is generated for the agent's immediate surroundings, enabling a local update of the environment representation. Second, the central coordinate of the original target object remains unchanged, ensuring continuity in the goal. The core innovation lies in the **variable sampling interval**: when re-planning the trajectory, sampling points are placed at smaller intervals near the beginning (head) of the trajectory, while maintaining the original sampling interval for the latter portion. Experimental results demonstrate that this method effectively helps agents navigate around obstacles and significantly improves the overall task success rate.\n\n4. Validation \\& Example: To better understand the Adaptive Error Corrector section (illustrated in the lower right corner of Figure 3 in the main text), please refer to the animation task in the far left panel of '**simulation-robot-demo.mp4**' (included in the attachments) at 0:01:08. The task instruction is \"**Organize sports equipment storage, set up a beverage station with drinkware, and maintain bathroom cleanliness with hygiene products**\". At t=34 seconds, the agent detects the target object in the bathroom within the panoramic view (note that not all panoramic views observed by the agent are shown in the animation). The agent initially plans a trajectory toward the target, but this path is blocked by a closed door. Upon detecting the collision, the system recognizes the action failure and triggers a replanning process. From the agent's current position, a new trajectory to the target object is generated. Thanks to a very small sampling interval for the head trajectory, the agent is able to first execute a turn maneuver, successfully navigating around the obstacle.\n\n## Clarity Questions from Weaknesses Section \n\n### **Response to Weaknesses 2**: \n\nGlobal Records (Long-Term Memory) stores visited information: a cumulative map of detected objects and their 2D poses, and a history of planned targets with their execution outcomes (success/failure), enabling progress tracking. Local Updates (Short-Term Memory) holds transient data for immediate decisions: local 3D point clouds extracted from the current panoramic view (8 perspectives), environmental feedback (e.g., from repeated execution failures), and the current subtask status.\n\n### **Response to Weaknesses 3**: \n\nWe sincerely thank the reviewer for pointing out this issue and we will revise this in the later version as follows: Equation (2), which formalizes the process described in Lines 185-194, will be explicitly referenced at that location. For Equation (1), corresponding to Lines 175-177, we will clearly define all variables correctly within the equation: $H_t$ (Historical Exploration Information, revised to $T_t$), $M_t$ (object memory bank, revised to $OM_t$), $State_t$ (current subtask list, revised to $Sub_{status}$), and $d_{sub}$ (subtask details, revised to $Subtasks$). These revisions will ensure that all symbols are properly introduced and the equations are fully understandable."}]

Author rebuttal:

Meta-review: This paper presents a novel benchmark and system for long-horizon navigation with multiple user sub-demands. The problem is relevant and important, addressing a clear research gap, and the proposed solution is technically sound. Post-rebuttal, concerns were resolved, and reviewers reached consensus for acceptance. The AC concurred and recommended acceptance.
Decision: Accept (poster)