Title: TP-MDDN: Task-Preferenced Multi-Demand-Driven Navigation with Autonomous Decision-Making
Venue: NeurIPS 2025
Accepted: yes
Keywords: Long-Horizon Task Planning

Continuous Vision and Language Navigation

Demand-Driven Navigation

Autonomous Decision-Making

Unseen Environment

Large Vision-Language Models
Abstract: In daily life, people often move through spaces to find objects that meet their needs, posing a key challenge in embodied AI. Traditional Demand-Driven Navigation (DDN) handles one need at a time but does not reflect the complexity of real-world tasks involving multiple needs and personal choices. To bridge this gap, we introduce Task-Preferenced Multi-Demand-Driven Navigation (TP-MDDN), a new benchmark for long-horizon navigation involving multiple sub-demands with explicit task preferences. To solve TP-MDDN, we propose AWMSystem, an autonomous decision-making system composed of three key modules: BreakLLM (instruction decomposition), LocateLLM (goal selection), and StatusMLLM (task monitoring). For spatial memory, we design MASMap, which combines 3D point cloud accumulation with 2D semantic mapping for accurate and efficient environmental understanding. Our Dual-Tempo action generation framework integrates zero-shot planning with policy-based fine control, and is further supported by an Adaptive Error Corrector that handles failure cases in real time. Experiments demonstrate that our approach outperforms state-of-the-art baselines in both perception accuracy and navigation robustness.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces TP-MDDN, a new benchmark for long-horizon embodied AI navigation that handles multiple sub-demands with explicit task preferences, reflecting real-world complexity where people have multiple related needs. The proposed AWMSystem, an autonomous decision-making system, is composed of BreakLLM (instruction decomposition), LocateLLM (goal selection), and StatusLLM (task monitoring). The framework is combined with a MASMap for efficient 3D-to-2D spatial memory and a Dual-Tempo action generation framework that integrates zero-shot planning with policy-based fine control. The approach also includes an Adaptive Error Corrector for real-time failure handling. Empirical results demonstrate superior performance of TP-MDDN across all metrics while maintaining computational efficiency compared to methods using LLMs at every step. The system successfully balances performance and efficiency for practical long-horizon navigation tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The introduced TP-MDDN benchmark advances beyond existing multi-object navigation by explicitly incorporating task preferences alongside multiple sub-demands, which addresses a real limitation where previous work, e.g., MO-DDN, would introduce ambiguous specification. Instead, the formalization of explicit task preferences better reflects real-world navigation complexity and provides clearer guidance for agent behavior.\n- The systematic decomposition into three specialized LLM modules represents a reasonable and novel approach to handling complex long-horizon tasks. Each module addresses a distinct aspect of the navigation problem, which is both technically sound and aligns well with how humans might approach such tasks. The modular design allows for targeted evaluation and optimization.\n- The Dual-Tempo action generation framework addresses a fundamental computational efficiency problem in LLM-based navigation by separating slow-tempo LLM reasoning for high-level planning and target selection from fast-tempo policy-based control for fine-grained actions near targets. Such design avoids the computational bottleneck of calling expensive LLMs at every step while maintaining the benefits of LLM reasoning for complex decision-making.\n\nWeaknesses\n- Regarding long-horizon instructions generation and split, according to the example illustrated in Fig. 3, the instructions are well-organized with straightforward sentence segmentation, where the BreakLLM simply breaks it into three parts split by comas. This is no more than directly feeding subtasks into the subsequent module. In the real-world scenario, a human instruction can be less clear and structured, although it may still clearly state the preference. Why not make the instructions more similar to human orders and take the advantage of LLM’s ability of language understanding? Moreover, the subtask completion status are uniformly set to False in BreakLLM. All these factors make the BreakLLM module trivial.\n- [L161] It is unclear what global records and local updates are, respectively.\n- There is no explicit reference to Eq. (1) and Eq. (2). Despite relevant, with out references, the two equations look independent of the text, especially for Eq. (1), where $H_t$, $M_t$, $State_t$ are never defined and explained in the text.\n- [L181] The design of including extra info prompt after failing consecutive times is not that elegant. Instead of blaming large model’s capability of understanding long historical sequences and using an enforced way to circumvent it, it would be better to dig into the failed cases and analyze the culprit more carefully, providing a more neat solution to avoiding the failure beforehand.\n\nQUESTIONS:\nPlease consider answering the following technical questions for clarification.\n- The authors mentioned in L206 that the output of slow-tempo is a sequence of continuous actions, but the output for fast-tempo/specialist policy are six discrete actions (as stated in L125 - L126), is that correct? If so, why is there such a discrepancy? Moreover, if the six actions for specialist policy are discrete, then the details about distance for MoveAhead, degrees for rotation and look actions are missing,\n- In Figure 2, the Adaptive Error Corrector is placed inside the module of slow-tempo, where the actions should be sequence of continuous ones. However, in L235, you mentioned the agent senses feedback of some discrete action (MoveAhead), which shows contradiction.\n- Still in L206, according to the authors, the sequence of actions are executed after decomposed from the navigation path. Does this mean the actions are executed in an open-loop manner? Does the agent gets feedback from the environment during this execution? If not, then there might be the situation where a collision or out-of-bound has occurred before the action sequence is completely executed.\n- Could the authors provide more details regarding re-planning, how it exactly \"starts with a finer interval  then gradually returns to the\nusual\"?\n- The questions regarding clarity in Weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces TP-MDDN, a benchmark addressing task-preferenced, multi-demand-driven long-horizon navigation scenarios. It proposes an integrated solution, AWMSystem, combining several distinct components: BreakLLM for decomposing complex instructions into subtasks, LocateLLM for goal selection, StatusLLM for monitoring task progress, MASMap for spatial memory management, and a Dual-Tempo action generation framework supported by an Adaptive Error Corrector to handle navigation errors dynamically.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is well-organized, clearly delineating the motivation, methodology, and results.\n- The motivation for addressing longer-horizon tasks with multiple subtasks is compelling and effectively articulated.\n- The comprehensive engineering effort to integrate multiple modules (LLMs for instruction handling, semantic mapping, policy-based navigation) into a cohesive system is impressive and demonstrates considerable practical innovation.\n\nWeaknesses: \n- The emphasis on \"task preferences\" appears conceptually promising, but operationally resembles detailed task criteria rather than true user preferences, such as choosing among multiple suitable objects based on subjective user criteria.\n- The solution seems heavily reliant on the integration of existing components, making the primary contribution more of an engineering achievement than a novel theoretical advancement or the identification of specific problems that inspire the need for different components.\n- The paper lacks a deeper discussion on computational efficiency and failure cases, which are critical for real-world applicability given the described \"high\" failure rates.\n\nOther comments:\nThere is redundancy in early sections where components are repeatedly introduced. Streamlining these descriptions would allow space for deeper analysis, such as expanded discussion of the ablation studies or detailed insights into the specific challenges addressed by each component.\n\nQUESTIONS:\n1. Could the authors clearly differentiate between operational \"preferences\" and simple subtask criteria or object categories?\n2. Is it possible to partially complete subtasks, or are subtasks strictly binary (fully successful or failed)? Clarifying this operational detail would significantly aid in understanding the task complexity.\n3. Could the authors provide more explicit analysis regarding computation time versus robot traversal time? \n4. Why are traditional temporal or precedence-aware planning algorithms not sufficient or applicable here? For example, after BreakLLM decomposes instructions into a formal specification, could a conventional planner (e.g., PDDL-based, temporal constraint solver) replace LocateLLM? If not, what specific limitations (e.g., reasoning over preferences, handling uncertain subgoals, dynamic error correction) prevent that? E.g. see https://ieeexplore.ieee.org/abstract/document/10611163\n5. Can the authors elaborate on why and how each component was selected for integration? More discussion of failure cases that motivated the components or justification for the chosen components would be beneficial. This could also be accomplished by more discussion on the ablation study.\n\nLIMITATIONS:\nThe authors partially address limitations, acknowledging mode switching instability and dependency on pre-trained models. However, explicit discussion of high failure rates and computation overheads would be beneficial. I recommend a dedicated discussion on these points.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 1, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a multi-demand object goal navigation benchmark TP-MDDN and an AWMSystem decision-making system, which can achieve SOTA performance on the benchmark. TP-MDDN is constructed based on AI2-THOR simulator on ProcThor scenes, where authors construct 100 task-preferenced multi-demand-driven unseen instructions. The AWMSystem is a complicated system which incorporates MASMap spatial memory scheme, Dual-Tempo action generation framework, and the adaptive error corrector.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: This paper is highly comprehensive, with significant contributions to both benchmarks and methods. I particularly appreciate the illustrations and the video, which effectively demonstrate the methodology and performance. The authors’ ability to make such a complex method work in both simulated and real-world environments is commendable. \n\nWeaknesses: \n\n- Justification for demand-driven vs. object-driven navigation: The pipeline attempts to decode instructions into a series of target objects and then perform object-goal navigation. This raises a key question: If the LLM is already effective at translating demands into target objects, why is a separate demand-driven navigation method necessary? This task could potentially be handled by object-goal navigation methods, which are well-established in the field. \n\n- Heuristic-based pipeline relying on LLMs and maps: While a heuristic-based pipeline that performs well is acceptable, the paper lacks methodological novelty and key insights. Authors could highlight the key technical contribution of this paper.\n\nQUESTIONS:\n- Line 119 states that the benchmark uses a distance threshold to determine whether the target is found. This is very unclear. What specific distance metric is used—Euclidean distance or geodesic distance? Additionally, should the target be observable by the agent, as in other navigation benchmarks (e.g., Habitat HM3D ObjectNav)?\n\n- The LLM is tightly coupled with the entire system. Does increasing the map size (and thus the number of prompts) lead to a performance drop in the LLM?\n\n- I note that the method supports looking up and looking down, but the real-world experiments do not include these actions. Is there any discussion about this discrepancy?\n\nLIMITATIONS:\nYes (Section 5, more detailed discussion will be appreciated.)", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Task-Preferenced Multi-Demand-Driven Navigation (TP-MDDN), a new benchmark for long-horizon navigation involving multiple sub-demands with explicit user preferences, addressing a key gap in existing research. To solve this, the authors propose the Autonomous Decision-Making World Model System (AWMSystem). This system uses three core LLM modules for reasoning: BreakLLM for decomposing instructions, LocateLLM for selecting goals, and StatusLLM for monitoring task completion. For spatial memory, it employs MASMap, which efficiently integrates 3D point clouds into a 2D semantic map. A key innovation is the Dual-Tempo action generator, which combines slower, high-level planning with fast, policy-based control to enhance efficiency and navigation robustness. An Adaptive Error Corrector further improves reliability by handling real-time navigation failures. Experimental results show the system significantly outperforms state-of-the-art baselines on the TP-MDDN benchmark\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. The method proposed in the paper is a very well developed modular system with very good scalability. The fast-slow system is a good solution to the problems of slow inference for large models and poor generalization of end-to-end models.\n2. The entire system is designed to be robust and safe with an Adaptive Error Corrector.\n3. Overall, the method designed by the authors is effective and exceeds the baseline in the experimental results.\n\n## Weaknesses\n\n1.  I feel a little bit strange, why merge several unrelated demands into one instruction? For example, in Fig 3, \"Organize sports equipment storage, set up a beverage station with drinkware, and maintain bathroom cleanliness with hygiene products\".  In a real-world environment, it doesn't seem to be a bother for a user to give several individual demand instructions. In addition, it does not seem to be a difficult task for LLM to break down these independently demand instructions nowadays. I think the author needs to explain the motivation for designing this benchmark\n2. There are some errors in the author's description of the MO-DDN near lines 33 to 35. The MO-DDN does not emphasize object names; like the DDN, it essentially satisfies a demand, so as long as an object satisfies a demand it does not require a specific object. \"Organize lunch\" implies that the agent needs to find the food first and then find the cooking tools , which is the meaning of its prefix MO (Multi-Object). That's because \"organize lunch\" can't be accomplished without finding food or without finding cooking tools.\n3. While the authors have designed an approach that exceeds the baseline, I feel there is some lack of novelty. As an example, Construction of long-range memory banks and affordance maps are used in InstructNav, and it is not the first time that the Dual-Tempo action generator has been used in navigation. I suggest that the authors elaborate on the differences and re-explain the novelties in their paper.\n\nQUESTIONS:\n1. Did the authors use the same segmentation model for the baselines? Because the performance of the segmentation model like in InstructNav can very much affect the navigation performance. Using the same segmentation model for all baselines is a fair comparison that demonstrates systematic superiority.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}