MO-DDN: A Coarse-to-Fine Attribute-based Exploration Agent for Multi-Object Demand-driven Navigation
Authors: Hongcheng Wang
Peiqi Liu
Wenzhe Cai
Mingdong Wu
Zhengyu Qian
Hao Dong
Venue: NeurIPS 2024 poster (2024)
Keywords: Mudolar Object Navigation
Demand-driven Navigation
Attribute Learning
Primary area: robotics
TL;DR: We propose a multi-object demand-driven navigation benchmark  and train an coarse-to-fine attribute-based exploration agent to solve this task.

Abstract: The process of satisfying daily demands is a fundamental aspect of humans' daily lives. With the advancement of embodied AI, robots are increasingly capable of satisfying human demands. Demand-driven navigation (DDN) is a task in which an agent must locate an object to satisfy a specified demand instruction, such as "I am thirsty." The previous study typically assumes that each demand instruction requires only one object to be fulfilled and does not consider individual preferences. However, the realistic human demand may involve multiple objects. In this paper, we introduce the Multi-object Demand-driven Navigation (MO-DDN) benchmark, which addresses these nuanced aspects, including multi-object search and personal preferences, thus making the MO-DDN task more reflective of real-life scenarios compared to DDN. Building upon previous work, we employ the concept of ``attribute'' to tackle this new task. However, instead of solely relying on attribute features in an end-to-end manner like DDN, we propose a modular method that involves constructing a coarse-to-fine attribute-based exploration agent (C2FAgent). Our experimental results illustrate that this coarse-to-fine exploration strategy capitalizes on the advantages of attributes at various decision-making levels, resulting in superior performance compared to baseline methods. Code and video can be found at https://sites.google.com/view/moddn.

MO-DDN: A Coarse-to-Fine Attribute-based Exploration Agent for Multi-object Demand-driven Navigation
Hongcheng Wang Peiqi Liu
Wenzhe Cai Mingdong Wu 1.3 Zhengyu Qian
Hao Dong 1,3f
CCFCS, School of CS, PKU "Scoool of BECS, PKU
3PKU-Agibot Lab School of Automation, Southeast University
Abstract
The process of satisfying daily demands is a fundamental aspect of humans' daily lives. With the advancement of embodied AI, robots are increasingly capable of satisfying human demands. Demand-driven navigation (DDN) is task in which an agent must locate an object to satisfy specified demand instruction, such "I am thirsty." The previous study typically assumes that each demand instruction requires only one object to fulfilled and does not consider individual preferences. However, the realistic human demand may involve multiple objects. In this paper, we introduce the Multi-object Demand-driven Navigation (MO-DDN) benchmark, which addresses these nuanced aspects, including multi-object search and personal preferences, thus making the MO-DDN task more reflective of real- life scenarios compared DDN. Building upon previous work, we employ the concept of "attribute" to tackle this new task. However, instead of solely relying attribute features in end-to-end manner like DDN, propose modular method that involves constructing a coarse-to- tove-to-fie fine attribute- based exploration agent (C2FAgent). Our experimental results illustrate that this coarse-to-fine exploration strategy capitalizes on the ad advantages of attributes at various decision -akiing levels, resulting in superior performance compared to baseline methods. Code and video can be found at https://sstess. http://itees.googlloooom// www..//o//www//o/o/nnnn
Introduction
In the field of psychology, the creation of demands directs the motivation for human behavior in the real world, and behavior ultimately leads the fulfillment of the demand [14. For example, an individual might crave sweet iced tea, prompting them to search for object such as tea, sugar, ice, and water, and then combine them to make sweet iced tea. Recently, with the rapid development of embodied AI and large language models, researchers are interested in using robots meet various human demands [5H]3] Demand-driven Navigation (DDN) 14 is a variant ObjectGoal Navigation (ON) 215.21). which requires agent to find object that satisfies given demand instruction. For example, when a user gives the agent an instruction such as am thirsty," the robot has to search the entire environment for objects such water, tea, coffee, etc., depending what is available within the environment.
In previous work [14), demand instructions often exhibit a low level of complexity, lack consideration for user preferences, and typically require only one object fulfill each instruction. However,
"Equal contribution.
!Coresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Ta
Depenrlag on the attilben
Boad
by rewdly
kide
nee yau organtio lanch and
ike will besuibea chaine seduet and p- TVE we up, Lanch served dising raum. and forkil takie.
anticttinn groat thuee ar
mouill aod
frtend grefee and bakeed feed. Their 3ddx avy peoter wase fce creans.
dl to caok Soot
Bed ice creona loons
Antritate
Warpoist
seper Beodjem
Labei
Tojevey
An Example of Multi-objeet Demand-driven Navigation. A user plans to host party at his new house and outlines some basic demands (highlighted in orange), along with specific preferences for different individuals (highlighted in red). The agent parses these demands and locates multiple objects in various locations in the scene to fulfill them. Despite not meeting the preferred "ice cream" demand, the agent successfully addresses basic demands, such as organizing lunch.
Figure 1:
real-world situations frequently involve more intricate instructions, necessitating the coordination of multiple objects to satisfy a demand. For example, in the BEHAVIOR-IK 1K [22], daily tasks often involve the interaction of multiple objects. Furthermore, individual users may have distinct preferences. For instance, when presented with the basic demand am thirsty," one user may prefer sparkling water while another may favor juice. However, satisfying preferred demands is obviously more difficult than satisfying basic demands, so the agent needs to be flexible in prioritizing goals depending both the situations of users and environments.
We introduce a new benchmark, Multi-object Demand-driven Navigation (MO-DDN), in which demand instructions are comprised of basic and preferred demand components. An agent must find a combination of objects (we call a "solution" later) that satisfies the demand. The agent should satisfy the user's preferred demands much as the situation allows. We use GPT-4 automatically generate and modify tasks and perform manual checks. Our settings are similar to Multi-object demand instruction to combine potentially multiple objects in a form that consistent with common sense and personal preferences list of object categories in MON. The number and category of objects in solution not known in advance but require an agent to reason on its own. Moreover, solutions that satisfy the same demand may have different number of objects. We argue that MO-DDN has similar advantages over MON as DDN has over ON.
Navigation (MON) 23 25), but
humans rather than a
We consider MO-DDN to be crucial preliminary step in task planning. Today, with sufficient meta-infonte information provided (e.g.. position, state, and category of all objects in the scene), large language models provide satisfactory task planning results (26-29 (5. However, maintaining and providing accurate meta-information can be challenging and inconvenient in real life. The user's interactions with the scene change the meta-information frequently. Moreover, since the user not necessarily omniscient about the scene, especially in unfamiliar scenes, it is possible that the user parses the demand himself and directly provides the desired objects, these desired objects may not exist in the scene. Even the user can provide comprehensive meta-information, delivering the meta-information the agent is time-consuming and inconvenient for the user. Therefore, the purpose of MO-DDN is to provide timely and accurate meta-information depending on a convenient demand instruction in natural language for later task planning or, more later, object manipulation.
we gpt-4-0125 -preview API this paper here and later, unless otherwise noted.

---

In the recently ObjectGoal Navigation work, benefiting from the development of Vision Language Models (VLMs) 36 and Large Language Models (LLMs)
[27 some ObjectGoal Navigation
methods return to a modular manner 45
45 50
rather than an end-to-end manner [19] 51-
58). While maintaining the core concept of the "attribute" from the end-to-end agent of the previous work 14], we modify the training of the attribute model to make it applicable to multi-object search, and propose a coarse-to-fin attribute-based exploration modular agent, C2FAgent. In the coarse exploration phase, similar to the modular methods in ObjectGoal Navigation, use depth camera to reconstruct the environment point clouds and an object detection module, GLEE [3I, to label the objects in the point clouds. Additionally, the point clouds compressed and segmented into several 2D rectangular blocks, and thus each detected object belongs to a block. For each block, we calculate the similarity of the objects' attribute features and the instruction's 's attribute features metric for choosing a waypoint. We calculate basic demand similarity scores and preferred demand similarity scores separately and select blocks according to weighted sum of the two scores. In the fine exploration phase (i.e,, when the agent arrives at the waypoint), we train an end-to-end attribute exploration module identify and report the target object to compose the solution.
The experimental results show that our proposed method outperforms the baselines, and the ablation study shows that the attribute model improves exploration efficiency in different phases. We argue that this coarse-to- ses-tto-fiin to-fine design allows for the incorporation of prior knowledge from external foundation models in the coarse exploration phase and task-relevant world-grounding exploration in the fine exploration phase. Morcover, the ablation study on the weighted sum of the basic and preferred similarity scores demonstrates that increasing the preferred weights allows the agent to prioritize searching for the preferred solution and increasing the basic weights allows the agent to prioritize searching for the basic solutions. Therefore, users can adjust the weights to influence the agent's behavior freely, which is a flexible way to handle personal preferences.
Our main contributions are listed as follows:
We propose a new benchmark, MO-DDN, which considers multi-object combinations as solutions and more complex and diverse demand instructions. MO-DDN can be regarded as a crucial preliminary step in task planning.
We extend the training process of the attribute model, enabling the attribute features to work well in multi-object setting. Based on the version of attribute features, We design coarse-to-fingg attribute-based exploration agent, C2FAgent, for this benchmark, allowing the attribute features play important role in different exploration phases.
The experimental results show that the attribute features do improve the efficiency of exploration, and the experimental results substantially surpass the baselines. Ablation study shows that attribute-basee exploration is more efficient than frontier-based exploration (59] and LLM--ased waypoint selection.
2
Related Work
2.1 Visual Navigation
Goal Description In general terms, visual navigation involves the continuous generation of ac- tions based on RGB-D and GPS+Compass inputs until a specified objective is reached [60] Visual navigation tasks vary in their goal description, such step-by-step instructions in Vision-Language Navigation (VLN) 61 68 audio in Audio-visual Navigation [69)74), object categories in Object- Goal Navigation (ON) (1) object category lists in MultiObject Navigation (MON) [43 23
and demand instructions in Demand-driven Navigation (DDN) 14). Our proposed benchmark, MO-DDN, can be viewed as a multi-object version of DDN. Although, like DDN, MO-DDN's inputs are demand instructions, MO-DDN's solutions are multi-object rather than singlc-object, as in DDN.
object Previous Method Overview In ON, with the rise of object detection models 31 30 75
76
segmentation models [7] and large language models, modular methods 43
44
45 50
are gradually being developed. They greatly improve navigation efficiency and success rate by building semantic maps and navigable maps and then planning paths on the maps. Meanwhile, end- to-end methods 19 51.58 focus more on learning associations object-object and object- to help reason about the potential location of target objects. In MON, previous work has focused on
3

---

studying how to quickly construct [25], memorize 244 43], and use semantic maps [8 or implicit representations [23] of scenes. In DDN, the concept of attributes is introduced as a way of expressing what an object shows when it fulfills demand, e.g., quenching thirst is an attribute of water in the context of demand "I am thirsty". In this paper, we extend the concept of attributes and propose a new method to train the attribute model. We combine the advantages of modular and end-to-end methods and propose a coarse-to-fine attribute-based exploration modular agent.
2.2 Foundation Large Model in Embodied Task
Foundation Large Models refer to self-supervised pre-trained models trained on large Internetsscale datasets, which have demonstrated promising capabilities across various embodied tasks. Large language models (LLMs) such as GPT-4 [22) LLaMA 83 40), and Gemma (84) exhibit performance
comparable to humans in task planning 22 85], common sense reasoning [86] and question
answering (8 90 34 I1]. Furthermore, researchers use LLMs to synthesize task datasets (14).
The CLIP model [92], which employs contrastive learning with image and text pairs, showcases strong semantic extraction abilities in navigation [4 93
68 94 Visual Language Models offer understanding of images such as image descriptions 95 98, object detection [31 30 75
21
776.. and object segmentation [77-80) for downstream embodied tasks. This paper uses LLMs, specifically GPT-44 to generate task instructions, solutions, and attribute examples for attribute learning. Furthermore, we employ GLEE (31), state-of-the-art object detection model, detect object categories within the field of view for attribute feature extraction.
3 Multi-object Demand-driven Navigation
Following the basic settings in DDN, let D denote a set of demand instructions, Se denotes a set of navigable scenes, and O denotes set of object categories. Let So denote set of solutions for demand instructions. An element in So (i.e., a solution) is a subset of O. In each episode, similar location within mapless environment and
to the DDN task, an agent is randomly initialized receives a demand instruction DI E D in natural language. In this benchmark, a demand instruction consists of two parts: one is basic demand instruction Dīg, and the other is preferred demand instruction DIp. e.g., need a comfortable place to play computer games, preferably with good lighting." " Each DI has two sets of solutions, i.e., basic solution So, and preferred solution Sop for Dí, and DI,, respectively. For example, (desk, soft chair) is an element in So, for the above example demand instruction, and (desk, soft chair, table lamp) an element in Sop.
Then, at each time step, the agent should choose an action from MoveAhead, RotateRight, RotateLeft, LookUp, LookDown, Find, and Done. The action Find is similar in MON, which automatically reports the objects in the field of view. To reduce the difficulty of the MO-DDN task and foeus it navigation, if the agent chooses Find, all objects in the field of view with distance below the threshold find will be recorded in a found list FI. instead of requiring the bounding box of the target object as in DDN. The found list FL is used to calculate the basic and preferred success rate later. When the agent chooses Done, or the number of choices Find reaches a threshold findd or the number of steps reaches a threshold natep, the episode ends and the success rate is calculated. For specific demand instruction DI, We calculate the basic success rate as follows:
N SR.
ΣOEFL 1oESs 1=1 $5ESo, Len(sE)
Σ max
N
where N donates the number of testing episodes, Sy donates a basic solution in the solution set of DI, FL donates the found list that the agent reports, o donates an object category. The preferred success rate SR is calculated similarly. To summarize, the success rate of an episode is the value with the highest percentage of satisfaction among all solutions. We also calculate the SPL [60] corresponding to SR, and RR. We generate 300 tasks, encompassing 358 object categories from the HSSD dataset 199. These tasks are referred to as HSSD's world world-groundin tasks, indicating that objects these tasks are in the HSSD dataset. We generate some language-groundd the attribute model afterward. hanguage-grounding everything that makes sense rather than restricting objects in HSSD. Please see the supplementary material for details about task generation A.I.I task metrics A.1.2 and task dataset statistics A.1.3
tasks to train means that the objects in the solutions can be

---

Method
4.1 Attribute Model Training
In this section, we describe how to train the MO-DDN's attribute model that extends from the attribute model in DDN. In DDN, the training of attribute features constrained by the assumption that each instruction requires only one attribute that can be satisfied by single object. Such attribute features may not work well under multi-object settings. This paper proposes directly mapping demand instructions and object categories into the same attribute feature space. Such mapping can learn multiple attribute features simultaneously to address multi-object search. Concretely, the core function of the attribute model is to map demand instruction in D or object category in O to several attribute features Rd R the set of real numbers, is the dimension of the attribute features) that are in the shared attribute feature space. In order to enable the alignment of the attribute features of instructions and objects, we design discrete codebook and five losses. The alignment means that for demand instruction and object in its solution, one of the attribute features of the instruction and one of the attribute features of the object have a high cosine similarity.
K-Means
Language grounding Attribute Features
GPE4
Clustering
Object
Encoder
Groasé-Trmt In Old
Initializacion
Antibone Feitures
Encoder
Encoder
MLP
Codebook
Decuder
Pealance
Altribat
Altribut Featurcs
Featurce
Figure 2: Attribute Model. This figure shows the architecture of the attribute model. Instructions and objects share the same model architecture. Instructions and items share the same model architecture. For parameters, they share only the parameters of the shared codebook, while the parameters of the MLP Encoder and Decoder are independent. Only the red with flames modules the figure will be trained while the blue with snowflakes CLIP model parameters will be frozen.
4.1.1 Codebook and Its Initialization
The reason for using a discrete codebook is similar to VQ-VAE, i.e., different attributes are inherently discrete from each other. Then using discrete codebook an attribute feature space is a better way to represent the relationships between attributes than a continuous attribute feature space. The codebook essentially some feature vectors, and in our experiments, and 768 as the vector dimension (same with CLIP ViT-L/14's text dimension); thus, the codebook is a x28x768 matrix. we CLIP- Text-exxxdddeeee (ViT-L/14) to encode the language-grounde generated in the supplementary material Sec in2oo
choose 128 as the number of vectors
attributes innto attribute features. These attribute features then clustered using K- K-means [100] to get 128 clustering centers. The feature vectors of these 128 clustering centers are used to initialize the codebook, making the codebook as a subspace of the CLIP feature space.
4.1.2 Definition of Losses
In our method, there are two attribute models that share the same architecture, the instruction attribute model AMmss and the object attribute model A.objs shown Fig. In both models, only MLP Encoder and CLIP Encoder will be used in C2FAgent, Codebook and MLP Decoder are only used to train ML.P Encoder. We obtain the k instruction attributes and ko object attributes (the green square in Fig- 2) by prompting GPT-4 for "what attributes can satisfy this instruction?" and "what attributes does this object have?"" respectively. For the five losses used to train the two attribute models, see the pseudo-code
5

---

Algorithm 1: Losses in Attribute Training
Input: An instruction and an object in the intruction's 's solution Output: Five losses
begin
// Variables in parenthesea are full names.
1. Ins Attributes AskGGPT-<(nsttrrttoon)
2. Obj Attributes AskGPT-4(Objetts)
GT-IAF (GT Ins Attribute Features)= CLIP Attributes) shape- (ki,
4. GT-OAF (GT Obj Attribute C Attributes) // shape- (122, d)
5. IF (Ins Feature) = CLIP((nstructioon) // shape= (1, d)
6. OF (Ohj Feature) = CL.IP(Ohjects) // shape=(1, d)
IAF (Ins Attrihute Features) Ins MI.P_ MI.P_MMMPPOOi_a Encoder([F) // shape= d)
OAF (Obj Attribute Features)= Obj MLP Encoder(OF) // shape= d)
9. Attribute Loss MEECT-IAA IAF) M.SE(GT-OAE, OAF)
10. Q-IAF (Quantized LAF) = Codebboo((AAF// shape= (k1, d)
11. Q-OAF (Quantized OAF) CodecoookOOAA/// shape- (kz, d)
12. Commitment Loss M E(AF, stop-grradeeni((-AAAF) + MSE(OAF, stop-gradien((Q-OAF))
13. VQ Loss M (Q-IAAF, stop noo-gradient((AFF)) MSE(Q-OAF, MAS(C
stop-gradien((AAF))
14. Recon IF = Ins MLP_ (L___Decoder(AF) // shape- (1, d)
15. Recon OF = Ins MLP Decoder((OAF) // shape=(1, d)
16. Reconstruction Loss MSReeeoon IF, IF) MRReoeoon OF, OF);
17. Matching Loss
(MSE(in IAF, jub OAF))
end
The full loss function is shown below:
Loss = A1 x Altrible Loss + X2 x VQ Loss+
À Commit Loss + X Recon Loss + As X Matching Loss
(2)
where A is 2.0, A2 1.0, A is 0.25, X is 1.0, and A5 is 1.0. Attribute Loss provides a direct loss for directing the MLP Encoder to learn the projections of IF and OF to IAF and OAF The next three items VQ Loss, Commitment Loss and Reconstruction Loss can be referred to VQ-VAE Loss [01]. Our motivation in using these three items is to provide indirect constraints that enable the IF and OF to be projected into the shared codebook's feature space. To provide a direct alignment of the attribute features of given instruction and objects that satisfy the given instruction, we design Matching Loss. An instruction and an object in the instruction's solution theoretically have at least one attribute that matches, i.e., an attribute of the object that necessarily satisfies part of this instruction: otherwise, the object should not be part of the instruction's 's solution. We consider the most similar pair of attribute features among the K IAF and the K2 OAF be a match, and therefore need reduce the error of this pair of attribute features.
In the end, only the instruction and object MLP Encoders and ELIP-Text-Encodee navigation, and other part like codebook and MLP Decoder are not used. Since this codebook is initialized by CLIP features and the ground-truth attribute features are also encoded by CLIP, the attribute feature space we get can actually be seen as subspace of the CLIP semantic space. We hope that this design will improve the generalization of attribute features, since the CLIP semantic
used in
space shows good generalization and performance on many tasks [94 93 102
4.2 Coarse-to-fine Exploration Agent
In this section, we describe how attribute features work in navigation. The agent switches between coarse and fine exploration phases back and forth until the number of Find executions reaches the upper limit find or the number of steps reaches the upper limit nstep. Fig. [ illlustrrats a general overview of the entire navigation policy and Fig. 4 and Fig. Sjsoww the details of the coarse and fine exploration phase, respectively. In both the coarse and fine exploration phase, we load the parameters of Ins MLP Encoder and Obj MLP Encoder from the attribute training for waypoint (i.e., block) selection and end-to-end fine exploration. We argue that this coarse-to-fine design allows for the incorporation of prior knowledge from external foundation large models the coarse exploration phase and task-relevant world world-grounding exploration in the fine exploration phase.
'The abbreviations in this section inherited from the pseudo-code |

---

Environment
Task Input
Besic &Preferred
RGB-D
Object Detection
Build Point Cloud (PC)
Action
Segment into Block
Plan Pall Selected Block
Action
Feature Encoder
Actor
When Arrived at Selected Block, Coarse to Fine
Calculate Select High
Transformer Encoder
LSTM
Block Score
Block
Coarse Exploration Phase
Fine Exploration Phase
When Action Find, Switch Fine to Coarse
Figure 3: Navigation Policy, The agent continuously switches between a coarse exploration phase and fine exploration phase until the Find count limit find is reached the total number of steps Natep is reached. See Sec. I and Sec. 4.2.2 for details about the two phases. In each timestep, the GLEE model is used to identity and label objects in the RGB and project them the point cloud.
Prijedic
Nerg
Clock
llack 1.5cc
Tes Make
CLIN-ES
Tirgit Marks
Inaractio
CLIF-I Tix
Ia MLP
lighe th tested ofses
Coarse Exploration Phase
GFTH
Figure 4: Coarse Exploration. This figure presents the process of building and labeling the point clouds, segmenting the blocks, and calculating the scores for each block.
4.2.1 Coarse Exploration Phase
In this phase (see Fig. 4) the agent receives the RGB-D input, pose, and the demand instruction. We then use camera parameters and depth map to compute partial point clouds of the current observation and merge them with the previously observed point clouds. We use object detection model, GLEE, to detect objects in the RGB image and project them into the depth map, labeling and square, where is 2 in experiments. Thus, each detected object belongs to block (according to the center of the object's point clouds). We use the instruction attribute features to query objects each attribute block, and each block will then get score. We use two different ways to generate instruction attribute features: the bottom in Fig. For the LLM branch in Fig. 4 we use GPT-4 to generate language-Iewe basic and preferred attributes separately and use CLIP-Texx-Encoder to obtain basic and preferred attribute features. These two attribute features can be used to calculate two scores, basic and preferred scores. Adjusting the weights of these scores control whether prioritize the search for basic preferred solutions. The formula for calculating the score (i.e., the process of query) is as follows:
the point clouds. We segment the point clouds into many rectangular blocks according y coordinates, and each block
the LLM branch and MLP branch at
Σ (Tp oEblock
max i= L.....kj=- =1:j=1..kk Fpref ins * ff{ + 2.1..ky:3 1..kg
max
fhasic f!)
(3)
Where f3 denote jeh attribute features of the object flasic denote ith basic attribute features of the instruction (so do f fpref_ime). ). denotes cosine similarity, and and rp adjustable weights for whether to find basic or preferred solutions. If deployed in real environment, these two weights
7

---

can be freely adjusted by the user to apply to different situations. We conduct an ablation study on these two weights and discuss in detail how they affect the basic and preferred solutions in the experimental section. For the MLP branch, we use the Ins MLP Encoder from attribute training to map the instruction features into 1 attribute features, and we use a similar query process to compute scores. MLP branch is lightweight alternative that neither requires remote LLM nor consumes computational resources to run local LLMs. If deployed in a real environment, the agent is free to choose one of the branches according to the current LLM availability. We report the results of the two branches separately the experimental section. Finally, we randomly choose point in the highest-scorie scoring and never-visited block as the waypoint. Then, the agent navigates to the point by path-plan paah-plann algorithm. Please see supplementary material for details about the coarse exploration module, the path-plane planninn algorithm and blocks' score visualizations A.3.1
4.2.2 Fine Exploration Phase
Actor
IATH Ohapat
Tikes
Compan
GPf
Actis
CES
Exeder
Deyth Fosterr
Dipit toueee
Transfrmerr Escoder
Oey MUP
Fundeer
Atrilah Funtes
2 dms
Attibaas Frstani
Instruction
Amiikass Fratun X
Aftibara Forters
Fine Exploration Phase
Figure 5: Fine Exploration. We employ imitation learning to train an end-to-end module in this phase. This module loads the Ins MLP Encoder and Obj MLP Encoder's parameters as initialization from attribute training, along with Transformer Encoder to integrate features. The output feature corresponding the CLS token is combined with GPS+Compass features and previous action embedding and passed through an LSTM to generate actions by an actor.
In this phase, we train an end-to-end module using imitation learning similar to DDN. The alignment capabilities of CLIP in the visual and textual domains allow Obj MLP Encoder still extract object attribute features through CLIP- -ocuaerrs 's object features. The self-attention mechanism in the Transformer Encoder learns the association between the attribute features of the objects in the current field of view and the attribute features of the instructions, which can implicitly determine whether these objects need to reported. Following BERT [[X]] and ViT [104, we add a CLS token as the output of the feature fusion. For more training and hyperparameter details, the supplementary material A.3.2
Experiment
5.1 Experimental Settings
We use habitat-sim and habitat-lab as our simulator and HSSD as our scene dataset. We randomly select 30 tasks as testing tasks and the remaining 270 tasks as training tasks(i.e. unseen task and seen task in Tab. [5.3] respectively). These training tasks are used to collect trajectories train the fine exploration module, VTN and ZSON. HSSD splits the scenes into val scenes and train scenes (i.e., unseen scenes and seen scenes in Tab. 5.3 respectively). In all experimental settings, the judgment distance of the found list dnf is one meter, and a maximum number of Find finind is five times, and the maximum step number Nstep 300. single RTX 4090 enough to run the experiments. More experimental settings available in the supplementary material A.4

---

Table I: Baseline Comparison, Values in parentheses represent standard deviations. represents the usage of ground truth semantic labels in the RGB image. The bold fonts represent optimal values.
Scen Somn
37%
SRg
52"
S2'p
376
SR,
5P2
572
Kundoon
(0,36))))
(.3070
4.10(0.45)
2,92 (0.04)
3.47
2.40
(0.26
(0.25
(3.0)
(..92))02)
4.19.0.16)
(..51
(.17))
(1.11.15)
(8.59)
5411 (0.99)
ZSON
6.751261)
3.79:0.82)
4.27 (1.5)
(082)
(.36(126))
308 (0.75)
(12(1
2.15(0.65)
DDN
8.1012.20)
6.62 (0.94)
5.95 (2.09)
(1.20)
(244
4.52
4.67
(0.96
MOP%+LLM
(3.75)
(1.99
3,00
0.38
(1.671444)
10.2
200 (0.13)
0.350.11)
FLE
13.25 (0.35)
9.59 (2.3)
3.9 (0.71)
3.24 (0.96)
10.00 (1 501
8.0 (2:23)
3.37 (0.54)
2,99 (0.73)
C2FAgon haunch
58
5.55
13.91
(0.87)
C2FAgen (LLM brunch)
(3899
(3.72
(1.30)
5.80 (1.0)
(1.93
8.63 (0.7))
621 (1.18
(..4
Unseen Soene
Method
Seen Task
Unseen Task
574
SR,
8P1
SPL,
SN.
SPLa
SPL.
Random
6.0000.67)
4.4770.47)
6,02 (1.11)
(0:400
4.63
(0.5))
346 (4600.40
4.18 (0.50)
3.18:0:425
VIN
3.01 (4.25)
(3.24
(1.96
3.64 (1.86)
6.33 (100)
(.751.35)
3:48 (0.15)
1,97 (0.50)
ZSON
(39(1.50)
5.0Y(1.71)
(.0291.19)
3,461199
(1.91
70.7
204
270
DUN
(2.00
5.50 (0.86)
7,89 (1.93)
4,86 (0.52)
(2.55
(170(1.1)
5337 (2,46)
(.44(
14.17 (941)
(0.53)
10.39
6.72
(0.23)
0.78
M
140074.77
8.0512.23)
(..66
2,99 (0.73)
9.67 (3.53)
(3,00
1.78 (0.58)
2.12 (0.68)
C27/gent IMLF brunch)
(2.77
(2.89)
5,49 (1.23)
3,77 (0.70)
12.17 (3.7))
6.92 (1.63)
3.51 (1.05)
2,46 (0.39)
C2PAgent (LLM branch)
(1.58)
15.98 (0.87)
(.32(
15.88 (2.63)
(1.30)
6.03 (0.77)
4,50
5.2 Baselines
an end-to-end closed-vocabulary ObjectGool Navigation method. ZSON (21 is an end-to-end open-vocabulary ObjectGioll Navigation method. MOPA+LLM [3) is modular-based based Multi-object Navigation method using LLM to select target objects. FBE+LLM [59] is a modular-based Multi-object Navigation method that uses frontier-based exploration (FBE) and LLM find objects. DDN (14) an end-to-end Single- object Demand-driven Navigation method. For more details about baselines in training and method modifications, please see supplementary material A.4.2
Random is method for randomly selecting actions. VTN [15
5.3 Baseline Comparison
The experimental results are in Tab. 5.3 Random reflects the difficulty of MO-DDN. The two end-to-end ON methods, VTN and ZSON, perform poorly, only slightly higher than Random. In comparison, though DDN uses attribute features designed for single object-instruction pairs. DDN's attribute features still provide some prior knowledge, outperforming VTN and ZSON. FBE+LLM and MOPA+LLM use different exploration strategies, but both use LLM to select waypoints. Both of them acquire the ground truth semantic label in the current RGB image when asking LLM whether or not to perform Find5) and they outperform the three end-to-end methods. MOPA+LLM obtains suboptimal results in many cases, better than FBE+LLM, which may stem from the fact 4 exploration strategy is superior to FBE, allowing more objects to be detected.
that its
Our method outperforms the baseline in the vast majority of settings. Attribute similarity scores good substitute for L.M for waypoint selection. Compared to MOPA+LLM, which requires 5.15 requests to the LLM per episode average, C2FAgent (LLM branch) only needs request the LLM to parse instructions into attributes once at the beginning and achieves better performance. Moreover, C2FAgent (MLP branch) does not need LLM parsing and gets results comparable MOPA+LLM. We also find that when no object meets the demand in the current detected object list, each block has low score, approximating randomly selected block to explore. In contrast, when there exists an object that meets the demand, the corresponding block has significantly higher score than the other blocks. This indicates that the attribute model can balance exploration and exploitation on its own.
5.4
Ablation Study
In this section, we would like to discuss the following four questions:
Q1: Is selecting waypoints by attribute feature similarity scores better than FBE, LLM and
CLIP features' similarity scores?
$ 'when other actions are performed, the ground truth semantic label will not be provided, and only the GLEE 31 detection will be provided.

---

Table 2: Ablation on Coarse Exploration (Q1)
Table 3: Ablation on Fine Exploration (Q2)
15%
S1p
SPLs
SR.
SPL,
CoomsnFine
Curmelitee
FBE
14.5655877 75.77
11,36 45.5
454
Couse 255N
0.74
02.0
(20)
1399449
947055
512
0.73
(000)
ConsevVI
16897430
1.60..10
CLIFitite
1231033)
7450220
452
313 (0:3))
ConterRandoon
3221222
3271212
Table 4: Ablation on Attribute Training (Q3)
Table 5: Ablation on Score Weights (Q4)
Method
5R
88.
S.P.t
S.L.
85%
8P'
2340
14.2 (372)
6,00.1.000
154551.58
7.91 (1.5g
Utn VIE
Oan cookbookk init
17.56
12,02 (2.54)
5.23 (1.14)
4.56 4.56.1..8)
28,10
(1
9.26 (2,17)
5.34
Q2: Do attribute features also work in the end-to-end fine exploration modules? How about replacing the fine exploration module with VTN and ZSON?
Q3: Do VQ-VAF losses and codebook initialization contribute to experimental results?
. Q4: Can adjusting the weights of basic and preferred scores affect agent behavior?
In the ablation study, we report the results in the seen tasks and seen scenes. Ours refers to C2FAgent (LLM branch). The experimental results for all four questions are in Tab. 5.4 For specific experimen- tal setups, please see supplementary materials A.4.3
For Q1, the experimental results demonstrate that attribute- -aaed coarse exploration outperforms rule- based FBE, commonsense-baxse LLM and CLIP-based exploration. For Q2, the experimental results show that the fine exploration module utilizes the prior in the attribute model well, outperforming the VTN, which has larger model parameters, and the ZSON, which has been pre-trained on 36M total episodes and fine-tuned on the same trajectory dataset with Ours. In addition, we note that Coarse+VTN exceeds VTN, suggesting that the coarse exploration module can steer the agent to the region where it is more likely to find objects that satisfy the demand. For Q3, find that the performance decreases after removing the VQ-VAE Loss or codebook initialization. Since the attribute model itself is trained on language-grounas evaluated, we can argue that the VQ-VAE Loss and the initialization contribute to the generalizability of attribute features. For Q4, adjusting the weights of two scores indeed affect the agent's behavior. By tuning up Tp. R and SP.,, increase, while SR, and SPL, decrease; and vice versa. This characteristic allows the user to freely decide whether to prioritize the search for the basic or preferred solution in the current situation. For example, when a user who likes Coke is very thirsty, he can increase ri allow the agent to satisfy the basic demand with higher success rate, while in the general case, Tp can be increased to allow the agent to try to satisfy the preferred demand.
tasks and thus agnostic to the task being
6 Conclusion and Discussion
In this paper. we propose a new benchmark, MO-DDN, which can be regarded as a multi-object version of DDN. Moreover, MO-DDN can be considered a crucial preliminary step in task planning. We also propose a coarse-to-fin attribute-based exploration agent, extending the concept of "attribute" " to a muuli--ojeect setting. The agent uses attribute features in different exploration phases. The experimental results show that our method outperforms the baselines, and the attribute features improve exploration efficiency. The ablation study also demonstrated the effectiveness of our method.
Limitations and Broader Societal Impacts In this paper, we assume that the number of attributes of both instructions and objects is fixed values (i.e., k1 and kg). Though this facilitates training and its experimental results outperform baselines, there are still some gaps with real life. Future work could consider more flexible attribute features. Please see A.3 for more limitation discussion. To the best of our knowledge, there are no observable adverse effects on society.
Acknowledgments and Disclosure of Funding
This work was supported by The National Youth Talent Support Program (8200800081), National Natural Science Foundation of China (No. 62376006) and National Natural Science Foundation of China (No. 62136001).
10

---

References
[1] R. J. Taormina and J. H. Gao, "Maslow and the motivation hierarchy: Measuring satisfaction of the needs," The American journal of psychology, vol. 126, no. 2, pp. 177-177, 2013.
[2] J. E. Gawel, "Herzberg's ' theory of motivation and maslow's ' hierarchy of needs," Practical Assessment, Research, and Evaluation, vol. 5, no. L, p. 2019.
[3] Maslow, Motivation And Personality: Motivation And Personality: Unlocking Your Inner Drive and Understanding Human Behavior by A. H. Maslow. Prabhat Prakashan, 1981. [Online]. Available: https://boook.ggogge wwt.ggll.coooooook/.oooo wwt.:boooos.idd-DoaDbDAAAAA
[4] A. H. Maslow, "A dynamic theory of human motivation." 1958.
[5] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakr- ishnan, K. Hausman el al., "Do as can, not as say: Grounding language in robotic affor- dances," arXiv preprint ar ar.iv:2204.01669
2022.
[6] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng, "Code policies: Language model programs for embodied control," in 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023, pp. 9493-9500.
[7] W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar el al., "Inner monologue: Embodied reasoning through planning with language models," arXiv preprint 2XXi::207.006005 2022.
[8] Brohan, N. Brown, Carbajal, Y. Chebotar, J. Dabis, Finn, Gopalakrishnan. K. Haus- man, A. Herzog, J. Hsu et al., "Rt-1: Robotics transformer for real-world control at scale," arXiv preprint arXiv:2212.005
2022.
[9] D. Driess, F. Xia, M. S. Sajjadi, C. Lynch. A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, Yu al., "Palm-e: An embodied multimodal language model," arXiv preprint 0rXiv:2303.03372 2023.
[10] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn el al., "Rt-2: Vision-languuag--cciionn models transfer web knowledge to robotic control," arXiv preprint arXiv:2307. /5818, 2023.
[11] W. Huang, C. Wang, R. Zhang, Y. Li, J. Wu, and L. Fei-Fei, "Voxposer: Composable 3d value maps for robotic manipulation with language models." arXiv preprint arXiv:2307.059733 2023.
[12] Wu and Zhao, "Vat-mart: Learning visual action trajectory proposals for manipulating 3d articulated objects," in International Conference on Learning Representations (ICIR), 2022, 2022.
[13] G. Zhou, Y. Hong, and Q. Wu, "Navgpt: Explicit reasoning in vision-ane and and-language navigation with large language models," " in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 7, 2024, pp. 7641-7649.
[14] H. Wang, A. G. H. Chen, X. Li, M. Wu, and H. Dong, "Find what you want: Learning object attribute space for demand -diven navigation," Advances in Neural Information Processing Systems, vol. 36, 2024.
[15] H. Du, X. Yu, and I.. Zheng, "Vtnet: Visual transformer network for object goal navigation," arXiv preprint
2021.
[16] W. Cai, S. Huang, G. Cheng, Y. Long, P. Gao, C. Sun, and H. Dong, "Bridging zero-shot object navigation and foundation models through pixel-guided navigation skill." arXiv preprint arXiv:2309. 10309, 2023.
[17] D. S. Chaplot, D. P. Gandhi, A. Gupta, and R. R. Salakhutdinov, "Object goal navigation using goal-oriented semantic exploration," Advances in Neural Information Processing Systems, vol. 33, pp. 4247-4258, 2020.
11

---

[18] F. Zhu, X. Liang, Y. Zhu, Q. Yu, X. Chang, and X. Liang, "Soon: Scenario oriented object navigation with graph-based exploration," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 12 689-126699.
[19] R. Fukushima, K. Ota, A. Kanezaki, Y. Sasaki, and Y. Yoshiyasu, "Object memory transformer for object goal navigation." " in 2022 International Conference on Robotics and Automation (ICRA). IEEE, 2022, pp. 11 288-11 11 294.
[20] K. Zhou, K. Zheng, C. Pryor, Y. Shen, H. Jin, L. Getoor, and X. E. Wang, "Esc: Explo- ration with soft commonsense constraints for zero-shot object navigation," in International Conference on Machine Learning. PMLR, 2023, pp. 42 829 42 842.
[21] A. Majumdar, G. Aggarwal, B. Devnani, J. Hoffman, and D. Batra, "Zson: Zero-shot object- goal navigation using multimodal goal embeddings." Advances in Neural Information Process- ing Systems, vol. 35, pp. 32 340-3233333 352, 2022.
[22] C. Li, R. Zhang. J. Wong, C. Gokmen, S. Srivastava, Martin-Martin, C. Wang, G. Levine, M. Lingelbach, J. Sun et al., "Behavior- lk: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation," " in Conference on Robot Learning. PMLR, 2023, pp. 80-93.
[23] P. Marza, L. Matignon, O. Simonin, and C. Wolf, "Multi-object navigation with dynami- cally learned neural implicit representations," " in Proceedings of the IEEF/CVF International Conference on Computer Vision, 2023, pp. 11 104-11015.
[24] S. Wani, S. Patel, U. Jain, A. Chang, and M. Savva, "Multion: Benchmarking semantic map memory using multi-object navigation," Advances in Neural Information Processing Systems, vol. 33, pp. 9700-9712, 12, 2020.
[25] P. Chen, D. Ji, K. Lin, W. Hu, W. Huang, T. Li, Tan, and C. Gan, "Learning active camera for multi-object navigation," Advances in Neural Information Processing Systems, vol. 35, pp. 28 670-28 682, 2022.
[26] X. Liu, H. Palacios, and C. Muise, "Egocentric planning for scalable embodied task achieve- ment," Advances in Neural Information Processing Systems, vol. 36, 2024.
[27] B. Y. Lin, C. Huang, Q. Liu, W. Gu, S. Sommerer, and X. Ren, "On grounded planning for embodied tasks with language models," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 11, 2023, pp. 13 192 13 200.
[28] Z. Wu, Z. Wang, X. Xu, J. Lu, and H. Yan, "Embodied task planning with large language models," arXiv preprint arXiv:2307.01888. 2023.
[29] J. Mendez-Mendez, L.. P. Kaelbling, and T. Lozano-Pérez, "Embodied lifelong learning for task and motion planning." in Conference on Robot Learning. PMLR, 2023, pp. 2134-2150.
[30] T. Cheng, L. Song, Y. Ge, W. Liu, X. Wang. and Y. Shan, "Yolo-world: Real-time open- vocabulary object detection," in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2024.
[31] J. Wu, Y. Jiang, Q. Liu, Z. Yuan, X. Bai, and S. Bai, "General object foundation model for images and videos at scale," 2024.
[32] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Milli- can, M. Reynolds al., "Flamingo: a visual language model for few-shot learning." Advances in neural information processing systems, vol. 35, pp. 23716-33 736, 2022.
[33] Z. Yang, L. Li. K. Lin, J. Wang, C.-C. Lin, Z. Liu, and L. Wang. "The dawn of Imms: Preliminary explorations with gpt-4y (ision)," arXiv preprint arXiv:2309. /7421, vol. 9, no. 1, p. 2023.
[34] J. Chen, D. Zhu, X. Shen, X. Li, Z. Liu, P. Zhang, R. Krishnamoorthi, V. Chandra, Y. Xiong, and M. Elhoseiny, "Minigpt-v2: large language model as unified interface for vision-language multi-task learning." arXiv preprint arXiv:2310.09477 2023.
12

---

[35] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, "Minigpt-4: Enhancing vision- language understanding with advanced large language models," arXiv preprint 2Xi1:23344.10500 2023.
[36] W. Wang, Z. Chen, X. Chen, J. Wu, X. Zhu, G. Zeng, P. Luo, T. Lu, J. Zhou, Y. Qiao et al., "Visionllm: Large language model is also an open-ended decoder for vision-centric tasks," Advances in Neural Information Processing Systems, vol. 36, 2024.
[37] W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang. Y. Hou. Y. Min, Zhang, J. Zhang. Z. Dong et al., "A survey of large language models," arXiv preprint arXiv:2303. 2023.
[38]
Chowdhery, S. Narang, J. Devlin. M. Bosma, G. Mishra, A. Roberts. P. Barham, H. W. Chung, C. Sutton, S. Gehrmann et al., "Palm: Scaling language modeling with pathways," Journal of Machine Learning Research, vol. 24, no. 240, pp. 1-113, 2023.
[39] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., "Language models are few-shot learners," Advances in neural information processing systems, vol. 33, pp. 1877-1901, 2020.
[40] H. Touvron, L. Martin, K. Stone, Albert, A. Almahairi, Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al., "I.lam 2: Open foundation and fine-tuned chat models," arXiv preprint arXiv:2307. 09288, 2023.
[41] Y. Shen, K. Song, X. Tan, D. Li, W. Lu, and Y. Zhuang, "Huggingept: Solving ai tasks with chatgpt and its friends hugging face," Advances in Neural Information Processing Systems, vol. 36, 2024.
[42] Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth et al., "Gemini: family of highly capable multimodal models," arXiv preprint arXiv:2312. 1805, 2023.
[43] S. Raychaudhuri, T. Campari, U. Jain, M. Savva, and A. X. Chang, "Mopa: Modular object navigation with pointgoal agents," in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2024, pp. 5763-5773.
[44] S. Gupta, J. Davidson, S. Levine, R. Sukthankar, and Malik, "Cognitive mapping and planning for visual navigation," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 2616-2625
[45] H. Luo, A. Yue, Z.-W. Hong, and P. Agrawal, "Stubborn: A strong baseline for indoor object navigation," " in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2022, pp. 3287-3293.
[46] S. Rudra, Goel, A. Santara, C. Gentile, L. Perron, Xia, V. Sindhwani, C. Parada, and G. Ag- garwal, "A contextual bandit approach for learning to plan in environments with probabilistic goal configurations," " 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023, pp. 5645-5652.
[47] Y. Liang. B. Chen, and S. Song, "Ssenav: Confidence-awand semantic scene completion for visual semantic navigation," in 2021 IEEE international conference robotics and automation (ICRA). IEEE, 2021, pp. 13 194-13 200.
[48] G. Georgakis, B. Bucher, K. Schmeckpeper, S. Singh, and K. Daniilidis, "Learning to map for active semantic goal navigation," arXiv preprint arXiv:2106. 15648, 2021.
[49] S. Y. Min, Y.-H. H. Tsai, W. Ding, A. Farhadi, R. Salakhutdinov, Y. Bisk, and J. Zhang, "Self- supervised object goal navigation with in-situ finetuning." " 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEF, 2023, pp. 7119-7126.
[50] Kumar, N. S. Shankar, H. Didwania, R. D. Roychoudhury, B. Bhowmick, and M. Krishna, "Geexp: Goal-conditioned exploration for object goal navigation," in 2021 30th IEEE International Conference on Robot Human Interactive Communication (RO-MAN). IEEE, 2021. pp. 123-130.
13

---

[51] W. B. Shen, D. Xu, Y. Zhu, L. J. Guibas, L. Fei-Fei, and S. Savarese. "Situational fusion of visual representation for visual navigation," in Proceedings of the IEEE/CVF international conference computer vision, 2019, pp. 2881-2890.
[52] A. Mousavian, A. Toshev, M. Fišer, J. Košecká, A. Wahid, and J. Davidson, "Visual represen- tations for semantic target driven navigation." " in 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019, pp. 8846-888 8846-8852.
[53] R. Druon, Y. Yoshiyasu, A. Kanezaki, and A. Watt, "Visual object search by learning spatial context," IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 1279-1286 2020.
[54] W. Yang, X. Wang, A. Farhadi, A. Gupta, and R. Mottaghi, "Visual semantic navigation using scene priors," " arXiv preprint arXiv: 181..06610005544 2018.
[55] A. Pal, Y. Qiu, and H. Christensen, "Learning hierarchical relationships for object-goal navigation," in Conference on Robot Learning. PMLR, 2021, pp. 517-528.
[56] B. Mayo, T. Hazan, and A. Tal, "Visual navigation with spatial attention," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 16 16888-16 907.
[57] R. Dang, Z. Shi, L. Wang. Z. He, C. Liu, and Q. Chen, "Unbiased directed object attention graph for object navigation." " in Proceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 3617-3627.
[58] J. Ye, D. Batra, A. Das, and E. Wijmans, "Auxiliary tasks and exploration enable objectgoal navigation," " in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 16 117-16 126.
[59] B. Yamauchi, "A frontier-based approach for autonomous exploration." in Proceedings 1997 IEEE International Symposium on Computational Intelligence in Robotics and Automation CIRA "9o.'Towards New Computational Principles for Robotics and Automation'. IEEE, 1997, pp. 146-151.
[60] P. Anderson, A. Chang, D. S. Chaplot, A. Dosovitskiy, S. Gupta, V. Koltun, Kosecka, J. Malik, R. Mottaghi, M. Savva et al., "On evaluation of embodied navigation agents." arXiv preprint arXiv: 807706777777. 2018.
[61] P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. Reid, S. Gould, and A. Van Den Hengel, Vision-and-languaal navigation: Interpreting visually- grounded navigation instructions in real environments," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 3674-3683.
[62] F. Zhu, Y. Zhu, X. Chang, and X. Liang, "Vision-language navigation with self-supervised auxiliary reasoning tasks," in Proceedings of the IEEE/CVF conference computer vision and pattern recognition, 2020, pp. 10012-10022.
[63] H. Wang, W. Wang, W. Liang, C. Xiong, and Shen, "Structured scene memory for vision- language navigation," in Proceedings of the IEEF/CVF conference on Computer Vision and Pattern Recognition, 2021, pp. 8455 8464.
[64] A. Majumdar, A. Shrivastava, Lee, P. Anderson, D. Parikh, and D. Batra, "Improving vision- and-language navigation with image-text pairs from the web," in Computer Vision-ECCo 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VI 16. Springer. 2020, pp. 259-274.
[65] X. Wang, Q. Huang, A. Celikyilmaz, J. Gao, D. Shen, Y.F. Wang. W. Y. Wang, and L. Zhang, "Vision-language navigation policy learning and adaptation." IEEE transactions on patlern analysis and machine intelligence, vol. 43, no. 12, pp. 4205-4216, 2020.
[66] Y. Long, X. Li, W. Cai, and H. Dong. "Discuss before moving: Visual language navigation via multi-expert discussions," arXiv preprint arXiv:2309./1382 2023.
[67] Y. Hong, Q. Wu, Y. Qi, C. Rodriguez-Opazo, and S. Gould, "Vin bert: A recurrent vision- and-language hert for navigation," " in Proceedings of the IEEF/CVF conference on Computer Vision and Pattern Recognition, 2021, pp. 1643-1653.
14

---

[68] D. Shah, B. Osiński, S. Levine et al., "Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action." Conference on robot learning. PMLR, 2023, pp. 492-504.
[69] C. Chen, Z. Al-Halah, and K. Grauman, "Semantic audio-visual navigation." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 516 525.
[70] C. Chen, U. Jain, C. Schissler, S. V. A. Gari, Z. Al-Halah, V. K. Ithapu, P. Robinson, and K. Grauman, "'oundspaces: Audio-visual navigation in 3d environments, in Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VI 16. Springer, 2020, pp. 17-36.
[71] C. Chen, S. Majumder, Z. Al-Halah, R. Gao, S. K. Ramakrishnan, and K. Grauman, "Learning to set waypoints for audio-visual navigation." arXiv preprint arXiv:2008. 09622, 2020.
[72] H. Wang, Y. Wang, F. Zhong, M. Wu, J. Zhang. Y. Wang, and H. Dong, "Learning semantic- agnostic and spatial-aware representation for generalizable visual-audio navigation," IEEE Robotics and Automation Letters, 2023.
[73] Y. Yu, W. Huang, F. Sun, C. Chen, Y. Wang, and X. Liu, "Sound adversarial audio-visual navigation," arXiv preprint 200i:22002.10100 2022.
[74] A. Younes, D. Honerkamp. T. Welschehold, and A. Valada, "Catch me if you hear me: Audio- visual navigation in complex unmapped environments with moving sounds," IEEE Robotics and Automation Letters, vol. 8, no. 2, pp. 928 935, 2023.
[75] M. Hussain, "Yolo-vl to yolo-v8, the rise of yolo and its complementary nature toward digital manufacturing and industrial defect detection." Machines, vol. 11, no. 7, p. 677. 2023.
[76] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, "End-to-end object detection with transformers," in European conference computer vision. Springer, 2020, pp. 213-229.
[77] Kirillov, F. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo et al., "Segment anything," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 4015-4026.
[78] L. Ke, M. Ye, M. Danelljan, Y.-W. Tai, C.-K. Tang, F. Yu et al., "Segment anything in high quality," Advances in Neural Information Processing Systems, vol. 36, 2024.
[79] S. Ren, F. Luzi, S. Lahrichi, K. Kassaw, L. M. Collins, K. Bradbury, and J. M. Malof, "Segment anything, from space?" in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2024, pp. 8355-8365.
[80] P. O. O Pinheiro, R. Collobert, and P. Dollár, "Learning to segment object candidates," Ad- vances in neural information processing systems, vol. 28, 2015.
[81] J. Kim, E. S. Lee, M. Lee, D. Zhang, and Y. M. Kim, "Sgolam: Simultaneous goal localization and mapping for multi-object goal navigation," arXiv preprint arXiv:2110.07177 2021.
[82] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., "Gpt-4 technical report," arXiv preprint arXiv:2303.08774 2023.
[83] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar et al., "Llama: Open and efficient foundation language models." arXiv preprint 23X2::3302.1977 2023.
[84] G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. S. Kale, J. Love et al., "Gemma: Open models based on gemini research and technology." arXiv preprint arXiv:2403.08299 2024.
[85] C. H. Song, J. Wu, C. Washington, B. M. Sadler, W.-L.. Chao, and Y. Su, "LIm-planner: Few-shot grounded planning for embodied agents with large language models," 2023.
15

---

[86] Z. Zhao, W. S. Lee, and D. Hsu, "Large language models commonsense knowledge for large-scale task planning." Advances Neural Information Processing Systems, vol. 36, 2024.
[87] Z. Shao, Z. Yu, M. Wang, and Yu, "Prompting large language models with answer heuristics for knowledge-basee visual question answering, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. pp. 14974-14988 183..
[88] S. Yu, J. Cho, P. Yadav, and M. Bansal, "Self-chained image-language model for video localization and question answering," Advances in Neural Information Processing Systems, vol. 36, 2024.
[89] J. Guo, Li, D. Li, A. M. H. Tiong, B. Li, D. Tao, and S. Hoi, "From images textual prompts: Zero-shot visual question answering with frozen large language models," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 10 8777
[90] J. Cui, Z. Li, Y. Yan, B. Chen, and L. Yuan, "Chatlaw: Open-source legal large language model with integrated external knowledge bases," " arXiv preprint arXiv:2306. 16092, 2023.
[91] Y. Zhuang, Y. Yu, K. Wang, H. Sun, and C. Zhang, "Toolqa: A dataset for I1m question answering with external tools," Advances in Neural Information Processing Systems, vol. 36, 2024.
[92] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., "Learning transferable visual models from natural language supervi- sion," in International conference on machine learning. PMLR, 2021, pp. 8748-8763,
[93] V. S. Dorbala, G. Sigurdsson, R. Piramuthu, J. Thoomaoon, and G. S. Sukhatme, "Clip-nav: Using clip for zero-shot lan-lan navigation." arXiv preprint arXiv:2211.16649 2022.
[94] A. Khandelwal, L. Weihs, R. Mottaghi, and A. Kembhavi, "Simple but effective: Clip embed- dings for embodied ai," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 14 849-14 838.
[95] P. Kinghorn, L. Zhang, and L. Shao, "A region-based image caption generator with refined descriptions," Neurocomputing, vol. 272, pp. 416 424, 2018.
[96] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, "Show and tell: A neural image caption generator," in Proceedings of the IEEE conference on computer vision and pattern recognition. 2015, pp. 3156-3164.
[97] X. Chen and C. Lawrence Zitnick, "Mind's eye: A recurrent visual representation for image caption generation." in Proceedings the IEEE conference on computer vision and pattern recognition, 2015, pp. 2422-24311
[98] T. Wang, J. Zhang, J. Fei, Y. Ge, H. Zheng, Y. Tang, Z. Li, M. Gao, S. Zhao, Y. Shan et al., "Caption anything: Interactive image description with diverse multimodal controls," arXiv preprint 2r05v 026777 2023.
[99] M. Khanna*, Y. Mao H. Jiang, S. Haresh, B. Shacklett, D. Batra, A. Clegg, E. Undersander, A. X. Chang, and M. Savva, "Habitat Synthetic Scenes Dataset (HSSD-200); An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigation," " arXiv preprint, 2023.
[100] S. Lloyd, "Least squares quantization in pem," IEEE transactions on information theory, vol. 28, no. 2, pp. 129-137, 1982.
[101] A. Van Den Oord, O. Vinyals et al., "Neural discrete representation learning." Advances in neural information processing systems, vol. 30, 2017.
[102] S. Y. Gadre, M. Wortsman, G. Ilharco, L. Schmidt, and S. Song, "Cows on pasture: Baselines and benchmarks for language-driven zero-shot object navigation." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 23 171-23 181.
[103] J. Devlin, "Bert: Pre-training of deep bidirectional transformers for language understanding." arXiv preprint arXiw:1810.04805, 2018
16

---

[104] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. De- hgghnn, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, "An image is worth 16x 16 words: Transformers for image recognition at scale," ICLR, 2021.
17

---

A
Appendix supplemental material
Multi-object Demand-driven Navigation Task A.1
Task Dataset Generation A.I.
Task Metrics A.1.2
Task Dataset Statistics A.1.3
Attribute Feature Training A.2
-Training Data Preparation A.2.1
AAttribute Training Details A.2.2
Details about Coarse-to-Fine Exploration A.3
Design Details and Visualizations for Coarse Exploration Module A.3.1
Training Details about Fine Exploration Module A.3.2
Experiments A.4
Details about Experimental Settings A.4.1
Details about Baselines AA.42
Details about Ablation Study A.4.3
Details about LLM's Prompt in Experiments A.4.4
More Limitations
A.1
Multi-object Demand-driven Navigation Task
A.1.1
Task Dataset Generation
We base our instruction generation process on the 466 object categories in the HSSD dataset 199] In short, we perform three steps while generating the task: LLM pre- pre-generation, LLM revision, and manual review (we use 111-4-0125-previee API in this paper). In the first step, we prompt the LLM with task definition, available object category list, task format, and generation guidelines to generate the raw tasks. In the second step, we prompt the LLM with previous raw tasks one by one. We let the LLM check whether the input raw task given object category list, and whether the demand instructions and the solutions correspond to each other, and let the LLM state the reason for the revision before providing the result of the revision. In the third step, manually check the generated tasks, removing overly far-fetched tasks and adding more obvious solutions.
the task format, whether the objects involved are the
Here are some examples. For each example, we only show part of the solution.
task instruction:
I need to display my photography collection, preferably with good lighting. basic solution: [picture frame, bookshelf]
preferred solution:
[picture frame, bookshelf, table lamp], [picture frame, bookshelf, ceiling lamp]
task instruction: I need sleeping arrangement for guest staying comfortable enough.
night; hope the bed is
basic solution: [single bed, blanket, pillow], [sofa, blanket]
preferred solution: [king bed, blanket, pillow]. [double bed, blanket, pillow]
task instruction: Train my small dog with water and treats, preferably it is for pet only. basic solution: [mixing bowl], [pet bowl], [bowl]
preferred solution: [pet bowl]
task instruction: I need to work writing project but prefer quiet and comfortable space. basic solution: [desk, swivel chair, notebook], [desk, straight chair, laptop] preferred solution: [desk, swivel chair, notebook, carphone, room divider]
We the following prompts to generate the raw tasks:
System Prompt: You are an Al assistant that can understand human demands and imagine what human demands can be met with existing object categories.
18

---

Prompt:
#i# Task Generation: Demand- Driven Navigation #i##
** **Objective:*** Create navigation task where an agent must locate an object within a specific category that satisfies given demands and preferences.
** **Category***
bed, sofa, cup, desktop....
*asa Requirements:**
** Basic Demand:** Describe the fundamental requirement for the object.
** Preference:** ** Detail any additional preferences that refine the object selection.
###Demand-drine Navigation Task Templat####
"task_ine" sass_instruction.. $basic_demanne demandS, Spreference
"basic_de fesiid_demaddd_eeee " "String"
"preferred_ee leriddddemmdddddeeee iinranddtnnttution". "String"
"basic_solution" [[object_a object_e], [object_bl]]
cefferred_solutioon [[object_a, object_c, object_d], [object_b, object_[]]
This should be in dict format.
basic cidic_sslutioot is list whose elements are lists, and each element represents solution that meets the basic aesic_demand instruction; each solution may consist of one or more objects.
preferred solution is list whose elements lists, and each element represents solution that meets both the basic_ ficiiddemadd_deeee innttructionnnn and preference; each solution may consist of one or more objects. "[object_gi" represents that just object_ can meet the demand. "[object_x, object_y, object_z] represents that only the combination of object y, and z can meet the demand.
### Example ### $Example Tasks$
###Additional Guideline ###
** **Avoid demands related to needing a specific place or location,
**Focus generating novel demands, possibly in the realm of entertainment, that don't require specific place.** **
** **Finalize the task using the provided template, ensuring it is concise and formatted correctly. * ** **eheeverr possible, generate task_ iss__instucttio that requires multiple combinations of objects to be met, such [object_a, object_e, object_d] and [object_b, object_e, object_e].**
### Previously Generated Task Instructions: ### SPrevious Task Instruction Example$
#### Process ####
Determine "basic demand instruction" and "preferred demand instruction" based on the object category.
Sequentially consider how each object or combination thereof meets the demands.
Finalize your response in the provided dict format, ensuring logical consistency between "ba- sic_ solution" and "preferred_soluttion solution."
Make the response more concise and clear and can be executed (tsssk". Please sequentially generate ten tasks split by different from the previously generated task instructions.
and make sure the task totally
We use the following prompts to revise the raw tasks:
System Prompt: You are an AI assistant that can help check whether objects meet the demands, equipped with common life knowledge. Your reply should be in JSON string format.
19

---

Prompt:
#i#f Task Verification and Modification ####
** **Objective:** Evaluate the provided task for logical consistency and feasibility. Modify it necessary to ensure all requirements are met accurately. ####bjecct Category#it# bed, sofa, cup, desktop....
### Demand-drivee Navigation Task Template##
"task instruction": $basic demandS, Spreference$ "basic demand instruction": "String" "preferred demand "basic cocticssoution:: [[object_a a, object_ci _jl [object_b)] b]]
_ inntrructittttiooo "String"
"preferred_soluteee [[object_aa a, object_c. object_d], d], [object b, object_[]] []]
This should be in dict format.
basic_solution is a list whose elements are lists, and each element represents a solution that meets the basic demand instruction; each solution may consist of one or more objects.
preferred solution is list whose elements are lists, and each element represents solution that meets both the ansic_demand instruction and preference; each solution may consist of or more objects, "[object_gi g]" represents that just object_ g can meet the demand. "[object_X" x, object_y, object_z] z]" represents that only the combination of object x, y, and z can meet the demand.
###To be modified Task in JSON String Format## $Task Dict$
** *Instructions:
1. Verify Object Categories: Ensure all items in 'basic solution' 'basic_oolution'
and preferred_solutiiee solution' are from the provided Object Category. Replace or remove any items that do not belong.
2. Expand Object Lists: Identify additional items within the Object Category that fulfill the lesii_ddemadd_c o and 'preferred demand instruction'. Add these items to "basic_solution and "preferredssolutioon'
solution' appropriate.
3. Validate Combination Solutions: Assess if each combination objects in 'basic solution' ' meets the "basic_de fecricddemnd___ee ieeemand_instuuctionn and if removing any object from these combinations makes them invalid. Apply the same verification for "preferred_solution concerning both "basic_dae seriie_demnd___eee iintraaniiitttttioon and "preferred demadddemdddd_deed demanddinstruution" iinstruuctton.....
4. Ensure Subset Relationship: Confirm prefeeredd _sluutio is subset of 'basic colutionootoo'.... If not, integrate missing combinations from 'preferred preferred_solutionn solution' into 'basic solution'.
** **Evaluation and Modification Steps:**
Begin by reviewing the JSON string of the current task for any logical inconsistencies or missing elements.
Identify and explain any aspects that are not rational or feasible within the context of the given Object Category.
Propose modifications, ensuring all suggested items are included in the Object Category and adhere to the Demand-driven Navigation Task Template.
F*nan Instruction:
Present a brief critique of the task's initial setup, highlighting any irrational elements. Offer a detailed plan for rectification, including specific changes to "basi__so solution' and "preferred_solutieee "olution" Ensure these modifications respect the original Object Category and meet the template requirements. Your response should conclude with revised JSON string format of the task, reflecting all necessary adjustments for coherence and completeness.
"basic_solution"
20

---

Table 6: A Comparison Between MO-DDN and DDN. The slash symbol "/" distinguishes between basic and preferred data. The left side of the slash indicates basic, and the right side indicates preferred.
MO-DDN
DDN
Preference
Multi-object
v
Number of Object Category
358
109
Average Instruction Length
17.44
7.5
Average Number of Solution
17.51/50,4
2.3
Average Number Object Per Solution
2.41/3.59
A.1.2
Task Metrics
1
N
ΣOEFL LOES Len(sb)
Sibasic
max ESS
N
1=1
An example of calculating Sabssic in an episode: when the agent finds object [a, b, c, d, e, f] and the solutions are object [a. b, c. y] or [d, c, m, n]. the Sibassc is maz len(a.b.e) len(d,e) 0.6.
(en(a.be, m..( ), i.e.
We calculate the SPL 60 using the following formula:
4
SPIbasic
SR mar(pi, 11)
(5)
N
EoSFl Roer 4 is the Len(es)
where SR is the success rate of ien episode, i.e., Sih maXe ESo length of the shortest path of an episode, and p is the length of the path taken by the agent in an episode.
A.1.3 Task Dataset Statistics
In MO-DDN, we generate 300 tasks, encompassing 358 object categories from the HSSD dataset. Notably, the average instruction length in MO-DDN is 17.44, significantly longer than the average length of 7.5 in DDN. On average, there are 17.51 basic solutions and 50.4 preferred solutions per task (note that preferred solutions also include basic demands so that the number will be higher than just basic solutions), much more than the average solution number of 2.3 in DDN, Each basic solution will contain 2.41 objects, and each preferred solution will contain 3.59 objects, while only one object is in the solution of DDN. In general, MO-DDN increases the complexity of instructions and the diversity of solutions. This makes MO-DDN more relevant real- life environments and more difficult. We create a table to compare the differences between DDN and MO-DDN shown in Tab.
A.2 Attribute Feature Training
A.2.1 Training Data Preparation
Due to the restricted scope of object categories within the HSSD dataset, the tasks are limited to 358 object categories, significantly smaller subset than those observed in real-world scenarios, which may cause overfitting on trained task and object categories. To address this limitation and enhance attribute feature generalization (i.e,, we would like the attribute feature be universal and not limited to the HSSD dataset), we introduce language-grounding without constraints on object categories and learn attribute features over these sagguage-groundinn tasks. In these tasks, broader array of objects and instructions is employed, thereby fostering improved diversity and generalization of attribute features. Then, for each uiguage-grounding task, we ask GPT-4 to answer the questions "What attributes are required to fulfill these demands?" and "What attributes are inherent to these objects?" This process yields language-level attributes for instructions and objects, which is used for training attribute features.
grounding tasks by prompting GPT-4 to generate tasks
21

---

We use the following prompts to generate grroungigg tasks:
System Prompt: You are a household task generator. Generate a list of ten daily household tasks or needs that a person might have. These should be tasks that are commonly encountered in daily living environment.
Prompt:
#### Task Introduction ####
Generate a list ten daily household tasks or needs that person might have. These should be tasks that commonly encountered in daily living environment.
### Task Example ###
$Some Task Examples$
### Task Format #### String: List[Lis[String]]
For example, task instruction: [["ObjectA", "ObjectB"), ["ObjectC", "ObjectD", "ObjectE"), ["ObjectF"]] ["ObjectG", "ObjectH"]]
#### Instructions ###
Generate list of 10 unique daily household tasks needs and separate them with
Generated ten tasks should be distinct from each other and Example.
- Each task should be clear, concise, and understandable for agent to execute or recognize.
Avoid repetitive tasks; ensure each one distinct. Consider a variety of household needs, including cleaning, maintenance, personal care, and organizational tasks.
For each task, generate ten different objects objects' combinations (c.g.. [ObjectA, ObjectB] is a combination, [ObjectC, ObjectD, ObjectE] is another combination, [ObjectF] a single object, [ObjectG, ObjectH] is another combination) that satisfy the demand be used complete the task.
Follow the format of the example provided in the Task Format.
Each Object should be enclosed in double quotes, for example, "ObjectA".
Here some language- grgguaiiggg tasks:
task instruction: want exercise home, focusing my abdominal muscles, preferably with minimal noise.
basic solution: (exercise bike).(yoga mat, dumbbell]
preferred Aauution[[yga mat]
task instruction: need secure way to store small valuables at home, preferably in method that's immediately obvious guests.
basic solution:
book]
preferred solution: [bookcase, book]
task instruction: need an efficient method to keep track of my daily hydration, preferably with a solution that's easy transport. basic solution: [water bottle), [[terrmos
preferred solution:[water bottle]
task instruction: want relax in my garden with chair that gently rocks, but prefer be able to withstand the weather conditions.
basic solution: [rocking chair, gazebo]. [camp chair, umbrella]
preferred solution: :ooukion:[rocking chair, gazebo]
We can see that the aagguage-groundin task uses some objects that do not appear in the HSSD dataset. then generate several attributes for each task's instructions and objects.
We use the following prompts to generate attributes for instructions:
22

---

You are household attribute generator. Generate list of attributes functions that can be used to satisfy the given instructions.
System Prompt:
Prompt:
#### Attribute Introduction ###
Every object item in the world has some attributes functions that be used to meet human demands. For example, water has the properties or functions of being drinkable, usable for washing clothes, capable of dissolving When a human need satisfied by a certain object or item, it is essentially the attributes of that object that meet this demand. Therefore, can deduce the required attributes or functions from specific demand.
solids, and suitable for bathing.
### Attribute Format ### String: List[String] For example, "instruction": ["Attributel", "Attribute2", "Attribute3", "Attribute4"]
### Example ### $Some ExamplesS
The Given Instruction ###
$InstructionS
### Instructions ###
for each demand, generate four attributes or functions that can be used to satisfy the given demand, separated by "======= Instructions and Attribute should be separated by ":
Each attribute function should be clear, concise, and understandable for agent execute recognize.
Avoid using words that too general vague.
The attribute or function can be a word or phrase.
Avoid using synonyms similar words that can be used interchangeably.
Consider a variety of attributes, including personal, organizational, and functional attributes. Follow the Attribute Format provided above.
Maintain the order of The Given Instruction same with the original.
Here are instructions and their attributes.
task instruction: want to exercise at home without assembling bulky equipment, preferring compact and versatile fitness tools.
attributes:
[resistance bands, adjustable dumbbells, foldable yoga mat, doorway pull-up bar]
task instruction: want to easily access my jewelry collection and minimize clutter, preferably by displaying the pieces.
attributes: hanging organizers, jewelry stands, "drawer dividers, wall-mounted display]
task instruction: I want to keep my cosmetics organized and easily accessible, preferably in a tidy arrangement.
attributes:[ compartmentalized storage, transparent containers, countertop design, cosmetic organiz- ers]
task instruction: want a simple way to track grocery needs, preferably one that the whole family can access.
attributes: shared list, real-time updating, accessible by multiple users, simple interface]
We use the following prompts to generate attributes for objects:
System Prompt: You are household attribute generator. Generate a list of attributes or functions that can be used to satisfy the given demand.
23

---

Prompt:
#### Attribute Introduction #####
Every object or item in the world has some attributes functions that can be used to meet human demands. For example, water has the properties or functions of being drinkable, usable for washing clothes, capable of dissolving some solids, and suitable for bathing. When a human need is satisfied by a certain object or item, it is essentially the attributes of that object that meet this demand. We can obtain the attributes and functions of an object from common sense or experience.
#### Attribute Format ### String: List[String] For example, Objectl: ("Attributel", "Attribute2", "Attribute3", "Attribute4")
#### Example Object ####
water: I"drinkable", "usable for washing clothes", "capable of dissolving some solids", "suitable for bathing]]=====
apple: l"sweet", "edible", "vitamin C". "vitamin -"|=====
book: ["readable", "hold information". "hold data," " "hold text"]== shirt: I"comfortablec. "cloth", "wearable", "can be worn"] "orm"]]==t""
### Example Attribute ### $ Example Attributes
### The Given Objects ###
$ Object Category$
### Guidance ###
For each object, generate four attributes or functions that can used to satisfy a demand, separated by
Object and Attribute should be separated by ":
Each attribute function should be clear, concise, and understandable for agent to execute recognize.
Avoid using words that too general vague.
The attribute or functions can be word or phrase.
Avoid using synonyms similar words that can be used interchangeably.
Consider a variety of attributes, including personal, organizational, and functional attributes.
Follow ### Attribute Format ### provided above. Follow the content format of ### Example Attribute ####
Maintain the Given object unchanged and only generate the attributes functions.
Here are objects and their attributes.
object: long-handled duster
attributes: extended reach cleaning, dust-attracting fibers, flexible head for tight spaces, washable and reusable]
object:
rocking chair attributes: relaxation, comfortable seating, aesthetic addition room, soothes babies]
object:
adjustable table
attributes: height customization, ergonomic design, collapsible for storage, multi-use surface]
object:
dusting brush
attributes: removes surface dust, gentle on delicate items, reaches tight spaces, ergonomic handle]
A.2.2
Attribute Training Details
The full loss function is shown below:
24

---

Loss X x Attribte Loss + X2 x Matching Loss+
X x VQ Loss + X4 Commit Loss + 25 x Recon Loss (6)
where A is 2.0, 22 is 1.0, A is 1.0, A4 is 0.25, and A5 is 1.0.
A.3 Details about Coarse-to-Fine Exploration
A.3.1 Design Details for Coarse Exploration Module
For the LLM branch, GPT-4 generates the attributes of the instruction in natural language. For example, "I need a comfortable place to read, preferably with natural light." has basic attributes of "comfortable seating". "quiet environment" and has preferred attributes of "natural light". Then, we use the CLI--Text-Enooder to encode these language-level attributes into the instruction attribute features. For the MLP branch, we use CLIP-Text-Ennoode to encode the instruction into the instruction features and then use Ins MLP Encoder to encode the instruction features into the instruction attribute features.
For the auth-planning algorithm, we use the habitat-sim built-in greedy planner. We ensure each step on an already explored point clouds to avoid navigating through unexplored points. this paper, for a fair comparison, baselines and ablations that involve building maps and traveling on them will all use habitat-sim's 's built-in greedy planner, including but not limited MOPA+LLM and FBE+LLM.
We provide some block score visualizations. A darker red color means higher cosine similarity between the attribute features of the objects and the instruction. The darkness of the color is based on the rank of the block score among all blocks, not the actual score. Since scores are not calculated for blocks that have been visited before, only a single coarse exploration is included in the visualization. Note that the block with the highest score (i.e., the darkest color) is likely to contain objects that can satisfy the demands. See Fig. 6. Fig. Fig. 8 Fig. block scores with different weights Fig.
Fig. TO
We also provide visualization of
A.3.2 Training Details about Fine Exploration Module
We use the standard transformer encoder from the official PyTorch 1.13.1 implementation, where model is 768, nhead 8, num_layerss of action is 64. The embedding dim of GPS+Compass is 32. The input dim of LSTM is 768+64+32, 1024, and its num layers The depth model a simple five-layer CNN model and two-layer MLP model.
and other parameters remain default. The embedding dim
its hidden_size
We collect about 50,000 trajectories train the fine exploration module under seen tasks and seen scenes settings in Experiments Sec. 5 using imitation learning. We collect trajectories by following steps: randomly select a scene and task, 2) initialize agent within two meters (i.e., block size in coarse exploration) of a target object, 3) use habitat-sim's 's built-in greedy planner to get the next step, when the distance to the target object less than 0.2 meters, turn left/right or look up/down according to the height and position of the object, 5) when the target object is in the field of view, execute Find and close this trajectory. The trajectories we collect here are relatively short and end with only one execution of Find. The average length of these trajectories is 9.19. This is because the fine exploration module only needs to take on exploration within a block, and therefore, the goal of learning only needs to be trajectory within block. Training with such short trajectories is very efficient and requires few computational resources (only a RTX 4090 graphics card needed). The standard deviation of the trajectory length is 5.62. The median trajectory length is 8. The plurality of trajectory lengths is 3. The maximum trajectory length 51. The minimum trajectory length is 2. We trained the model on single RTX 4090 using imitation learning and cross-entropy loss, i.e., considering the action prediction as a classification task, consuming about 12h.
We keep switching between the two exploration phases until the number of Find reaches n finds and then execute Done. When Find executed in the fine exploration phase, switch back to the coarse exploration phase to continue selecting the next waypoint.
25

---

Computer Armchair Table
Figure 6: Block Visualizations. Instruction: I need take quick notes during meeting, preferably with device that saves them digitally. Solution: Computer, Armchair, Table.
A.4 Experiments
A.4.1 Details about Experimental Settings
The effect of each action is as follows:
MoveAhead: Move forward 0.25 meters.
RotateRight: Turn right 30 degrees.
RotateLeft: Turn left 30 degrees.
LookUp: Turn the camera up 30 degrees.
LookDown: Turn the camera down 30 degrees.
Find: Record objects with distance less than dffind in the current field of view the found list.
Done:
End the current episode and report the success rate and SPL.
Note that we want to focus the benchmark on navigation. When executing Find, the simulator will automatically add objects to the found list that have distances less than dfind in the current field of view, regardless of whether or not they are recognized by the object detection module GLEE. Such detection is done by the simulator, so the agent does not get the ground truth semantic label, except when Find determinations are made by MOPA+LLM and FBE+LLM.
Our method and baselines can be trained on single RTX 4090, which will take about one day for each method. We report the mean and standard deviation for least three seeds. We the agent for 100 epochs in each seed, which takes about hours.
26

---

Cabinet
Wall shelf
Figure 7: Block Visualizations. Instruction: need to store collection of fine china, preferably in way that displays them elegantly, Solution: Wall shelf, Cabinet.
A.4.2 Details about Baselines
Since many baselines policies in single object navigation setting, we make some modifications to their action space. In their original policy, Done is modified to Find, and Done is automatically output when the number of Find exccutions reaches the upper limit findd
For Random, we let the agent randomly select an action other than Done and execute When the number of Find reaches the maximum number of executions fidd or the number of steps reaches the limit, execute Done.
For VTN, trained in the original paper in two stages, imitation learning without LSTM and reinforcement learning with LSTM. We remove the reinforcement learning phase here because find that in the MO-DDN task, using reinforcement learning to train VTN leads to performance degradation. We, therefore, chose to collect 20,000 trajectories and train the VTN using imitation learning with an LSTM module. We collect trajectories by following these steps: 1) randomly select scene and task, initialize agent random position, set a target object according to the solution, 4) use habitat-sim' 's built-in greedy planner to get the next step, 5) when the distance to the target object is less than 0.2 meters, turn left/right or look up/down according to the height and position of the object, 6) when the target object is in the field of view, execute Find, 7) go back to step until objects in the solution have been searched for, and then execute Done. The average length of these trajectories is 49.15. We replace the VTN's s original goal description with the CLIP features of the demand instruction. During evaluating, the agent executes Done when the number of Find reaches the maximum number of executions fnd or the number of steps reaches the limit.
27

---

Dryer
Figure 8: Block Visualizations. Instruction: I need to quickly dry batch of laundry, but prefer an fast and energy-efficient method. Solution: Dryer.
For ZSON, we use imitation learning to fine-tune the official ZSON's trained weights using the trajectories collected the VTN. We take the CLIP features the demand instruction as ZSON's goal descriptions. During evaluating, the agent executes Done when the number of Find reaches the maximum number of executions n find or the number of steps reaches the limit.
For DDN, we train the agent using the trajectories collected in the VTN, For training the attribute model DDN, we concatenate all objects in a solution at the language level and encode them with Tene-eer--neeeer as object features. If two solutions satisfy the same demand instruction, then the object features corresponding to these two solutions are positive samples. During evaluating, the agent executes Done when the number of Find reaches the maximum number of executions find or the number of steps reaches the limit.
For MOPA+LLM, we use map-building and object-detection models that are the same as our method, and use LLM make selections for currently recognized objects. When the LIM selects object as a target, we use habitat-sim's 's built-in greedy planning to walk near it; if the LLM does select any object, chooses a random point on the already explored map. Compared to the original MOPA implementation, we replace the point goal agent with the built-in greedy planner of habitat-sim. When the agent reaches the target, prompt the LLM decide whether perform the Find action. The prompts we use are in Sec. A.4.4
the FBE the exploration module and then let the LLM choose which objects that have been explored meet the demand and serve as targets for the habitat-sim's 's built-in greedy planner walk around. If the L.M does select an object, the agent continues explore using the FBE method. When the agent reaches the target, we prompt the L.M to decide whether to perform the Find action. The prompts use are in Sec. A.4.4
For FBE+LLM,
A.4.3 Details about Ablation Study
Ablation on Coarse Exploration For FBE+Fine, we use FBE to select waypoints and then use the habitat-sim's built-in greedy planner to get to the waypoints. We use the fine exploration module to locate the target objects when arriving at the waypoints. For LLM+fine, use LLM to select
28

---

Ceiling lamp Side table
Sofa
Figure 9: Block Visualizations. Instruction: I need to find a comfortable place to read for my study group, preferable with good lighting. Solution: Sofa, Side table, Ceiling lamp.
waypoints, and everything else is the same as FBE+Fine. The prompts we use are in Sec. A.4.4 For CLIP+fine, we replace attribute features with CLIP features to compute cosine similarity as block scores.
Ablation on Fine Exploration For Coarse+ZSON/TNN, to train the fine exploration module to train ZSON and VTN and then replace the fine exploration module with ZSON/VTN. For Coarse+Random, we replace the fine exploration module with randomly selecting an action except for Done.
the trajectory dataset used
Ablation on Attribute Training For Ours w/o VQ-VAE, set the weight of VQ Loss, Commit Loss, and Recon Loss to zero and only maintain Attribute Loss and Matching Loss. For Ours w/o initialization, we do not load the clustering centers to codebook.
Ablation on Score Weights We modify the attribute query to adjust the weighting of the basic and preferred scores. See for the exact formula.
A.4.4 Details about LLM's Prompt in Experiments
In our baseline experiments and ablation study, use LLM select the waypoint (e.g., MOPA+LLM. FBE+LLM, and LLM+Fine) and decide whether to perform the Find action or not (e.g., MOPA+LLM and FBE+LLM).
We use the following prompts to let LLM select the waypoint:
System Prompt: You are Al assistant that can understand human demands and imagine what human demands can be met with some combinations of objects. Now, need you to assist in selecting positions on the map based on current location and recorded object locations.
29

---

Dining table Straight chair
Figure 10: Block Visualizations. Instruction: need to organize small evening gathering but want to avoid any accidents with real candles. Solution: Dining table, Straight chair.
Prompt:
#### Task ####
Now, I hope you help more efficiently select areas in the scene for exploration using your knowledge. Specifically, I will provide you with the current records of objects available for exploration, along with their names, positions, and distances in ### Explored Object List#iH,, You need to choose one of them and proceed to explore the area nearby (if you determine that none of the objects the ### Explored Object List ### suitable, you use Frontier Base Exploration in this round). My goal to find a set of objects that can complement each other to meet my demands. will give you the objects have already found (in ### Found Object List ##H), and your task is to consider the objects already found (if not empty) and efficiently search for the missing ones to meet my requirements. #### Demand Instruction ###
choose output FBE, and will
SInstructionS
#### Found List ###
$Found List$ ### Explored Object List ### $Object in Point Clouds$
30

---

AemChair
Shad
Sarl
Bed
= and Ty 0
Tb 2
and
Figure 11: The Effect of rb and rp. This is an example used only to demonstrate scoring blocks. The darker the color in the figure, the higher the block score. When the task is "Find place sit, prefering a comfortable and soft place", lowering Tp (left) ignores preferences, so ArmChair, Sofa and Stool get high score (dark red); raising Tp (right) focuses preferences, Sofa and Bed can get a high score.
### Reply Format ####
"I choose:
The placeholder must be included in the response; is used identify the answer. You must strictly output the names from the ### Explored Object List ####; you cannot invent other objects or provide multiple objects. For example, if you think the most probable object explore next is "apple", you should output "I choose: apple". If you think none of the objects in the list are suitable, you can choose to output "I choose: FBE"
#### Requirement ###
You need to carefully consider this: order meet my demand, besides the objects mentioned in #### Found Object List ##Hti, what other types of objects need to be searched for, and which of these objects are most likely to appear near an object in the ### Explored Object List ### In the end, only need you to output the name of the most probable object from the ### Explored Object List #и# as the area you should prioritize exploring next. Please strictly follow the format declared in ### Reply Format ### Please think carefully and select the relatively best object the next exploration target. Only when you can't think of any good choices should you output FBE.
We the following prompts to let LLM decide whether perform the Find action not:
System Prompt: You are an Al assistant that can understand human demands and imagine what human demands can be with some combinations of objects.
31

---

Prompt:
#### Task ###
### Surrounding Object List #ifff declares the objects currently around you. You need carefully inspect them to see if there are any that are relevant to the demand stated in #i## Demand Instruction ###, and that can be used conjunction with previously found objects (in ### Found Object List ###). If any of the objects in ### Surrounding Object List ### might be potentially useful, report "Find". there are no objects that could be useful relevant, report "Skip". ### Demand Instruction ###
SInstructionS
### Found List ###
SFound List$
###f Surrounding Object List#Hi#
Sground truth semantic label in the current field of view #### Reply Format #### will execute $ActionS.
You should replace the Action with Find Skip, "S$" identifier used specify the range of the answer; please make sure you will not use it in other places.
### Requirement ###
Exploration steps reflect the progress of exploration. In total, you can perform Smax Smax_step$ exploration steps, and currently, Scurrent stepptsttepppttttpepeee steps have been completed. You can execute 'Find' total of 5 times, and so far, you have executed it Scurrent find time times. Please consider both the progress of exploration and the number of explorations performed decide whether to execute 'Find' or 'Skip' now. Feel free to use your imagination and try to find any object that might be potentially useful. Don't waste your find opportunities. You can take your time to consider whether the surrounding objects can help me meet my demand and also think about which objects among the surrounding ones can complement those already found. However, you must strictly use the answer format declared in ### Reply Format ###
A.5 More Limitations
In our method, we do not the agent decide choose Done until the number of Find reaches threshold or the number of steps reaches threshold. This leads to the possibility that the objects the found list already fully satisfy the demand instruction, yet the agent will still enter the coarse exploration module select the next waypoint, resulting in low SPL value. Future work could explore the attribute features used for decision-men maaiing in Done.
32

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA] is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can found.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We demonstrate the conclusions in the Experiment Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations in the Conclusion Section 6
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
33

---

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specing well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tes system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
This paper does not have theoretical results
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Answer: [Ycs]
Justification: A full description of the model and prompts for large language models are provided in our main paper 5 and supplementary materialA. We will release all the code and dataset if this paper accepted.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
34

---

If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
But we will release all the code and dataset if this paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips.cc cc/ httpc//ppwwccccuieeeeeeeeeeeeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Detalls
35

---

Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Please the main paper and and supplemental material A.4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We report the standard deviation results in Tab. 5.3
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: A single RTX4090 is enough to train and test our agent. It takes one day to train the agent and two days test. Please the supplemental material for details A.4.I
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
36

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Question:
Answer: [Yes]
Justification: We do.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: by [Yes]
Justification: We discuss them in the Conclusion Section
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We focus on navigation. To the best of our knowledge, does not have a high risk.
Guidelines:
The answer NA means that the paper poses such risks.
37

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We follow the protocol for the dataset.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com /dtaaset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide some examples of our task dataset in the supplemental mate- rial A.I.1
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
38

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
39


Reviews:
[{"review_id": "vZ2Wn9tH7m", "reviewer": "Reviewer_XjVA", "summary": "The paper introduces a new task called Multi-Object Demand Driven Navigation where an agent is tasked with searching for multiple-objects that satisfy a specified demand instruction by a user. The demand instruction is a natural language instruction specified by a user which might directly or indirectly ask for a specific object. For example, a instruction “I’m thirsty” requires an agent to search for some object in the environment that can be a drink (softdrink or water) that can address the query from the user. In MO-DDN the authors extend the task of DDN to scenarios where an instruction might require the agent to search for multiple objects. In addition, the paper proposes a modular method that involves building a coarse to fine attribute based exploration agent by leveraging SLAM and multimodal models like CLIP. The key idea of the method is to finetune CLIP to map a demand instruction to features that capture relevant attributes (one or more) to satisfy the goal. For example, feature vector for instruction “Find me a comfortable spot to work on report” should contain relevant features for objects required for such a instruction which could be “desk, lounge chair, laptop/book”. To learn such feature space authors propose a attribute based finetuning scheme by leveraging synthetic data generated using GPT-4. Finally, the authors show that using this method enables learning effective MO-DDN agents", "questions": "Mentioned in weaknesses section", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is well written and easy to follow.\n2. The proposed task of formulating multi-object search as a demand driven navigation where the objects required for tasks are not necessarily explicitly specified is a interesting task at the intersection of common sense and embodied AI and is valuable to study.\n3. The proposed method of distilling demand based attributes into CLIP feature space is intuitive, simple and effective based on results shown in table 1.", "weaknesses": "1. The experiments section is missing one important baseline where we evaluate how good the pretrained CLIP features do zero-shot on this task without any attribute based finetuning. It is important to establish that baseline to show effectiveness of the proposed method that leverages CLIP finetuning for the task. I’d appreciate if authors can add those results.\n2. The authors mention about flexibility of the approach which allows to switch from basic solution to preferred solution for a demand instruction easily. However the part about basic vs preferred solution and when is the agent expected to execute the preferred solution in unclear in the manuscript. It’d be good if authors can clarify if any of the basic/preferred solution is acceptable for each task in the dataset or there are specific instructions which require the agent to execute the preferred solution. If yes, how is that conveyed to the agent? Is that part of the instruction or the agent needs to implicitly infer it?\n3. The details about fine exploration policy is unclear. Is the fine exploration policy only called for low-level navigation action execution when a object of specific attribute is detected on the map? Authors need to add more details in section 4.2.2. to clarify the role of this module\n\nDue missing experiment mentioned in 1 and clarifications needed for 2 & 3 I am currently giving the paper a borderline reject but I'd be happy to increase my rating if authors address my concerns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are very grateful for your time and effort in reviewing our paper! We are also very appreciative that you have recognized the writing, the task setting, and the method proposed. We hope the following clarification will ease your concerns, and hope to hear back from you if you have further questions!\n\n>Q1: The experiments section is missing one important baseline\n\nA1: Thanks for the valuable advice. We supplement this baseline experiment named **CLIP Exploration**. In CLIP Exploration, CLIP-Text-Encoder ViT-L/14 is used to obtain CLIP features for instructions and objects, and the sum of feature similarities is used to select the block in the coarse phase. Please see Common Response's attached PDF for the results of the experiment. The experimental results show that computing block scores with CLIP features is not as good as computing them with attribute features. \n\n\n\n>Q2: The authors mention about flexibility of the approach which allows to switch from basic solution to preferred solution for a demand instruction easily.\n\nA2: Thanks for pointing this out. In our task settings, both basic and preferred demands are informed to the robot at the beginning of the episodes. Then GPT-4 generates the basic and preferred attributes which are encoded into basic attribute features and preferred attribute features. These two kind of attribute features will be used to calculating the block scores in the equation (2) of the paper. The flexibility we mention is that when the robot actually serves a specific user, the user can tell the robot to prioritize basic or preferred demands by **easily and flexibly adjusting the two hyper parameters, $r_b$ and $r_p$ in the equation (2)** of the paper (please see Table 5 in the paper for the effects of the two hyper parameters). $r_b$ and $r_p$ can be set at the beginning of the episodes by users (of course they are adjustable during navigation, but we didn't try this in our experiments). The higher the value of $r_p$, the more the robot tends to look for preferred solutions.  In Table 1, we set $r_b=r_p=1$ for our main experiment. However, it can be noticed from Table 1 and Table 5 that finding the preferred solution is more difficult compared to the basic solution; this flexible design allows users to adjust the probability of being satisfied according to their own situation. For example, a user who prefers to drink coke can increase $r_p$ to motivate the robot to find the preferred coke. However, when he is **very** thirsty, This user can  adjust $r_p$ down and $r_b$ up , so that the robots can find basic solutions (e.g., water) with a higher success rate. Future work can consider letting the robot automatically select the appropriate $r_b$ and $r_p$ from the user's instruction content and intonation. For example,  when the robot detects that the user is anxious by the user's intonation, the robot can automatically turn up $r_b$ and turn down $r_p$. Please see **Common Response 3 for Perference** for more detailed description.\n\n\n>Q3: Authors need to add more details in section 4.2.2. to clarify the role of this module.\n\nA3: Thank you very much for your advice. The fine exploration phase actually consists of an end-to-end navigation model. Once a target block is selected in the coarse exploration phase based on block scores, a habitat-sim built-in path planner generates a sequence of actions (such as $\\mathrm{MoveAhead}$, $\\mathrm{RotateRight}$ and  $\\mathrm{RotateLeft}$) for the robot to reach the selected target block (we make sure that the next action of the path planner is on the explored region). **When the robot reaches inside the selected target block, it enters the fine exploration phase.** The end-to-end model continuously generates actions based on the current RGB-D image. The actions are given to the robot to execute. When this end-to-end model outputs $\\mathrm{Find}$, the habitat simulator records a list of objects in the RGB image, and then the robot returns to the coarse exploration phase to select the next block to explore. Such switches between the coarse and fine exploration phase do not stop until the $\\mathrm{Find}$ count reaches the upper limit $c_{find}=5$ (we also discuss the limitations of doing so in Appendix A.5).  Please see **Common Response 1 for Method** for more detailed description.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "He1iQjIoRX", "reviewer": "Reviewer_9oyu", "summary": "The paper presents \"MO-DDN,\" a novel benchmark and approach for Multi-object Demand-driven Navigation (MO-DDN), where an agent needs to find multiple objects to satisfy complex, user-specific demand instructions. The proposed approach leverages a coarse-to-fine attribute-based exploration strategy. The method includes the training of an attribute model using CLIP features and a VQ-VAE loss, followed by a dual-phase exploration process combining modular and end-to-end techniques. The experimental results demonstrate that this method outperforms several baselines on the HM3D ObjectNav datasets.", "questions": "Scalability: How does the method scale with increasing complexity of demand instructions and the number of objects required to satisfy them? Is there a performance degradation when the complexity of the scene increases?\nAttribute Feature Space: The paper mentions using a discretized codebook for attribute features. How does the choice of the number of vectors (128) and their dimensions (768) affect performance? Would a different configuration yield better results?\nAdaptability to Dynamic Environments: How does the approach handle dynamic environments where objects might move or new objects might appear? Is the model capable of real-time adaptation in such scenarios?\nRobustness to Noise: How robust is the method to noisy or incomplete demand instructions? For example, how does it handle ambiguous or partially incorrect instructions from users?\nEnergy and Time Efficiency: Given the dual-phase exploration strategy, what are the energy and time efficiencies of the proposed method compared to the baselines? Are there any trade-offs between computational cost and navigation performance?", "limitations": "n/a", "rating": 4, "confidence": 1, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "Novel Benchmark: The introduction of MO-DDN as a benchmark addresses real-life complexities in demand-driven navigation, considering multi-object searches and user preferences.\nCoarse-to-Fine Strategy: The paper presents an innovative coarse-to-fine exploration strategy that effectively combines the benefits of modular and end-to-end methods, optimizing both efficiency and performance.\nComprehensive Evaluation: The method is evaluated rigorously against multiple baselines, showing significant improvements in success rates and navigation efficiency.\nAttribute Model: The use of attribute features trained with a discretized codebook and VQ-VAE loss is well-motivated and shows promising results in aligning demand instructions and object features.", "weaknesses": "Complexity and Implementation: The coarse-to-fine exploration strategy and the dual-phase approach add significant complexity to the implementation. This complexity might hinder the practical deployment and replication of the method in other settings.\nGeneralization to Unseen Environments: While the method shows good performance on the HM3D dataset, its generalization to other datasets or real-world environments with different characteristics is not extensively evaluated.\nFixed Attribute Numbers: The assumption of fixed attribute numbers (k1 and k2) for instructions and objects simplifies training but may limit the model's flexibility and applicability to real-world scenarios where attributes can vary widely.\nDependency on Pre-trained Models: The method heavily relies on pre-trained models like CLIP and GPT-4 for attribute extraction and task generation, which might pose limitations in environments where these models do not perform well or are not available.\nLimited Discussion on Limitations: The paper provides limited discussion on the potential limitations of the proposed method and the challenges that might arise in different applications or under varying conditions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your time and effort in reviewing our paper! We really value your recognizing the novelty of our benchmark and method and the comprehensive evaluation. We hope the following clarification will ease your concerns, and hope to hear back from you if you have further questions! \n\n\n>Q1: Complexity and Implementation\n\nA1: Our modular method is easy to deploy, with multiple models connected to each other by a simple pipeline. \nMoreover, the computational resource consumption of our proposed method in the deployment is low, requiring only a consumer-grade graphics card with 10G memory (such as a GTX1080Ti or a RTX3080) for complete deployment. We can also replace the object detection model with the more lightweight model in some computing resource constrained situations. Training consumption is also very low throughout the method, requiring only training the end-to-end agent in the fine exploration phase, which only requires about 12 hours of training on an RTX4090 graphics card.\n\n>Q2: Generalization to Unseen Environments\n\nA2: Thank you so much for pointing this out. We first need to point out that the scene dataset we use is HSSD, not HM3D. HSSD divides the scenes into training and testing scenes, and we follow their settings. We completely evaluate the generalizability of our method including unseen environments and unseen task instructions in Table 1. To test on other scene datasets we first need to build the set of task instructions for the corresponding dataset, which is very difficult due to time constraints. Generalization is an issue that needs to be addressed for all navigation tasks. We do not purposely design modules to improve generalizability in this paper, which will be left for future research. \n\n>Q3: Fixed Attribute Numbers\n\nA3: Thanks for the advice. This is indeed a limitation, and we have discussed this limitation in Section 6 of the paper. We argue that four attributes are sufficiently expressive for common demands and objects. The more flexible options of $k_1$ and $k_2$ are left for future research. \n\n>Q4: Dependency on Pre-trained Models\n\nA4: Thank you for pointing this out. The use of pre-trained models is common in navigation, e.g. EmbCLIP[1], LM-Nav[2], MOPA[3].  These pre-trained models generalize well in most scenarios after being trained on large-scale datasets. We are also proposing **MLP branch** in the coarse exploration phase, which can be run completely locally when GPT-4 is not available or computational resources are limited to run a LLM locally. \n\n>Q5: Scalability\n\nA5: Thank you for pointing this out. We argue that as the complexity of the demand instructions and the number of objects increase, the performance degradation is possible. Due to the limitation of the categories of objects in the dataset, it is difficult for us to construct more complex demand tasks in a short period of time, so verifying this fact may need to be done on more complex scene datasets in the future. \n\n>Q6: Attribute Feature Space\n\nA6: Thank you for pointing this out. We did some simple experiments (which were not put in the paper) for the number of vectors. In this simple experiment, we tested 64, 128, 256, 512 as the number of vectors. The metrics chosen are similar to the cosine similarity expressed by Line 310 in the paper. In the end, 128 achieved the best cosine similarity. 768 was chosen because we wanted this code book to be a subspace of the CLIP feature space, and therefore chose dimensions consistent with the CLIP ViT-L/14 version. Since code book only serves as a non-direct alignment between object attributes and instruction attributes, we did not perform complete ablation experiments on different configuration. Our ablation experiments demonstrate that the absence of VQ-VAE Loss and codebook initialization degrades the expression of attribute features, which in turn leads to degraded navigation performance. Please see **Common Response 2 for Experiment** for OpenCLIP ViT-H-14's results.\n\n>Q7: Adaptability to Dynamic Environments\n\nA7: Thank you for pointing this out. The research on dynamic environments is necessary and important in the real world but beyond the scope of this paper. We assume, as most Object Navigation tasks do, that the scenes are static during navigation.\n\n>Q8: Robustness to Noise\n\nA8: In the LLM branch, the reasoning of instructions is left to LLMs such as GPT-4, whose powerful reasoning capability also handles ambiguous instructions well. And our tasks are designed with the assumption that all the demand instructions are correct, otherwise it is difficult to design the correct solutions. We add the following experiments for illustrating the robustness of our method to noise. We add Gaussian noise to the RGB-D camera (add N(0,3) to RGB and N(0, 0.03m) to Depth). The experimental results show that our method does not suffer much performance degradation under noise, showing robustness to noise. Please see the attached PDF file in the Common Response for results.\n\n\n>Q9: Energy and Time Efficiency \n\nA9: As we said in A1, the model can be deployed on a consumer-grade graphics card. During our testing, inference with the RTX4090 resulted in an FPS of around 5 or so, which is acceptable for navigation (even in the real world). Compared to end-to-end baselines (VTN, DDN, ZSON), our method does increase the computational consumption, but it also brings large performance gains. Compared to FBE+LLM and MOPA+LLM, the computational consumption is almost the same, but our method performs better.\n\n## References\n\n[1] Simple but Effective: CLIP Embeddings for Embodied AI\n\n[2] LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action\n\n[3] MOPA: Modular Object Navigation with PointGoal Agents", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lOP2HLodAH", "reviewer": "Reviewer_xf5s", "summary": "The paper presents Multi-object Demand-driven Navigation (MO-DDN) which extends the DDN task to work with multi-object search and personal preference. DDN task aims to find an object in a navigation setting based on a given demand instruction. The paper proposes a new attribute model for multi-object DDN where the demand instructions and object categories are encoded into the same feature space. The attribute model is a VQ-VAE like model that uses attributes generated from GPT-4 as input. The attribute features are later used with a coarse-to-fine exploration agent for navigation and obtaining the solutions. The proposed method is evaluated on habitat with the HSSD dataset and the quantitative results show that both attribute training and coarse-to-fine exploration improve the performance.", "questions": "1. For results in Table 1, are $r_b$ and $r_p$ in Equation 2 being adjusted to prioritize basic or preferred solutions? And how are the metrics being evaluated (Success rate for basic/preferred) considering the baseline only works on one solution? \n2. The authors proposed attribute loss and matching loss in the attribute training. I am wondering how efficient are they and why is the weight of attribute loss larger than other loss terms.\n3. In line 137, a simpler version of \"Find\" action is used in MO-DNN compared to DNN. Does this adjustment contribute to the performance improvement over the DNN method?\n4. Why does the MLP branch have a similar or worse performance compared to MOPA+LLM? And how is the branch chosen between MLP and LLM during navigation?", "limitations": "The authors discussed the limitations of their works.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper has good motivation for the MO-DDN task. Ideally for robot navigation, the robot should be able to consider all possible solutions and prioritize the most feasible one. \n2. Coarse-to-fine exploration is interesting and seems to be a good solution to the DDN task.", "weaknesses": "1. In line 41, the authors claimed that personal preferences are considered. However, I don't think it was addressed in the paper.\n2. It is still unclear to me that, how MO-DDN is superior to DDN other than the solutions are multi-object rather than single-object. If we train a DDN with preferred solutions, would it have superior performance on $SR_p$?\n3. Overall, I feel like section 4, especially section 4.2 is a bit disconnected which makes the section hard to follow. Also, Figure 3 and 4 are hard to understand too. \n4. It would be more helpful to show more examples of basic and preferred attribute features, and some corner cases where priotizing becomes challenging.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are very grateful for your time and effort in reviewing our paper! We appreciate your kind endorsement of our benchmark and method. We hope the following clarification will ease your concerns, and hope to hear back from you if you have further questions! \n\n\n>Q1: In line 41, the authors claimed that personal preferences are considered.\n\nA1: Thanks for pointing this out. Due to character limitations, please see **Common Response 3 for Perference** .\n\n>Q2: how MO-DDN is superior to DDN other than the solutions are multi-object rather than single-object\n\nA2: Thanks for your advice! A complete comparison between MO-DDN and DDN benckmark can be found in Table 6 in the Appendix. Our method can leverage knowledge from external large models to **evaluate the small areas** where objects are most likely to be found, and use end-to-end models to efficiently and quickly **explore within the small areas**. DDN, on the other hand, needs to rely on an end-to-end model to **explore in a large area**. What's more, the attribute features used by DDN are one-to-one between instructions and objects, which cannot be well applied to multi-object search. Using preferred solutions slightly improves the $SR_p$ of DDN, but does not exceed our proposed method. Please see **Common Response 2 for Experimet** for the DDN trained with preferred solutions. \n\n>Q3: Overall, I feel like section 4, especially section 4.2 is a bit disconnected which makes the section hard to follow.\n\nA3: Thank you for pointing this out. We apologize for not making the method clear. We'll go into more detail about section 4.2 in the **Common Response 1 for Method**. We have provided a video in the supplemental material that may hopefully provide a better understanding. We will add summarizing paragraphs in the paper to illustrate the connections between each section.\n\n>Q4: It would be more helpful to show more examples.\n\nA4: Thanks for the suggestion. We **visualize two point clouds in the attached PDF**. Each block is colored by its score, with darker colors representing higher scores. We find that adjusting the values of $r_b$ and $r_p$ changes the scores of the blocks, which in turn affects the behavior of the agent. $r_b$ and $r_p$ are hyper parameters that control the priority of basic and preferred solutions (See A1 and **Common Response 3**).\n\n>Q5: For results in Table 1,xxx\n\nA5: Thanks for pointing this out. Table 1 shows the results under $r_b=1$ and $r_p=1$. In a real deployment, these two hyper parameters can be modified by the user to flexibly prioritize the basic and preferred solutions. See Appendix A.1.2 for calculation of the success rate. We have made some modifications to baselines to make them trainable and testable on our task, see Appendix A.4.2 for how this was done. Briefly, for the fairness of the test, we modify all the baselines' action space to Moveforward, RotateLeft, RotateRight, LookUp, LookDown, and Find. It is note that the baselines' policy does not output Done and Done is automatically executed only after the number of Find reaches the $c_{ind}=5$ limit.\n\n>Q6: Attribute loss and matching loss in the attribute training.\n\nA6: Thanks for pointing this out.  Attribute loss directly train the mapping of instructions and objects to attribute features. Matching loss directly guides the alignment of the attribute features of objects and instructions. In attribute feature training, we have two objectives: first, to train two MLPs (Ins MLP Encoder and Obj MLP Encoder), which serve to map the CLIP features of instructions and objects, to the CLIP features of attributes (referred to as attribute features), and second, to align the attribute features of instructions and objects to the same feature space. Attribute loss serves to train the first objective, while all other losses are in the training the second objective. We argue that the first objective's to be more important because the alignment only makes sense when the mapping is correct. Therefore, the weight of the attribute loss should be greater than the other losses. Without attribute loss and matching loss, attribute features would degenerate into CLIP features, since VQ-VAE Loss would only map features to the codebook's feature space which is initialized by CLIP features. We add an experiment named **CLIP Exploration** to show that CLIP features are not as good as attribute features in coarse exploration phase, proving the effectiveness of the two losses.\n\n>Q7: a simpler version of \"Find\" action is used in MO-DNN compared to DNN.\n\nA7: Thanks for pointing this out. This simpler version of Find action is used in all baselines and our method, including DDN. All baselines do not require the output of the bounding box of the target objects. As we said in A5, we keep the fairness in the action space. Moreover, the simpler Find action just removes the requirements of outputting the bounding box and does not affect the navigation success metrics.\n\n\n>Q8: Why does the MLP branch have a similar or worse performance compared to MOPA+LLM?\n\nA8: Thanks for pointing this out. We argue that MLP branch is weaker than MOPA+LLM largely because we provide ground truth semantic labels for MOPA+LLM for GPT-4 to make decisions on whether to perform Find. The powerful inference capability of GPT-4 and the availability of ground truth labels greatly enhance the decision correctness. MLP branch does not use GPT-4 at any time and does not use ground truth labels for decision making. MLP branch is designed with the intention of abandoning dependence on external LLM resources and focusing on lightweight, purely local running. In Table 1, we test the performance of the two branches separately, so that the branch chosen among an episodes is predetermined (of course they can also be switched freely during navigation, but we did not test this). When deployed in real life, one of the branches can be freely chosen to run according to compute resources and LLM availability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2YZxbJftYi", "reviewer": "Reviewer_AuxK", "summary": "The paper introduces a new benchmark Multi-object Demand-driven Navigation and trains various models on it.", "questions": "typo on line 82 incorrect capitailzation\ntypo on line 240 missing a\nWhy was that specific version of CLIP chosen?\nFor the LLM component, why was the specific version chosen", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper introduces a new benchmark, which are historically undervalued\n1. The ablation section is very clear and well written\n1. There is adequate coverage of baselines to my knowledge, although my knowledge may not be current on this area.\n1. The paper does achieve SOTA performance on many of the tasks relative", "weaknesses": "1. Some of the figures are a bit hard to read\n1. Requires pretrained foundation models for the task\n1. The method section can be tough to follow at times\n1. Frequent gramatically errors and typo\n1. They are a variety of CLIP encoders available now, and it would be interesting to compare to newer models or other variants of OpenClip", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We highly appreciate the time and effort you put into reviewing our paper! We are very grateful to you for appreciating our benchmarks and experiments! We hope the following clarification will ease your concerns, and hope to hear back from you if you have further questions! \n\n>Q1: Some of the figures are a bit hard to read\n\nA1: We apologize for the confusion caused by the figures. We will modify the content of these figures to make them easier to understand. \nFigure 1 shows an example of a MO-DDN task. The robot visits five different locations and finds multiple objects that fulfill the user's demand. Figure 2 shows how we train the attribute features. The ends of the arrows in the figure represent the targets for loss computation, and the colors of the arrows represent different ways of loss computation. Please read Figure 2 and Section 4.1.3 jointly. \nWe have drawn a new figure in the PDF attached at Common Response that describes the switches between the coarse exploration phase and the fine exploration phase, and we hope that this will help you to understand the relationship between Figure 3 and 4. Please see **Common Response 1 for Method** for detailed description. We draw a new figure in the attached PDF to describe the switches between the coarse (Figure 3) and fine (Figure 4) exploration phase.\n\n>Q2: Requires pretrained foundation models for the task\n\nA2: Thanks for pointing this out. The usage of pre-trained models is common in navigation, e.g. EmbCLIP[1], LM-Nav[2], MOPA[3]. These pre-trained models have good generalization due to training on large-scale datasets. \n\n>Q3: The method section can be tough to follow at times\n\nA3: We apologize for the difficulty in understanding the method section. We will add more articulation and section summaries to enhance readability. Please see **Common Response 1 for Method** for detailed description and **Revision Plan** at the bottom of Common Response. We will also add the above modification in the video in the supplemental material that hopefully provide a better understanding. \n\n\n\n>Q4: Frequent gramatically errors and typo\n\nA4: We appreciate you pointing out these typo and grammatical errors. We'll revise the issues you've mentioned and double-check the entire paper to correct grammatical issues and typos.\n\n>Q5: they are a variety of CLIP encoders available now, and it would be interesting to compare to newer models or other variants of OpenClip. Why was that specific version of CLIP chosen?\n\nA5: Thank you very much for your advice. We are using the official model provided by OpenAI, ViT-L/14, which is **the most popular and downloaded** model on the hugging face website among all OpenAI's CLIP models. We argue that the shared semantic space of vision and text provided by the CLIP model can effectively migrate attribute features to vision, which is important for the end-to-end model in the fine exploration phase (a similar conclusion in DDN). We add an experiment named **Ours (ViT-H-14 Encoder)**, which uses the OpenCLIP's ViT-H-14 model as you suggested. See the attached PDF in Common Response for experimental results. Experimental results show that the larger CLIP model does slightly improve navigation performance, but the time and computational resources required for training are also increased. These experimental results still support the conclusion in our paper that attribute features can improve navigation performance at two different levels of the exploration phase.\n\n>Q6: For the LLM component,\n\nThere seems to be unfinished questions here, and we look forward to your suggestions for LLM!\n\n\n## References\n\n[1] Simple but Effective: CLIP Embeddings for Embodied AI\n\n[2] LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action\n\n[3] MOPA: Modular Object Navigation with PointGoal Agents", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Common Response

We are very grateful to all the reviewers and AC for their time and effort. We highly thank the reviewers for their appreciation of our writing, benchmark, methods, and experiments. "The paper is well written and easy to follow.""The proposed task .... is a interesting task at the intersection of common sense and embodied AI and is valuable to study."(XjVA) "The paper presents an innovative coarse-to-fine exploration strategy that effectively combines the benefits of modular and end-to-end methods, optimizing both efficiency and performance"(9oyu) "The paper has good motivation for the MO-DDN task." "Coarse-to-fine exploration is interesting and seems to be a good solution to the DDN task."(xf5s) "The ablation section is very clear and well written" "The paper does achieve SOTA performance on many of the tasks relative" (AuxK)

**(Common Response 1 for Method)** However, we notice that some reviewers (XjVA, xf5s, AuxK) have some confusion about our method and the figures.  We next summarize our method section at a high level. 

In Secion 4.1, We train two MLPs (Ins MLP Encoder and Obj MLP Encoder in Fig.2) for mapping **the CLIP features of instructions and objects**, to **the CLIP features of their corresponding attributes** (referred to as attribute features). The purpose of Attribute Loss is to allow the MLPs to correctly map CLIP features to ground truth attribute features. The purpose of the other four losses is to align the attribute features of instructions and objects, guiding them to be in the same feature space. 

In Section 4.2, We describe the coarse-to-fine exploration phase. Specifically, the robot label the object categories in the RGB and builds a scene point cloud as it walks. The point cloud will then be segmented into a number of squares, called Blocks, based on the xy coordinates of the points. At the beginning of the episodes, the robot asks the GPT-4 about the task instructions to get the basic and preferred attributes of the instructions, which will then be encoded into the basic attribute features and preferred attribute features for the next computation of block scores. The robot calculates the score of blocks using equation (2) and chooses a highest-scoring and never-visited block as the target block. We use habitat-sim's built-in path planner to generate a series of action sequences that navigate the robot to the target block (we make sure that the next action of the path planner is in the explored area). **Please see our video in the supplementary material or appendix A.3.1 for the Block Visualizations.** Once the robot reaches the target block, it enters the fine exploration phase, where the robot invokes an end-to-end navigation model (Section 4.2.2) to generate a sequence of actions based on the current RGB-D. When the robot performs a Find action in the fine exploration phase, the objects in the current RGB are recorded by the habitat simulator. The robot then switches to the coarse exploration phase to plan the next target block. The switches between these two phases do not stop until the number of Find action executions reaches the upper limit $c_{find}=5$, followed by the execution of Done and the end of the current episode. **We draw a new figure in the attached PDF to describe the switches between the coarse and fine exploration phase.**

**(Common Response 2 for Experiment)** We also note that the reviewers (XjVA, 9oyu, xf5s, AuxK) make valuable suggestions about our experiments. We add the following four experiments: (1) **DDN (Preferred Trajectories)**: training the DDN using trajectories with the preferred solutions, (2) **CLIP Exploration**: computing block scores using the CLIP features rather than attribute features, (3) **Ours (with Noise)**: adding Gaussian noise to the RGB-D camera (add N(0,3) to RGB and N(0, 0.03m) to Depth), and (4) **Ours (ViT-H-14 Encoder)**: replacing the OpenAI's CLIP ViT-L-14 model with the OpenCLIP's ViT-H-14 model in the Method Section in the LLM branch.  See attached PDF for the results. 

**(Common Response 3 for Perference)** Some reviewers (XjVA, xf5s) have questions about the way we address personal preferences. We apologize for not explaining clearly the roles of the two hyper parameters $r_b$ and $r_p$.
In our benchmark, we propose that personal preferences affect the behavior of the robot. In our method, we use two hyper parameters $r_b$ and $r_p$ to adjust the prioritization of the basic and the preferred solutions in the equation (2). The two hyper parameters will be set at the beginning of episodes. In Table 1, we set $r_b=r_p=1$ for our main experiment. In Table 5, we study the effect of these two hyper parameters on robot behavior. We make a visualization example for illustrating how $r_b$ and $r_p$ affect decision making in the attached PDF. From Table 1 and 5 we can note that, in general, the success rate of satisfying the basic demands is higher than the success rate of satisfying the preferred demands. When deployed in a real environment, the two hyper parameters can be flexibly adjusted by users, allowing users to decide whether to prioritize basic or preferred demands at the moment. For example, a user who prefers to drink coke can increase $r_p$ to motivate the robot to find the preferred solution. When he is **very** thirsty, This user can  adjust $r_p$ down and $r_b$ up , so that the robots can find basic solutions (e.g., water) with a higher success rate. Future work can consider letting the robot automatically select the suitable $r_b$ and $r_p$ from users' instruction and intonation.

# Revision Plan:
1. We will add a description of the switches between the coarse and fine exploration phases and an associated pipeline figure in section 4.2. We will explain in more detail how preferences are addressed in our method.

2. We will again double-check the paper for grammar and typos.

3. In the experiment section, we will add our supplementary ablation and baseline experiments.

Meta-review: This paper received divergent initial opinions with two reviewers in support of acceptance and two reviewers expressing concerns.  The reviewer concerns revolved around missing a zero-shot baseline with pretrained CLIP features, lack of clarity about the distinction of basic vs preferred solutions and about the exploration policy, and overall complexity of the approach limiting its practical deployment.  The rebuttal responded to reviewer concerns, and one initially negative reviewer found the clarifications to alleviate their concerns, thus raising their opinion to accept. The second initially negative reviewer did not engage in discussion despite repeated prompting by the AC.  Moreover, this latter negative reviewer's review appears to be cursory and somewhat contradictory in parts. The AC does not find a basis to overrule the positive overall opinion of all the other reviewers.  The AC thus recommends acceptance and strongly encourages the authors to incorporate clarifications and revisions to improve the final manuscript.
Decision: Accept (poster)