Find What You Want: Learning Demand-conditioned Object Attribute Space for Demand-driven Navigation
Authors: Hongcheng Wang
Andy Guan Hong Chen
Xiaoqi Li
Mingdong Wu
Hao Dong
Venue: NeurIPS 2023 poster (2023)
Keywords: Visual Navigation
Demand-Driven Navigation
TL;DR: We propose a demand-driven navigation task, which requires an agent to find objects that satisfy human demands, and propose a novel method to solve this task.

Abstract: The task of Visual Object Navigation (VON) involves an agent's ability to locate a particular object within a given scene. To successfully accomplish the VON task, two essential conditions must be fulfiled:  1) the user knows the name of the desired object; and 2) the user-specified object actually is present within the scene. To meet these conditions, a simulator can incorporate predefined object names and positions into the metadata of the scene. However, in real-world scenarios, it is often challenging to ensure that these conditions are always met. Humans in an unfamiliar environment may not know which objects are present in the scene, or they may mistakenly specify an object that is not actually present.  Nevertheless, despite these challenges, humans may still have a demand for an object, which could potentially be fulfilled by other objects present within the scene in an equivalent manner. Hence, this paper proposes Demand-driven Navigation (DDN), which leverages the user's demand as the task instruction and prompts the agent to find an object which matches the specified demand. DDN aims to relax the stringent conditions of VON by focusing on fulfilling the user's demand rather than relying solely on specified object names. This paper proposes a method of acquiring textual attribute features of objects by extracting common sense knowledge  from a large language model (LLM). These textual attribute features are subsequently aligned with visual attribute features using Contrastive Language-Image Pre-training (CLIP). Incorporating the visual attribute features as prior knowledge, enhances the navigation process. Experiments on AI2Thor with the ProcThor dataset demonstrate that the visual attribute features improve the agent's navigation performance and outperform the baseline methods commonly used in the VON and VLN task and methods with LLMs. The codes and demonstrations can be viewed at https://sites.google.com/view/demand-driven-navigation.

Find What You Want: Learning Demand-conditioned Object Attribute Space for Demand-driven Navigation
Hongcheng Wang
Andy Guan Hong Chen School of EECS, Peking University cghandy@pku. edu. on
CFCS, School of CS, Peking University whe. 19990pku, edu. en
Xiaoqi Li
Mingdong Wu
CFCS, School of CS, Peking University clorisli@stu pku. edu. cn
CFCS, School of CS, Peking University wmingd0pku, edu. cn
Hao Dong"
CFCS, School of CS, Peking University hao dooddo@g@pu...mm cn
Abstract
The task of Visual Object Navigation (VON) involves agent's ability to locate a particular object within a given scene. To successfully accomplish the VON task, essential conditions must be fulfiled: 1) the knows the the desired object; and 2) the user-specified object actually is present within the scene. To meet these conditions, simulator can incorporate predefined object names and positions into the metadata of the scene. However, in real-world scenarios, it is often challenging to ensure that these conditions are always met. Humans in an unfamiliar environment may not know which objects are present in the scene, or they may mistakenly specify an object that not actually present. Nevertheless, despite these challenges, humans may still have a demand for an object, which could potentially be fulfilled by other objects present within the scene in an equivalent manner. Hence, this paper proposes Demand-driven Navigation (DDN), which leverages the user's demand the task instruction and prompts the agent to find object which matches the specified demand. DDN aims relax the stringent conditions of VON by focusing on fulfilling user's demand rather than relying solely on specified object names. This paper proposes method acquiring textual attribute features of objects by extracting common sense knowledge from a large language model (LLM). These textual attribute features are subsequently aligned with visual attribute features using Contrastive Language- Image Fre-training (CLIP). Incorporating the visual attribute features prior knowledge, enhances the navigation process. Experiments on A12Thor with the ProcThor dataset demonstrate that the visual attribute features improve the agent's navigation performance and outperform the baseline methods commonly used the VON and VLN task and methods with LLMs. The codes and demonstrations can be viewed at
of
"Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

nood pick
Coifee
auttin
will fid sometingg that quench Sliss
thimly
am thinty
QU3Tch thist
dban sth
Faant deat fands
Demand-oondiooed Altribute Space
Ineed
the bale
Tond
Figure I: Demand-driven Navigation and Demand-conditionn Attribute Space. The left side shows a many-to-many mapping between demands and objects, i.e., demand can be satisfied by more than one object, and an object can satisfy more than one demand: but in any case, objects that can satisfy the same demand will have similar attributes. On the right side, user at home provides the agent with an instruction "I am so thirsty". An agent determines what objects can satisfy this user's s demand. Although the user does not indicate which specific object they want, the agent interprets that the object he wants must have the attribute of being able to quench thirst.
Introduction
Visual Object Navigation (VON) 1:155 which is considered fundamental for home service agents, refers to the task in which agent is required to find user-specified object within provided scene. Previous VON studies can be broadly categorised into two groups. The first category, known as glosed-vocabular navigation [0 6-10 involves a predetermined set of target object categories so that the agent is restricted to selecting objects from this predefined set. On the other hand, open- vocabulary navigation 11 12] encompasses more flexible approach where the target objects described in natural language. meaning the agent is not limited to specific predefined categories.
In real-world applications of VON, users are typically required to provide the name of the object they want the agent to locate, regardless of whether it falls under the closed- cooodd-ocabulary or open- open-vocabulary group. This requirement introduces two conditions that must be fulfilled: 1) the user must possess knowledge of the object's name in the given environment; and 2) the specified object exists within the environment. These conditions scenarios. In common simulators (such as A12Thor [6] and Habitat 17 18), these two conditions can be easily satisfied. As direct access to the scene's 's metadata is available in the simulators, the existence of target objects can be checked using the metadata, or the object names in the metadata can be directly used as target objects. However, in the real world, where the user may not be omniscient of the environment (e.g., at friend's house), the two conditions described above may not be met, which will result in failure to satisfy the user's demand for an object. Additionally, it possible that when the user specifies an object to search for the agent is able to find an object with a similar function within the scene, when the specified object does not exist. For example, an object such as a bottle of water may be specified which has a related demand, to quench thirst. In the case where a bottle of water is not present in the scene, this demand can be adequately met by a cup of tea or apple juice found by the agent, however this would fail the VON task. This challenge illustrates key difficulty VON faces in real environments.
necessary for successful object navigation tasks in practical
From a psychological point of view (12 this paper considers the user's motivation to make the agent search for an object. In the VON task the user initially has a certain demand, then thinks of an object to satisfy it, and finally asks the agent to search for that specific object. Then, would it not be more consistent with the constraints of the real world environment to omit the second step and directly use the demand as an instruction to ask the agent to find an object that satisfies the demand? This paper argues that utilising user demands as instructions has the following benefits: 1) Users no longer need to consider the objects in the environment, but only their own demands. This demand-driven description of the task more in line with the psychological perspective of human motivations. 2) The mapping between demands and objects is many-to-many (i.e., a demand instruction can be satisfied by multiple objects, and an object can potentially satisfy multiple demand instructions), which can theoretically increase the probability of an agent satisfying user demands in

---

comparison to VON. 3) The demands. when described in natural language have a wider descriptive space than j specifying an object, and the tasks are thus described more naturally and in accordance with the daily habits of humans.
Therefore, this paper proposes the task of Demand-Driven Navigation (DINN). In the DDN task, the agent receives a demand instruction described in natural language (e.g.. I am thirsty), and then the agent must search for an object within the provided environment to satisfy this demand. Our task dataset is generated semi-automatically: it is initially generated by GPT-3 [23 (In the entire paper, we use gpt-3.5-turte
version), and then manually filtered and supplemented. The DDN task presents new challenges for embodied agents: 1) Different real-world environments contain different objects and, even within a fixed environment, the categories of objects can change over time due to human interaction with the environment, resulting in agents potentially looking for different objects when attempting to satisfy the same demand. 2) The many-to-many mapping between instructions and objects requires agents to reason depending on common sense knowledge, human preferences and scene grounding information: to identify what objects potentially exist in the scene to satisfy the user's demand, and where these objects are most likely to be located. 3) The agent needs to determine whether the current objects within the field of view satisfy the user's demand from the objects' visual geometry. There exists the possibility that this reasoning is not only based on the visual features, but stems from the functionality of the object, necessitating the agent having common sense knowledge about the object. In summary, the core challenge of DDN is how to common sense knowledge, human preferences and scene grounding information to interpret demand instructions and efficiently locate and identify objects that satisfy the user's demands.
noted that the essence of an object satisfying a demand is that certain attributes of the object fulfill the demand instructions. For example, for the demand of "I am thirsty." the object to be sought only needs to have attributes such as "potable" and "can quench thirst". Both bottle of water and a cup of tea are objects that have these attributes. To address the challenge of mapping attributes and demand instructions while considering their relevance to the current scene, this paper proposes a novel method that extracts the common sense knowledge from LLMs to learn the demand-conditioned textual attribute features of objects. Then the proposed method aligns these textual attribute features with visual attribute features using the multi-modal model CLIP (24). The learned visual attribute features contain common sense knowledge and human preferences from LLMs, and also obtain scene grounding information via CLIP. This paper also trains demand- based visual grounding model to output the bounding box of objects in the RGB input that match the demand when an episode ends. Experiments are conducted on the A12Thor simulator (16) with the ProcThor dataset (25). The proposed method is compared with closed-vocabular object navigation, open- open-vocabulary object navigation, and their variants. This paper also evaluates the performance of GPT-3 and multi-modal large language (MdML(MM)LL.M),, MiniGPT-4 [26]. operated agents which accomplish this task via attribute features extracted from the LLM and aligned with CLIP effectively assist in navigation and outperform baselines. In summary, this paper's 's contributions are listed as follows:
prompt engineering. The experiments demonstrate that the
This paper proposes the task of Demand-Driven Navigation (DDN), that requires the agent to find an object to meet the user's 's demand. The DDN task relies heavily on common sense knowledge, human preferences, and scene grounding information.
This paper evaluates two different categories of VON algorithms and their variants combined with GPT-3, as well as the performance of large language models (e.g., GPT-3 and MiniGPT- 4) on the DDN task. The results demonstrate that existing algorithms have difficulty solving the DDN task.
This paper provides a novel method to tackle the DDN task and benchmark for this. The proposed method extracts the common sense knowledge from LLMs to learn textual attribute features and uses CLIP to align the textual and visual attribute features. The experiments demonstrate that the demann-condittonee attribute features extracted from the L.L and aligned with CLIP effectively assist in navigation and outperform the basclines.
3

---

2 Related Work
2.1 Visual Navigation
The task of visual navigation requires the agent to use visual information [27] 28 to reach target location, such as visual object navigation (VON) [HI5 29), visual language navigation (VLN) [30
[38. and visual audio navigation (VAN) 43 The proposed DDN task can be considered as combination of VON and VLN in terms of describing the target: the demand is described using natural language and the agent is asked to find objects that match the demand. The VLN task requires the agent to follow step-by-step instructions to navigate in a previously unseen environment. In contrast to the VLN task, the proposed DDN task provides high-level demand instructions and requires the agent to infer which objects satisfy the demand instructions within the current scene. The VON task object navigation and open-vocabulary
can be broadly classified into two groups: (zero-shot) object navigation. In vooaau--vccaulary object navigation, the target object categories are pre-determined. For example, in the 2023 Habitat ObjectNav Challenge [45), there are 6 target object categories. These categories are usually represented as one-hot vectors. Therefore, many studies establish semantic maps [O] or scene graphs [13 to solve the closed-vocabulary VON task. In open-vocabule vocabulary object navigation, the range of object categories is unknown and is usually given in the form of word vectors [T] [2]. Many studies [T] 5 use pre-trained language models obtain word embedding vectors to achieve generalisation in navigating unknown objects. Additionally, CLIP on Wheels [5] uses the CLIP model for object recognition. Different from the VON task, our proposed DDN task does not focus on the category of objects, but chooses "human demand" as the task context.
2.2 Large Language Models in Robotics
Recently, large language models (LLMs) 44 49] have attracted attention for their performance on various language tasks (c.g., text classification and commonsense reasoning). LLMs can exhibit human level of common sense knowledge and can even be on par with human performance in seeking to use the knowledge in LLMs to control assist robots in performing range of tasks. LM-Nav [37] accomplishes outdoor navigation without any training by combining multiple large pre-trained models and ViNG (55)). In SayCan (51), the authors use LLM to interpret high- level human instructions to obtain detailed low-level instructions, and then use pre-trained low-level instruction skills to enable the robot to complete tasks. However, SayCan assumes that the agent can access the object positions in navigation and only conduct experiments on one scene. PaLM-E [52] goes a step further by projecting visual images into the same semantic space as language, enabling the robot to perceive the world visually, but PaLM-E exhibits control that is either limited to table-Ievel tasks or scene-level tasks with a map. Different from these approaches, the proposed method does not use LLMs for providing reasoning about instructions directly. but instead uses LLMs to learn attribute features about the objects. Such attribute features can help the agent learn an effective navigation policy in mapless scenes. This paper also uses LLMs to generate the required dataset.
some specialised domains. There is an increasing amount of research [37] 50 54
(GPT-3 233. CLIP 24
3 Problem Statement
In the DDN task, an agent is randomly initialised at a starting position and orientation in mapless and unseen environment. The agent is required to find an object that meets a demand instruction described in natural language (e.g., "I am thirsty"). Formally, let 'D denote set of demand instructions, denote a set of navigable scenes, and denote a set of object categories that cxist in the real world. To determine whether found object satisfies the demand, let G D x O + {0, 1} denote discriminator, outputting if the input object satisfies the demand, or if it does not. In the real world, G is effectuated by the user, but to quantitatively evaluate the proposed method and the basclines, G is effectuated by the DDN dataset collected in Sec.
At the beginning of each episode, the agent is initialised at scene S with initial pose Po (i.e., position and orientation), and provided natural language demand instruction d =< d1, d2 .....,d.. >E D. where I. is the length of the instruction and d; is single word token. In the demand instruction d, the task is made more challenging by not providing specific object names. This to ensure the agent learns to reason in scenarios where these objects are not specified by the user. Then the agent is re-

---

quired to find an object o E O to satisfy the demand instruction d and only take RGB images as sensor inputs. The agent's 's action space is MoveAhead, RotateRight, RotateLeft, LookUp, LookDown. and Done. When the agent selects the action Done, the agent is also required to output bounding box b to indicate the object that satisfies the demand within the current field of view RGB image. Then it is determined whether the success criteria have been satisfied. The success criteria for the DDN task include two conditions. 1) The navigation success criterion: this requires there is an object in the field of view that satisfies the demand instruction and the horizontal distance between the agent and the object is less than a threshold Gravi- 2) The selection success criterion: this requires that given that the navigation is successful, the intersection over union (IoU) [56] between the output bounding box and the ground truth bounding box of an object that satisfies the demand instruction is greater than threshold Caele- There is 100 step limit to succeed during the episode.
4 Demand-Driven Navigation Dataset
Although mappings between objects and instructions will vary with the environment, a fixed mapping between demand instructions and objects is required to check selection success for training the agent in specific environment, referred to as world world-grounding mappings (WG mappings). The modifier wworld-grounding emphasises the connection the specific environment of the real world and will vary with the environment. For example, the ProcThor dataset [25) and the Replica dataset [57] will have different WG mappings. To simulate real-world scenarios, it is assumed, unless otherwise stated, that the agent does not have access to the metadata of the object categories in the environment or the WG mappings that are used to determine selection success during training, validation and testing. The set of world- weeld-grounding mappings constructed in this paper form the demand -diiven navigation dataset.
To construct the WG mappings, the metadata regarding object categories in the environment is obtained and GPT-3 is used establish fixed set of WG mappings between demand instruc- tions and objects F D + O,, where O is subset of O. For example, F( am thirsty") = (Water, Tea, AppleJuice). Concretely, prompt engineering used to inform GPT-3 of the object categories that may be present in the experimental environment. Then GPT-3 is used to determine which demands these objects can satisfy and returns this information in the form of demand in- struction and statement of which objects present satisfy this demand instruction. It has been found that the WG mappings generated by GPT-3 are not absolutely accurate. Due the presence of some errors in the generation, manual filtering and supplementation is then used to correct and enhance the dataset. These WG mappings F are used during training and testing, only to discriminate the selection success (i.e., to effectuate discriminator 9). In total, approximately 2600 WG mappings are generated. Please see supplementary material for details of the generation process, the prompts, and the statistical features of the dataset.
5 Demand-Driven Navigation Method
is important to note that if an object can fulfill a demand, it is because one of its functions or attributes can satisfy that demand. In other words, if multiple objects can fulfill the same demand, the objects should have similar attributes. For example, if your demand is to heat food, the corresponding microwave, which all have similar
objects that can satisfy the demand attributes such as heating and temperature control. The relationship between attributes and instructions not specific a particular environment, but instead is universal common sense knowledge. Thus, this common sense knowledge can be extracted from LLMs. This universality turns the multi- object-target search into tare---ttrtuutttrreeee search: the similarity between attributes reduces the complexity of policy learning. Therefore, we expect to extract the attribute features of the objects conditioned on the given demand from the LLM, i.e.. demand-conditionee textual attribute features.
stove, oven
In this section, how the textual attribute features are learned from LLMs (Sec. 5.1 how they are aligned from text to vision (Sec. 5.2) and how to train the model (Sec. 5.3
5.1 Textual Attribute Feature Learning
Knowledge Extraction Given that the relationship between attributes and demand instructions common sense knowledge, many mappings between demand instructions and objects established
5

---

D1. am hungry.
Quary
D3.1 want to sleep.
D2. am so cold.
BERT
D3 Feature
Closer
Langugge-grounnen Demand
Demand BERT
Feature
Toosi
Push
Attribute
Away
Module
CC31
TOL lematie, stuaits reddeed. Tio uuiag omar1 nanwers
CLIP
Language-grounding
Demand-object Feature
Demand-conditoned
conditioned Attribute Space
GLIP-lextul Feature
Attribte Feature
Demand-condittone
Knowledge Extraction
Contrastive Learning
Figure 2: Textual Attribute Feature Learning. This diagram illustrates the process for textual attribute feature learning, where BERT and CLIP features are combined via concatenation and the attribute module is trained using contrastive learning. Green arrows and minus signs represent negative sample pairs. Red arrows and plus signs represent positive sample pairs. Models this blue represent the use of pre-trained weights with frozen parameters.
Copy
place to eat food World-gruuddng
BERT
Demend BERT Fealure
Tranaformer
Fealure
Box
Demand BERT Feature
PO1-
A1-4 EXTTETEE
Visual Grounding
D014
Module
D01-d
Frooder
Demand-Obiect Feature
Damang- conditooed
Blox Logis
Done
Nhor
Atribute Feature
Imitation Learning
World-grounding
CLIP-iiual Feature
Aclin
RGB
Imsge
Global Visual
Feeture
Visual-textual Attribute Alignment
Policy Learning
Figure 3: Policy Learning with Visual Attribute Features. This diagram illustrates the process for policy learning using visual attribute features, where BERT and CL.P features are combined via concatenation. The demand-conditionel attribute features obtained from the Attribute Module are fed into transformer model together with BERT features and global image features. Models in blue represent the use of pre-trained models with frozen parameters.
with GPT-3, for learning demand-conditioned textual attribute features, referred to as language- grounding muppings (LG mappings). The modifier mappings can obtained from LLMs without the need for any connection real-world situations. independent of the real world and the experimental environment, they
emphasises that these
As LG mappings suitable to be used in training the model. Concretely, as shown in Fig. 2 GPT-3 used to generate a multitude of demand instructions in advance, called language-grounding demands (LG demands), and then GPT-3 is queried as to which objects can satisfy these demands. GPT-3 generates the resulting objects called urguage-groundiigg objects compose the LG mappings. The BERT model is used to encode the LG demand instructions and obtain demand BERT features. The Tep--Tex--Encoder is used encode LG objects and obtain CLIP-textual features. Then the demand BERT features and CLIP-textual features are concatenated together obtain demand-object features (these features named DO(X)-(Y), with X representing the demand label and Y representing the object label).
objects (LG objects). The LG demands and the LG
Demand-conditioned Contrastive Learning Due the fact that objects which satisfy the same demand show similar attributes and vice versa, contrastive learning is suitable to be used to train the Auribute Module. The way positive and negative samples in contrast learning are defined is next illustrated using the object and demand satisfaction relationships in Fig. 2 object a, b and satisfy demand DI, object and satisfy demand D2, and object g, and satisfy demand D3. two objects can satisfy the same demand, then the two demand-object features corresponding these

---

attribute features obtained during two objects and the demand are positive sample pairs. For example, DO1-a and DO1-b is a positive sample pair. There are three categories of negative sample pairs. 1) For the same demand. if one object can satisfy it and the other cannot, the corresponding Dem-Ohj features are negative sample pairs. For example, DOI-a and DO1-d a negative sample pair. 2) For the same object, if it satisfies one demand but not the other, then the corresponding demand-object features are also negative sample pairs. For example, DOI-a and DO2-a is a negative sample pair. 3) The demand-object features with different objects and demands are also negative samples pairs. For example, DO1-a and DO2-c is negative sample pair. Then demand-object features are used as the input to the Attribute Module to obtain the label and Y representing the object label). The demand-condee conditioned attribute features corresponding to positive sample pairs should be as close as possible in the Gemand-conditiones attribute feature space, and negative sample pairs should be as far away as possible. Therefore InfoNCE Loss [58] is used train the Attribute Module. We use 6-layer Transformer Encoder Attribute Module.
attribute features (named A(X)-(Y), with X representing the demand
(59 to implement the
5.2 Textual-Visual Alignment via CLIP
In Sec. 5.1 the cemand-conditioned textual attribute features are learned using language- grounding
language-grounding
mappings. However, the SL.I-SSemanticSpaaa is used as a shared space for vision and text to To address this problem, the allow the Attribute Module to obtain scene grounding information. The DETR [60] model is first used to segment object patches (referred to as world-gre groudddg objects, WG object) in the field of view during navigation, and then the HeaP-Image-Eneee
textual attribute features are still at the language-Ievel
is used to project these object patches into the CLIP-Semantic-Spael to obtain CLIP-Visual features. Both CLIP-visual features and CLIP-textual features are projected to the same CLIP-Semantic- ------mttt--ppccee and thus they are aligned. The Demand BERT feature and CLIP-Visual features are concatenated (i.e., demand-object features) and used as inputs to the Attribute Module trained in 5.1 Note that the CLIP-Visual features come from the wvene-groundigg visual input. Therefore, the navigation involve scene grounding information.
To summarise, in Sec. 5.1 contrast learning and are used to allow the Attribute Module to learn to extract demand conditcconnitoonedd attribute features of ob- jects, which contain common sense knowledge and human preferences; Sec.55.2 CLIP's shared visual-textual semantic space is used to enable the Attribute Module to receive grene-grounding information during navigation.
mappings generated by GPT-3
5.3 Policy Learning and Visual Grounding Model
The features that are fed to Transformer [59 are processed in similar way to VTN 6 Demand- conditioned features are concatenated with bounding box and logits and input into a Transformer Encoder, then passed into a Transformer Decoder with Demand BERT features and global visual
features (encoded by a pre-trained Vision Transformer 66. (2)). ). Imitation learning is used to train the proposed model. Approximately 27,000 trajectories generated by the A algorithm 633 and used for training.
are collected
The Visual Grounding Model (VG model) is essentially classifier built with Transformer Encoder. The DETR model is used to segment the object instances on the current image, and then the features of the last layer of DETR for the highest & patches of DETR logits, the global image features encoded by RESNET-18 (64, the Demand BERT features and a CLS token, are combined and fed to the VG model, and then the output features corresponding to the CLS token used for classification. The VG model is used for all the baselines that perform DDN and the proposed method.
For more details about policy learning, the fine-tuning of the image encoder and DETR, the pipeline and training procedure of the demand-based visual- groundii model, and hyperparameters, please see the supplementary material.
7

---

Table I: Quantitative comparision baselines and ablations. NSR: navigation success rate.
NSPL: navigation success rate weighted by the path length. SSR: selection success rate. indicates that the algorithm leverages scene metadata. The parentheses show the sample standard deviation.
Meilad
SSR
NSPL.
NSR
NSPL
Random
5220.72)
2,006600
5000,80)
3.700.70)
2,00.80
2.300.3)
3:300,413
2,80.88
3,500.8)
1.00.15)
3,400
VIN demand
1.20..3
3221.3)
5,20033)
3.00.75)
2M00511
3:2:1:1)
2.21.21
6.001
VIN-CLIP-demind
12,00500
5.00.51
52223)
10.744.1
3310.5
5,001.0
10.01500
36313)
4.03.00
9.35533
3900.11
4.0.3.0
VTN -TN-
1601.
0.500.70
000
1.40065)
6460.57)
0.500355
1.300.70
0.200.20
0.3/0.4)
0.900.20
0.400.34
0.5:0.35
ZSON demand
4221.23
2.700.5
1901.
4.62200
3.1(1.3
2.006
4,300.5
2.900
1.200.45
3,5/0.5
2.400.7
1.10066
ZSON GPT
400235
L302.4)
0.300.25
190256
27300
2501.21
0. 0.700:4
0.300.2)
1.500.11
02 0.200221
C1 J.I.-Na-MinniPPP
48
20
2.1
30
18
223
3.7
27
5.3
5.11
3:17
NLPP-Nax-GPPT
50
50
4.1
60
5.5
533
55
53
400
4.0
3.11
2.0
FBE-MiuGPPT4
3.5
38
22
3.5
32
2.0
3.5
3.5
2.0
4.0
4.1
FBE-GPT
4.
50
355
43
45
5.5
5:1
0.01
012
60
035
MinilPP-4
Ours w/o Jur transoommr
156/10.3
6.90.99
6.0.00
15,30.60
8.912.))
7.3/2.3)
12330.3
4,5/2.)
4.72231
11,30.3)
5.46655
2,700.3
Dun, wio allr recinin
438200
0-1133)
5322.33
12.71-33
5700000
6.3151
1.005575
Oun_w WRRT
122/1222
4.333:5)
S308.)
3,4033
2.8/2.3
4.03335
7.8010.7)
25:2333
22003
2.71241
1.1177
Ouns
21.9:300
9.(1
75013)
19.30.3
9.41.5)
4,51188
14222.77
6.41.00)
571.81
16.1:1.5
6.01221
Experiments
6.1 Experimental Environment
The A12Thor simulator and the ProcTThr dataset 25
are used to conduct the experiments. Through- the experiments, 200 scenes in each of ProcThorr train/validationntest sets selected for total of 600 scenes. There are 109 categories of objects that can be used in the environments to satisfy the demand instructions. 200 WG mappings are selected for training and 300 for testing from our collected DDN dataset. The WG mappings used for validation are the same as those used for training. Our experiments can all be done on 2 A 100 40G, 128-core CPU, 192G RAM. To complete training for a main experiment takes 7 days.
6.2 Evaluation Method and Metrics
closed-vocabulary object navigation algorithm, VTN 66. and an open-vocabulary object navigation algorithm, ZSON [1] are chosen as the original VON baselines. To fit these baselines to the proposed DDN task, several modifications are made to the algorithms as follows and the results are shown in the form of a postfix shown in Tab. 5.3 The VTN-CLIP means that we replace the DETR features used in VTN with CLIP features. The "demand" suffix means the original one-hot vectors (closed-vocabulary)) or word vectors (open-vocauulary) are replaced with the demand BERT features. The "GPT" suffix means that the GPT-generated the parsing of the demand instruction, and then the parsing results are used as the input for object navigation. These VON baselines and their variants should demonstrate the difficulty of the DDN task and the fact that DDN tasks not easily solved by utilising language models for reasoning. Finally, five variants of the VON algorithm are introduced: VTN-CLIP CLIP-demand VIN-demand, VTN-GPT, ZeoN-demand, ZSON-GPT.
aagguage-grounding
objects are used as
The popular large language model GPT-3 and the open source multi modtll large language model MiniGPT-4 are also used for navigation policies and recognition policies, resulting in two baselines: MiniGPT-4 and GPT-3-Prompi GPT-3 and MiniGPT-4's test results, should provide insight into the strengths and weaknesses of large language models for scene-level tasks. Random (iee, randomly select an action in the action space) is also used as a baseline for showing the difficulty of the DDN task. We also employ risual-language navigation algorithm. CLIP-Nav [65] and heuristic exploration algorithm, FBE [66] as navigation policies, with GPT-3 and MiniGPT-4 serving as recognition policies, resulting in four baselines: CLIP-Nav-GPT, HUP--Na-MiniGPEE FBE-GPT, and FBE-MinGGPT-.
The MIN-CLIP-demand VTN-GPT, and -eN-demand algorithms are trained using the same col- lected trajectory as our method. indicates that the algorithm leverages environmental metadata. Navigation success rate, navigation SPL [67, and selection success rate are used metrics for comparison. For more details on metrics and baselines (about the metadata usage of the algorithms, prompts used in GPT-3 and MiniGPT-4, etc), please see the supplementary materials.
8

---

6.3 Baseline Comparison
Detailed and thorough experiments are conducted to demonstrate the difficulty of the DDN task for the baselines and the superiority of the proposed method, shown in Tab. 5.3
The results of VTN-demand are slightly better than Random, indicating that VTN-demand has learned to some extent the ability to infer multiple potential target objects simultaneously. However, ZSON- demand has lower performance than Random. Considering that ZSON-demand uses CLIP for visual and text alignment, this lower result indicates that CLIP does not perform well on aligmment between instructions and objects. VTN-GPT and ZSON-GPT demonstrate very poor performance. This is likely to be due to the fact that the not to be present in the current environment, which leads to a meaningless search by the agent. VTN-GPT and ZSON-GPT also show that converting DDN to VON through a pre-defined mapping between instructions to objects is not effective. The superior performance of VTN-CLIC CLIP-demand compared to VTN-demand underscores the contribution of CLIP in effectively extracting features of objects. The two variants about CLJP-Nav do not perform well, and we argue that this may be attributed to significant disparity in content between the step-by-step instructions in VLN and the demand instructions in DDN. Since ProcThor a large scene consisting of multiple rooms, the results show that heuristic search FBE is not efficient.
objects given by GPT-3 have a high likelihood
The performance of GPT-3+Prompp is due to the lack of visual input to the GPT-3+Prompp method and relying only on manual image descriptions resulting in insufficient perception of the scene. Moreover, during the experiment, is observed that GPT-3+ Prrmppt rarely performs the Done action, and approximately 80% of the failures are due to exceeding the step limit, suggesting that GPT-3+Prompt lacks the ability to explore the environment which leads to its inability to discover objects that can satisfy the given demand. With visual inputs, MiniGPT-4 outperforms GPT-3+Prommp despite its relatively small model size, indicating that visual perception (i.e., scene grounding information) is of high significance in scene-level tasks. MiniGPT-4 still does not perform as well as Random. By observing MiniGPT-4's trajectories, two characteristics are noticed: 1) the agent tends to turn its body left and right in the original place and turn the camera up and down, and does not often move; 2) the selection success rate is very close to the navigation success rate. It is therefore considered that MiniGPT-4'ss task strategy is as follows: observe the scene in place and then choose to perform the Done action if the agent identifies objects in sight that can satisfy the demand, and output the selected object. By the fact that the selection success and navigation success rates are close, it can be found that MiniGPT-4 satisfactorily recognises the semanties of objects and infers whether they meet the user's demand. However, the low navigation success rate implies that MiniGPT-4 cannot determine whether the distance between it and the object is less than the threshold value Cravi-
significantly lower than Random. This lack of performance
6.4 Ablation Study
We conduct three ablation experiments, with specific details as follows:
Ours w/o attr pretrain: we use the transformer network in the Attribute Module without demand-conditioned contrastive learning to pretain it.
Ours w/o attr transformer: we replace the transformer network with a MLP layer in the Attribute Module.
Ours w/o BERT: we replace the BERT endocer with a ML. layer.
Ours surpasses all baselines. The Attribute Module turns multi-object-targe goal search into single- attribute-target search, which makes the agent no longer need to find potentially different objects in the face of the same demand instruction, but consistent attribute goals. The consistency between demands and attributes reduces the policy learning difficulty. In Ours w/o BERT, we observe significant performance drop, especially in the setting with unseen instructions. This finding indicates that the features extracted by BERT are highly beneficial for generalisation on instructions. Ours w/o attr pretrain shows that our proposed improvement to performance. After demand-conditionee contrastive learning, the Attribute Module extracts generalised attribute features from LLMS and LLMs provide sufficient common sense knowledge and interpretation of human preferences for the Attribute Module to learn. Once the common sense knowledge extracted from the LLMs is lost, the Attribute Module can only learn
coiaandd-oodiioonedd contrastive learning has significant

---

the association between attributes and instructions from the sparse navigation signals of imitation learning. However, Ours w/o attr pretrain still surpasses all baselines (including VTN with a similar model structure), which is probably due to the fact that the object semantic information 668 contained in the CI.Il-visual features helps the Attribute Module to understand the demand instructions and learn the relationship between objects and instructions to some extent through imitation learning. The performance decrease observed in Ours w/o attr transformer compared to Ours suggests that the transformer network is more effective than MLP in capturing the attribute features of objects.
7 Conclusion and Discussion
This paper proposes Demand-drived Navigation which requires an agent to locate an object within the current environment to satisfy user's demand. The DDN task presents new challenges for embodied agents, which rely heavily on reasoning with common sense knowledge, human preferences and scene grounding information. The existing VON methods modified to fit to the proposed DDN task and they have not been found to have satisfactory performance on the DDN task. The LLM GPT-3 and the open source MM-LLM, MiniGPT-4 also show some drawbacks on the DDN task, such as being unable to determine the distance hetween the object and the agent from visual inputs. This paper proposes to learn demand-conditioned object attribute features from L.MS and align them to visual navigation via CLIP. Such attribute features not only contain common sense knowledge and human preferences from LLMs, but also obtain the scene grounding information obtained from CLIP. Moreover, the attribute features are consistent across objects (as long as they satisfy the same demand), reducing the complexity of policy learning. Experimental results on A12thor with the ProcThor dataset demonstrate that the proposed method is effective and outperforms the baselines.
Limitations and Broader Societal Impacts As we do not have access GPT-4 with visual inputs and PaLM-E, we can only use the open source MM-LLM (i.e., MiniGPT-44 to test on the proposed DDN task, so we only have the possibility to conjecture the strengths and weaknesses of MM-LLMs on our task from the performance of MiniGPT-4. We expect that our work will advance the performance of MM-LLMs on scenario-level tasks. The proposed DDN task in the field of visual navigation for finding objects that satisfy human demands. To the best of our knowledge, there are no observable adverse effects on society.
Acknowledgments and Disclosure of Funding
This work was supported by National Natural Science Foundation of China General Program (62376006) and The National Youth Talent Support Program (820080008)) We also thank Benjamin Redhead for correcting and polishing the grammar and wording of our paper.
References
[1] F. Zhu, X. Liang, Y. Zhu, Q. Yu, X. Chang, and X. Liang, "Soon: Scenario oriented object navigation with graph Computer Vision and Pattern Recognition, 2021, pp. 12 689-12699,
graph-based based exploration." in Proceedings of the IEEE/CVF Conference on
[2] A. Staroverov, D. A. Yudin, I. Belkin, V. Adeshkin, Y. K. Solomentsev, and A. I. Panov, "Real- time object navigation with deep neural networks and hierarchical reinforcement learning," " IEEE Access, vol. 8, pp. 195 605-195688 621, 2020.
[3] D. S. Chaplot, D. P. Gandhi, A. Gupta, and R. R. Salakhutdinov, "Object goal navigation using goal-oriented semantic exploration," Advances in Neural Information Processing Systems, vol. 33, pp.
4258, 2020.
[4] H. Du, X. Yu, and I.. Zheng, "Learning object relation graph and tentative policy for visual navigation," " in Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VII 16. Springer, 2020, pp. 19-34.
[5] S. Y. Gadre, M. Wortsman, G. Ilharco, L. Schmidt, and S. Song, "Clip on wheels: Zero-shot object navigation as object localization and exploration," arXiv preprint arXiv:2203. 10421, 2022.
10

---

[6] H. Du, X. Yu, and L. Zheng, "Viet: Visual transformer network for object goal navigation," in International Conference on Learning Representations.
[7] A. Pal, Y. Qiu, and H. Christensen, "Learning hierarchical relationships for object- naviga- tion," Conference on Robot Learning. PMIR, 2021, pp. 517-528.
[8] Y. Qiu, A. Pal, and H. I. Christensen, "Learning hierarchical relationships for object-goat navigation." arXiv preprint arXiv:2003.n 067499 2020.
[9] S. K. Ramakrishnan, D. S. Chaplot, Z. Al-Halah, J. Malik, and K. Grauman, "Poni: Poten- tial functions for objectgoal navigation with interaction-free learning," " in Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, 2022, pp. 18 890-18 900.
[10] Wani, S. Patel, U. Jain, A. Chang, and M. Savva, "Multion: Benchmarking semantic map memory using multi-object navigation," Advances in Neural Information Processing Systems, vol. 33, pp. 9700-9712, 2020.
[11] A. Majumdar, G. Aggarwal, B. S. Devnani, J. Hoffman, and D. Batra, "Zson: Zero-shot object-goal navigation using multimodal goal embeddings," in Advances in Neural Information Processing Systems.
[12] Q Zhao, L. Zhang, B. Hc. H. Qiao, and Z. Liu, "Zero-shot object goal visual navigation," " arXiv preprint Xiiv:2206.074233 2022.
[13] W. Yang, X. Wang. A. Farhadi, A. Gupta, and R. Mottaghi, "Visual semantic navigation using scene priors," in 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. [Online]. Available: http:///weeeeiiieeeeee/uumo/aaea
[14] D. S. Kiran, K. Anand, C. Kharyal, G. Kumar, N. Gireesh, S. Banerjee, R. D. Roychoudhury, M. Sridharan, B. Bhowmick, and M. Krishna, "Spatial relation graph and graph convolutional network for object goal navigation," in 2022 IEEE 18th International Conference on Automation Science and Engineering (CASE). IEEE, 2022, pp. 1392-1398.
[15] J. Guo, Z. LU, T. Wang, W. Huang, and H. Liu, "Object goal visual navigation using semantic spatial relationships." in Artificial Intelligence: First CAAI International Conference, CICAI 2021, Hangzhou, China, June 5-6, 2021, Proceedings, Part I. Springer, 2022, pp. 77-88.
[16] E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, D. Gordon, Y. Zhu, A. Gupta, and A. Farhadi, "AI2-THOR: An Interactive 3D Environment for Visual AI," arXiv, 2017.
[17] M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik, D. Parikh, and D. Batra, "Habitat: A Platform for Embodied AI Research," in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
[18] A. Szot, A. Clegg. E. Undersander, E. Wijmans, Y. Zhao, Turner, N. Maestre, M. Mukadam, D. Chaplot, O. Maksymets, A. Gokaslan, V. Vondrus, S. Dharur, F. Meier, W. Galuba, A. Chang, Z. Kira, V. Koltun, J. Malik, M. Savva, and Batra, "Habitat 2.0: Training home assistants to rearrange their habitat," in Advances in Neural Information Processing Systems (NeurIPS), 2021.
[19] F. B. Goldstein, Cognitive psychology: Connecting mind, research and everyday experience. Cengage Learning, 2014.
[20] J. Kuhl, "Motivation and information processing: A new look at decision making, dynamic change, and action control." 1986.
[21] M. Van Der Meer, 7. Kurth-Nelson, and A. D. Redish, "Information processing in decision- making systems," " The Neuroscientist, vol. 18, no. 4, pp. 342-359, 2012.
[22] J. S. Brown, Motivation of behavior. Prabhat Prakashan, 1961.

---

[23] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., "Language models are few-shot learners," Advances in neural information processing systems, vol. 33, pp. 1877-1901, 2020.
[24] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., "Learning transferable visual models from natural language supervi- sion," " in International conference on machine learning. PMLR, 2021, pp. 7748-8763
[25] M. Deitke, E. VanderBilt, A. Herrasti, L. Weihs, J. Salvador, K. Ehsani, W. Han, E. Kolve, A. Farhadi, A. Kembhavi, and R. Mottaghi, "ProeTHOR: Large-Scale Embodied Al Using Procedural Generation," in NeurlPS, 2022, outstanding Paper Award.
[26] D. Zhu, Chen, X. Shen, X. Li, and M. Elhoseiny, "Minigpt-4: Enhancing vision-language understanding with advanced large language models," arXiv preprint arXiv:2304. /0592, 2023.
[27] KETI, C. M. Kim, K. Goldberg. A. Kanazawa, and M. Tancik, "Lerf: Language embedded radiance fields," arXiv preprint arXiv:2303.00 0.5553 2023.
[28] R. Ding. J. Yang, C. Xue, W. Zhang, S. Bai, and X. Qi, "Pla: Language-driven open-vocabulary 3d scene understanding." in Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 7010-7019.
[29] C. Huang, O. Mees, A. Zeng, and W. Burgard, "Visual language maps for robot navigation," in 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023, pp.
[30] Y. Qi, Z. Pan, Zhang, A. van den Hengel, and Q Wu, "Object-and-action aware model for visual language navigation." in Computer Vision-ECCV ECCV 2020: 16th European Conference, Glasgow, UK, August 23 23-28, 2020, Proceedings, Part 16. Springer, 2020, pp. 303-317.
[31] A. Pashevich, C. Schmid, and C. Sun, "Episodic transformer for vision-and and---nnd-a- land-la navigation," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 15 942-159999 952.
[32] R. Hu, D. Fried, A. Rohrbach, D. Klein, T. Darrell, and K. Saenko, "Are you looking? grounding to multiple modalities in dision-and-languaal navigation," in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019, pp. 6551-6557.
[33] A. Majumdar, A. Shrivastava, S. Lee, P. Anderson, D. Parikh, and D. Batra, "Improving vision- and-language navigation with image-text pairs from the web," in Computer Vision-ECCO ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VI 16. Springer, 2020, pp. 259-274.
[34] P.-L. Guhur, M. Tapaswi, S. Chen, I. Laptev, and C. Schmid, "Airbert: In-domain pretraining for vision-and-language navigation." " in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 1634-1643.
[35] C.WW. Kuo, C.-Y. Ma, Hoffman, and Kira, "Structure-encodiee auxiliary tasks for improved visual representation in vision-and-langaae navigation," in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2023, pp. 1104-1113.
[36] H. Huang, V. Jain, H. Mehta, A. Ku, G. Magalhaes, J. Baldridge, and E. Ie, "Transferable representation learning in vision-and-language navigation." " in Proceedings of the IEEE/CVF International conference Computer Vision, 2019, pp. 7404-7413.
[37] D. Shah, B. Osiński, Levine el al., "Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action," in Conference on Robol Learning. PMLR, 2023, pp. 492-504.
[38] P. Chen, J. K. Lin, R. Zeng, T. Li, M. Tan, and C. Gan, map learning for vision-and- languuuge navigation," Advances in Neural Information Processing Systems, vol. 35, pp. 38 149-38 161, 2022.
gulti-granularity
12

---

[39] C. Chen, U. Jain, C. Schissler, S. V. A. Gari, Z. Al-Halah, V. K. Ithapu, P. Robinson, and K. Grauman, "Soundspaces: Audio-visual navigation in 3d environments. in Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VI 16. Springer, 2020, pp. 17-36.
[40] C. Chen, S. Majumder, Z. Al-Halah, R. Gao, K. Ramakrishhan, and K. Grauman, "Learning to set waypoints for audio-visual navigation," arXiv preprint arXiv:2008. 09662,, 2020.
[41] C. Chen, Z. Al-Halah, and K. Grauman, "Semantic audio-visual navigation." in Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2021, pp. 15 516 15 525.
[42] C. Gan, Y. Zhang, J. Wu, B. Gong, and J. B. Tenenbaum, "Look, listen, and act: Towards audio-visual embodied navigation," in 2020 JEEE International Conference on Robotics and Automation (ICRA). IEEE, 2020, pp. 9701-9707.
[43] Y. Yu, W. Huang, Sun, C. Chen, Y. Wang, and Liu, "Sound adversarial audio-visual navigation." arXiv preprint arXiv:2202.10%0 2022.
[44] H. Wang, Y. Wang, F. Zhong, M. Wu, J. Zhang, Y. Wang, and H. Dong, "Learning semantic- agnostic and spatial-aware representation for generalizable visual-audin navigation," IEEE Robotics and Automation Letters, 2023.
[45] K. Yadav, J. Krantz, R. Ramrakhya, S. K. Ramakrishnan, J. Yang, A. Wang. J. Turner, A. Gokaslan, V.-P. Berges, R. Mootaghi, Maksymets, A. X. Chang, M. Savva, A. Clegg, D. Chaplot, and D. Batra, "Habitat challenge 2023," https:/ahhbbiia..ggdalleeggg//// 2023.
[46] OpenAI, "Gpt-4 technical report," 2023.
[47] J. D. M.-W. C. Kenton and L. K. Toutanova, "Bert: Pre-training of deep bidirectional transform- ers for language understanding." in Proceedings of NAACL-HLT, 2019, pp. 4171-4186.
[48] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann el al., "Palm: Scaling language modeling with pathways," arXiv preprint arXiv:2204.02311 2022.
[49] R. Thoppilan, D. De Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H.-T. Cheng, A. Jin, T. Bos. L. Baker, Y. Du et al., "Lamda; Language models for dialog upplications," arXiv preprint arXiv:2201.08239 2022.
[50] Z. Wang. S. Cai, A. Liu, X. Ma, and Y. Liang, "Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents," arXiv preprint arXiv:2302.01 01.30020.6600 5600, 2023.
[51] A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian et al., "Do as can, not as i say: Grounding language in robotic affordances." in Conference Robot Learning. PMLR, 2023, pp. 287-318.
[52] D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, W. Huang, Y. Chebotar, P. Sermanet, D. Duckworth, S. Levine, V. Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng, I. Mordatch, and P. Florence, "Palm-e: An embodied multimodal language model," in arXiv preprint arXiv: 2023.
[53] W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng. J. Tompson, I. Mordatch, Y. Chebotar et al., "Inner monologue: Embodied reasoning through planning with language models," in 6th Annual Conference on Robot Learning.
[54] I. Singh, V. Blukis. A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg, "Progprompt: Generating situated robot task plans using large language models," in Workshop on Language and Robotics at CoRL. 2022.
[55] D. Shah, B. Eysenbach, G. Kahn, N. Rhinehart, and S. Levine, "Ving: Learning open-world nav- igation with visual goals," " 2021 IEEF International Conference on Robotics and Automation (ICRA). IEEE, 2021, pp. 13 13225-13222.
13

---

[56] H. Rezatofighi, N. Tsoi, J. Gwak, A. Sadeghian, I. Reid, and S. Savarese, "Generalized in- tersection over union: A metric and a loss for bounding box regression." " 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2019, pp. 658-666.
[57] J. Straaub T. Whelan, L. Ma, Y. Chen, E. Wijmans, S. Green, J. J. Engel. R. Mur-Artal, C. Ren, S. Verma, A. Clarkson, M. Yan, B. Budge, Y. Yan, X. Pan, J. Yon, Y. Zou. K. Leon, N. Carter, J. Briales, T. Gillingham, E. Mueggler, L. Pesqueira, M. Savva, D. Batra, H. M. Strasdat, R. D. Nardi, M. Goesele, S. Lovegrove, and R. Newcombe, "The Replica dataset: digital replica of indoor spaces," arXiv preprint
2019.
[58] A. d. Oord, Y. Li, and O. Vinyals, "Representation learning with contrastive predictive coding," arxiv preprini arXiv:1807.0888 1807.03748, 2018.
[59] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention all you need," Advances in neural information processing systems, vol. 30, 2017.
[60] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, "End-to-end object detection with transformers," " in Computer FisvonEECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16. Springer, 2020, pp. 213-229.
[61] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., "An image is worth 16x16 words: Transformers for image recognition at scale," arXiv preprint arXiv:2010.11 11900 2020.
[62] K. He, X. Chen, Xie, Li, P. Dollár, and R. Girshick, "Masked autoencoders are scalable vision learners," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. pp. 16 000-16009,
[63] F. Hart, J. Nilsson, and B. Raphael, "A formal basis for the heuristic determination of minimum cost paths," IEEF Transactions on Systems Science and Cybernetics, vol. 4, no. 2, pp. 100-107, 1968.
[64] K. He, X. Zhang, S. Ren, and Sun, "Deep residual learning for image recognition." " in 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770-778.
[65] V. S. Dorbala, G. Sigurdsson, R. Piramuthu, Thomason, and G. S. Sukhatme, "Clip-nav: Using clip for zero-shot vision-and--anguage navigation," arXiv preprint arXiv:2211.1666 2022.
[66] B. Yamauchi, "A frontier-based approach for autonomous exploration," in Proceedings 1997 IEEE International Symposium on Computational Intelligence in Robotics and Automation CIRA '97. "Towards New Computational Principles for Roboties and Automation', 1997, pp- 146-151.
[67] P. Anderson, A. Chang, D. S. Chaplot, A. Dosovitskiy, S. Gupta, V. Koltun, J. Kosecka, J. Malik, R. Mottaghi, M. Savva et al., "On evaluation of embodied navigation agents," arXiv preprint arXiv: 2018.
[68] A. Khandelwal, L. Weihs, R. Mottaghi, and A. Kembhavi, "Simple but effective: Clip embed- dings for embodied ai," in Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 14 829-14
[69] L. Van der Maaten and G. Hinton, "Visualizing data using t-sne." Journal of machine learning research, vol. 9, no. 11, 2008.
[70] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, "Proximal policy optimization algorithms," arXiv preprint arXiv: 1707 770.006347 2017.
14


Reviews:
[{"review_id": "WDkxZlPXU3", "reviewer": "Reviewer_JLnR", "summary": "This paper proposes the novel task of demand-driven navigation, where a robot must navigate towards a goal object that satisfies the human user's demand (e.g., for the demand \"I am thirsty\", the robot has to find water/juice/tea, etc.) Citing limitations of navigation methods used for other object-goal navigation variants, the paper also proposes a novel architecture that relies on extracting attribute features conditioned on the demand text features and object features for visible objects. The attribute features extracted are indicative of what physical / semantic properties are fulfilled by a given object to satisfy the demand (e.g., the object \"water bottle\" has the property to \"quench thirst\"). These features are learned using a contrastive learning objective on prior-knowledge extracted from a large-language model (GPT-3). Experiments on the ProcThor dataset demonstrate the difficulty of the demand-driven navigation task and the superiority of the proposed method over prior navigation methods.", "questions": "I'd appreciate it if the authors can clarify the questions raised in weaknesses. It will help me arrive at my final decision.", "limitations": "Yes, the limitations are discussed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The idea of demand-driven navigation is interesting and novel. While the goal specification is natural language (similar to prior work), it focuses on a very different concept of \"demand\" where multiple functionally equivalent objects can satisfy a given demand.\n* The problem statement is well motivated and the paper writing clarity is largely good (see weaknesses for some questions).\n* The supplementary material provides necessary implementation details and details about the dataset for reproducibility.\n* The experimental design is good. Recent baselines for open and closed vocabulary navigation are considered. Multiple experimental trials have been performed to show statistical significance. The proposed method significantly outperforms the baselines.", "weaknesses": "# Post-rebuttal update\n\nI thank the authors for addressing my questions and concerns in the rebuttal. It is clear to me that this paper presents a valuable new direction in this space of navigation tasks and the experiments are sufficiently strong to recommend acceptance. The motivation can be clarified further and the rebuttal responses need to be reflected in the final paper. With the understanding that this will be done, I am increasing my rating to accept (7).\n\n---------------------------------------------------------------\n\n## Task motivation good, but practical implications are not clear\n\nI liked the task itself, but a key question that concerns me is *\"how often does an object demanded not be present in the scene, and therefore, a functionally equivalent object was needed to satisfy the demand?\"*. That is, how often do demands for objects become infeasible because the object was missing in the scene? E.g., if the demand is \"Get me a water bottle because I am thirsty\", I would expect most scenes would contain water bottles. The task definition avoids addressing this issue by having a generic demand itself as the input.\n\nOf course, it is possible that there are other objects that can be functionally equivalent **in addition** to the object demanded, but that's not the scenario motivated in the introduction. Additionally, as a user, if I request a water bottle, that's exactly what I'd want unless it is nowhere to be found. So the proposed task makes more sense in the absence of the primary object.\n\n\n## Dataset is underwhelming\n\nL63 - 64 - \"mapping between demands and objects is many-to-many\" ---This is only partly true. Based on statistics in Figure 4 supp., only 2.3 objects, on average, correspond to a given instruction. Calling this \"many\" is underwhelming. The other direction is still true though, i.e., there are many instructions satisfied by a given object (Figure 5 supp).\n\n## Task definition fails to consider whether an object instance satisfies the demand\nL89 - \"Both a bottle of water and a cup of tea\" can \"quench thirst\" --- this is only true if the bottle has water and cup is filled with tea. Does the dataset / task differentiate between bottle/cup instances that contain liquids vs. those that are empty?\n\n## Approach clarifications needed\n* L175 - \"only take RGB images as sensor inputs\" --- is the GPS+compass information also included here or does the model learn to localize on its own?\n* L185 - why is the time step limit only 100? That seems very short for navigation in large environments.\n* L257 - why is the attribute module a transformer (e.g., why not just an MLP)? Is self-attention across demand-object features needed?\n* L281 - why use only imitation learning and not reinforcement learning?\n\n## Experiment clarifications needed\n* Table 1 - why is the performance on seen scene, seen instruction so low? I'd expect close to 100% success due to overfitting.\n* L306 - how are the object categories derived from demand inputs for the *-object methods?\n* L317 - why are there variants of \"Random\" if the policy only selects actions randomly? \n* L336 - 338 - \"likely to be due to the fact that ... have a high likelihood not to be present ... meaningless search\" --- can we empirically quantify this? How often is it the case that objects predicted by GPT-3 are missing in the scene?\n* L371 - 373 - \"surpasses all baselines ... CLIP-visual features helps ...\" - it seems to me an unfair advantage to use CLIP visual features only for the proposed method and not the baselines. How do baselines perform when equipped with CLIP features?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "REPLY: Thank you! We have clarified some questions, address your concerns, and hope to hear back from you if you have further questions!\n\n**Q1**: how often does an object demanded not be present in the scene, and therefore, a functionally equivalent object was needed to satisfy the demand? \n\nA1: Thank you for pointing this out! We carefully analyzed the scenario you pointed out, wherein the Procthor contains multiple objects (e.g., objects A, B, and C) that fulfill the demand instruction, but not all of them are simultaneously present within a specific scene. Our statistical calculations revealed a probability of 54.4% for this situation to occur. Correspondingly, there is a 45.6% chance that all objects satisfying the demand instruction are indeed present in the scene.\n\n**Q2**: Additionally, as a user, if I request a water bottle, that's exactly what I'd want unless it is nowhere to be found.\n\nA2: Thank you very much for your valuable suggestion. Due to Rebuttal character limitations, we put the relevant discussions into **Common Response 3**. we would appreciate it if you could read them.\n\n**Q3**: Dataset is underwhelming. only 2.3 objects, on average, correspond to a given instruction\n\nA3: Thank you very much for pointing this out. We argue that the nature of DDN tasks is many-to-many. We acknowledge that the limited number of object categories available in the ProcThor dataset might have influenced the presentation of this nature, resulting in an average of only 2.3 object categories per instruction being fulfilled. In real life, a demand can be satisfied by a much larger number of object categories, for example, we can easily generate 10 different categories of objects for each instruction using GPT-3 ( please see language-grounding mappings in supp 8.2.1). In the future work, we will focus on more diverse scenes and objects to generate a more complex DDN dataset.\n\n**Q4**: Task definition fails to consider whether an object instance satisfies the demand. Does the dataset / task differentiate between bottle/cup instances that contain liquids vs. those that are empty?\n\nA4: Thank you for raising this important point. Yes, the dataset differentiates them.  For instance, in the demand \"I am thirsty,\" the required object is \"water,\" rather than specifying a particular container like \"bottle\" or \"cup.\" On the other hand, in the demand \"I need a container to hold water,\" the required object is explicitly defined as \"bottle.\" We can identify them using the simulator's metadata easily.\n\n**Q5**: Approach clarifications\n\nL175:\n\nNo, because many previous visual navigation works have used depth and GPS, we want to emphasize that we only used RGB.\n\nL185:\n\nOur decision to set the time step limit to 100 is based on the analysis of our traject dataset. The dataset, collected using the A* algorithm, revealed that 90% of the trajectory lengths are less than 50 and the average length is 27.24. Consequently, we opted for a time step limit that is twice times longer than 50.\n\nL257:\n\nThank you for your insightful question and valuable suggestions. We chose to implement the attribute module as a transformer because transformers have demonstrated remarkable effectiveness in various domains, such as NLP and computer vision. \nTo show the transformer's ability for learning attribute features, we added experiments by replacing the transformer with a 4-layer MLP in the attribute module.  **Due to Rubuttal character limitations, the results of the experiment are shown in the attached PDF in Common Response.** The results clearly showed that the MLP was not superior to the transformer in learning attribute features. Thus, we found that self-attention across demand-object features, provided by the transformer, is essential for achieving optimal performance in our task.\n\nL281: \n\nThank you for your insightful suggestion! We agree that integrating our method with reinforcement learning (RL) could potentially lead to further performance improvements, conceptually. However, due to the size of our model (even several times larger than VTN), the reward signal from RL turns out to be weak, rendering RL less effective for our method. Consequently, we initially focused on exploring an imitation learning (IL)-based method, which yielded significant performance gains over baselines. Nonetheless, we recognize the value and promise of integrating RL into our method as a valuable future direction for further research.\n\n**Q6**:Experiment clarifications\n\nTable 1:\n\nThe performance that is not 100% on seen scenes and instructions can be attributed to the limited size of our trajectory dataset. We only collect up to 3 trajectories with different initial positions for each instruction and each room. Since each room contains hundreds or even thousands of initial positions, each corresponding to different final objects found, the trajectory dataset we collect is relatively small compared to the vast number of all possible trajectories, even less than 1%. This limited dataset size makes it challenging for our model to overfit to the specific seen scenes and instructions, resulting in performance below the expected 100% success rate. \n\nL306: \n\nThe object categories are obtained by asking GPT-3 for the current demand instruction and letting GPT-3 provide the categories of objects that fulfill the demand. Subsequently, we convert the GPT-3 generated answers into formats acceptable by VTN and ZSON, respectively, using different methods. For detailed information on the conversion process, please refer to the supplementary materials Section 8.3.2.\n\nL317:\n\nPlease see **Common Response 2**.\n\nL336-338: \n\nAs we described in Q1, there is a 54.4% probability that the object given by GPT-3 does not exist in the current scene, and there are other objects that satisfy the given demand instruction.\n\nL371-373:\n\nThank you for your suggestion. We added the experiments VTN+CLIP-demand. The results are in the attached PDF file in **Common Response**. \n\n"}, {"review_id": "0hdnI8rQpf", "reviewer": "Reviewer_nwVS", "summary": "This paper proposes a Demand-driven Navigation (DDN) problem to leverages the user’s demand as the task instruction and prompts the agent to find an object which matches the specified demand. Then the authors proposed a method by learning demand-conditioned object attribute features from LLMs and align them to visual navigation via CLIP. The experiment shows the efficiency of the proposed method. However, I have some concerns about this paper. My detailed comments are as follows.", "questions": "My main concerns are the lack of an important baseline and the analysis of the experimental results.", "limitations": "The authors have adequately addressed the limitations", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes a novel Demand-Driven Navigation task to explore the navigation with only the user’s demand as the task instruction. This task is practical and worth more research, especially with the development of open-vocabulary foundation models.\n\n2. The proposed attribute module is interesting and helpful for extracting attributes of objects.\n\n3. This paper provides a method to tackle the DDN task by extracting common sense from LLMs to learn textual attribute features and uses CLIP to align the textual and visual attribute features. The results obviously outperform the baselines.\n", "weaknesses": "1.\tOne important baseline is missing. The agent could explore the environment using a heuristic algorithm like FBE. At each time step, the agent detects all objects in observation and ask LLM whether these objects can satisfy the human demand. \n\n2.\tWhat are the differences between common sense knowledge and human preferences mentioned in the paper?\n\n3. It is not clear why the results of random-object are different from random-demand. Are they both execute random actions? More explanations are needed.\n\n4.\tIn Table1, the ZSON-demand performs better the ZSON-object. Does it indicate that CLIP performs better in understanding high-level abstract demand compared to concrete objects? This result seems to conflict with the claim that “CLIP does not perform well on alignment between instructions and objects” in Line 335.\n\n5.\tSome related works that try to solve open-vocabulary navigation[1,2] or scene understanding[3] are missed. It would be better to add and discuss them in the related work part for the sake of completeness.\n\n[1] Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigation, NeurIPS 2022.\n\n[2] Visual Language Maps for Robot Navigation, ICRA 2023.\n\n[3] LERF: Language Embedded Radiance Fields, ArXiv 2023.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "REPLY: Thank you! We have clarified some questions, address your concerns, and hope to hear back from you if you have further questions!\n\n**Q1**: One important baseline is missing. The agent could explore the environment using a heuristic algorithm like FBE. At each time step, the agent detects all objects in observation and ask LLM whether these objects can satisfy the human demand.\n\nA1: Thank you very much for your suggestion. We supplement two experiments with FBE as an exploratory module and MiniGPT-4 and GPT-3 as recognition modules with the following results. Due to Rubuttal character limitations, the results are provided in the attached PDF file in **Common Response**. Since ProcThor is a large scene consisting of multiple rooms, the results show that heuristic search is not efficient. Our model structure mimics VTN, a structure that can learn associations between objects, and learns the attribute features of the objects with Contrastive Learning, which in a way improves the search efficiency by using the semantics of the objects. \n\n**Q2**: What are the differences between common sense knowledge and human preferences mentioned in the paper?\n\nA2: Thank you for your valuable commont. In the context of the paper, common sense knowledge refers to the general knowledge and understanding that humans possess about the world and its functioning. This knowledge includes basic facts and principles that are commonly accepted and expected in everyday life. On the other hand, human preferences refer to the individual preferences, desires, and choices that vary from person to person. These preferences can be influenced by personal experiences, cultural background, and subjective judgments, leading to variations in how individuals perceive and prioritize different options. In our DDN dataset, we did not explicitly express personal preferences, but instead added some modifiers to further specify the target category and reflect personal preferences, such as \"I want a place to rest\" vs \"I want a soft place to rest.\"\n\n**Q3**: It is not clear why the results of random-object are different from random-demand. Are they both execute random actions? More explanations are needed.\n\nA3: Thank you for pointing that out. Indeed, both random-object and random-demand baselines execute actions randomly from the action space provided. However, the key difference lies in the tasks assigned to them and the criteria used to evaluate success, leading to varied results. For random-object, the baseline is given **a specific category of objects** and is tasked with finding an object of that category within the environment. On the other hand, random-demand is given **a demand instruction** and is asked to find an object that satisfies that particular demand. In some scenarios, a single demand instruction may be satisfied by **multiple categories of objects** present in the environment. Due to this broader range of possible successful outcomes, the success criteria for random-demand are more relaxed compared to random-object, resulting in higher success rates for random-demand.\n\n**Q4**: In Table1, the ZSON-demand performs better the ZSON-object. Does it indicate that CLIP performs better in understanding high-level abstract demand compared to concrete objects? This result seems to conflict with the claim that “CLIP does not perform well on alignment between instructions and objects” in Line 335.\n\nA4: Thank you for raising this concern. It is essential to note that ZSON-demand and ZSON-object represent different task settings. In ZSON-demand, the objective is to find the category of objects that fulfills a given demand instruction, where multiple object categories in the scenes may satisfy the demand. In contrast, ZSON-object entails locating a specific object category. The difference in task settings makes it challenging to draw direct comparisons between ZSON-demand and ZSON-object performance.\n\nRegarding why we claim that CLIP does not perform well on alignment between instructions and objects, we need to first explain the difference in testing and task input between ZSON-object and ZSON-demand. Both ZSON-object and ZSON-demand were trained using Image Navigation (task input is CLIP-vision-feature of the target image), but for testing, ZSON-object was tested with Object Navigation (with the task input being CLIP's text feature of the target object's name), while ZSON-demand was tested with the DDN task (with the task input being CLIP's text feature of the demand instruction). ZSON 's motivation is to rely on CLIP's ability to align between text and vision to accomplish zero-shot object navigation. This alignment ability is reflected in the cosine similarity of the features.The average cosine similarity between **object image features** and **object name features** in ZSON-Object is 0.28; whereas the average cosine similarity between **object image features** and **instruction features** in ZSON-demand is only 0.22. So we argue that the alignment between instructions and objects in DDN task is not as good as in Object Navigation task. \n\n**Q5**: Related work\n\nA5: Many thanks for the papers you have provided. We think these papers are very relevant to our work, so we will add them all to the related work section and discuss them. \n\n"}, {"review_id": "OVZlQKHftD", "reviewer": "Reviewer_7yMw", "summary": "This paper introduces a new task called Demand-Driven Navigation (DDN) that, unlike previous Visual Object Navigation (VON) tasks that evaluate the ability of an agent to find a specific object in an unknown environment, considers fulfilling the demand of a human. This new task is motivated by the lack of real-world grounding of current VON tasks that either require an agent to find an instance of an object category from a pre-defined fixed vocabulary or a language-specified object in an open-vocabulary fashion. However, in a real environment, a specific object might not be present or, if thinking from the point of view of fulfilling a human’s demand, other objects might be equally as good as the targeted one. As a result, authors suggest querying an autonomous agent with a language demand instruction. This allows one to be more flexible with respect to environments but also requires common sense knowledge, an understanding of how objects can be used, and where they are likely to be located in a scene. \n\nThe paper evaluates several baselines inspired by previous literature to show the introduced task is hard and cannot be solved with currently known methods. A new approach to solving the DDN task is thus introduced in this paper. The main goal is to learn a mapping between human demands and attributes of objects that can fulfill them. At training time, a GPT-3 model is thus used to generate a series of demands and objects that can meet each of them. Each demand is encoded by a BERT model and each object is encoded by a CLIP text encoder. For each demand, a demand-object vector is created by concatenating the demand representation and an object representation. An attribute module takes the demand-object vector as input and is trained with contrastive learning to extract representations that are as close as possible for different pairs sharing the same demand. This attribute module is then used in the final policy that is composed of a Transformer model and is trained with imitation learning. The paper shows this new method outperforms other baselines.", "questions": "Four questions (1.-4.) were already asked in the \"Weaknesses\" section. I would like authors to address these concerns. I also add two additional questions (5., 6.), that do not appear to me as paper weaknesses but rather required clarifications:\n\n1.-4. See \"Weaknesses\" section.\n\n5. [Major] When describing the *Random* baseline, authors say it is about randomly selecting an action in the action space. I thus do not understand the difference between *Random-object* and *Random-demand*. Further clarifications are needed.\n6. [Major] Performance for *GPT-3+Prompt** and *MiniGPT-4* in Table 1 is always the same independently from the scene (seen/unseen) and instructions (seen/unseen). This should either be corrected or explained.", "limitations": "The paper mentions limitations regarding the drawn conclusions about the performance of current language models on the DDN tasks. Authors explain they did not have access to the code for recent methods such as GPT-4 with visual inputs or PaLM-E, and thus were not able to evaluate these methods. It is a good thing to mention this limitation, but their experiments were already conducted with many recent models, which might still allow them to draw relevant conclusions (see “Weaknesses” section for remarks about the drawn conclusions, which are however orthogonal to the mentioned limitations in this section).\n", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The introduced task is very interesting: querying an agent with a human demand seems more aligned with what is needed when deploying a robot in a real environment to help humans. Authors have properly motivated the caveats of current VON tasks in the literature.\n2. The extraction of object attributes seems very relevant and well-motivated. Learning attributes is indeed a way to guide the learning of common sense knowledge and experiments showcase the gain in performance it allows to reach.\n3. Authors propose diverse baselines to evaluate the performance of current language models in robotics/navigation tasks.", "weaknesses": "1. [Major] The introduced method is composed of many different modules and pre-trained models (GPT-3, BERT, CLIP text encoder, CLIP vision encoder, Attribute Module, DETR, Image Encoder, Policy Transformer, Visual Grounding model). It is not clear what parts are the most important and whether the overall method could maintain the same performance without some of these building blocks. Additional ablation studies would be very interesting.\n2. [Major] Most baselines except *Ours* showcase very low and close performance. However, section 6.3 discussing the experimental results is quite long and detailed. I am not convinced authors can draw as many conclusions as they do when we consider how close all baselines are in terms of average performance (even more when considering the standard deviation).\n3. [Major] This remark is very related to the previous one. All concurrent baselines reach very low performance. Could it be that these methods were not trained to convergence or would simply require much more training? The authors mention 1.8M training frames in the paper, which seems rather small compared with the number of frames required to train baselines in other navigation tasks (generally closer to 100M training frames). \n4. [Minor] When presenting baselines in section 6.2, a lot of information is missing. The paper refers to details given in the supplementary material. When reading this supplementary information, we can understand the introduced baselines. I would suggest including this information in the main paper directly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "REPLY: Thank you! We have clarified some questions, address your concerns, and hope to hear back from you if you have further questions!\n\n**Q1**: Additional ablation.\n\nA1: Thank you very much for your advice. We show ablation experiments for pre-training of attribute modules in the main paper. Now we additionally add two ablation experiments on BERT and Attribute Module's network: replacing BERT with MLP and AttributeTransformer with a 4-layer MLP, and the results of the experiments are shown **in the attched PDF file in Common Response**. The results reveal that BERT features are essential for learning a robust navigation policy, especially on unseen instructions. The ablation results on Attribute Module's network show that the transformer outperforms the MLP in learning attribute features.\n\nSince we need to rely on CLIP's ability to align text and vision, we cannot perform ablation experiments on CLIP. DETR, VG Model and GPT-3 are just tools we use to make the pipeline complete, they can be replaced by any model with similar functionality. For example, DETR can be replaced by Faster-RCNN, VG model can be replaced by SAM. We regret not being able to provide ablation experiments on the policy transformer and image encoder due to computational and time constraints during the rebuttal period. However, our network follows the structure of VTN, where you may find some related ablation experiments.\n\n**Q2**: I am not convinced authors can draw as many conclusions.\n\nA2: Thank you for bringing up this concern. Our conclusions were derived from a comprehensive analysis of the agent's trajectory and decision-making data. We acknowledge that we fell short in providing this crucial information and apologize for the oversight.\n\nTo explain the claim in L334-335, the difference between ZSON-object and ZSON-demand needs to be explained. Both ZSON-object and ZSON-demand were trained using Image Navigation (task input is CLIP-vision-feature of the target image), but for testing, ZSON-object was tested with Object Navigation (with the task input being CLIP's text feature of the target object's name), while ZSON-demand was tested with the DDN task (with the task input being CLIP's text feature of the demand instruction). ZSON 's motivation is to rely on CLIP's ability to align between text and vision to accomplish zero-shot object navigation. This alignment ability is reflected in the cosine similarity of the features.The average cosine similarity between object image features and object name features in ZSON-Object is 0.28; whereas the average cosine similarity between object image features and instruction features in ZSON-demand is only 0.22. So we argue that the alignment between instructions and objects is not good.\n\nOur contention in L336-338 is rooted in rigorous statistical analysis. Our findings indicate a 54.4% probability that the object suggested by GPT-3 may not even exist within the current environment.\n\nThe reason we claim in L343-345 is that we counted the distribution of GPT-3+Prompt's actions and the number of the episodes that exceed the step limit, and found that 66.41% of the steps were spent rotating in place or adjusting the camera, whereas these actions accounted for only 32.41% of the expert's data in our traject dataset; and also that 80% of its episodes of failure were due to exceeding the 100-step limit. \n\nRegarding MiniGPT-4, we statistically obtained that \"MoveAhead\" accounts for only 15% of the total number of actions, while rotating and adjusting the camera in place accounts for 66% (\"MoveAhead\" accounts for 66.41% and rotating and adjusting the camera in place accounts for 32.41% in the expert trajectories); and the average episode length of MiniGPT-4 is 4.38 (the average trajectory length of the expert trajectories is 27.24). This suggests that MiniGPT-4 does not tend to move around looking for objects, but rather observes in place and then quickly decides on the target object to end the episode.\n\n**Q3**: Could it be that these methods were not trained to convergence or would simply require much more training?\n\nA3: We set up the validation set for model selection (picking the model with the highest NSR on validation). We found that long before the 1.8M step, the baselines' performance on the validation set has been decreasing. After trading off training time and computational resources, we chose 1.8M as the training step size for RL. While further training might yield improvements, we argue that the chosen step size provided a reasonable balance on computing resources, time, and performance.\n\n**Q4**:  I would suggest including this information in the main paper directly.\n\nA4: Thank you for your suggestion. We will make the necessary changes to the main paper to include a more detailed baseline description. Additionally, we will provide a comprehensive account of the training and testing processes, with other essential information, in the supplementary material. \n\n**Q5**: the difference between Random-object and Random-demand.\n\nA5: Due to the characters limitation, please see **Common Response 2**.\n\n**Q6**: Performance for GPT-3+Prompt* and MiniGPT-4 in Table 1  \n\nA6: Thanks for pointing that out. The results for GPT-3+Prompt* and MiniGPT-4 in Table 1 being the same across all scene and instruction settings can be attributed to several factors. Firstly, both GPT-3 and MiniGPT-4 were not  trained on any scene or instruction. As a result, there is no distinction between seen and unseen scenes/instructions for these models.  Secondly, to ensure the reliability of the results, we conducted extensive testing with thousands of episodes using MiniGPT-4 and GPT-3. Over the course of testing, the results stabilized across all four settings, leading to consistent performance values. Lastly, when presenting the results in the table, we rounded the values, which might lead to data that appears the same in the table but might differ at multiple decimal places.\n\n\n"}, {"review_id": "cbTdabmPzm", "reviewer": "Reviewer_i2zg", "summary": "This paper presents a new visual navigation setting, where the goal is not specified by objects or images but described by a sentence. The sentence encodes the essential information to search for specific objects during navigation. Different from VLN, the task is able to analyze the demand within each sentence rather than step-by-step language guidance. Common sense and knowledge will be explicitly extracted from a large language model. To align the visual and attribute features, CLIP is employed. Overall, the motivation of this work is reasonable and interesting. \n\n# Post rebuttal\nThe authors have addressed most of my concerns. If the authors can provide some visualization results, that would be more convincing. Therefore I changed my score to accept.", "questions": "My questions mainly focus on two tasks:\n\nIn this comparisons, the results of some baseline methods are significantly lower than the results reported by their original papers. Therefore, I am wondering how the authors adapt their methods to this setting? \n\nA few works leverage CLIP for VLN or VN tasks. \nCLIP-Nav: Using CLIP for Zero-Shot Vision-and-Language Navigation\nVision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments\nThese works can be adopted for comparison.\n\nAs the searched objects have strong association with the demand, what if some demands cannot be processed properly? For example, I am thirsty but I cannot drink cold. Simply providing demands and their corresponding objects would lead to overfitting.", "limitations": "There is no negative societal impact.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The motivation of this work is interesting. Providing a demand description to an agent would enable the agent to search for not only one object in order to complete the specified goals. \n\nThe authors also provide semi-automatically generated data for this new task. This would be complementary to the existing VLN or VN tasks.", "weaknesses": "The introduction part is overly lengthy. The authors exert three pages to describe the motivations of this work, making reading quite tedious. I highly suggest the authors could trim the introduction part a bit.\n\nThe natural questions come to this task is whether the proposed method can complete object-goal navigation after training? For example, after the network is trained based on demand driven sentences, whether it can be used as a object-goal navigation agent?\n\nThe common knowledge or sense is pre-defined. In the illustrative figures, it seems a demand may correspond to three different objects. Whether this would restrict the options?\n\nIn L189, the WG mappings are different depending on the environment. I am not sure whether this implies that these WG mappings need to be specified manually. If so, this may contradict with the original motivation of this work, where humans may not know the environment in advance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "REPLY: Thank you! We have clarified some questions, address your concerns, and hope to hear back from you if you have further questions!\n\n**Q1**: The introduction part is overly lengthy.\n\nA1: Thank you for your valuable feedback. We understand your concern, and we agree that the introduction may have become lengthy due to the detailed descriptions of the new task (DDN) and the comparison with the old task (VON). To streamline the introduction, we will remove redundancies and avoid duplicating information that will be discussed in detail in later sections. Additionally, we will consider moving some of the descriptions of the DDN task to a separate section to maintain a better flow in the paper.\n\n**Q2**: The natural questions come to this task is whether the proposed method can complete object-goal navigation after training?\n\nA2: Thank you for raising this question. Theoretically, our proposed method can be utilized for object-goal navigation, by translating the target object into a demand instruction.  We use the template \"I want a $object$\" to transform an object name into the form of a demand instruction, and then test our trained DDN model in the Object Navigation setting (consistent with the setting of the -object suffix in our paper). The results of the experiment are 9.0/9.0 (SR/SPL) in seen scene and 8.5/8.3 (SR/SPL) in unseen scene. Compared to VTN-object and ZSON-object, the results demonstrate that our method also performs well on object navigation.\n\n\n**Q3** : The common knowledge or sense is pre-defined. In the illustrative figures, it seems a demand may correspond to three different objects. Whether this would restrict the options?\n\nA3: Thank you so much for pointing that out. We apologize for any confusion. To clarify, common knowledge or sense is not pre-defined; rather, it arises from human consensus and understanding of the world. In the illustrative figures, we provided an example explanation of the DDN task and our method, showing that a demand may correspond to three different objects. However, this example is not intended to limit the number of objects that can satisfy a demand. In real-life scenarios, a demand can indeed be fulfilled by numerous objects, and the number of possible options can vary significantly depending on the environments.To demonstrate the flexibility and diversity of the DDN task, we use GPT-3 to generate 10 object categories for each demand instruction in a DDN dataset, showcasing the multiple possibilities that can exist for a single demand (please refer to the description of LG mapping in the Supplementary Material 8.2.1).\n\n**Q4**: If so, this may contradict with the original motivation of this work, where humans may not know the environment in advance.\n\nA4: Thanks for pointing that out. During the training process in a simulator, we require an \"expert\" to assess whether the object found by the agent satisfies the demand or not, which allows the \"expert\" to provide the agent with correct training signals. The WG mapping serves as this \"expert\" during training.\nSince different environments can have distinct categories of objects, different WG mappings are needed for training in different environment. However, it is crucial to note that when the agent is fully trained and deployed in a real-world environment, it no longer requires WG mapping. Once a user gives a demand instruction, the agent will locate an object and present it to the user. In this case, the user does not need to know the details of the scene or the objects present; they only need to judge whether the object found by the agent satisfies their demand or not.\n\n**Q5**: Therefore, I am wondering how the authors adapt their methods to this setting?\n\nA5: Thank you for raising this question. To ensure a fair and comprehensive benchmark of the baseline methods, we employed different adaptation protocols for each one. For the -demand suffix baselines, we directly replaced the original VON input with the BERT feature of the demand instruction. The -GPT suffix baselines involve asking GPT-3 what objects satisfy a given demand instruction and then providing the answered object as input to the VON baselines. We have described the training and testing protocols for each baseline in detail in the Supplementary Material 8.3.2, providing a comprehensive explanation of how we adapted the VON method to suit the DDN task.\n\n**Q6**: VLN can be adopted for comparison.\n\nA6: Thank you for your insightful suggestion. We have taken your advice into account and included two additional baselines in our experiments. These baselines utilize CLIP-Nav as the navigation policy and GPT-3 and MiniGPT-4 as recognition policies, respectively. It is important to note that while the task instructions for vision-language navigation (VLN) are typically step-by-step, the instructions for the Demand-Driven Navigation (DDN) task revolve around the concept of \"demand\" for describing an object. As a result, we have adapted CLIP-Nav for the DDN task without utilizing its instruction breakdown. **Due to Rubuttal character limitations, the results of the experiment are shown in the attached PDF in Common Response**.\n\n**Q7** : As the searched objects have strong association with the demand, what if some demands cannot be processed properly? \n\nA7:  Thank you very much for your valuable suggestion. What you are describing is a more refined instruction, and we have some examples in our dataset, such as \"I need a place to rest\" and \"I need a soft place to rest\". The former can be \"bed, wooden chair, sofa\" while the latter cannot be \"wooden chair\" but \"bed, sofa\". In future work, we will add some explicit personal preferences as well as more complex demand instructions, including negation of object samples, combining multiple demands, and prioritising between objects.\n\n\n\n"}, {"review_id": "OhF2zQP2Pr", "reviewer": "Reviewer_FyRT", "summary": "This paper proposes Demand-driven Navigation (DDN), which leverages the user’s demand as the task instruction and prompts the agent to find an object which matches the specified demand. This paper also proposes a method of first acquiring textual attribute features of objects by extracting common sense knowledge from a large language model. These textual attribute features are subsequently aligned with visual attribute features using Contrastive Language-Image Pre-training (CLIP). They experimented on AI2Thor with the ProcThor dataset and demonstrated that the visual attribute features improve the agent’s navigation performance and outperform the baseline methods. ", "questions": "1) Can we handle a search composed with multiple demands, with the current method?\n2) Why other benchmark dataset are not used for evaluation? is there any justification?", "limitations": "The authors discussed both limitations and the societal impact.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper flows nicely with good motivation of the novel task and its challenges clearly stated.\nThe proposed method makes sense and comparison is good.", "weaknesses": "1.\tL40, the second condition is a bit questionable. I don’t think it is a necessary requirement of only searching objects that are in the scene. It is probably the case in current benchmark setup, however it is not a constraint for the research task. Subsequently, it might also impact the definition of “navigation failure” at L48. Asking the robot to search for a non-existent object in the scene and the robot could not find it. This should not be defined as a failure from my point of view. Rather, if you ask a robot to look for some non-existent object and it reports a finding, this should be counted as a failure as it is a clear false positive detection. Nevertheless, the new task DDN does not necessarily remove ‘requirement’ 2) as you might anyway asks for a demand that no object in the scene can satisfy.\n2.\tFrom L171, my impression is that the problem is constrained by only one demand d at a time. In reality, one could have multiple demands to further constrain what object they might want to search. For example, something “quench thirst” and “contains caffeine”.\n3.\tIt will be more convincing with more benchmark datasets. In addition to AI2Thor, there is Habitat Challenge on ObjectNav which can be exploited, plus it is based on 3D scan of real scenes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "REPLY: Thank you! We have clarified some questions, address your concerns, and hope to hear back from you if you have further questions!\n\n**Q1**: L40, the second condition is a bit questionable.\n\nA1: Thank you very much for your insightful suggestion. We understand the concern about the second condition in L40. In the current VON task, the robot is typically asked to find an object that is guaranteed to exist in the current scene. However, considering real-world applications, users may request the robot to find an object that they are unsure exists in the scene. In terms of satisfying the user, the robot does fail to find it, which is considered a failure. But we also agree with you that it's not really a navigation failure because the object doesn't exist in the scene; we'll revise the statement here to say  \"failure to satisfy the user's demand\" rather than \"navigation failure\".\n\n**Q2** : From L171, my impression is that the problem is constrained by only one demand d at a time. Can we handle a search composed with multiple demands, with the current method?\n\nA2: Thank you very much for your valuable suggestion. What you are describing is a more refined instruction, and we have some similar examples in our dataset, such as \"I need a place to rest\" and \"I need a soft place to rest\". The former can be \"bed, wooden chair, sofa\" while the latter cannot be \"wooden chair\" but \"bed, sofa\". The latter contains two demands in its instruction: \"soft\" and \"available for rest\".\n In future work, we will add some explicit personal preferences as well as more complex demand instructions, including negation of object samples, combining multiple demands, and prioritising between objects. Theoretically our method works for any demand expression including multiple demands, since any demand corresponds to some attributes that match the demand.\n\n**Q3**:It will be more convincing with more benchmark datasets.\n\nA3: Thank you very much for your reminder. Our dataset generation process and navigation methods can be easily migrated to any other dataset, including the scene dataset used in the Habitat Challenge. We will later generate some DDN dataset (e.g., Matterport3D) for the current mainstream scene datasets for training and testing.\n"}]

Author rebuttal:
## Common Response ##
We thank all reviewers for appreciating our DDN task, method and experiments.  "The idea of demand-driven navigation is interesting and novel." (JLnR) "The proposed attribute module is interesting and helpful for extracting attributes of objects." (nwVS) "Authors propose diverse baselines to evaluate the performance of current language models in robotics/navigation tasks." (7yMw) "The motivation of this work is interesting." (i2zg) "The paper flows nicely with good motivation of the novel task and its challenges clearly stated. The proposed method makes sense and comparison is good." (FyRT) 

**(Common Response 1)** However, we notice that some reviewers (JLnR, nwVS, 7yMw, i2zg) have some suggestions about our experiments, such as missing some important baselines and ablation experiments. We supplement with five baselines (FBE+MiniGPT-4, FBE+GPT-3, CLIP-Nav+MiniGPT-4, CLIP-Nav+GPT-3, VTN+CLIP) and two ablation experiments (Ours_w/o_BERT, Ours_w/o_Attribute_Transformer) as they suggested. The original ablation Ours_w/o_attr in the main paper is renamed Ours_w/o_Attribute_Pretrain. Due to time and computational resource constraints, we conducted only one round of each experiment. We will conduct more rounds of experiments using different random seeds if the paper is accepted. **The supplementary experimental results are shown in the attached PDF file in Common Response**. 

**(Common Response 2)** We also notice that some reviewers (JLnR, nwVS, 7yMw) have some questions on our Random baselines. The inclusion of variants of "Random" in our baselines is motivated by previous research [1,2,3], where "Random" is used as a baseline to reflect the task's difficulty.
Random-object and Random-demand are different in terms of task settings. Random-object's task is to find **a given object category** whereas Random-demand's task is to find an object that satisfies a given demand instruction. In some scenarios, a single demand instruction may be satisfied by **multiple object categories** present in the environment. Due to this broader range of possible successful outcomes, the success criteria for Random-demand are more relaxed compared to Random-object, resulting in higher success rates for random-demand. Therefore, theoretically the result of Random-demand should be better. However, because these two baselines have different task settings, it is difficult to draw valuable conclusions from a comparison between them.

**(Common Response 3)** Some reviewers (JLnR, i2zg, FyRT) have valuable suggestions on the content of our DDN dataset. They suggested that we could add more fine-grained demand instructions such as "I am thirsty but I cannot drink cold", "something quench thirst and contains caffeine". We have actually included some similar examples in our DDN dataset, such as "I need a place to rest" and "I need a soft place to rest". The former can be "bed, wooden chair, sofa" while the latter cannot be "wooden chair" but "bed, sofa". The latter contains two demands in its instruction: "soft" and "available for rest". In future work, we'll take their suggestions into serious consideration and add some explicit personal preferences as well as more complex demand instructions, including negation of object samples, combining multiple demands, and prioritising between objects.

We truly appreciate the time all the reviewers, AC and SAC have taken to carefully review our work. What follows are the revision plan and  point-to-point responses, and we hope that our responses addresses your concerns. In the attached PDF file, we summarise the results of all experiments (both baseline and ablation experiments) including the experiments we supplemented in the rebuttal period. Thanks again for all valuable comments and suggestions.


## Revision Plan ## 
As suggested by reviewer i2zg, we will reduce some of the redundant expressions in the Introduction section to make it more concise and focused.

In the Related Work section, we will add some papers recommended by reviewer nwVS and discuss them.

In the Experiment section, we will add the results of our supplementary baseline and ablation experiments to Table.1. We wiil also add more detailed description about baselines.

### References ### 

[1] Du, H., Yu, X., & Zheng, L. (2021). VTNet: Visual transformer network for object goal navigation. ICLR 2021

[2] Anderson, P., Wu, Q., Teney, D., Bruce, J., Johnson, M., Sünderhauf, N., ... & Van Den Hengel, A. (2018). Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. CVPR 2018

[3] Chen, C., Jain, U., Schissler, C., Gari, S. V. A., Al-Halah, Z., Ithapu, V. K., ... & Grauman, K. (2020). Soundspaces: Audio-visual navigation in 3d environments.  ECCV 2020


Meta-review: The reviewers initially had some concerns regarding practical implications, missing baselines and ablations, clarity of experiments and concepts, connection with object-goal navigation, etc. These aspects were comprehensively addressed in the rebuttal, ultimately resulting in an increase in the ratings.

Reviewer i2zg recommends rejection while the others are on the accept side. While the concerns from Reviewer i2zg are valid and the insights are valuable, the AC believes the provided rebuttal alleviates the concerns. Therefore, the AC follows the recommendation of the majority of the reviewers and recommends acceptance.
Decision: Accept (poster)