Title: Find What You Want: Learning Demand-conditioned Object Attribute Space for Demand-driven Navigation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Visual Navigation

Demand-Driven Navigation
Abstract: The task of Visual Object Navigation (VON) involves an agent's ability to locate a particular object within a given scene. To successfully accomplish the VON task, two essential conditions must be fulfiled:  1) the user knows the name of the desired object; and 2) the user-specified object actually is present within the scene. To meet these conditions, a simulator can incorporate predefined object names and positions into the metadata of the scene. However, in real-world scenarios, it is often challenging to ensure that these conditions are always met. Humans in an unfamiliar environment may not know which objects are present in the scene, or they may mistakenly specify an object that is not actually present.  Nevertheless, despite these challenges, humans may still have a demand for an object, which could potentially be fulfilled by other objects present within the scene in an equivalent manner. Hence, this paper proposes Demand-driven Navigation (DDN), which leverages the user's demand as the task instruction and prompts the agent to find an object which matches the specified demand. DDN aims to relax the stringent conditions of VON by focusing on fulfilling the user's demand rather than relying solely on specified object names. This paper proposes a method of acquiring textual attribute features of objects by extracting common sense knowledge  from a large language model (LLM). These textual attribute features are subsequently aligned with visual attribute features using Contrastive Language-Image Pre-training (CLIP). Incorporating the visual attribute features as prior knowledge, enhances the navigation process. Experiments on AI2Thor with the ProcThor dataset demonstrate that the visual attribute features improve the agent's navigation performance and outperform the baseline methods commonly used in the VON and VLN task and methods with LLMs. The codes and demonstrations can be viewed at https://sites.google.com/view/demand-driven-navigation.

Find What You Want: Learning Demand-conditioned Object Attribute Space for Demand-driven Navigation

Abstract

The task of Visual Object Navigation (VON) involves an agent's ability to locate a particular object within a given scene. To successfully accomplish the VON task, two essential conditions must be fulfiled: 1) the user knows the name of the desired object; and 2) the user-specified object actually is present within the scene. To meet these conditions, a simulator can incorporate predefined object names and positions into the metadata of the scene. However, in real-world scenarios, it is often challenging to ensure that these conditions are always met. Humans in an unfamiliar environment may not know which objects are present in the scene, or they may mistakenly specify an object that is not actually present. Nevertheless, despite these challenges, humans may still have a demand for an object, which could potentially be fulfilled by other objects present within the scene in an equivalent manner. Hence, this paper proposes Demand-driven Navigation (DDN), which leverages the user's demand as the task instruction and prompts the agent to find an object which matches the specified demand. DDN aims to relax the stringent conditions of VON by focusing on fulfilling the user's demand rather than relying solely on specified object names. This paper proposes a method of acquiring textual attribute features of objects by extracting common sense knowledge from a large language model (LLM). These textual attribute features are subsequently aligned with visual attribute features using Contrastive LanguageImage Pre-training (CLIP). Incorporating the visual attribute features as prior knowledge, enhances the navigation process. Experiments on AI2Thor with the ProcThor dataset demonstrate that the visual attribute features improve the agent's navigation performance and outperform the baseline methods commonly used in the VON and VLN task and methods with LLMs. The codes and demonstrations can be viewed at https://sites.google.com/view/demand-driven-navigation.

I need to wipe

the table

I want to clean

my hands

I am so thirsty

I am tired and

need a pick

me up

quench thirst

contain caffeine

wipe sth.

Demand-conditioned

clean sth.

I  am  thirsty.

I will find something that

can quench thirst.

Towel

Water

Wet Tissue

Tea

Coffee

1 Introduction

Visual Object Navigation (VON) [1-15], which is considered fundamental for home service agents, refers to the task in which an agent is required to find a user-specified object within a provided scene. Previous VON studies can be broadly categorised into two groups. The first category, known as closed-vocabulary navigation [1-4, 6-10], involves a predetermined set of target object categories so that the agent is restricted to selecting objects from this predefined set. On the other hand, openvocabulary navigation [5, 11, 12] encompasses a more flexible approach where the target objects are described in natural language, meaning the agent is not limited to specific predefined categories.

In real-world applications of VON, users are typically required to provide the name of the object they want the agent to locate, regardless of whether it falls under the closed-vocabulary or open-vocabulary group. This requirement introduces two conditions that must be fulfilled: 1) the user must possess knowledge of the object's name in the given environment; and 2) the specified object exists within the environment. These conditions are necessary for successful object navigation tasks in practical scenarios. In common simulators (such as AI2Thor [16] and Habitat [17, 18]), these two conditions can be easily satisfied. As direct access to the scene's metadata is available in the simulators, the existence of target objects can be checked using the metadata, or the object names in the metadata can be directly used as target objects. However, in the real world, where the user may not be omniscient of the environment ( e.g. , at a friend's house), the two conditions described above may not be met, which will result in failure to satisfy the user's demand for an object. Additionally, it is possible that when the user specifies an object to search for the agent is able to find an object with a similar function within the scene, when the specified object does not exist. For example, an object such as a bottle of water may be specified which has a related demand, to quench thirst. In the case where a bottle of water is not present in the scene, this demand can be adequately met by a cup of tea or apple juice found by the agent, however this would fail the VON task. This challenge illustrates a key difficulty VON faces in real environments.

From a psychological point of view [19-22], this paper considers the user's motivation to make the agent search for an object. In the VON task the user initially has a certain demand, then thinks of an object to satisfy it, and finally asks the agent to search for that specific object. Then, would it not be more consistent with the constraints of the real world environment to omit the second step and directly use the demand as an instruction to ask the agent to find an object that satisfies the demand? This paper argues that utilising user demands as instructions has the following benefits: 1) Users no longer need to consider the objects in the environment, but only their own demands. This demand-driven description of the task is more in line with the psychological perspective of human motivations. 2) The mapping between demands and objects is many-to-many ( i.e. , a demand instruction can be satisfied by multiple objects, and an object can potentially satisfy multiple demand instructions), which can theoretically increase the probability of an agent satisfying user demands in

comparison to VON. 3) The demands, when described in natural language have a wider descriptive space than just specifying an object, and the tasks are thus described more naturally and in accordance with the daily habits of humans.

Therefore, this paper proposes the task of Demand-Driven Navigation (DDN). In the DDN task, the agent receives a demand instruction described in natural language ( e.g. , I am thirsty), and then the agent must search for an object within the provided environment to satisfy this demand. Our task dataset is generated semi-automatically: it is initially generated by GPT-3 [23] (In the entire paper, we use gpt-3.5-turbo version), and then manually filtered and supplemented. The DDN task presents new challenges for embodied agents: 1) Different real-world environments contain different objects and, even within a fixed environment, the categories of objects can change over time due to human interaction with the environment, resulting in agents potentially looking for different objects when attempting to satisfy the same demand. 2) The many-to-many mapping between instructions and objects requires agents to reason depending on common sense knowledge, human preferences and scene grounding information: to identify what objects potentially exist in the scene to satisfy the user's demand, and where these objects are most likely to be located. 3) The agent needs to determine whether the current objects within the field of view satisfy the user's demand from the objects' visual geometry. There exists the possibility that this reasoning is not only based on the visual features, but stems from the functionality of the object, necessitating the agent having common sense knowledge about the object. In summary, the core challenge of DDN is how to use common sense knowledge , human preferences and scene grounding information to interpret demand instructions and efficiently locate and identify objects that satisfy the user's demands.

It is noted that the essence of an object satisfying a demand is that certain attributes of the object fulfill the demand instructions. For example, for the demand of 'I am thirsty,' the object to be sought only needs to have attributes such as 'potable' and 'can quench thirst'. Both a bottle of water and a cup of tea are objects that have these attributes. To address the challenge of mapping attributes and demand instructions while considering their relevance to the current scene, this paper proposes a novel method that extracts the common sense knowledge from LLMs to learn the demand-conditioned textual attribute features of objects. Then the proposed method aligns these textual attribute features with visual attribute features using the multi-modal model CLIP [24]. The learned visual attribute features contain common sense knowledge and human preferences from LLMs, and also obtain scene grounding information via CLIP. This paper also trains a demand-based visual grounding model to output the bounding box of objects in the RGB input that match the demand when an episode ends. Experiments are conducted on the AI2Thor simulator [16] with the ProcThor dataset [25]. The proposed method is compared with closed-vocabulary object navigation, open-vocabulary object navigation, and their variants. This paper also evaluates the performance of GPT-3 and multi-modal large language model(MM-LLM), MiniGPT-4 [26], operated agents which accomplish this task via prompt engineering. The experiments demonstrate that the demand-conditioned attribute features extracted from the LLM and aligned with CLIP effectively assist in navigation and outperform baselines. In summary, this paper's contributions are listed as follows:

This paper proposes the task of Demand-Driven Navigation (DDN), that requires the agent to find an object to meet the user's demand. The DDN task relies heavily on common sense knowledge, human preferences, and scene grounding information.

This paper evaluates two different categories of VON algorithms and their variants combined with GPT-3, as well as the performance of large language models ( e.g. , GPT-3 and MiniGPT4) on the DDN task. The results demonstrate that existing algorithms have difficulty solving the DDN task.

This paper provides a novel method to tackle the DDN task and a benchmark for this. The proposed method extracts the common sense knowledge from LLMs to learn textual attribute features and uses CLIP to align the textual and visual attribute features. The experiments demonstrate that the demand-conditioned attribute features extracted from the LLM and aligned with CLIP effectively assist in navigation and outperform the baselines.

2 Related Work

2.1 Visual Navigation

The task of visual navigation requires the agent to use visual information [27, 28] to reach a target location, such as visual object navigation (VON) [1-15, 29], visual language navigation (VLN) [3038], and visual audio navigation (VAN) [39-44]. The proposed DDN task can be considered as a combination of VON and VLN in terms of describing the target: the demand is described using natural language and the agent is asked to find objects that match the demand. The VLN task requires the agent to follow step-by-step instructions to navigate in a previously unseen environment. In contrast to the VLN task, the proposed DDN task provides high-level demand instructions and requires the agent to infer which objects satisfy the demand instructions within the current scene. The VON task can be broadly classified into two groups: closed-vocabulary object navigation and open-vocabulary (zero-shot) object navigation. In closed-vocabulary object navigation, the target object categories are pre-determined. For example, in the 2023 Habitat ObjectNav Challenge [45], there are 6 target object categories. These categories are usually represented as one-hot vectors. Therefore, many studies establish semantic maps [3, 10] or scene graphs [13] to solve the closed-vocabulary VON task. In open-vocabulary object navigation, the range of object categories is unknown and is usually given in the form of word vectors [11, 5, 12]. Many studies [11, 5] use pre-trained language models to obtain word embedding vectors to achieve generalisation in navigating unknown objects. Additionally, CLIP on Wheels [5] uses the CLIP model for object recognition. Different from the VON task, our proposed DDN task does not focus on the category of objects, but chooses 'human demand' as the task context.

2.2 Large Language Models in Robotics

Recently, large language models (LLMs) [46, 23, 47-49] have attracted attention for their performance on various language tasks (e.g., text classification and commonsense reasoning). LLMs can exhibit a human level of common sense knowledge and can even be on par with human performance in some specialised domains. There is an increasing amount of research [37, 50-54] seeking to use the knowledge in LLMs to control or assist robots in performing a range of tasks. LM-Nav [37] accomplishes outdoor navigation without any training by combining multiple large pre-trained models (GPT-3 [23], CLIP [24] and ViNG [55]). In SayCan [51], the authors use a LLM to interpret highlevel human instructions to obtain detailed low-level instructions, and then use pre-trained low-level instruction skills to enable the robot to complete tasks. However, SayCan assumes that the agent can access the object positions in navigation and only conduct experiments on one scene. PaLM-E [52] goes a step further by projecting visual images into the same semantic space as language, enabling the robot to perceive the world visually, but PaLM-E exhibits control that is either limited to table-level tasks or scene-level tasks with a map. Different from these approaches, the proposed method does not use LLMs for providing reasoning about instructions directly, but instead uses LLMs to learn attribute features about the objects. Such attribute features can help the agent learn an effective navigation policy in mapless scenes. This paper also uses LLMs to generate the required dataset.

3 Problem Statement

In the DDN task, an agent is randomly initialised at a starting position and orientation in a mapless and unseen environment. The agent is required to find an object that meets a demand instruction described in natural language ( e.g. , 'I am thirsty'). Formally, let D denote a set of demand instructions, S denote a set of navigable scenes, and O denote a set of object categories that exist in the real world. To determine whether a found object satisfies the demand, let G : D × O →{ 0 , 1 } denote a discriminator, outputting 1 if the input object satisfies the demand, or 0 if it does not. In the real world, G is effectuated by the user, but to quantitatively evaluate the proposed method and the baselines, G is effectuated by the DDN dataset collected in Sec. 4.

At the beginning of each episode, the agent is initialised at a scene s ∈ S with a initial pose p 0 ( i.e. , position and orientation), and provided a natural language demand instruction d = < d 1 , d 2 · · · d L > ∈ D , where L is the length of the instruction and d i is a single word token. In the demand instruction d , the task is made more challenging by not providing specific object names. This is to ensure the agent learns to reason in scenarios where these objects are not specified by the user. Then the agent is re-

quired to find an object o ∈ O to satisfy the demand instruction d and only take RGB images as sensor inputs. The agent's action space is MoveAhead , RotateRight , RotateLeft , LookUp , LookDown , and Done . When the agent selects the action Done , the agent is also required to output a bounding box b to indicate the object that satisfies the demand within the current field of view RGB image. Then it is determined whether the success criteria have been satisfied. The success criteria for the DDN task include two conditions. 1) The navigation success criterion: this requires there is an object in the field of view that satisfies the demand instruction and the horizontal distance between the agent and the object is less than a threshold c navi . 2) The selection success criterion: this requires that given that the navigation is successful, the intersection over union (IoU) [56] between the output bounding box b and the ground truth bounding box of an object that satisfies the demand instruction is greater than a threshold c sele . There is a 100 step limit to succeed during the episode.

4 Demand-Driven Navigation Dataset

Although mappings between objects and instructions will vary with the environment, a fixed mapping between demand instructions and objects is required to check selection success for training the agent in a specific environment, referred to as world-grounding mappings (WG mappings). The modifier 'world-grounding' emphasises the connection to the specific environment of the real world and will vary with the environment. For example, the ProcThor dataset [25] and the Replica dataset [57] will have different WG mappings. To simulate real-world scenarios, it is assumed, unless otherwise stated, that the agent does not have access to the metadata of the object categories in the environment or the WGmappings that are used to determine selection success during training, validation and testing. The set of world-grounding mappings constructed in this paper form the demand-driven navigation dataset.

To construct the WG mappings, the metadata regarding object categories in the environment is obtained and GPT-3 is used to establish a fixed set of WG mappings between demand instructions and objects F : D → O ∫ , where O ∫ is a subset of O . For example, F (' I am thirsty ′′ ) = { Water , Tea , AppleJuice } . Concretely, prompt engineering is used to inform GPT-3 of the object categories that may be present in the experimental environment. Then GPT-3 is used to determine which demands these objects can satisfy and returns this information in the form of a demand instruction and a statement of which objects present satisfy this demand instruction. It has been found that the WG mappings generated by GPT-3 are not absolutely accurate. Due to the presence of some errors in the generation, manual filtering and supplementation is then used to correct and enhance the dataset. These WG mappings F are used during training and testing, only to discriminate the selection success ( i.e. , to effectuate discriminator G ). In total, approximately 2600 WG mappings are generated. Please see supplementary material for details of the generation process, the prompts, and the statistical features of the dataset.

5 Demand-Driven Navigation Method

It is important to note that if an object can fulfill a demand, it is because one of its functions or attributes can satisfy that demand. In other words, if multiple objects can fulfill the same demand, the objects should have similar attributes. For example, if your demand is to heat food, the corresponding objects that can satisfy the demand are a stove, an oven or a microwave, which all have similar attributes such as heating and temperature control. The relationship between attributes and instructions is not specific to a particular environment, but instead is universal common sense knowledge. Thus, this common sense knowledge can be extracted from LLMs. This universality turns the multiobject-target search into single-attribute-target search: the similarity between attributes reduces the complexity of policy learning. Therefore, we expect to extract the attribute features of the objects conditioned on the given demand from the LLM, i.e., , demand-conditioned textual attribute features.

In this section, how the textual attribute features are learned from LLMs (Sec. 5.1), how they are aligned from text to vision (Sec. 5.2), and how to train the model (Sec. 5.3).

5.1 Textual Attribute Feature Learning

Knowledge Extraction Given that the relationship between attributes and demand instructions is common sense knowledge, many mappings between demand instructions and objects are established

D1. I am hungry.

D2. I am so cold.

D3. I want to sleep.

Language-grounding

Demand

GPT-3

TO1. bread(a), pizza(b), noodle(c).

TO2. clothes(d), blanket(e), quilt(f).

TO3. sofa(g), bed(h), cushion(i).

Language-grounding

Object

BERT

D1 Feature

Demand BERT

Feature

CLIP

Text

Encoder

TO1 Feature

TO2 Feature

a

b

d

e

g

h

CLIP-textual

Feature

DO1-a

DO1-b

+

DO1-d

DO2-a

-

Demand-object

Feature

Satisfied

Unsatisfied

Demand-conditioned

Attribute Feature

A1-a

A1-b

A1-d

A2-a

Push

Away

Pull

Closer

Demand-conditioned

Attribute Space

Knowledge Extraction

Demand-conditioned

Contrastive Learning

Query

Answer

TO3 Feature

c

f

i

DO1-c

DO2-d

DO2-e

DO2-f

DO3-g

DO3-h

DO3-i

A1-c

A2-d

A2-e

A2-f

A3-g

A3-h

A3-i

+

+

-

-

Attribute

Module

D2 Feature

D3 Feature

D1. I want a

place to eat

food

BERT

Copy

DETR

CLIP-

Visual-

Encoder

World-grounding

Object

Demand BERT

Feature

D1 Feature

D1 Feature

D1 Feature

D1 Feature

VO1 Feature

a

b

c

d

CLIP-visual

Feature

（

a

）

（

b

）

（

c

）

（

d

）

Demand BERT

Feature

DO1-a

Demand-Object

Feature

A1-a

A1-d

A1-b

A1-c

Cat.

Image

Encoder

Demand BERT

Feature

Copy

Transformer

Global Visual

Feature

Imitation

Learning

Action

Environment

Done

Visual

Grounding

When

Bounding

Box

Visual-textual Attribute Alignment

Policy Learning

Attribute

Module

conditioned

Attribute Feature

Bbox &

Logits

DO1-b

DO1-c

DO1-d

World-grounding

Demand

with GPT-3, for learning demand-conditioned textual attribute features, referred to as languagegrounding mappings (LG mappings). The modifier 'language-grounding' emphasises that these mappings can be obtained from LLMs without the need for any connection to real-world situations. As LG mappings are independent of the real world and the experimental environment, they are suitable to be used in training the model. Concretely, as shown in Fig. 2, GPT-3 is used to generate a multitude of demand instructions in advance, called language-grounding demands (LG demands), and then GPT-3 is queried as to which objects can satisfy these demands. GPT-3 generates the resulting objects called language-grounding objects (LG objects). The LG demands and the LG objects compose the LG mappings. The BERT model is used to encode the LG demand instructions and obtain demand BERT features. The CLIP-Text-Encoder is used to encode LG objects and obtain CLIP-textual features. Then the demand BERT features and CLIP-textual features are concatenated together to obtain demand-object features (these features are named DO{X}-{Y}, with X representing the demand label and Y representing the object label).

Demand-conditioned Contrastive Learning Due to the fact that objects which satisfy the same demand show similar attributes and vice versa, contrastive learning is suitable to be used to train the Attribute Module. The way positive and negative samples in contrast learning are defined is next illustrated using the object and demand satisfaction relationships in Fig. 2: object a, b and c satisfy demand D1, object d, e and f satisfy demand D2, and object g, h and i satisfy demand D3. If two objects can satisfy the same demand, then the two demand-object features corresponding to these

two objects and the demand are positive sample pairs. For example, DO1-a and DO1-b is a positive sample pair. There are three categories of negative sample pairs. 1) For the same demand, if one object can satisfy it and the other cannot, the corresponding Dem-Obj features are negative sample pairs. For example, DO1-a and DO1-d is a negative sample pair. 2) For the same object, if it satisfies one demand but not the other, then the corresponding demand-object features are also negative sample pairs. For example, DO1-a and DO2-a is a negative sample pair. 3) The demand-object features with different objects and demands are also negative samples pairs. For example, DO1-a and DO2-c is a negative sample pair. Then demand-object features are used as the input to the Attribute Module to obtain the demand-conditioned attribute features (named A{X}-{Y}, with X representing the demand label and Y representing the object label). The demand-conditioned attribute features corresponding to positive sample pairs should be as close as possible in the demand-conditioned attribute feature space, and negative sample pairs should be as far away as possible. Therefore InfoNCE Loss [58] is used to train the Attribute Module. We use a 6-layer Transformer Encoder [59] to implement the Attribute Module.

5.2 Textual-Visual Alignment via CLIP

In Sec. 5.1, the demand-conditioned textual attribute features are learned using language-grounding mappings. However, the demand-conditioned textual attribute features are still at the language-level. To address this problem, the CLIP-Semantic-Space is used as a shared space for vision and text to allow the Attribute Module to obtain scene grounding information. The DETR [60] model is first used to segment object patches (referred to as world-grounding objects, WG object) in the field of view during navigation, and then the CLIP-Image-Encoder is used to project these object patches into the CLIP-Semantic-Space to obtain CLIP-Visual features. Both CLIP-visual features and CLIP-textual features are projected to the same CLIP-Semantic-Space, and thus they are aligned. The Demand BERT feature and CLIP-Visual features are concatenated ( i.e. , demand-object features) and used as inputs to the Attribute Module trained in Sec 5.1. Note that the CLIP-Visual features come from the scene-grounding visual input. Therefore, the demand-conditioned attribute features obtained during navigation involve scene grounding information.

To summarise, in Sec. 5.1, contrast learning and language-grounding mappings generated by GPT-3 are used to allow the Attribute Module to learn to extract demand-conditioned attribute features of objects, which contain common sense knowledge and human preferences ; in Sec. 5.2, CLIP's shared visual-textual semantic space is used to enable the Attribute Module to receive scene-grounding information during navigation.

5.3 Policy Learning and Visual Grounding Model

The features that are fed to a Transformer [59] are processed in a similar way to VTN [6]. Demandconditioned features are concatenated with a bounding box and logits and input into a Transformer Encoder, then passed into a Transformer Decoder with Demand BERT features and global visual features (encoded by a pre-trained Vision Transformer [61, 62]). Imitation learning is used to train the proposed model. Approximately 27,000 trajectories generated by the A ∗ algorithm [63] are collected and used for training.

The Visual Grounding Model (VG model) is essentially a classifier built with a Transformer Encoder. The DETR model is used to segment the object instances on the current image, and then the features of the last layer of DETR for the highest k patches of DETR logits, the global image features encoded by RESNET-18 [64], the Demand BERT features and a CLS token, are combined and fed to the VG model, and then the output features corresponding to the CLS token are used for classification. The VG model is used for all the baselines that perform DDN and the proposed method.

For more details about policy learning, the fine-tuning of the image encoder and DETR, the pipeline and training procedure of the demand-based visual-grounding model, and hyperparameters, please see the supplementary material.

6 Experiments

6.1 Experimental Environment

The AI2Thor simulator and the ProcThor dataset [25] are used to conduct the experiments. Throughout the experiments, 200 scenes in each of ProcThor's train/validation/test sets are selected for a total of 600 scenes. There are 109 categories of objects that can be used in the environments to satisfy the demand instructions. 200 WG mappings are selected for training and 300 for testing from our collected DDN dataset. The WG mappings used for validation are the same as those used for training. Our experiments can all be done on 2 × A100 40G, 128-core CPU, 192G RAM. To complete training for a main experiment takes 7 days.

6.2 Evaluation Method and Metrics

Aclosed-vocabulary object navigation algorithm, VTN [6], and an open-vocabulary object navigation algorithm, ZSON [11], are chosen as the original VON baselines. To fit these baselines to the proposed DDN task, several modifications are made to the algorithms as follows and the results are shown in the form of a postfix shown in Tab. 5.3. The VTN-CLIP means that we replace the DETR features used in VTN with CLIP features. The 'demand' suffix means the original one-hot vectors (closed-vocabulary) or word vectors (open-vocabulary) are replaced with the demand BERT features. The 'GPT' suffix means that the GPT-generated language-grounding objects are used as the parsing of the demand instruction, and then the parsing results are used as the input for object navigation. These VON baselines and their variants should demonstrate the difficulty of the DDN task and the fact that DDN tasks are not easily solved by utilising language models for reasoning. Finally, five variants of the VON algorithm are introduced: VTN-CLIP-demand , VTN-demand , VTN-GPT , ZSON-demand , ZSON-GPT .

The popular large language model GPT-3 and the open source multi-modal large language model MiniGPT-4 are also used for navigation policies and recognition policies, resulting in two baselines: MiniGPT-4 and GPT-3-Prompt* . GPT-3 and MiniGPT-4's test results, should provide insight into the strengths and weaknesses of large language models for scene-level tasks. Random ( i.e. , randomly select an action in the action space) is also used as a baseline for showing the difficulty of the DDN task. We also employ a visual-language navigation algorithm, CLIP-Nav [65] and a heuristic exploration algorithm, FBE [66] as navigation policies, with GPT-3 and MiniGPT-4 serving as recognition policies, resulting in four baselines: CLIP-Nav-GPT , CLIP-Nav-MiniGPT-4 , FBE-GPT , and FBE-MiniGPT-4 .

The VTN-CLIP-demand, VTN-GPT, and VTN-demand algorithms are trained using the same collected trajectory as our method. ∗ indicates that the algorithm leverages environmental metadata. Navigation success rate, navigation SPL [67], and selection success rate are used as metrics for comparison. For more details on metrics and baselines (about the metadata usage of the algorithms, prompts used in GPT-3 and MiniGPT-4, etc), please see the supplementary materials.

6.3 Baseline Comparison

Detailed and thorough experiments are conducted to demonstrate the difficulty of the DDN task for the baselines and the superiority of the proposed method, shown in Tab. 5.3.

The results of VTN-demand are slightly better than Random , indicating that VTN-demand has learned to some extent the ability to infer multiple potential target objects simultaneously. However, ZSONdemand has lower performance than Random . Considering that ZSON-demand uses CLIP for visual and text alignment, this lower result indicates that CLIP does not perform well on alignment between instructions and objects. VTN-GPT and ZSON-GPT demonstrate very poor performance. This is likely to be due to the fact that the language-grounding objects given by GPT-3 have a high likelihood not to be present in the current environment, which leads to a meaningless search by the agent. VTN-GPT and ZSON-GPT also show that converting DDN to VON through a pre-defined mapping between instructions to objects is not effective. The superior performance of VTN-CLIP-demand compared to VTN-demand underscores the contribution of CLIP in effectively extracting features of objects. The two variants about CLIP-Nav do not perform well, and we argue that this may be attributed to a significant disparity in content between the step-by-step instructions in VLN and the demand instructions in DDN. Since ProcThor is a large scene consisting of multiple rooms, the results show that heuristic search FBE is not efficient.

The performance of GPT-3+Prompt is significantly lower than Random . This lack of performance is due to the lack of visual input to the GPT-3+Prompt method and relying only on manual image descriptions resulting in insufficient perception of the scene. Moreover, during the experiment, it is observed that GPT-3+Prompt rarely performs the Done action, and approximately 80% of the failures are due to exceeding the step limit, suggesting that GPT-3+Prompt lacks the ability to explore the environment which leads to its inability to discover objects that can satisfy the given demand. With visual inputs, MiniGPT-4 outperforms GPT-3+Prompt despite its relatively small model size, indicating that visual perception ( i.e. , scene grounding information) is of high significance in scene-level tasks. MiniGPT-4 still does not perform as well as Random . By observing MiniGPT-4 's trajectories, two characteristics are noticed: 1) the agent tends to turn its body left and right in the original place and turn the camera up and down, and does not often move; 2) the selection success rate is very close to the navigation success rate. It is therefore considered that MiniGPT-4 's task strategy is as follows: observe the scene in place and then choose to perform the Done action if the agent identifies objects in sight that can satisfy the demand, and output the selected object. By the fact that the selection success and navigation success rates are close, it can be found that MiniGPT-4 satisfactorily recognises the semantics of objects and infers whether they meet the user's demand. However, the low navigation success rate implies that MiniGPT-4 cannot determine whether the distance between it and the object is less than the threshold value c navi .

6.4 Ablation Study

We conduct three ablation experiments, with specific details as follows:

Ours w/o attr pretrain: we use the transformer network in the Attribute Module without demand-conditioned contrastive learning to pretain it.

Ours w/o attr transformer: we replace the transformer network with a MLP layer in the Attribute Module.

Ours w/o BERT: we replace the BERT endocer with a MLP layer.

Ours surpasses all baselines. The Attribute Module turns multi-object-target goal search into singleattribute-target search, which makes the agent no longer need to find potentially different objects in the face of the same demand instruction, but consistent attribute goals. The consistency between demands and attributes reduces the policy learning difficulty. In Ours w/o BERT , we observe a significant performance drop, especially in the setting with unseen instructions. This finding indicates that the features extracted by BERT are highly beneficial for generalisation on instructions. Ours w/o attr pretrain shows that our proposed demand-conditioned contrastive learning has a significant improvement to performance. After demand-conditioned contrastive learning, the Attribute Module extracts generalised attribute features from LLMs and LLMs provide sufficient common sense knowledge and interpretation of human preferences for the Attribute Module to learn. Once the common sense knowledge extracted from the LLMs is lost, the Attribute Module can only learn

the association between attributes and instructions from the sparse navigation signals of imitation learning. However, Ours w/o attr pretrain still surpasses all baselines (including VTN with a similar model structure), which is probably due to the fact that the object semantic information [68] contained in the CLIP-visual features helps the Attribute Module to understand the demand instructions and learn the relationship between objects and instructions to some extent through imitation learning. The performance decrease observed in Ours w/o attr transformer compared to Ours suggests that the transformer network is more effective than MLP in capturing the attribute features of objects.

7 Conclusion and Discussion

This paper proposes Demand-driven Navigation which requires an agent to locate an object within the current environment to satisfy a user's demand. The DDN task presents new challenges for embodied agents, which rely heavily on reasoning with common sense knowledge, human preferences and scene grounding information. The existing VON methods are modified to fit to the proposed DDN task and they have not been found to have satisfactory performance on the DDN task. The LLM GPT-3 and the open source MM-LLM, MiniGPT-4 also show some drawbacks on the DDN task, such as being unable to determine the distance between the object and the agent from visual inputs. This paper proposes to learn demand-conditioned object attribute features from LLMs and align them to visual navigation via CLIP. Such attribute features not only contain common sense knowledge and human preferences from LLMs, but also obtain the scene grounding information obtained from CLIP. Moreover, the attribute features are consistent across objects (as long as they satisfy the same demand), reducing the complexity of policy learning. Experimental results on AI2thor with the ProcThor dataset demonstrate that the proposed method is effective and outperforms the baselines.

Limitations and Broader Societal Impacts As we do not have access to GPT-4 with visual inputs and PaLM-E, we can only use the open source MM-LLM ( i.e. , MiniGPT-4) to test on the proposed DDN task, so we only have the possibility to conjecture the strengths and weaknesses of MM-LLMs on our task from the performance of MiniGPT-4. We expect that our work will advance the performance of MM-LLMs on scenario-level tasks. The proposed DDN task is in the field of visual navigation for finding objects that satisfy human demands. To the best of our knowledge, there are no observable adverse effects on society.

Acknowledgments and Disclosure of Funding

This work was supported by National Natural Science Foundation of China - General Program (62376006) and The National Youth Talent Support Program (8200800081). We also thank Benjamin Redhead for correcting and polishing the grammar and wording of our paper.

References

F. Zhu, X. Liang, Y. Zhu, Q. Yu, X. Chang, and X. Liang, 'Soon: Scenario oriented object navigation with graph-based exploration,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 12 689-12 699.

A. Staroverov, D. A. Yudin, I. Belkin, V. Adeshkin, Y. K. Solomentsev, and A. I. Panov, 'Realtime object navigation with deep neural networks and hierarchical reinforcement learning,' IEEE Access , vol. 8, pp. 195 608-195 621, 2020.

D. S. Chaplot, D. P. Gandhi, A. Gupta, and R. R. Salakhutdinov, 'Object goal navigation using goal-oriented semantic exploration,' Advances in Neural Information Processing Systems , vol. 33, pp. 4247-4258, 2020.

H. Du, X. Yu, and L. Zheng, 'Learning object relation graph and tentative policy for visual navigation,' in Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VII 16 . Springer, 2020, pp. 19-34.

S. Y. Gadre, M. Wortsman, G. Ilharco, L. Schmidt, and S. Song, 'Clip on wheels: Zero-shot object navigation as object localization and exploration,' arXiv preprint arXiv:2203.10421 , 2022.

H. Du, X. Yu, and L. Zheng, 'Vtnet: Visual transformer network for object goal navigation,' in International Conference on Learning Representations .

A. Pal, Y. Qiu, and H. Christensen, 'Learning hierarchical relationships for object-goal navigation,' in Conference on Robot Learning . PMLR, 2021, pp. 517-528.

Y. Qiu, A. Pal, and H. I. Christensen, 'Learning hierarchical relationships for object-goal navigation,' arXiv preprint arXiv:2003.06749 , 2020.

S. K. Ramakrishnan, D. S. Chaplot, Z. Al-Halah, J. Malik, and K. Grauman, 'Poni: Potential functions for objectgoal navigation with interaction-free learning,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 18 890-18 900.

S. Wani, S. Patel, U. Jain, A. Chang, and M. Savva, 'Multion: Benchmarking semantic map memory using multi-object navigation,' Advances in Neural Information Processing Systems , vol. 33, pp. 9700-9712, 2020.

A. Majumdar, G. Aggarwal, B. S. Devnani, J. Hoffman, and D. Batra, 'Zson: Zero-shot object-goal navigation using multimodal goal embeddings,' in Advances in Neural Information Processing Systems .

Q. Zhao, L. Zhang, B. He, H. Qiao, and Z. Liu, 'Zero-shot object goal visual navigation,' arXiv preprint arXiv:2206.07423 , 2022.

W. Yang, X. Wang, A. Farhadi, A. Gupta, and R. Mottaghi, 'Visual semantic navigation using scene priors,' in 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019. [Online]. Available: https://openreview.net/forum?id=HJeRkh05Km

D. S. Kiran, K. Anand, C. Kharyal, G. Kumar, N. Gireesh, S. Banerjee, R. D. Roychoudhury, M. Sridharan, B. Bhowmick, and M. Krishna, 'Spatial relation graph and graph convolutional network for object goal navigation,' in 2022 IEEE 18th International Conference on Automation Science and Engineering (CASE) . IEEE, 2022, pp. 1392-1398.

J. Guo, Z. Lu, T. Wang, W. Huang, and H. Liu, 'Object goal visual navigation using semantic spatial relationships,' in Artificial Intelligence: First CAAI International Conference, CICAI 2021, Hangzhou, China, June 5-6, 2021, Proceedings, Part I . Springer, 2022, pp. 77-88.

E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, D. Gordon, Y. Zhu, A. Gupta, and A. Farhadi, 'AI2-THOR: An Interactive 3D Environment for Visual AI,' arXiv , 2017.

M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik, D. Parikh, and D. Batra, 'Habitat: A Platform for Embodied AI Research,' in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2019.

A. Szot, A. Clegg, E. Undersander, E. Wijmans, Y. Zhao, J. Turner, N. Maestre, M. Mukadam, D. Chaplot, O. Maksymets, A. Gokaslan, V. Vondrus, S. Dharur, F. Meier, W. Galuba, A. Chang, Z. Kira, V. Koltun, J. Malik, M. Savva, and D. Batra, 'Habitat 2.0: Training home assistants to rearrange their habitat,' in Advances in Neural Information Processing Systems (NeurIPS) , 2021.

E. B. Goldstein, Cognitive psychology: Connecting mind, research and everyday experience . Cengage Learning, 2014.

J. Kuhl, 'Motivation and information processing: A new look at decision making, dynamic change, and action control.' 1986.

M. Van Der Meer, Z. Kurth-Nelson, and A. D. Redish, 'Information processing in decisionmaking systems,' The Neuroscientist , vol. 18, no. 4, pp. 342-359, 2012.

J. S. Brown, Motivation of behavior . Prabhat Prakashan, 1961.

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al. , 'Language models are few-shot learners,' Advances in neural information processing systems , vol. 33, pp. 1877-1901, 2020.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al. , 'Learning transferable visual models from natural language supervision,' in International conference on machine learning . PMLR, 2021, pp. 8748-8763.

M. Deitke, E. VanderBilt, A. Herrasti, L. Weihs, J. Salvador, K. Ehsani, W. Han, E. Kolve, A. Farhadi, A. Kembhavi, and R. Mottaghi, 'ProcTHOR: Large-Scale Embodied AI Using Procedural Generation,' in NeurIPS , 2022, outstanding Paper Award.

D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, 'Minigpt-4: Enhancing vision-language understanding with advanced large language models,' arXiv preprint arXiv:2304.10592 , 2023.

J. Kerr, C. M. Kim, K. Goldberg, A. Kanazawa, and M. Tancik, 'Lerf: Language embedded radiance fields,' arXiv preprint arXiv:2303.09553 , 2023.

R. Ding, J. Yang, C. Xue, W. Zhang, S. Bai, and X. Qi, 'Pla: Language-driven open-vocabulary 3d scene understanding,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 7010-7019.

C. Huang, O. Mees, A. Zeng, and W. Burgard, 'Visual language maps for robot navigation,' in 2023 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2023, pp. 10 608-10 615.

Y. Qi, Z. Pan, S. Zhang, A. van den Hengel, and Q. Wu, 'Object-and-action aware model for visual language navigation,' in Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part X 16 . Springer, 2020, pp. 303-317.

A. Pashevich, C. Schmid, and C. Sun, 'Episodic transformer for vision-and-language navigation,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 15 942-15 952.

R. Hu, D. Fried, A. Rohrbach, D. Klein, T. Darrell, and K. Saenko, 'Are you looking? grounding to multiple modalities in vision-and-language navigation,' in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , 2019, pp. 6551-6557.

A. Majumdar, A. Shrivastava, S. Lee, P. Anderson, D. Parikh, and D. Batra, 'Improving visionand-language navigation with image-text pairs from the web,' in Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VI 16 . Springer, 2020, pp. 259-274.

P.-L. Guhur, M. Tapaswi, S. Chen, I. Laptev, and C. Schmid, 'Airbert: In-domain pretraining for vision-and-language navigation,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 1634-1643.

C.-W. Kuo, C.-Y. Ma, J. Hoffman, and Z. Kira, 'Structure-encoding auxiliary tasks for improved visual representation in vision-and-language navigation,' in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , 2023, pp. 1104-1113.

H. Huang, V. Jain, H. Mehta, A. Ku, G. Magalhaes, J. Baldridge, and E. Ie, 'Transferable representation learning in vision-and-language navigation,' in Proceedings of the IEEE/CVF International conference on Computer Vision , 2019, pp. 7404-7413.

D. Shah, B. Osi´ nski, S. Levine et al. , 'Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action,' in Conference on Robot Learning . PMLR, 2023, pp. 492-504.

P. Chen, D. Ji, K. Lin, R. Zeng, T. Li, M. Tan, and C. Gan, 'Weakly-supervised multi-granularity map learning for vision-and-language navigation,' Advances in Neural Information Processing Systems , vol. 35, pp. 38 149-38 161, 2022.

C. Chen, U. Jain, C. Schissler, S. V. A. Gari, Z. Al-Halah, V. K. Ithapu, P. Robinson, and K. Grauman, 'Soundspaces: Audio-visual navigation in 3d environments,' in Computer VisionECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VI 16 . Springer, 2020, pp. 17-36.

C. Chen, S. Majumder, Z. Al-Halah, R. Gao, S. K. Ramakrishnan, and K. Grauman, 'Learning to set waypoints for audio-visual navigation,' arXiv preprint arXiv:2008.09622 , 2020.

C. Chen, Z. Al-Halah, and K. Grauman, 'Semantic audio-visual navigation,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 15 51615 525.

C. Gan, Y. Zhang, J. Wu, B. Gong, and J. B. Tenenbaum, 'Look, listen, and act: Towards audio-visual embodied navigation,' in 2020 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2020, pp. 9701-9707.

Y. Yu, W. Huang, F. Sun, C. Chen, Y. Wang, and X. Liu, 'Sound adversarial audio-visual navigation,' arXiv preprint arXiv:2202.10910 , 2022.

H. Wang, Y. Wang, F. Zhong, M. Wu, J. Zhang, Y. Wang, and H. Dong, 'Learning semanticagnostic and spatial-aware representation for generalizable visual-audio navigation,' IEEE Robotics and Automation Letters , 2023.

K. Yadav, J. Krantz, R. Ramrakhya, S. K. Ramakrishnan, J. Yang, A. Wang, J. Turner, A. Gokaslan, V.-P. Berges, R. Mootaghi, O. Maksymets, A. X. Chang, M. Savva, A. Clegg, D. S. Chaplot, and D. Batra, 'Habitat challenge 2023,' https://aihabitat.org/challenge/2023/, 2023.

OpenAI, 'Gpt-4 technical report,' 2023.

J. D. M.-W. C. Kenton and L. K. Toutanova, 'Bert: Pre-training of deep bidirectional transformers for language understanding,' in Proceedings of NAACL-HLT , 2019, pp. 4171-4186.

A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann et al. , 'Palm: Scaling language modeling with pathways,' arXiv preprint arXiv:2204.02311 , 2022.

R. Thoppilan, D. De Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H.-T. Cheng, A. Jin, T. Bos, L. Baker, Y. Du et al. , 'Lamda: Language models for dialog applications,' arXiv preprint arXiv:2201.08239 , 2022.

Z. Wang, S. Cai, A. Liu, X. Ma, and Y. Liang, 'Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents,' arXiv preprint arXiv:2302.01560 , 2023.

A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian et al. , 'Do as i can, not as i say: Grounding language in robotic affordances,' in Conference on Robot Learning . PMLR, 2023, pp. 287-318.

D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, W. Huang, Y. Chebotar, P. Sermanet, D. Duckworth, S. Levine, V. Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng, I. Mordatch, and P. Florence, 'Palm-e: An embodied multimodal language model,' in arXiv preprint arXiv:2303.03378 , 2023.

W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar et al. , 'Inner monologue: Embodied reasoning through planning with language models,' in 6th Annual Conference on Robot Learning .

I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg, 'Progprompt: Generating situated robot task plans using large language models,' in Workshop on Language and Robotics at CoRL 2022 .

D. Shah, B. Eysenbach, G. Kahn, N. Rhinehart, and S. Levine, 'Ving: Learning open-world navigation with visual goals,' in 2021 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2021, pp. 13 215-13 222.

H. Rezatofighi, N. Tsoi, J. Gwak, A. Sadeghian, I. Reid, and S. Savarese, 'Generalized intersection over union: A metric and a loss for bounding box regression,' in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2019, pp. 658-666.

J. Straub, T. Whelan, L. Ma, Y. Chen, E. Wijmans, S. Green, J. J. Engel, R. Mur-Artal, C. Ren, S. Verma, A. Clarkson, M. Yan, B. Budge, Y. Yan, X. Pan, J. Yon, Y. Zou, K. Leon, N. Carter, J. Briales, T. Gillingham, E. Mueggler, L. Pesqueira, M. Savva, D. Batra, H. M. Strasdat, R. D. Nardi, M. Goesele, S. Lovegrove, and R. Newcombe, 'The Replica dataset: A digital replica of indoor spaces,' arXiv preprint arXiv:1906.05797 , 2019.

A. v. d. Oord, Y . Li, and O. Vinyals, 'Representation learning with contrastive predictive coding,' arXiv preprint arXiv:1807.03748 , 2018.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, 'Attention is all you need,' Advances in neural information processing systems , vol. 30, 2017.

N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, 'End-to-end object detection with transformers,' in Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16 . Springer, 2020, pp. 213-229.

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al. , 'An image is worth 16x16 words: Transformers for image recognition at scale,' arXiv preprint arXiv:2010.11929 , 2020.

K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, 'Masked autoencoders are scalable vision learners,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 16 000-16 009.

P. E. Hart, N. J. Nilsson, and B. Raphael, 'A formal basis for the heuristic determination of minimum cost paths,' IEEE Transactions on Systems Science and Cybernetics , vol. 4, no. 2, pp. 100-107, 1968.

K. He, X. Zhang, S. Ren, and J. Sun, 'Deep residual learning for image recognition,' in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2016, pp. 770-778.

V. S. Dorbala, G. Sigurdsson, R. Piramuthu, J. Thomason, and G. S. Sukhatme, 'Clip-nav: Using clip for zero-shot vision-and-language navigation,' arXiv preprint arXiv:2211.16649 , 2022.

B. Yamauchi, 'A frontier-based approach for autonomous exploration,' in Proceedings 1997 IEEE International Symposium on Computational Intelligence in Robotics and Automation CIRA'97. 'Towards New Computational Principles for Robotics and Automation' , 1997, pp. 146-151.

P. Anderson, A. Chang, D. S. Chaplot, A. Dosovitskiy, S. Gupta, V. Koltun, J. Kosecka, J. Malik, R. Mottaghi, M. Savva et al. , 'On evaluation of embodied navigation agents,' arXiv preprint arXiv:1807.06757 , 2018.

A. Khandelwal, L. Weihs, R. Mottaghi, and A. Kembhavi, 'Simple but effective: Clip embeddings for embodied ai,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 14 829-14 838.

L. Van der Maaten and G. Hinton, 'Visualizing data using t-sne.' Journal of machine learning research , vol. 9, no. 11, 2008.

J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, 'Proximal policy optimization algorithms,' arXiv preprint arXiv:1707.06347 , 2017.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Demand-driven Navigation (DDN), which leverages the user’s demand as the task instruction and prompts the agent to find an object which matches the specified demand. This paper also proposes a method of first acquiring textual attribute features of objects by extracting common sense knowledge from a large language model. These textual attribute features are subsequently aligned with visual attribute features using Contrastive Language-Image Pre-training (CLIP). They experimented on AI2Thor with the ProcThor dataset and demonstrated that the visual attribute features improve the agent’s navigation performance and outperform the baseline methods.\n\nSTRENGTHS:\nThe paper flows nicely with good motivation of the novel task and its challenges clearly stated.\nThe proposed method makes sense and comparison is good.\n\nWEAKNESSES:\n1.\tL40, the second condition is a bit questionable. I don’t think it is a necessary requirement of only searching objects that are in the scene. It is probably the case in current benchmark setup, however it is not a constraint for the research task. Subsequently, it might also impact the definition of “navigation failure” at L48. Asking the robot to search for a non-existent object in the scene and the robot could not find it. This should not be defined as a failure from my point of view. Rather, if you ask a robot to look for some non-existent object and it reports a finding, this should be counted as a failure as it is a clear false positive detection. Nevertheless, the new task DDN does not necessarily remove ‘requirement’ 2) as you might anyway asks for a demand that no object in the scene can satisfy.\n2.\tFrom L171, my impression is that the problem is constrained by only one demand d at a time. In reality, one could have multiple demands to further constrain what object they might want to search. For example, something “quench thirst” and “contains caffeine”.\n3.\tIt will be more convincing with more benchmark datasets. In addition to AI2Thor, there is Habitat Challenge on ObjectNav which can be exploited, plus it is based on 3D scan of real scenes.\n\nQUESTIONS:\n1) Can we handle a search composed with multiple demands, with the current method?\n2) Why other benchmark dataset are not used for evaluation? is there any justification?\n\nLIMITATIONS:\nThe authors discussed both limitations and the societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new visual navigation setting, where the goal is not specified by objects or images but described by a sentence. The sentence encodes the essential information to search for specific objects during navigation. Different from VLN, the task is able to analyze the demand within each sentence rather than step-by-step language guidance. Common sense and knowledge will be explicitly extracted from a large language model. To align the visual and attribute features, CLIP is employed. Overall, the motivation of this work is reasonable and interesting. \n\n# Post rebuttal\nThe authors have addressed most of my concerns. If the authors can provide some visualization results, that would be more convincing. Therefore I changed my score to accept.\n\nSTRENGTHS:\nThe motivation of this work is interesting. Providing a demand description to an agent would enable the agent to search for not only one object in order to complete the specified goals. \n\nThe authors also provide semi-automatically generated data for this new task. This would be complementary to the existing VLN or VN tasks.\n\nWEAKNESSES:\nThe introduction part is overly lengthy. The authors exert three pages to describe the motivations of this work, making reading quite tedious. I highly suggest the authors could trim the introduction part a bit.\n\nThe natural questions come to this task is whether the proposed method can complete object-goal navigation after training? For example, after the network is trained based on demand driven sentences, whether it can be used as a object-goal navigation agent?\n\nThe common knowledge or sense is pre-defined. In the illustrative figures, it seems a demand may correspond to three different objects. Whether this would restrict the options?\n\nIn L189, the WG mappings are different depending on the environment. I am not sure whether this implies that these WG mappings need to be specified manually. If so, this may contradict with the original motivation of this work, where humans may not know the environment in advance.\n\nQUESTIONS:\nMy questions mainly focus on two tasks:\n\nIn this comparisons, the results of some baseline methods are significantly lower than the results reported by their original papers. Therefore, I am wondering how the authors adapt their methods to this setting? \n\nA few works leverage CLIP for VLN or VN tasks. \nCLIP-Nav: Using CLIP for Zero-Shot Vision-and-Language Navigation\nVision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments\nThese works can be adopted for comparison.\n\nAs the searched objects have strong association with the demand, what if some demands cannot be processed properly? For example, I am thirsty but I cannot drink cold. Simply providing demands and their corresponding objects would lead to overfitting.\n\nLIMITATIONS:\nThere is no negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new task called Demand-Driven Navigation (DDN) that, unlike previous Visual Object Navigation (VON) tasks that evaluate the ability of an agent to find a specific object in an unknown environment, considers fulfilling the demand of a human. This new task is motivated by the lack of real-world grounding of current VON tasks that either require an agent to find an instance of an object category from a pre-defined fixed vocabulary or a language-specified object in an open-vocabulary fashion. However, in a real environment, a specific object might not be present or, if thinking from the point of view of fulfilling a human’s demand, other objects might be equally as good as the targeted one. As a result, authors suggest querying an autonomous agent with a language demand instruction. This allows one to be more flexible with respect to environments but also requires common sense knowledge, an understanding of how objects can be used, and where they are likely to be located in a scene. \n\nThe paper evaluates several baselines inspired by previous literature to show the introduced task is hard and cannot be solved with currently known methods. A new approach to solving the DDN task is thus introduced in this paper. The main goal is to learn a mapping between human demands and attributes of objects that can fulfill them. At training time, a GPT-3 model is thus used to generate a series of demands and objects that can meet each of them. Each demand is encoded by a BERT model and each object is encoded by a CLIP text encoder. For each demand, a demand-object vector is created by concatenating the demand representation and an object representation. An attribute module takes the demand-object vector as input and is trained with contrastive learning to extract representations that are as close as possible for different pairs sharing the same demand. This attribute module is then used in the final policy that is composed of a Transformer model and is trained with imitation learning. The paper shows this new method outperforms other baselines.\n\nSTRENGTHS:\n1. The introduced task is very interesting: querying an agent with a human demand seems more aligned with what is needed when deploying a robot in a real environment to help humans. Authors have properly motivated the caveats of current VON tasks in the literature.\n2. The extraction of object attributes seems very relevant and well-motivated. Learning attributes is indeed a way to guide the learning of common sense knowledge and experiments showcase the gain in performance it allows to reach.\n3. Authors propose diverse baselines to evaluate the performance of current language models in robotics/navigation tasks.\n\nWEAKNESSES:\n1. [Major] The introduced method is composed of many different modules and pre-trained models (GPT-3, BERT, CLIP text encoder, CLIP vision encoder, Attribute Module, DETR, Image Encoder, Policy Transformer, Visual Grounding model). It is not clear what parts are the most important and whether the overall method could maintain the same performance without some of these building blocks. Additional ablation studies would be very interesting.\n2. [Major] Most baselines except *Ours* showcase very low and close performance. However, section 6.3 discussing the experimental results is quite long and detailed. I am not convinced authors can draw as many conclusions as they do when we consider how close all baselines are in terms of average performance (even more when considering the standard deviation).\n3. [Major] This remark is very related to the previous one. All concurrent baselines reach very low performance. Could it be that these methods were not trained to convergence or would simply require much more training? The authors mention 1.8M training frames in the paper, which seems rather small compared with the number of frames required to train baselines in other navigation tasks (generally closer to 100M training frames). \n4. [Minor] When presenting baselines in section 6.2, a lot of information is missing. The paper refers to details given in the supplementary material. When reading this supplementary information, we can understand the introduced baselines. I would suggest including this information in the main paper directly.\n\nQUESTIONS:\nFour questions (1.-4.) were already asked in the \"Weaknesses\" section. I would like authors to address these concerns. I also add two additional questions (5., 6.), that do not appear to me as paper weaknesses but rather required clarifications:\n\n1.-4. See \"Weaknesses\" section.\n\n5. [Major] When describing the *Random* baseline, authors say it is about randomly selecting an action in the action space. I thus do not understand the difference between *Random-object* and *Random-demand*. Further clarifications are needed.\n6. [Major] Performance for *GPT-3+Prompt** and *MiniGPT-4* in Table 1 is always the same independently from the scene (seen/unseen) and instructions (seen/unseen). This should either be corrected or explained.\n\nLIMITATIONS:\nThe paper mentions limitations regarding the drawn conclusions about the performance of current language models on the DDN tasks. Authors explain they did not have access to the code for recent methods such as GPT-4 with visual inputs or PaLM-E, and thus were not able to evaluate these methods. It is a good thing to mention this limitation, but their experiments were already conducted with many recent models, which might still allow them to draw relevant conclusions (see “Weaknesses” section for remarks about the drawn conclusions, which are however orthogonal to the mentioned limitations in this section).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a Demand-driven Navigation (DDN) problem to leverages the user’s demand as the task instruction and prompts the agent to find an object which matches the specified demand. Then the authors proposed a method by learning demand-conditioned object attribute features from LLMs and align them to visual navigation via CLIP. The experiment shows the efficiency of the proposed method. However, I have some concerns about this paper. My detailed comments are as follows.\n\nSTRENGTHS:\n1. This paper proposes a novel Demand-Driven Navigation task to explore the navigation with only the user’s demand as the task instruction. This task is practical and worth more research, especially with the development of open-vocabulary foundation models.\n\n2. The proposed attribute module is interesting and helpful for extracting attributes of objects.\n\n3. This paper provides a method to tackle the DDN task by extracting common sense from LLMs to learn textual attribute features and uses CLIP to align the textual and visual attribute features. The results obviously outperform the baselines.\n\nWEAKNESSES:\n1.\tOne important baseline is missing. The agent could explore the environment using a heuristic algorithm like FBE. At each time step, the agent detects all objects in observation and ask LLM whether these objects can satisfy the human demand. \n\n2.\tWhat are the differences between common sense knowledge and human preferences mentioned in the paper?\n\n3. It is not clear why the results of random-object are different from random-demand. Are they both execute random actions? More explanations are needed.\n\n4.\tIn Table1, the ZSON-demand performs better the ZSON-object. Does it indicate that CLIP performs better in understanding high-level abstract demand compared to concrete objects? This result seems to conflict with the claim that “CLIP does not perform well on alignment between instructions and objects” in Line 335.\n\n5.\tSome related works that try to solve open-vocabulary navigation[1,2] or scene understanding[3] are missed. It would be better to add and discuss them in the related work part for the sake of completeness.\n\n[1] Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigation, NeurIPS 2022.\n\n[2] Visual Language Maps for Robot Navigation, ICRA 2023.\n\n[3] LERF: Language Embedded Radiance Fields, ArXiv 2023.\n\nQUESTIONS:\nMy main concerns are the lack of an important baseline and the analysis of the experimental results.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the novel task of demand-driven navigation, where a robot must navigate towards a goal object that satisfies the human user's demand (e.g., for the demand \"I am thirsty\", the robot has to find water/juice/tea, etc.) Citing limitations of navigation methods used for other object-goal navigation variants, the paper also proposes a novel architecture that relies on extracting attribute features conditioned on the demand text features and object features for visible objects. The attribute features extracted are indicative of what physical / semantic properties are fulfilled by a given object to satisfy the demand (e.g., the object \"water bottle\" has the property to \"quench thirst\"). These features are learned using a contrastive learning objective on prior-knowledge extracted from a large-language model (GPT-3). Experiments on the ProcThor dataset demonstrate the difficulty of the demand-driven navigation task and the superiority of the proposed method over prior navigation methods.\n\nSTRENGTHS:\n* The idea of demand-driven navigation is interesting and novel. While the goal specification is natural language (similar to prior work), it focuses on a very different concept of \"demand\" where multiple functionally equivalent objects can satisfy a given demand.\n* The problem statement is well motivated and the paper writing clarity is largely good (see weaknesses for some questions).\n* The supplementary material provides necessary implementation details and details about the dataset for reproducibility.\n* The experimental design is good. Recent baselines for open and closed vocabulary navigation are considered. Multiple experimental trials have been performed to show statistical significance. The proposed method significantly outperforms the baselines.\n\nWEAKNESSES:\n# Post-rebuttal update\n\nI thank the authors for addressing my questions and concerns in the rebuttal. It is clear to me that this paper presents a valuable new direction in this space of navigation tasks and the experiments are sufficiently strong to recommend acceptance. The motivation can be clarified further and the rebuttal responses need to be reflected in the final paper. With the understanding that this will be done, I am increasing my rating to accept (7).\n\n---------------------------------------------------------------\n\n## Task motivation good, but practical implications are not clear\n\nI liked the task itself, but a key question that concerns me is *\"how often does an object demanded not be present in the scene, and therefore, a functionally equivalent object was needed to satisfy the demand?\"*. That is, how often do demands for objects become infeasible because the object was missing in the scene? E.g., if the demand is \"Get me a water bottle because I am thirsty\", I would expect most scenes would contain water bottles. The task definition avoids addressing this issue by having a generic demand itself as the input.\n\nOf course, it is possible that there are other objects that can be functionally equivalent **in addition** to the object demanded, but that's not the scenario motivated in the introduction. Additionally, as a user, if I request a water bottle, that's exactly what I'd want unless it is nowhere to be found. So the proposed task makes more sense in the absence of the primary object.\n\n\n## Dataset is underwhelming\n\nL63 - 64 - \"mapping between demands and objects is many-to-many\" ---This is only partly true. Based on statistics in Figure 4 supp., only 2.3 objects, on average, correspond to a given instruction. Calling this \"many\" is underwhelming. The other direction is still true though, i.e., there are many instructions satisfied by a given object (Figure 5 supp).\n\n## Task definition fails to consider whether an object instance satisfies the demand\nL89 - \"Both a bottle of water and a cup of tea\" can \"quench thirst\" --- this is only true if the bottle has water and cup is filled with tea. Does the dataset / task differentiate between bottle/cup instances that contain liquids vs. those that are empty?\n\n## Approach clarifications needed\n* L175 - \"only take RGB images as sensor inputs\" --- is the GPS+compass information also included here or does the model learn to localize on its own?\n* L185 - why is the time step limit only 100? That seems very short for navigation in large environments.\n* L257 - why is the attribute module a transformer (e.g., why not just an MLP)? Is self-attention across demand-object features needed?\n* L281 - why use only imitation learning and not reinforcement learning?\n\n## Experiment clarifications needed\n* Table 1 - why is the performance on seen scene, seen instruction so low? I'd expect close to 100% success due to overfitting.\n* L306 - how are the object categories derived from demand inputs for the *-object methods?\n* L317 - why are there variants of \"Random\" if the policy only selects actions randomly? \n* L336 - 338 - \"likely to be due to the fact that ... have a high likelihood not to be present ... meaningless search\" --- can we empirically quantify this? How often is it the case that objects predicted by GPT-3 are missing in the scene?\n* L371 - 373 - \"surpasses all baselines ... CLIP-visual features helps ...\" - it seems to me an unfair advantage to use CLIP visual features only for the proposed method and not the baselines. How do baselines perform when equipped with CLIP features?\n\nQUESTIONS:\nI'd appreciate it if the authors can clarify the questions raised in weaknesses. It will help me arrive at my final decision.\n\nLIMITATIONS:\nYes, the limitations are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}