R2H: Building Multimodal Navigation Helpers that Respond to Help Requests
Authors: Yue Fan
Jing Gu
Kaizhi Zheng
Xin Eric Wang
Venue: EMNLP 2023 Main (2023)
Keywords: Multimodal Understanding
Embodied AI
Vision-and-language Navigation
TL;DR: This paper proposes a new benchmark, Respond to Help Requests (R2H), to build multimodal navigation-helpers that can respond to help requests, based on existing dialog-based embodied datasets.

Abstract: Intelligent navigation-helper agents are critical as they can navigate users in unknown areas through environmental awareness and conversational ability, serving as potential accessibility tools for individuals with disabilities. In this work, we first introduce a novel benchmark, Respond to Help Requests (R2H), to promote the development of multi-modal navigation helpers capable of responding to requests for help, utilizing existing dialog-based embodied datasets. R2H mainly includes two tasks: (1) Respond to Dialog History (RDH), which assesses the helper agent's ability to generate informative responses based on a given dialog history, and (2) Respond during Interaction (RdI), which evaluates the effectiveness and efficiency of the response during consistent cooperation with a task performer. Furthermore, we explore two approaches to construct the navigation-helper agent, including fine-tuning a novel task-oriented multi-modal response generation model that can see and respond, named SeeRee, and employing a multi-modal large language model in a zero-shot manner. Analysis of the task and method was conducted based on both automatic benchmarking and human evaluations.

R2H: Building Multimodal Navigation Helpers that Respond to Help Requests
Yue Fan, Jing Gu, Kaizhi Zheng, Xin Eric Wang
University of California, Santa Cruz [yfnn71, kzheng31, jgul 10, xwang366 @ @uesc.edu
Abstract
Deliver packaage
Intelligent navigation-hee helper agents critical as they can navigate users in unknown areas through environmental awareness and conver- sational ability, serving potential accessibil- ity tools for individuals with disabilities. In this work, we first introduce novel bench- mark, Respond to Help Requests (R2H), to pro- mote the development multi-modal naviga- tion helpers capable of responding requests for help, utilizing existing dialog-based em- bodied datasets. R2H mainly includes two tasks: (1) Respond to Dialog History (RDH), which assesses the helper agent's ability to gen- erate informative responses based on a given dialog history, and (2) Respond during Inter- action (RdI), which evaluates the effective- ness and efficiency of the response during consistent cooperation with task performer. Furthermore, we explore two approaches to construct the navigation-helpee agent, includ- ing fine-tuning a novel task-oriented multi- modal response generation model that can see and respond, named SeeRee, and employ- ing a multi-modal large language model in zero-shot manner. Analysis of the task and method was conducted based on both automatic benchmarking and human evaluations. Project website: https: / //s.////tttteeeeeeeeeeeeeeeeeees google. com/view/ response2helpreeqesst////
Inquiry (help requesi]: Which dirction should go?
Response:
Fallow the kollwnyy right through the opened door wifl a blue
Task performe
History
Destiraton
Figure I: Example of a helper agent. A navigation helper provides responses help a task performer who is delivering a package. The helper has access to oracle information that not available to the task performer. such the location of the destination and map of the environment.
about directions and receive responses that are tai- lored to visual information about the current sur- roundings.
Building such a helper agent poses significant challenges. It requires understanding the visual environment and the task performer's inquiries and leveraging oracle information to provide effective responses. Evaluating these agents also requires a task performer to show how well the helper agent performs in the real collaborative real scenario, where the task performer follows instructions and further sends inquiries when needed the helper with the goal of completing tasks. Using a human as an oracle task performer would be the most in- tuitive setting, but is impractical due to the high cost and low efficiency.
1 Introduction
Assisting humans in real-world scenarios is a fun- damental capability that AI agents should possess. An AI helper, communicating with humans in nat- ural language based on visual observation in the environment and oracle information could signifi- cantly enhance work productivity and serve as accessibility tool for individuals with disabilities. Figure/ illustrates an example of a delivery per- son seeking assistance navigating through an unfamiliar building. With the help navigation- helper agent, the delivery person can ask questions
In this work, introduce the Respond Help Requests (R2H) benchmark, designed automati- cally evaluate conversational multi-modal naviga- tion helpers in a cooperative dynamic with another agent as the task performer. The R2H benchmark

---

incorporates pre-trained performer agents fol- low the responses from the helper agent, and the helper agent's performance is then reflected in the performance of the fixed task performer. Lever- aging three existing vision-and-dialog navigation datasets, CVDN (Thomason et al., 2020), AIFRED (Shridhar et al., 2020) and AVDN (Fan et al., 2022), our R2H benchmark introduces two novel tasks: the Respond Dialog History task (RDH) and the Respond during Interaction task (RdI). the RDH task, the helper agent generates response to the inquiry in the dialog history from humans, aiming at facilitating the task completion of the performer agent. In the Rdl task, the helper agent needs to generate multiple responses from the start of the navigation process with no prior dialog history till the task's 's success. To this end, R2H benchmark offers pragmatic evaluation of the response from helper agents both single- and multi-turn helper- performer cooperation.
We present the Respond to Help Requests (R2H) benchmark as a test-bed for automat- ically evaluating the capabilities of multi- modal conversational ievigaationhhlper agent, that helps task performers complete tasks by providing natural language responses to in- quiries based on environment information.
We build two task helper agents, novel task- oriented multi-modal helper agent, SeeRee, utilizing the Conditional Optimized Sparse (COS) attention mask and noise-free step-by- step instructions (Parse by Step) and multi- modal LLM helper with mPLUG-OWl (Ye et al., 2023).
Our experiments on the R2H benchmark and human evaluations of two helper agents over baseline also indicate that closer linguistic resemblance to human helpers does not auto- matically translate into more effective versational helper agent.
We also present multi-modal helper agent SeeRee for R2H benchmark, which leverages the oracle knowledge about the task and environment such as the destination location in a navigation task, generate responses to inquiries from the task per- former. SeeRee employs pre-trained vision and language models to handle multi-modal inputs. To manage long input sequences, SeeRee leverages a novel Conditional Optimized Sparse (COS) at- tention mask. Morcover, we introduce Parse by Step, which leverages Large Language Model to transform ground-truth human responses into struc- tured step-by-step navigation instructions. Those parsed instructions (instead of human responses) serve as a better training source with improved per- formance in helping the task performer. In experi- ments, SeeRee surpasses the baseline in generating effective responses and validates the COS atten- tion mask and Parse by Step method. SeeRee's responses have been evaluated through human as- sessments, demonstrating high accuracy and sig- nificant improvement in task success rate compared the baseline.
2 The R2H Benchmark
Significant differences exist between building a helper agent and developing task performer agents for dialog-based multi-modal navigation tasks. Task performers, such as ones in CVDN (Thoma- son et al., 2020), DialFRED (Gao et al., 2022), and AVDN (Fan al., 2022), are challenged command followers, evaluated based their per- formance of following dialog histories from human annotations, as shown Figure 2i. However, the helper agent works as a supportive role, responds to questions from the task performer to facilitate task success. Therefore, the evaluation of the helper agent's 's performance could not be solely dependent on the agent itself, but on how the task performer benefited from the response. Therefore building the helper agent requires evaluations collaborative setting with task performers.
Involving humans as task performers to evalu- ate the helper agent is ideal but expensive. Alter- natively, inspired by Padmakumar et al. (2022), Nguyen and Daumé III (2019) and Roman et al. (2020) that build helper and performer agents to collaborate as shown in Figure 2ii, we introduce the Respond Help Requests (R2H) benchmark, involving task performer in the evaluation process as shown Figure 2iii). In this way, the helper agent can be assessed comprehensively and real- istically. R2H benchmark tests the agent's ability
Additionally, we ask human testers to rate the faithfulness and naturalness of the responses evalu- ate the response from helper agents with automatic scores. As result, our experiments indicate that higher language similarity human helpers does not necessarily lead to a more successful conversa- tional helper agent.
The main contributions are concluded as follows:

---

Task performers under test
Helper-performer agent pairs under test
Helper agents under test
pre-trained & fixed performer agent
Human ennotation
Environment
Environment
Environment
(i) Benchmark for task performers.
(ii) Benchmark for helper- performer agent pairs.
(iii) Benchmark for helper agents.
Figure 2: Comparison between different dialog-based embodied benchmark types. The benchmark either (i) evaluates task performers, where performers follow instructions in human annotations, (ii) evaluates helper- performer pairs, where the helper and performer agents need to learn and be evaluated together jointly, (iii) evaluates helper agents only (our R2H benchmark), where helpers need to provide appropriate instructions according to performer and environment information.
to respond effectively in wide range of scenarios, including two novel tasks, the Respond to Dialog History (RDH) task and the Respond during In- teraction (Rdl) task, building upon three existing vision-and-dialog navigation datasets. RDH task evaluates helper agents in situation where par- tial human dialog history is provided and the RdI task aims at challenging the helper agents with real collaborative scenarios.
2.2 Respond during Interaction Task
The Respond during Interaction (RdI) task chal- lenges the ability of helper agents to cooperate consistently with the task performer. Similar to the RDH task, the Rdl task involves pre-trained task performer agent predicting navigation actions based on dialog histories. However, unlike the RDH task, no dialog history is initially provided in the RdI task, and the helper agent needs to spond to the navigation inquiries from the task performer constantly during the navigation. The task performer agent initiates inquiries q for help when needed and navigates based on the ongo- ing dialog between itself and the helper agent h = fo..f...... di f.). where f. is the real-time responses from the helper agent to g. The dialog i as result of interaction serves as the primary source of guidance for the task performer agent, making the consistent quality of the helper agent's responses crucial for successful navigation. Addi- tionally, since there multi-turn helper-performer cooperation involved in the Rdl task, the helper's s communication efficiency can be evaluated by the number of conversation turns required for fixed performer's outcome.
2.1 Respond to Dialog History Task
task, I think that is a stand alone task. Respond to Dialog History (RDH) Task focuses on eval- uating the accuracy and completeness of the re- sponse from helper agents. The helper agent is challenged with understanding the dialog history and responds help the task performer based on information about the task and environment the form of image sequences. We developed environment-specific scripts to generate the image sequence which is introduced in Section 2.4. Af- ter the responses f; generated, they will be con- catenated with all available human dialog history h {go, To ... 91-1, r;-1 } in the correspond- ing trajectory before the inquiries y from human task performers. As a result, the generated response from the helper agent forms new dialog history h = (/4-1, qú, } which becomes the input to the task performer.
2.3 Our R2H benchmark requires task performer agents to form helper-performer cooperation. In the RDH task, the task performer agent predicts nav- igation actions based on dialog history in specific
Task Performer Agent

---

environments. As for the Rdl task, the task per- former also needs to generate navigation inquiries to accomplish the navigation task better.
sequences. Script-based samplers are designed for each environment to generate the image sequences by leveraging oracle information. The sampler out- puts image sequences showing the task performer's s views on the shortest path to the destination. Es- pecially for the CVDN dataset, following the data collection process, connectivity graph for view- points is used to generate the shortest path and therefore the image sequence length is variable but limited to views within viewpoints of the current position. Examples are shown in the appendix, For the AVDN dataset, since the allocentric direction description, such as "turn south" could be involved, we keep the image sequence all oriented with the north at the top and indicate the drone direction with a red arrow. As result, helper agents can be evaluated in different datasets with the same input format, which enhanced the benchmark's 's versatil- ity to adapt further new datasets.
R2H benchmark adopts state-of-the-art open- sourced task performer agents for vision-and- language navigation datasets. The task performer agent is pre-trained the original training set with human dialogs h; including the response from the human helper ry and predicts actions based on h for completing the task. Therefore, the progress and success made by the task performer can be seen as a test of the accuracy and completeness of the responses generated by the helper agent.
2.4 Adapting Existing Datasets
R2H benchmark establishes the same tasks across different datasets.
Datasets R2H benchmark is built upon existing vision-and-diale dialo navigation datasets with dialogs between task performers and helpers.
2.5 Metries
Since aim to evaluate how capable the response generated by helper agents is in helping the task performer, we adopt the primary metrics for task completion from each dataset: Goal Progress (GP) in CVDN evaluates the distance of the progress made towards the destination, where is computed as the trajectory length, deducted by the remaining trajectory from the current location to the destina- tion viewpoint; Success Rate (SR) in DialFRED, shows the ratio of tasks being completed success- fully: Success weighted by inverse Path Length (SPL) (Anderson et al., 2018) as the dominant met- ric in AVDN measures the Success Rate weighted by the total length of the navigation trajectory.
CVDN (Thomason et al., 2020) is situated in the Matterport3D simulator (Chang al., 2017) with photo-realistic scenes. The dataset records the collaboration between the human task performer and the human helper to com- plete navigation tasks of finding target objects in different indoor environments.
DialFRED (Gao et al., 2022) is built on Ai2- thor (Kolve et al., 2017) simulator with syn- thetic views. Similar to the CVDN, the helper and task performer collaborate to navigate to targets. However, each trajectory only corre- sponds one pair of inquiry and response, making unsuitable for Rdl tasks.
3 Models
AVDN (Fan et al., 2022) is an aerial vision- and-language dataset that includes dialogs, navigation trajectories, and visual observation between the helper and task performer. The dataset is annotated upon continuous state photo-realistic drone simulator where the goal of the task performer to control the drone in the simulator with a top-down view to navi- gate to a certain destination.
3.1 SeeRee
In this section, we introduce the helper agent that can see and respond, named SeeRee. SeeRee generates responses to task-oriented navigation in- quiries from the task performer. As is illustrated in Figure 3, our helper agent SeeRee generates natural language responses to the task performer's 's inquiries based on the task and environment infor- mation that is not aware by the task performer. The image sequences are padded
Environment-sppecfic Adaption Given the vari- ability of available information across different datasets and environments, the R2H benchmark designed with a harmonizing approach, converting the environment-specift information into image
fixed length (Lin et al., 2022), encoded by Video Swin Transformer (Liu et al., 2022) and then concatenated with BERT text embeddings (Devlin et al., 2019). Finally, the embeddings are fed into a multi multi-modal transformer

---

Input inquiry
[MSK]
Image sequence w/ Env. Info.
[MSK] tokens are predicted.
3.1.2
Conditional Optimized Sparse (COS)
VidSwin
Attention Mask
One challenge in generating responses for dialog- based embodied tasks is effectively modeling the long input image sequence, reducing the redun- dancy in the repetitive images but keeping the crit- ical details. To this end, we introduce a Condi- tional Optimized Sparse (COS) attention mask for the multi-modal transformer, shown in Figure 3. The mask can be divided into three row sections, corresponding to what range the embeddings for input inquiry, the response generated, input image sequence can attend to, respectively. The first row section shows that the language embedding (LE) for input inquiry can attend to itself and the input visual embedding (VE). The second row section means the LE for the generated response can attend to the VE and all the previous LE, allowing unidi- rectional text generation (Devlin al., 2019). The third row section indicates that the VE can attend to partial itself and the LE of input inquiry. Especially, instead of being fulling binary and pre-defined, a learnable conditional mask C' that is non-binary and sparse is adopted in the third row section of the COS attention mask, controlling the self-attention of the VE. C is conditioned on VE, and the map- ping modeled by:
1000 0||ل Langunge embadding (LE)
embaoding (VE)
Multimodal Transformer
a
C=TIVE
lenemall conditioal mask C
(Non-bhary)
K4
Conditional Optimized Sperse (COS) Attentien Mask
Natural langusge response
Figure 3: Overview of SeeRee. The visual and text inputs are encoded and fed into the multi-modal trans- former, where Conditional Optimized Sparse (COS) attention mask is applied. The COS attention mask has fixed binary values except for learnable conditional mask for visual embedding (VE) that is conditioned VE itself. Yellow, blue, and green colored rows cor- respond to the attention masks for input inquiry and generated response and VE, respectively.
C=
which generates the natural language response in an auto-regressive way. SeeRee is trained end-to- end with Mask Language Modeling (Devlin et al., 2019) and please refer to the appendix for the train- ing detail.
where (x) I+e- and multi-layer per- ceptron. In this way, our COS attention mask uses conditional optimizing strategy that optimizes the attention mask based on the image sequence. As result, COS attention mask enables better en- coding and understanding of long visual input and improves the response generation result.
3.1.1 Multi-modal Transformer
Following prior multi-modal language generation studies (Hu et al., 2020; Lin et al., 2022), our multi- modal transformer takes as input the embedding containing both text and image information and generates natural language responses in a unidi- rectional requence-to-sequeee sequence generation process. We treat the input inquiries as prompts for gener- ating the response, and a special token [CLS] is added to the end of the inquiry.
3.1.3 Response Preprocessing with Parse by Step
Training helper agent to imitate the responses the human dialog directly may not be optimal as they may be unorganized and include utterances irrelevant to the task. Structured step-by-ster in- structions are easier for the helper agent to learn.
Therefore, inspired by the idea of in-context Iearn- ing (Brown al., 2020; Kojima et al.), pro- pose Parse by Step method that prompts GPT-3 (Brown al., 2020) to preprocess the ground-truth responses in the training data. We detail the de- signed prompts and examples in the appendix. Af-
At inference time, the text is generated in an auto-regressive manner, where we insert multiple [MSK] tokens after the [CLS] token and predict tokens to replace [MSK]tokens one by one unidi- rectionally until the prediction is [EOS] or all the

---

CVDN
DialFRED
AVDN
4.2
RDH Task
Seen
Unseen
Seen
Unseen
Seen
Unseen SPL
Helper
Task Performer Agents For RDH task on
GP
1PL
Human Annotator
6.9
5.1
49.1
33.4
14.7
16.5
CVDN dataset (Thomason et al., 2020), we use HAMT (Chen et al., 2021), pre-trained on the orig- inal Navigation from Dialog History (NDH) task in CVDN the task performer agent. For AVDN (Fan et al., 2022) and DialFRED (Gao et al., 2022) datasets, we leverage the task performer in the orig- inal work, i.e., HAA-transformer and DialFRED model 2, We further detail the task performers the appendix.
RMM 5
4.7
2.8
46.5
32.1
2,4
3.6
Multimodal LLM
5.3
3.6
47.0
33.8
0.7
SooKee
4,9
33.1
Table 1: Results of Respond to Dialog History (RDH)
task on CVDN, DialFRED, and AVDN dataset (Thoma- son et al., 2020; Gao al., 2022: Fan et al., 2022). We replace the original response in validation sets with responses from different helper agents. With fixed task performer agent, a better performance of the per- former agent represents a more effective response from the helper agent.
Result As indicated in Table 1, the task per- former agent attains the best overall performance across all three datasets when guided by responses generated by the SeeRee helper agent. On both CVDN and DiaFRED datasets, SeeRee demon- strates performance levels that are strikingly sim- ilar to those of human helpers. This is likely at- tributable to the fact that the environments used in the CVDN are visually akin to the data employed during the pre-training phase of SeeRee's encoders. Such similarity facilitates efficient fine-tuning and leads to heightened performance. In addition, for the DialFRED dataset, due to the predominance template baased human utterances, the challenge in response generation is simplified. Mcanwhile, the AVDN dataset is more challenging due to the drone simulator's 's adoption of continuous control space, and complex visual information leads to difficulty in reaching the precise destination. Still, SeeRee outperforms the baseline by a large margin. We present results from more LLM baselines and case studies of the RDH task in Appendix and C.
ter Prase by Step, the preprocessed training data is parsed in a step-by-step manner with a stream- lined language pattern. As a result, the learning objectives of SeeRee is the preprocessed human response Y = P(R), where P is the Parse by Step method, and R is the original human response in the dialog of the training set.
3.2 Multi-modal Large Language Model
Besides our SeeRee model, we introduce another swwigation-helpe agent constructed from a multi- modal large language model (LLM). Specifically, we employ mPLUG-OWl (Ye et al., 2023), a State- Of-The-Art multi-modal LLM that is able to take as input an image sequence and text for language generation. mPLUG-Owl is originally trained with a large amount of uni-modal and multi-modal data in two stage paradigm including instruction tuning. To leverage the extensive knowledge that mPLUG- Owl accumulated through training and avoid the potential issue associated with limited task-specific training data, we adopt mPLUG-OWl in a zero- shot manner. The build of a helper agent based on mPLUG-Owl serves as valuable comparison for SeeRee and sheds light on the potential of leverag- ing LLMs in building navigation-heli helper agents.
4.3 Rdi Task
Task Performer Agents We utilize the task per- former agent from the pioneering work RMM (Roman al., 2020) deployed CVDN dataset (Thomason et al., 2020), which combines two com- ponents: the LSTM-based RMM Navigator model (A), responsible for predicting navigation actions, and RMM Questioner model (2), generating in- quires to the helper agent after every four naviga- tion actions. Despite its comparatively simple struc- ture, the RMM task performer stands out as the sole task performer developed that possesses both language and navigation capabilities: generating navigation actions and crafting natural language
4 Experiments
4.1 Setup
We initialize the encoders and multi-moda, trans- formers in SeeRee with weights from SwinBert (Lin et al., 2022) to benefit from its image sequence understanding ability and then finetune SeeRee on the training set of each dataset separately. We adopt the RMM Guide model (g) (Roman et al., 2020) as the baseline model in our experiments. Please refer to the appendix for more implementation details.
'https://githi:i cow/csghizh//VLNHHN
'Due to unavailable model weights, we reproduced the model with comparable results as in the original paper.

---

COS
Seen Validation
Unseen Validation
5.5
RMM
Helper
attention mask
Parse by Step
Mult-mmodal LLM
GP
B2
R
GP
B2
R
SeeRee
4.5
Annotator
7.3
5.1
9.9
21.8
10.0 22.1
SceRee
14,
25,5
13,9 25.
3.5
13.8
24.2
4. 13.2 24.1
Table 2: Ablation study of the SeeRee agent CVDN dataset (Thomason et al., 2020) based on RDH task GP and language generation metries, BLUE2 (B2) and ROUGE-L (R). We directly apply our Parse by Step method to the original validation data created by hu- mans, showing that Parse by Step maintains essential task-related information. The results show the effec- tiveness of COS attention mask and Parse by Step in task-oriented response generation.
3.0
2.5
2.0
10
Coversation Turn
Figure 4: Results of Respond during Interaction (Rdl) task on CVDN dataset (Thomason et al., 2020). The mean Goal Progress (GP) of the same task performer collaborating with different helper agents is plotted with respect to the number conversation turns happened. Error bars show the relative range of the GP. Multi- modal LLM enables better but noisier performance of task performer than SeeRee.
navigation inquiries. Future works for building such language- and navigation-capable task per- formers in AVDN and DialFRED environments are needed. Furthermore, given the flexibility of our system, should a superior model be developed, the task performer could be effortlessly swapped.
Subjective Response Evaluation
Completion
Helper
GP
Narumlness 1
Faithfulness ?
Performer
helpor
RMM G
6.32
73/100
59/100
Human
Multi-mddal
75/100
60/100
Seckte
9,89
72/100
75/100
Table Results of human evaluation on Rdl task on the CVDN (Thomason et al., 2020). The human tester plays the task performer role via an interface interacting with the helper agents. Task completion and subjective response evaluation are collected. The response from SeeRce most effective despite being less natural.
Result The Rdl task is conducted with a max- imum 20 turns of conversation between the task performer and helper on the unseen validation set of the CVDN dataset. As depicted in Figure 4, we plot the mean GP of the task performer agent concerning the dialog turns happened during the navigation. As the dialog turn between helper and performer increases chronologically. more informa- tion provided to the task performer. The navi- gation stops once the task performer reaches the maximum steps, 20, or stop action is generated where the goal progress remains constant thereafter. Based on the result, the multi-modal LLM as the helper agent facilitates the most effective perfor- mance from the task performer at the end of the maximum conversation turns. However, it's worth noting that the response from SeeRee is shown to be less noisy on its effect and therefore is poten- tially more reliable over extended dialogue turns whereas multi-modaa LLM tends to generate re- sponses with more varied effeetiveness. More- over, with less than half the maximum dialogue turns, SeeRee's responses yield a superior result that is 90% close the navigation result at the end the maximum turn. This underscores SeeRee's greater communication efficiency.
4.4 Ablation Study
Based on CVDN dataset (Thomason et al., 2020), we conduct two ablation studies to explore the Parse by Step method and COS attention mask for SeeRce. We first let the task performer infers on the human dialog from the original dataset, but the response within is processed by our Parse by Step method. Then we create ablated SeeRee model by repeating the same training on SeeRee model twice, but incrementally replacing naive fixed attention mask with the COS attention mask and applying our Parse by Step method.
Response Effectiveness Result The first ablation study that evaluate of Parse by Step method the original validation serves the result Table shows that human responses processed with Pase by Step keep the information contained in the original response and they are over- all equally capable as the original response. Addi-
sanity check. As

---

tionally in the second ablation study, the GP of the same task performer cooperated with different ab- lated SeeRee shows that both COS attention mask and Parse by Step leads to major improvements to the effectiveness of the response in facilitating task success.
helper (Roman et al., 2020; Hahn al., 2020; Pad- makumar al., 2022). For instance, the CVDN (Thomason et al., 2020) evaluates a task performer to navigate to a desired room by dialog histories. Gao et al. (2022) developed a dialogue-enabled embodied instruction following benchmark, Dial- FRED, based on the ALFRED benchmark (Shrid- har et al., 2020) and presented a task performer framework. Further, there is wide range of ac- tivities studied in these tasks, such as navigating to a specific location (Roman et al., 2020), locat- ing positions (Hahn et al., 2020), and interacting objects (Padmakumar et al., 2022). Compared to these benchmarks, our R2H benchmark aims for better helper agents and is the only benchmark for sole helper evaluation.
Language Generation Similarity Result As shown in Table 2, we also evaluate the ablated models with language generation metrics, BLUE2 (Papineni et al., 2002) and ROUGE-L (Lin and Och, 2004). BLUE2 score and ROUGH-L score drops when Parse by Stop method is applied, but GP in the RDH task receives major increase. This indicates that high language similarity to human responses does not necessarily equate more effective conversational helper agent.
Multimodal-based Language Generation Building helper agents is in line with a growing collection of methods applied to the visual guestion-ansswrring task. A unified vision-and- language framework can be trained to handle a variety of tasks, including the question-answering problem, using single objective (Cho et al., 2021; Wang al., 2022). Fu et al. (2021) tackled the video question question-answert answering problem by adopting a video transformer to model the temporal dynamics of video inputs explicitly. One problem shared by these works is the input frames to the model are limited in quantity, whereas the helper agent has to take long image sequence as input to include adequate information. Lin et al. (2022) developed learnable sparse attention mask for video caption that enables long frame sequence as input. However, the learned sparse attention mask is fixed after training, lacking generalization ability compared to the COS attention mask of SeeRee.
4.5 Human Evaluation
To further evaluate the performance of helper agents, we conduct human evaluations based on the RdI task. Human participants act as task per- formers navigating 60 randomly selected trajecto- ries in validation sets of CVDN (Thomason et al., 2020). During the simulation, participants can con- trol their movement in the environment using their keyboards and ask questions to the helper agent whenever needed. We evaluate both settings where the helper agent exists or not. Human participants are also asked to rate the naturalness and faithful- ness of the response from the helper. The aver- age GP and subjective rating of the responses are shown in Table 3. The result shows that SeeRee pro- vides the best results in terms of task completion. Through subjective evaluation, we find that SceRee achieves significantly higher scores in terms of re- sponse faithfulness. Despite being trained with data preprocessed by Parse by Step, where the training supervision is no longer the original human utter- ances, it still achieves a close score of naturalness compared to the baseline model trained on original human responses. Through these evaluations, show the ability of SeeRee to help human users complete embodied tasks.
Conclusion
In this paper, we introduce the Respond to Help Re- quests (R2H) benchmark with two tasks, Respond to Dialog History (RDH) and Respond during In- teraction (RdI), assessing a helper agent's guidance capabilities. With the R2H benchmark, we build and evaluate two navigation- and Multi-modal LLM model. The results show that they both outperformed the bascline model. Through further ablation study and human evalu- ation, we prove the effectiveness of SeeRee in as- sisting humans with tasks and argue that the effec- tiveness of responses cannot be determined by their linguistic resemblance to human dialogue alone.
agents, SeeRee
5 Related Work
Dialog-based Multi-modal Embodied Bench- marks Previous dialog-based multi-modal em- bodied benchmarks usually focus on evaluating either task performers ( (Thomason et al., 2020; Gao et al., 2022; Shi et al., 2022; Gu et al., 2022) or the corresponding pairs of task performer and

---

Limitations
ronments. International Conference on 3D Vision (3DV).
R2H benchmark presents platform for helper agents that create natural language to address queries from task performers. Yet, the assessment of such an helper agent mandates a capable task performer agent that can not only navigate but also communicate, given that the efficacy of the helper can be gauged through final task fulfillment, and the task performer shouldn't it act as a constraint. Also, the complexity of the real world surpasses that of simulated environment, thereby impos- ing additional prerequisites for the task performer. Furthermore, the lack of abundant dialog-base based embodied datasets also restricts the progression of both performer and helper agents.
Shizhe Chen, Pierre-Louis Guhur, Cordelia Schmid, and Ivan Laptev. 2021. History aware multimodal transformer for vision-and- -anaanuad navigation. In NewrlPS.
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. 2021, Unifying vision- andoon-nn--ae and-language tasks via text genera- tion. In International Conference on Machine Learn- ing, pages 1931-1942, PMLR.
Jacob Devlin, Ming-War Wei Chang. Kenton Lee, and Kristina Toutanova, 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing.
Yue Fan. Winson Chen, Tongzhou Jiang, Chun Zhou, Yi Zhang, and Xin Eric Wang. 2022. Aerial ausion-and-dialog navigation. arXiv preprint arXiv:2205.1222
Ethies Statement
The human evaluation part of this project is classi- fied as exempt by Human Subject Committee vis IRB protocols. Additionally, we recognize the po- tential ethical issues related to training language generation models using human dialog data, such as the possibility of the model learning harmful language without understanding its context. To mit- igate this concern, our proposed data preprocess- ing method, Parse by Step, converts the responses in the training data into more structured and task- specific instructions, effectively reducing the pres- ence of noise and profanity the training data. As a result, the likelihood of our model generating inappropriate responses is greatly minimized.
Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang. Lijuan Wang, and Zicheng Liu. 2021. Violet: End-to-end ton video-language trans- formers with masked visual-token modeling, arXiv preprint arXiv:2111. 12681.
Xiaofeng Gao, Qiaozi Gao, Ran Gong, Kaixiang Lin, Govind Thattai, and Gaurav S Sukhatme. 2022. Dial- fred: Dialogue-enablln enabled agents for embodied instruc- tion following, arXiv preprint (xXX::2222.13333
Jing Gu, Eliana Stefani, Qi Wu, Jesse Thomason, and Xin Wang. 2022. Assionaand-languag navigation: A survey of tasks, methods, and future directions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7606-7623.
References
Meera Hahn, Jacob Krantz, Dhruv Batra, Devi Parikh, James Rehg. Stefan Lee, and Peter Anderson. 2020. Where are you? localization from embodied dialog. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 806-822.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. 2018. Vision- and-language navigation: Interpreting visually- grounded navigation instructions in real environ- ments. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3674- 3683.
Xiaowei Hu, Xi Yin, Kevin Lin, Lijuan Wang. Lei Zhang, Jianfeng Gao, and Zicheng Liu. 2020. Vivo: Surpassing human performance in novel object cap- tioning with visual vocabulary pre-training.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Advances in Neu- ral Information PreiiioooPrrraeee Processing Systems.
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli Van- derBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. 2017. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv:1712.004474
Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng. and Yinda Zhang. 2017. Matter- port3d: Learning from RGB-D data in indoor envi-

---

Jumnan Li. Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping anguuae-image pre- training with frozen image encoders and large lan- guage models. arXiv preprint ar arXiv2231.11559)
Mohit Shridhar, Jesse Thomason, Daniel Gordon. Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zee, and Dieter Fox. 2020. Alfred: A bench- mark for interpreting grounded instructions for ev- eryday tasks. In Proceedings of the IEEE/CVF con- férence on computer vision and pattern recognition, pages 10740-10749.
Yew Lin and Franz Josef Och. 2004. Auto- matic evaluation of machine translation quality using longest common subsequence and Skip-bigran statis- tics. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL- 04), pages 605-612.
Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. 2020. Vision-and-dialo navi- gation. In Conference on Robot Learning, pages 394-406. PMLR.
Kevin Lin, Linjie Li, Chung- Lin, Faisal Ahmed, Zhe Gan, Zicheng Liu, Yumao Lu, and Lijuan Wang. 2022. Swinbert; End-to-end transformers with sparse attention for video captioning. In Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition, pages 17949-1795 17958.
Jianfeng Wang, Zhengyuan Yang. Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. 2022. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205.144 141000
Qinghao Yc, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan. Yiyang Zhou, Junyang Wang, An- wen Hu, Pengcheng Shi, Yaya Shi, 2023. mplug-owl: Modularization empowers large lan- guage models with multimodality. arXiv preprint arXiv:2304.144 14178.
Liu, Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. 2022. Video swin trans- former. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3202-3211.
Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
Khanh Nguyen and Hal Daumé III. 2019. Help, anna! visual navigation with natural multimodal as- sistance via retrospective tation learning. arXiv preprint 1909..:19090016.
imi-
Aishwarya Padmakumar, Jesse Thomason, Ayush Shri- vastava, Patrick Lange, Anjali Narayan- Chen, Span- dana Gella, Robinson Piramuthu, Gokhan Tur, and Dilek Hakkani-Tur. 2022. Teach: Task-driven em- bodied agents that chat. In Proceedings of the AAAI Conference Artifiial Intelligence, volume 36, pages 2017-2025.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th anmual meeting of the Association for Computa- tional Linguistics, pages 31 1-318.
Alexander Pashevich, Cordelia Schmid, and Chen Sun. 2021. Episodic transformer for vision-and-language navigation. In Proceedings of the IEEF/CVF Interna- tional Conference on Computer Vision, pages 15942- 15952.
Homero Roman Roman, Yonatan Bisk, Jesse Thomason, Asli Celikyilmaz, and Jianfeng Gao. 2020. Rmm: recursive mental model for dialogue navigation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1732-1745.
Zhengxiang Shi, Yue Feng, and Aldo Lipani. 2022. Learning to execute actions ask clarification ques- tions. In Findings of the Association for Computa- tional Linguistics: NAACL 2022, pages 2060-2070.

---

A Implementation Details
RMM RMM is an LSTM-based model that generates natural language response based on the input image sequence and dialog history. We train RMM & from scratch with the same data used for fine-tuning SeeRee, using a batch size of 8 and a learning rate of le=4.
A.1 Helper Model
SeeRee For the training of SeeRee, we apply Mask Language Modeling (MLM) (Devlin et al., 2019) to the response where 80% tokens are masked with [MSK] tokens, and 10% tokens are changed randomly. Cross entropy loss applied to predictions for the masked tokens:
A.2 Task Performer Agents
We aim leverage the best available task perform- ers in R2H benchmark.
IMEM = ExL-CrossSttrrpp (Vi, Di);
(2)
For CVDN dataset, we select History Aware Multimodal Transformer (HAMT) (Chen et al., 2021) as the task performer agent in RDH task. HAMT model is designed for Vision-and- Language Navigation (VLN) tasks. The HAMT model incorporates long-horizon history into multi-modal decision making. It efficiently en- codes all past panoramic observations via a hierar- chical vision transformer and then combines text, history, and current observation to predict the next action. The model is first trained end-to-end using several proxy tasks, including single-step action prediction and spatial relation prediction, and then reinforcement learning is used improve the navi- gation policy further. As the time we conduct the experiment, HAMT has achieved state-of-the-an results on a broad range of VLN tasks, including CVDN (Thomason et al., 2020). However, HAMT model is only capable for navigation action pre- diction and cannot generate questions for asking help. Therefore, we adopt RMM Navigator model (N) with RMM Questioner model (Q) (Roman et al., 2020) in RdI task, which is the only task performer model to the best of our knowledge that is designed to navigate while communicating in natural language. Both RMM N and Q are Istm based models and the RMM Questioner model is designed to generate questions at a fixed interval. It takes into account the Navigator's perspective in the environment and the dialog history.
where y/ is the masked token at position i and Øs is the prediction. Additionally, in order to let the COS attention mask attend to the specific details of the Visual Embedding (VE) that are most relevant to the task, we enforce C to be sparse, letting the VE to sparsely attend to itself using a sparsity loss (Lin et al., 2022):
M
M
LSPARSE = -
Cisl,
(3)
where À is regularization hyperparameter and Cz.
is the value of the learnable conditional mask C. SeeRee trained on CVDN (Thomason al., 2020). DiaDFRED (Gao et al., 2022) and AVDN (Fan et al., 2022) datasets individually using AdamW optimizer (Loshchilov and Hutter, 2018) for 20k iterations with a batch size of 6 and learning rate of 14, The training data used are converted from the original training set from each dataset, with the environment-speccffe scripted sampler that generates image sequences with oracle environ- ment information. We select the trained weights based on the RDH task evaluation result. Training takes about 12 hours on one NVIDIA A6000 GPU.
Multi-modal Large Language Model We uti- lized mPLUG-Owl (Ye et al., 2023) as a represen- tative method for Large Language Model in this paper. mPLUG-Owl takes images and text as input and output natural language. It achieved state-of- the-art performance various multi-modality tasks. Providing the further path in the form of images and a proper prompt, mPLUG-OW Owl directly outputs the guidance for the performer agent. Table 4 shows prompt templates. The QUESTION is relevant to the current task, and it not strictly formed. For example, it could be "Should I continue forward?* in CVDN, "What does the object look like?" for DialFRED, "I am on top of a building block, Can I see the destination?" for AVDN.
For AVDN and DialFRED datasets (Fan et al., 2022: Gao et al., 2022), we use Tranforrnfrremer and DialFRED model proposed along with the dataset as the task performer agent because they are still leading the leaderboard 3 44, DialDRED model and Tra-nrrasfrrmer model are both based on Episodic Transformer model (Pashevich et al., 2021). The DialFRED model trained DialFRED
httts://eel..iieebbbhlee////eeee 159
page/2049/overview

---

Prompt for CVDN: According to the given images, which way to go? Please only tell me the moving direction. Do not introduce the room details.
SQUESTION
Prompt for DialFRED:
The images first show sequenee of first person view of robot. Based on the image sequence, please answer the question
about SQUESTION TION
Prompt for AVDN:
The images lirst show the drone view and red arrow shows current direction of the drone. How could the drose reach the destination in green bounding box? Note, north up. Give instruction to the drone on SQUESTION! Concisely describe landmarks but do not
mention red arrow and green
Table 4: Text prompts for mPLUG-Ow/ In RDH task, SQUESTION is the human inquires from the validation set. In Rdl task, SQUESTION is generated by task performer agent in real-time.
Prompt for CVDN:
Commander says "Yes the kitchen. Go the left the fireplace and then all the up the stairs.' Step by step: I. Yes. 2. go to the kitchen 3. go the left of the fireplace, 4. upstairs. Commander says:
Step by step: 1.
Prompt for AVDN:
Commander says: 'Hi drone, if you go four o' c'olock, and go to small building, go forward the bigger building, your destination." Step by step: Go four o' clock. 2. fl to small building. destination the bigger building.
Commander says: 'eey drome, if fly over the white building."
Step by step: fly over the white building. Commander says:
Step by step:
Dafasel
Original response
Prucessed response
Yeah.
CVDN
Yeah keep going around the outside rill you to the end. And sonry about the mixup at first.
2. Keep going amoind the outside.
3. Get to the end.
head southwest.
AVDN
Hey drone, head southwest, through a small vegetation and arrive house that the final destination.
go through small vegetation.
3. final destination is house.
Table 5: Examples of our Parse by Step method on the CVDN dataset (Thomason et al., 2020) and AVDN dataset (Fan et al., 2022). We fill the original response to the blank of the prompt and input to GPT-3 for sentence completion. The output from the GPT-3 becomes the processed response with mainly task related instructions kept and organized in steps. Through Parse by Step, we preprocess the response in the training set.
dataset focuses on three types of questions: loca- tion clarification, appearance clarification, and di- rection clarification. The TAAA-Tranffomer model trained on the AVDN dataset can predict both navi- gation waypoints and human attention.
ble 6. Especially, the datasets we have adopted cover a wide range of environment types, includ- ing both indoor (CVDN and DialFRED) and out- door (AVDN) settings, well as synthetic (Dial- FRED) and photo-realistic environments (CVDN and AVDN). This diversity adds depth and robust- ness to our benchmark.
A.3 Prompts and Examples for Parse by Step
We design different prompts for applying Parse by Step on CVDN dataset (Thomason et al., 2020) and AVDN dataset (Fan et al., 2022). Table 5 shows the designed prompt and some example pre- processed results. To process the training set re- sponses, we first insert the original response into the blank of the prompt, which is then inputted into GPT-3 for sentence completion. The output from GPT-3, primarily retaining task-related instructions arranged in steps, is subsequently treated as the processed response. Furthermore, we also elimi- nate the item numbers to streamline the instructions further. This approach aids in preserving the essen- tial task-related information while enhancing the response's 's readability and conciseness.
C RDH Task Examples
In this section, we show examples of our RDH task on all three datasets, CVDN dataset (Thomason et al., 2020), DialFRED dataset (Gao et al., 2022) and AVDN dataset (Fan et al., 2022). As shown in Figure 8, 7 and 6 each example includes the input inquiry from human, the sampled input image sequence from the environment oracle information, the response generated by SeeRee and the human ground truth response.
D Results of RDH Task on More Metrics
Table 1 shows the performance of RDH task on the main metric. Here Table 8 exhibits the results on more metries for thorough performance under- standing.
B Additional Benchmark Analysis
We conduct additional analyses on our R2H bench- mark across the three datasets as shown in Ta-
Goal Progress (GP) evaluates the distance of the progress made towards the destination. Success

---

# navigation
querries Irajectory
Avg length of bumaa respoese (words)
Avg length of human query (words)
Avg images in input image sequence of helper agents
trajectories
CVDN
14.9
DiaPRRED
120958
6.0
10.8
AVDN
5372
1.8
19.7
9.0
9.1
Table 6: Statistic analysis on R2H benchmark across the three datasets. The columns from left to right are the number of navigation trajectories; the number of queries per trajectory, which is the same the number of responses per trajectory; the average length of responses and queries provided by human annotators and the average number of images in the image sequences sampled from the environment by script-based samplers, which serve as input to the helper agent.
Rate (SR) shows the ratio tasks being completed successfully. Success weighted by inverse Path Length (SPL) and Path Weighted Success Rate (PWSR) measure the Success Rate weighted by the total length of the navigation trajectory. Here we could draw similar conclusions with Table 1. Note that AVDN dataset (Fan et al., 2022) more challenging due the drone simulator's adoption continuous control space and complex visual information. The mistakes made by the helper are likely to be amplified if the task performer misses the target location and overshoots, or if the initial direction provided by helper agent is inaccurate. Even though all methods have negative impact the goal progress. SeeRee minimized the influence while achieving the highest Success Rate.
language pretraining. Due to the model's limitation of accepting only a single image input, we stack the input image sequences into a single image arranged in four columns. The text prompt used is the same as the prompt for Multimodal-LLM (mPLUG-OWl) as shown in Table 4.
F Human Evaluation Details
In our human evaluation study, we recruited five students from the university and paid with at least $1 7/h as the human task performer. They consented to participate and contribute their data used for this work. These participants were given instructions on how use the simulator before the evaluation began and had the same prior knowledge about the project. The participants are randomly assigned a navigation-helper agent and CVDN data including the initial instruction, starting, and target location. An interface, as shown in Figure 5, is built where an image box will show the image returned from the simulator and text box allows both navigation control commands and sending natural language in- quires for helper agents. It enables an easy interac- tion among the human task performer, helper agent, and simulator so that the participants can experi- ence a real-time collaboration between helper and performer with only a screen and keyboard. Upon completion of the task, we assess task success and goal progress in the same way in Thomason al. (2020) and we ask the participant rate the natu- ralness and faithfulness of the response from the helper in the last task session.
E Additional LLM Baselines
Comprehensive baselines are essential for vali- dating our approach, particularly in light of the growing focus on Large Language Models (LLMss). Given the complexity of our tasks, the multimodal LLM (mPLUG-OWl) serves as an adequate LLM baseline as it inherently supports our task require- ments by natively accommodating both image se- quences and text for language generation. Addition- ally, we have also explored two additional baselines that leverage the power of LLMs as detailed below and the result is shown Table 7.
Multimodal-LLM + ChatGPT We experiment with prompting the Multimodal-LLM (mPLUG- Owl) to generate only captions for the image se- quences. Subsequently, we use ChatGPT to serve as the helper agent, generating responses based on the captions and queries from the task performer.
BLIP2 Model with stacked images input We employ the BLIP2 model (Li et al., 2023) in zero- shot manner. BLIP2 model benefits from a generic and efficient pre-training strategy that combines pretrained vision models with LLMs for vision-

---

CVDN
CVDN validation unseen
DialFRED validation
DialPRED unseen
AVON
AVDN
validation
seen
validation unseen
GP
GP
SR
SR
SPI.
SPI.
Mulrimodal LIM baseline (
366
47.0
33.8
0.7
1.5
mPLL'G-Ow+= ChaiGPT
BLIP2
566
34
38.6
24.7
3,8
5,0
Table 7: Evaluation of multimodal LLM baselines the RdH task. Additional LLM baselines (bottom two rows) are compared with regard to the multimodal LLM baseline using mPLUG-OWl.
viewpoint in front
vicwpoint
Prese contirm your hout
weupoit
Sntir your irput.
left
loop' stop.
tum
followed wit question sak help.
T tunn ett.
stop to slop a' fellowe with ouestoo on to ask helo.
Goal: Find the nightstand
Dialog history:
Find
You: where
go?
Dialog history:
Helper: Turn around and exit the room.
Should go down here?
You: Is destination in room on
Helper: Yes down stairs and turn left.
Helper: No, turn right and down stairs.
(i) Example
(ii) Example
Figure 5: Examples of interface used during human evaluation based on Rdl task with CVDN (Thomason et al., 2020) dataset.

---

(c)
(g)
(h)
(k)
Figure 6: An example of data in RDH task CVDN dataset. Human inquiry: Shall continue assuming this is not the bathroom? Ground truth human response: Yes leave the bedroom turn right go into the bedroom the left.
(a)
(b)
(c)
(d)
(c)
g
(h)
Figure 7: An example of data in RDH task on the AVDN dataset.
Human inquiry: see some warehouses in my view. Am the destination? How go destination? Ground truth human response: You are not close to your destination yet, head north and you will reach the complex gray warehouse office.

---

(b)
(c)
(d)
(+)
(I)
(8)
(h)
(i)
(k)
(1)
(m)
Figure 8: An example of data in RDH task on DialFRED dataset.
Human inquiry: What the color of the sink basin? Ground truth human response: The sink basin is Grey.

---

Seen
Unseen
Task
Helpper
GP
SPL
GP
SPL
SR
Performer
Haman Annotator
6.9
17.6
20.7
3.1
11.2
17.1
& Multimodal LLM
HAMT
5.3
14.7
17.5
3.6
8.2
13.1
ScoRee
(a) CVDN dataset
Seen Validation
Unseen Validation
Task
Helper
SR
PWSR
SR
PWSR
Performer
Human Annotator
49.1
39.1
33.4
14.6
DID
RMM
46.5
35.3
32.1
13.8
Multimodal L.M
Model
47.0
32.5
33.8
14.2
SocRoe
(b) DialFRED dataset
Validation
Validation
Helper
Task Performer
SPL
GP f
Human Annotator
56.3
14.7
17.3
55.2
16.5
20.4
RMM G Mullimodal LLM
BAA-
37,0
2.4
26.6
4.9
Transfommer
-96.5
0.7
0.8
.95.
1.5
222
SoeRee
-29.6
-249
(c) AVDN dataset
Table 8: Results of Respond to Dialog History (RDH)
on CVDN dataset (a), DialFRED dataset (b) and AVDN dataset (Thomason et al., 2020; Gao et al., 2022; Fan et al., 2022). We replace the response validation with responses from different helper agents. Since the task performer agent is trained with human dialog and is fixed, it relies on the information contained in the response complete the task. Thus, the better the task performer agent performs in the task, the more effective the response is.


Reviews:
[{"review_id": "FLPhdWkrWV", "reviewer": "Reviewer_yJ8D", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review comments. In the following response, we have organized our answers to address each point raised in your comments. We are committed to incorporating additional results in the future revision of the paper based on your feedback.\n\n**Challenges in building helper agents**\n\nDue to the challenges posed by real-world complexity and dataset scarcity, the task of building helper agents becomes not just difficult but also critically important. It is precisely these challenges that underscore the urgent need for focused research in this area. Our work aims to advance this important direction by developing helper agents that can be beneficial in real-life applications.\n  \n\nTo mitigate the issue of data scarcity, we have leveraged three existing datasets that were originally developed for benchmarking task performers. By doing so, we not only make efficient use of available resources but also demonstrate the broader applicability of our model.\n\n  \n\nOur benchmark is the first to focus specifically on the development of multimodal navigational helper agents. We believe that this will help to foster more future works in this direction, thereby gradually addressing the challenges related to real-world complexity and dataset scarcity.\n\n\n\n**Additional baselines**\n\nWe agree that comprehensive baselines are essential for validating our approach, particularly in light of the growing focus on Large Language Models (LLMs). Given the complexity of our tasks, the multimodal LLM (mPLUG-Owl) serves as an adequate LLM baseline as it inherently supports our task requirements by natively accommodating both image sequences and text for language generation. Additionally, we have also explored two additional baselines that leverage the power of LLMs:\n\n1. Multimodal-LLM + ChatGPT: We also experimented with prompting the Multimodal-LLM (mPLUG-Owl) to generate only captions for the image sequences. Subsequently, we used ChatGPT to serve as the helper agent, generating responses based on the captions and queries from the task performer.\n2. BLIP2 Model with stacked images input: We employed the BLIP2 model [1] in a zero-shot manner. BLIP2 model benefits from a generic and efficient pre-training strategy that combines pretrained vision models with LLMs for vision-language pretraining. Due to the model's limitation of accepting only a single image input, we stacked the input image sequences into a single image arranged in four columns. The text prompt used is the same as the one we adopted for the Multimodal-LLM (mPLUG-Owl) in Table 4 of our paper. \n\nThe result of additional baselines is shown in the following table with the same format as in Table 1 from the paper.\n  \n|                                         | CVDN validation seen GP | CVDN validation unseen GP | DialFRED validation seen SR | DialFRED validation unseen SR | AVDN validation seen SPL | AVDN validation unseen SPL |\n|-----------------------------------------|-------------------------|---------------------------|-----------------------------|-------------------------------|--------------------------|----------------------------|\n| Multimodal LLM baseline reported in the paper (mPLUG-Owl)     | 5.3                     | 3.6                       | 47.0                        | 33.8                          | 0.7                      | 1.5                        |\n| Additional baseline mPLUG-Owl + ChatGPT | 5.3                     | 3.5                       | 43.5                        | 31.8                          | 1.2                      | 1.6                        |\n| Additional baseline BLIP2               | 5.6                     | 3.4                       | 38.6                        | 24.7                          | 3.8                      | 5.0                        |\n\n\nOverall, our additional baseline mPLUG-Owl + ChatGPT shows performance comparable to the multimodal LLM baseline mPLUG-Owl reported in the paper. The BLIP2 baseline is weak on the DialFRED dataset but performs particularly well on the AVDN dataset. \n\n\n**Reference:**\n[1] Li, Junnan, et al. \"Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.\" arXiv preprint arXiv:2301.12597 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The R2H benchmark could become a standard for evaluating and comparing models in the domain of multi-modal navigation, fostering further research and innovation.\n2. The paper explores two approaches to construct navigation helper agents: fine-tuning a task-oriented multi-modal response generation model named SeeRee, and employing a multi-modal large language model in a zero-shot manner.\n", "reasons_to_reject": "1. The challenges related to real-world complexity and dataset scarcity may limit the broader applicability of the proposed model.\n2. Authors may need to use other more LLMs for experimental comparisons.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces the Respond to Help (R2H) benchmark for developing intelligent navigation helper agents. It includes two tasks: Respond to Dialog History (RDH) and Respond during Interaction (RdI). The authors present two approaches, including a novel model named SeeRee, and a multi-modal large language model. Meanwhile the authors conducted experiments on existing vision-and-dialog navigation datasets to demonstrate the effects.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Sp2MRkX7KK", "reviewer": "Reviewer_DxYF", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review comments. In the following response, we have organized our answers to address each point raised in your comments. We are committed to incorporating additional results in the future revision of the paper based on your feedback.\n\n**Limited choices of baseline methods**\n\nWe acknowledge the reviewer's concern regarding the limited experimental evaluation and comparisons with existing methods. However, it's important to note that the scarcity of directly comparable baseline methods is precisely why our work focuses on this new direction. Previous works have either concentrated on benchmarking task performers (Figure 2 i) or on benchmarking helper-performer agent pairs (Figure 2 ii). Our work proposes a new benchmark specifically for helper agents (Figure 2 iii), filling a gap in the existing literature.\n\n\n\n\n**Additional baselines**\n\nWe agree that comprehensive baselines are essential for validating our approach, particularly in light of the growing focus on Large Language Models (LLMs). Given the complexity of our tasks, the multimodal LLM (mPLUG-Owl) serves as an adequate LLM baseline as it inherently supports our task requirements by natively accommodating both image sequences and text for language generation. Additionally, we have also explored two additional baselines that leverage the power of LLMs:\n\n1. Multimodal-LLM + ChatGPT: We also experimented with prompting the Multimodal-LLM (mPLUG-Owl) to generate only captions for the image sequences. Subsequently, we used ChatGPT to serve as the helper agent, generating responses based on the captions and queries from the task performer.\n2. BLIP2 Model with stacked images input: We employed the BLIP2 model [1] in a zero-shot manner. BLIP2 model benefits from a generic and efficient pre-training strategy that combines pretrained vision models with LLMs for vision-language pretraining. Due to the model's limitation of accepting only a single image input, we stacked the input image sequences into a single image arranged in four columns. The text prompt used is the same as the one we adopted for the Multimodal-LLM (mPLUG-Owl) in Table 4 of our paper. \n\nThe result of additional baselines is shown in the following table with the same format as in Table 1 from the paper.\n  \n|                                         | CVDN validation seen GP | CVDN validation unseen GP | DialFRED validation seen SR | DialFRED validation unseen SR | AVDN validation seen SPL | AVDN validation unseen SPL |\n|-----------------------------------------|-------------------------|---------------------------|-----------------------------|-------------------------------|--------------------------|----------------------------|\n| Multimodal LLM baseline reported in the paper (mPLUG-Owl)     | 5.3                     | 3.6                       | 47.0                        | 33.8                          | 0.7                      | 1.5                        |\n| Additional baseline mPLUG-Owl + ChatGPT | 5.3                     | 3.5                       | 43.5                        | 31.8                          | 1.2                      | 1.6                        |\n| Additional baseline BLIP2               | 5.6                     | 3.4                       | 38.6                        | 24.7                          | 3.8                      | 5.0                        |\n\n\nOverall, our additional baseline mPLUG-Owl + ChatGPT shows performance comparable to the multimodal LLM baseline mPLUG-Owl reported in the paper. The BLIP2 baseline is weak on the DialFRED dataset but performs particularly well on the AVDN dataset. \n\n\n**Reference:**\n[1] Li, Junnan, et al. \"Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.\" arXiv preprint arXiv:2301.12597 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper addresses the fundamental capability of AI agents to assist humans in real-world scenarios. This is an essential and practical area of research, as AI systems that can communicate naturally with humans and provide assistance based on visual observations hold the potential to significantly enhance productivity and accessibility, particularly for individuals with disabilities.\n2. The authors introduce the Respond to Help (R2H) benchmark, which serves as a crucial contribution to the field. This benchmark is designed to automatically evaluate conversational multi-modal navigation helper agents in dynamic cooperative interactions with other agents acting as task performers.\n3. The paper introduces a novel Conditional Optimized Sparse (COS) attention mask and a Parse by Step approach to transform ground-truth human responses into structured, noise-free step-by-step instructions. These contributions enhance the training process and improve the performance of the navigation helper agent.", "reasons_to_reject": "The experimental evaluation of the proposed approaches is somewhat limited, lacks proper comparisons with existing methods, which may raise doubt about the reliability and effectiveness of the proposed solution. A strong experimental validation is crucial to establish the viability of the proposed techniques.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces a new benchmark called Respond to Help (R2H), aimed at facilitating the development of multi-modal navigation assistants. This benchmark leverages existing dialog-based embodied datasets to encourage the creation of navigation helpers capable of responding effectively to help requests. R2H consists of two tasks: Respond to Dialog History (RDH), assessing the agent's capacity to provide informative responses based on past dialog, and Respond during Interaction (RdI), which gauges response effectiveness during cooperative interactions. Two strategies for constructing the navigation helper agent are explored: fine-tuning a task-oriented multi-modal model called SeeRee, designed for perceiving and responding, and using a large multi-modal language model in a zero-shot manner. The research entails both automatic benchmarking and human evaluations to analyze the tasks and methods. The authors plan to release the code and data associated with this study.\n\n\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0RRRgnGdpB", "reviewer": "Reviewer_6i43", "summary": "", "questions": "Have you investigated whether different task performers affect the results? Appendix A.2 says the use of different task performers for different datasets. It would be helpful to see the results of using different task performers within the same dataset.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review comments. In the following response, we have organized our answers to address each point raised in your comments. We are committed to incorporating additional results in the future revision of the paper based on your feedback.\n\n\n**The unique focus of our R2H benchmark: building helpers**\n\nOur work focuses on the development and importance of multimodal navigation helper agents in real-world, practical scenarios. While much of the existing literature has been centered on improving task performers either independently (Figure 2 i) or in a two-agent collaboration setting (Figure 2 ii), there is substantial untapped potential in specifically advancing helper agents for collaborative tasks. First, creating helper agents is a valuable research endeavor in its own right, with real-world applications such as guiding humans in package delivery or room-finding tasks. Second, even in the two-agent collaboration setting, optimizing the helper agent is an under-explored direction, and our study showcases how these optimized helper agents can be readily integrated into two-agent collaboration, thereby enhancing the effectiveness of the task performers.\n\nMoreover, existing dialog-based embodied datasets like CVDN, DiaFRED, and AVDN offer a unique opportunity. These datasets are compiled through human-to-human interactions where humans perform both the helper and performer roles. Consequently, these datasets are not exclusively geared towards evaluating two-agent settings (Figure 2 ii). They also serve as robust platforms for the assessment of either task-performing agents (Figure 2 i) or helper agents (Figure 2 iii) in isolation. Leveraging the rich human-to-human interactions captured in these datasets, we have established a new benchmark tailored for the evaluation of helper agents in scenarios where the task performer is either a fixed model or a human participant.\n\nIn sum, our work not only distinguishes itself by focusing on independent helper agents but also pioneers new pathways for research and practical applications in this relatively unexplored domain.\n\n**Additional benchmark analysis**\n\nWe appreciate the reviewer's call for a more detailed benchmark analysis. To address this, we have conducted additional analyses on our R2H benchmark across the three datasets included in the following table. Specifically, we calculated:\n\n-   The number of navigation trajectories;\n    \n-   The number of queries per trajectory, which is the same as the number of responses per trajectory;\n    \n-   The average length of responses and queries provided by human annotators;\n    \n-   The average number of images in the image sequences sampled from the environment by script-based samplers, which serve as input to the helper agent.\n    \n|          | # navigation trajectories | # querries per trajectory | Avg length of human response (words) | Avg length of human query (words) | Avg # images in the input image sequence to helper agents |\n|----------|---------------------------|---------------------------|--------------------------------------|-----------------------------------|-----------------------------------------------------------|\n| CVDN     | 2050                      | 2.3                       | 14.9                                 | 9.7                               | 11.8                                                      |\n| DialFRED | 120958                    | 1.0                       | 6.0                                  | 7.0                               | 10.8                                                      |\n| AVDN     | 5372                      | 1.8                       | 19.7                                 | 9.0                               | 9.1                                                       |\n  \n\nWe would like to emphasize that the datasets we've adopted cover a wide range of environment types, including both indoor (CVDN and DialFRED) and outdoor (AVDN) settings, as well as synthetic (DialFRED) and photo-realistic environments (CVDN and AVDN). This diversity adds depth and robustness to our benchmark.\n\nFor further clarity, we have provided examples from each dataset in Figure 6, 7 and 8 of our paper's Appendix, allowing readers to fully grasp the intricacies of our benchmark.\n\n\n\n\n**Language similarity result and effectiveness of the conversational helper agents**\n\n  \n\nFirst, we want to point out that a single query from the task performer could elicit multiple valid responses, and there is no single \"ground truth\" response for every given query. For instance, if a generated response closely resembles a human response but reverses a crucial directional word (e.g., \"left\" is replaced with \"right\"), the effectiveness of the response for aiding the task performer in navigation would be compromised.\n\n  \n\nBesides the intuitive analysis, we also draw conclusions from quantitative results. We use BLUE2 and ROUGE-L to evaluate the similarity of generated responses compared with human responses, and the GP to measure how well the task performer moves towards the destination, indicating the effectiveness of the response in aiding the task performer. In Table 2 of our paper (the bottom two rows), we present evidence that language similarity metrics, BLUE2 and ROUGE-L, do not consistently correlate with the GP metric. For instance, when BLUE2 decreases from 14.4 to 13.8 and ROUGE-L decreases from 25.5 to 24.2, the GP actually increases from 5.9 to 6.5. This quantitative result underscores that a high language similarity to human responses is not a reliable indicator of an effective conversational helper agent.\n\n  \n\nTo further substantiate this point, we conducted an experiment where we deliberately altered human responses in the CVDN dataset. Specifically, we removed the directional words \"left\" and \"right\" and also eliminated the last two words in all responses, which usually indicate the target. We compare the evaluation of these altered human responses with responses generated from our SeeRee helper in the table below. Despite achieving high language similarity scores, the altered human responses lead to significantly lower task performer's GP scores. demonstrating that language similarity is not a reliable indicator of the effectiveness of a conversational helper agent.\n|                         | Seen Validation GP | Seen Validation BLUE2 | Seen Validation ROUGE-L | Unseen Validation GP | Unseen Validation BLUE2 | Unseen Validation ROUGE-L |\n|-------------------------|--------------------|-----------------------|-------------------------|----------------------|-------------------------|---------------------------|\n| Altered human responses | 6.2               | **83.6**              | **95.6**                | 4.2                 | **83.5**                | **95.9**                  |\n| SeeRee                 | **6.5**            | 13.8                  | 24.2                    | **4.9**              | 13.2                    | 24.1                      |\n\n\n\n\n**Conditional Optimized Sparse attention mask**\n\nFirstly, we would like to point out the ablation study result based on CVDN dataset presented in Table 2, showing that the COS attention mask enhances the ability of the helper model to assist the task performer in reaching the destination. Besides the CVDN dataset, we also conducted an ablation study on the COS attention mask using the AVDN dataset. Same as the ablation study in Section 4.4 and Table 2, we train the SeeRee with a fixed COS attention mask (all ones) on the AVDN dataset, and the evaluation is done with the same task performer as used in the paper, the HAA-Transformer. We list the results in the table below. \n\n|         helper                      | Seen validation SPL | Unseen validation SPL |\n|-------------------------------|---------------------|-----------------------|\n| SeeRee w/o COS attention mask | 3.18                | 3.88                  |\n| SeeRee                        | 4.6                 | 4.44                  |\n\nThe result shows quantitative evidence as in the paper, underscoring the effectiveness of our proposed COS attention mask.\n\n  \n\nTo further understand the behavior of the COS mask, we conducted a qualitative analysis through visualization of the COS attention mask. We observe that the COS mask learns a hybrid strategy. For different input image sequences, there is a generally similar mask pattern. However, adjustments are made, particularly when the length of the input image sequence varies. These variations between COS masks for different sequence lengths are quite significant. Moreover, we randomly sampled a few examples in the validation set of AVDN, and for each token in the video embedding, we computed the averaged corresponding value in the COS attention mask. Then since the video embedding has dimensions T/2 * H/32 * W/32, we reshaped the corresponding COS attention mask and observed that the learned COS attention mask shows a preference for the edge regions of every image. We believe this is because, in the CVDN dataset, the edge regions of the input images are more coherent and thus more informative for determining the direction of movement.\n\n\n\n**Different task performers within the same dataset**\n\nIn the paper, we select HAA-Transformer as the task performer for AVDN dataset. Here we ran additional experiments using a different task performer, HAA-LSTM, within the same AVDN dataset to evaluate helper agents on the RDH task. The new results on AVDN unseen validation set are shown in the following table. \n|       Helper         | SPL of HAA-LSTM  | SPL of HAA-Transformer   |\n|----------------|--------------------------|----------------------------|\n| RMM G          | 2.8                    | 3.4                       |\n| Multimodal LLM | 3.6                    | 3.7                       |\n| SeeRee         | 4.9                     | 4.4                       |\n\n\n\nOur new results show that the evaluation outcome is consistent with different task performers. This consistency reinforces the robustness of our approach, indicating that it is not heavily influenced by the choice of task performer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written and presents the information clearly.\n2. The Conditional Optimized Sparse attention masks for reducing redundancy in repetitive images while retaining critical details is intuitive and effective.", "reasons_to_reject": "1. The main concern lies in the motivation behind the R2H benchmark. It only evaluates the helper agent, while some datasets require collaboration between both helper and performer agents (Fig 2 ii). The explanation provided in Line 161-164 is not sufficiently convincing.\n2. The paper needs more benchmark analysis. While the benchmark is built on three existing datasets, readers need more details to understand its intricacies fully. \n3. The paper discusses the Language Generation Similarity Result in Line 529-532: a high language similarity to human responses may not necessarily indicate a more effective conversational helper agent. Further analysis is required to understand the underlying reasons for this situation.\n4. More analysis of Conditional Optimized Sparse attention mask is needed to prove its effectiveness, such as what does the learnable conditional mask C capture, and how does it appear on the test set.", "questions_to_authors": "Have you investigated whether different task performers affect the results? Appendix A.2 says the use of different task performers for different datasets. It would be helpful to see the results of using different task performers within the same dataset.", "paper_topic_and_main_contributions": "This paper presents the Respond to Help (R2H) benchmark as a testbed for automatically evaluating the capabilities of multi-modal conversational navigation helper agent. The helper agent helps task performers complete tasks by providing natural language responses to inquiries based on environment information. \nThe benchmark comprises two tasks: (1) Respond to Dialog History (RDH), which evaluates the helper agent's ability to generate informative responses from given dialog histories, and (2) Respond during Interaction (RdI), which assesses the effectiveness and efficiency of the agent's responses during consistent cooperation with a task performer. The paper explores two approaches to constructing the helper agent: (1) fine-tuning a task-oriented multi-modal response generation model capable of seeing and responding, and (2) employing a multi-modal large language model in a zero-shot manner.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a benchmark that is a valuable contribution in that it could/should become a standard for evaluating navigation agents on the more specific characteristics introduced here (responding to help, keeping track of history, cooperating with the helper). The insights and novel attention-mask schemes are also good contributions to building better multi-modal navigation agents. The authors provided detailed answers to alleviate most reviewer concerns, added additional baselines to compare to, and added more analysis of the datasets used to address questions and concerns of the reviewers. This paper, and the evaluation benchmark that comes with it, is a good contribution to the language/RL/navigation field.
Decision: Accept-Main