Towards Versatile Embodied Navigation
Authors: Hanqing Wang
Wei Liang
Luc Van Gool
Wenguan Wang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Versatile Embodied Navigation
Multitask Multimodal Embodied Navigation

Abstract: With the emergence of varied visual navigation tasks (e.g., image-/object-/audio-goal and vision-language navigation) that specify the target in different ways, the community has made appealing advances in training specialized agents capable of handling individual navigation tasks well. Given plenty of embodied navigation tasks and task-specific solutions, we address a more fundamental question: can we learn a single powerful agent that masters not one but multiple navigation tasks concurrently? First, we propose VXN, a large-scale 3D dataset that instantiates~four classic navigation tasks in standardized, continuous, and audiovisual-rich environments. Second, we propose Vienna, a versatile embodied navigation agent that simultaneously learns to perform the four navigation tasks with one model. Building upon a full-attentive architecture, Vienna formulates various navigation tasks as a unified, parse-and-query procedure: the target description, augmented with four task embeddings, is comprehensively interpreted into a set of diversified goal vectors, which are refined as the navigation progresses, and used as queries to retrieve supportive context from episodic history for decision making. This enables the reuse of knowledge across navigation tasks with varying input domains/modalities. We empirically demonstrate that, compared with learning each visual navigation task individually, our multitask agent achieves comparable or even better performance with reduced complexity.

Towards Versatile Embodied Navigation
Hanqing Wang 1.2
Wei Liang 1.4
Luc Van Gool
Wenguan Wang"
Beijing Institute of Technology *Computer Vision Lab, ETH Zurich
ReLER AAII, University of Technology Sydney
Yangtze Delta Region Academy of Beijing Institute of Technology, Jiaxing
Project page: https:/ haaps:///ithbbb www/hanngingwwaaaa//Wa.
Abstract
With the emergence of varied visual navigation tasks (e.g.. image-/object//audio goal and rision-language navigation) that specify the target in different ways, the community has made appealing advances in training specialized agents capable of handling individual navigation tasks well. Given plenty of embodied navigation tasks and task-specife solutions, we address a more fundamental question: can we learn single powerful agent that masters not one but multiple navigation tasks concurrently? First, we propose VXN, a large-scale 3D dataset that instantiates four classic navigation tasks in standardized, continuous, and audiovisual-rich environ- ments. Second, we propose VIENNA, versatile embodied navigation agent that simultancously learns to perform the four navigation tasks with one model. Building upon full-attentive architecture, VIENNA formulates various navigation tasks as unified, parse-and-quee procedure: the target description, augmented with four task embeddings, is comprehensively interpreted into a set of diversified goal vec- tors, which are refined as the navigation progresses, and used as queries to retrieve supportive context from episodic history for decision making. This enables the reuse of knowledge across navigation tasks with varying input domains/modalitier We empirically demonstrate that, compared with learning each visual navigation task individually, our multitask agent achieves comparable or even better perfor- mance with reduced complexity.
Introduction
As a fundamental research topic, visual navigation has attained extensive attention across many disci- plines, including robotics [1], computer vision [2,3], and natural language processing [4]. Consider a typical navigation scenario (Fig. 1), in which a human intends to direct a robot agent to navigate to target- buzzing washer. The target can be specified by photo of the washer (i.e., P image-goal nav. [5]), or the buzzing sound (i.e., 49 audio-goal nav. [6]), the corresponding semantic tag washing machine (i.e., @object-goal nav. [7]), or linguistic instructions - "go to the end of this corri- dor, turn left and enter the laundry-room" (i.e., vision-language nav. [8]). Naturally, the agent is expected to be smart enough to execute all these kinds of navigation tasks involving varying modali- ties/domains (i.e., image, audio, semantic tag, text) with different optimal policies. Contrary to expectation, almost all existing navigation agents are specifically designed/trained for one specific task "versatile" agent capable of mastering multiple navigation tasks remains far beyond reach.
Besides its great value in practice, investigating embodied navigation in multitask scenarios can help better understand human intelligence. First, we humans can learn multiple tasks in a parallel ad hoc
"Corresponding authors.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

washer
image-goal nav.
audio-goal naw.
washing machine
go to the end of this corridor, enter laundry 190m.
&
object goal nav.
vision- langun-lngggggeeee nav.
Figure I:
Rather than existing efforts training specialized agents on individual navigation tasks, we build single powerful agent that undertake multiple tasks, i.e., P image-god nav., <8 audio-goal nav., 4 object-goal nav., and vision-language nav. in visually and acoustically realistic environments.
manner, and benefit from commonalitics across related tasks [9]. Second, we accomplish tasks by processing and combining signals from different modalities. Evidences from cognitive psychology indicate that our senses are functioning together and multisensory integration is a central tenant of human intelligence [10, 11]. Though the idea of multitask learning [12] was widely explored in com- puter vision field [1 [13], prior attempts are often made in unsupervised and supervised learning settings; in the context of multitask reinforcement learning (MTRL) [14], not much is done for visually-ries embodied navigation scenarios. One possible reason is the lack of a suitable dataset, compounded by considerable costs involved in data collection, as multiple navigation tasks should be supported.
In response, a large-scale 3D dataset, VXN, is established to investigate multitask multimodal embodied navigation in audiovisual complex indoor environments. VXN allows simulated robot agents to concur- rently learn four tasks, i.e., image-goal nav., audio-goal nav, object-goal nav, and vision-language mwwstiicall--rallstti and perceptually-riee world Based on a throoghuuu
nav. in continuous, simulator [15], VXN instantiates different navigation tasks in unified environments following the same physical rules. It equips the agents with multimodal sensors to gather information from 360 RGBD and audio observations. Taken all together, VXN provides a realistic testbed for multitask navigation.
With VXN, we further develop VIENNA, a versatile embodied navigation agent that jointly learns to solve the four navigation tasks using one single model without switching among different models. Based on Transformer encoder-decoder architecture [16]. VIENNA encodes the full episode history of multisensory inputs (i.e., RGB, depth, and audio) and navigation actions, and absorbs common knowledge across different navigation tasks with a shared decoder. Target signals (i.e., goal picture, target class, aural cues, linguistic instruction) are parsed into queries, and the supportive context retrieved from the encoded history is fed to corresponding policy for task-specific decision making. With such a fully- attentvveee model design, VIENNA able to comprehend multimodal observations, conduct long-to tergteermiie reasoning, and, more essentially, exploit tosss-task knowledge.
By contrasting our VIENNA to several single-task counterparts on VXN, we empirically demonstrate i) Better performance. Through exploiting cross-task relatedness, VIENNA outperforms independent task training. ii) Reduced model-size. Training four tasks together using a single VIENNA achieves about four times model size compression, compared with training them individually. iii) Improved generalization. VIENNA performs robust on unseen environments, through learning task-shared, general representations. iv) More is better. The above conclusions are typically true when we train VIENNA on more navigation tasks. v) Multisensory integration does matter. Both visual (RGB and depth) and aural information are crucial building blocks for general -urpose navigation robot creation.
Related Work
Embodied Navigation. As a fundamental element in building intelligent robots, navigation has long been the focus of the scientific community [17]. The availability of building-scale 3D datasets [| [18-21]
Sttrietyy speaking, as we synthesize visually and acoustically realistic environments, the classic image-godl goal nav. object-goal nav. and vision language nav. tasks in our VXN dataset are extended as image-goe risual-audio nav, audio object- goal nav., and visual-auda language nav., respectively.

---

and perfoormaf simulation platforms [15,22-24] led to a plethora of reproducible research of navigation in large-scale, visually-rich environments. Depending on how to specify the target goal. diverse navigation tasks are proposed to let an agent i) navigate to target coordinates
goill nav. [15D) ii) find an instance of given object category (ohject- nav. (7]), iii) search for target photos (image-god) nav. [5]), iv) locate sound sources (audio-godl nav. [6]). follow navigation instru- ctions (vision- Dassion- nav. [8. 25)). Aside from these battlefields, there are some more complicated embodied tasks, such as embodied question answering [26], vision-dialog nav. [27-29], and multiagent nav. [30]. The community also made great strides in improving reinforcement learning (RL) algorithms capable of fulfilling specific navigation tasks, by using, for example, recurrent neural networks [8, 15, 311, map building [3, 3332-41], path planning [42-45], cross-modas attention [41,46-49 46-49], synthesized or unlabeled data [50-55], and external knowledge [7.56]. However, though the learning algorithm is general - RI., each solution is not; each navigation agent can only handle the one task it was trained on.
With various navigation tasks and task- specifie navigation solutions, a critical question arises whether we can build a single general agent that works well for multiple navigation tasks. In response, we make two unique contributions. First, we build a large-scale 3D dataset that supports four representative navigation tasks in continuous and realistic environments. In contrast, prior navigation datasets are built upon different platforms and with certain assumpiion/cconfiuratioo
(e.g., sparse navigation graphs [8], discrete world representation [6]). making them hard to explore different navigation tasks in unified and standardized environments. Second, we create generalist agent which is capable of undertaking a set of navigation tasks of different modalities/domains and is equipped with multimodal sensors (i.e.. RGB, depth, audio) to better address real-worls scenarios. However, existing navigation agents are trained one task at the time, each new task requiring to train a new agent instance.
Multitask I Learning (MTL). MTL [12], inspired by the human ability to transfer knowledge across different tasks [57], has led to wide success in computer vision [58-60] and natural language pro- cessing [61]. Related efforts were made along three directions [13]: i) architecture design (i.e., how to partition the model into task-specific and shared components) [62-65], ii) optimization (i.e., how to balance learning between different tasks) [66-71], and iii) task relationship learning (i.e., how to learn and utilize task relationships to improve learning) [72-74]. In the field of MTRL [75-81], recent solutions explored knowledge transfer [82]. modular networks 183,84], and policy distillation [85,8] 86]. A few robotics benchmarks [87-89] are also proposed for MTRL. However, most of these efforts were based upon low- dimensioon observations, e-g., grrd like or game environments. To the best of our knowledge, there are two prior work [48, 90] that addressed multitask navigation, but they only consider two closely-related, language-guin guided navigation tasks with the same input modalities.
Drawing inspiration from these efforts, we seek for a "universal" agent that can complete multiple navigation tasks with a single agent instance, and distinguish ourselves by i) joint learning of four navigation tasks with diverse input modalities, ii) visually complex and acoustically realistic operation space, iii) multisensory integration, and iv) fully-
architecture based parse-and and-qurry regime.
Auxiliary Learning in Embodied Navigation. There are a group of algorithms that exploit comple- mentary objectives from auxiliary tasks to facilitate navigation policy learning. Specifically, super- vised auxiliary tasks expose privileged information to the agent (e.g., depth [91], surface normals [92], semantics [26], etc.). Self-supervised auxiliary tasks derive free supervisory signals from the agent's 's own experience (e.g.. next-step visual feature prediction [93], predictive modeling [94], loop closure prediction [9]], temporal distance estimation [95], navigation progress estimation (96,97) etc.).
Although auxiliary learning based navigation models are also trained on set of tasks, their ideas are far away from ours. These models still focus on only a single "main" navigation task with extra aid of auxiliary intermediate objectives. while we aim capture and utilize common knowledge of a collection of different navigation tasks to enhance the performance on all the tasks. Moreover, their auxiliary tasks, in principle, can be utilized by our agent, but they cannot handle our task setting.
Transformer in Embodied Navigation and MTL. Inspired by the great success of Transformer [16] in sequence transduction tasks, a few recent methods applied Transformer for certain navigation tasks [40, 98-. Rather than sharing similar advantages in long-term memory and cross-modal information fusion, our method further formulates different navigation tasks as a unified process of active goal parsing and supportive information query. Through cleverly encoding all task-specific embeddings into goal parsing, our agent is able to explicitly leverage cross-task knowledge to boost different navigation tasks. There are also a few notable studies that exploit Transformer- network architectures for MTL [103-106], while none of them addresses embodied visual tasks.
3

---

Table 1: Data splits and the number of navi- gation episodes in our VXN dataset ($3.2).
Table 2: Comparison ($3.2) of navigation datasets (MT: multitask; CS: continuous space; VR AR: visual/audio realistic; PA: panorama).
Navigation Task
train
seen
unseen (1I environments) episodes
(58 environments)
cavirommenss)
Navigation Datiset
Year
MT
C8
VR
KR
Pa
Audio-goul
2.0M episodes
episodes
Hobitar- PountGoall [15]
EQA [26]
2019
2018
Vision-langee longuagg
10,819 episodes
778 episodes
1,899 episades
RIR
Object-gool
2.6M episodes
500 episodes
2,195 episodes
VLN-CE [25]
2020
V
Image gool
5.0M episodes
495 episodes
495 episodes
Gibson-Image ImageGGaal [7]
SoundSpaccss (6)
2020
2020
Total
9.6M episodes
2,273 episodes
5,029 episodes
Pretraining in Embodied Navigation. A series of methods decompose embodied tasks into visual (and linguistic) representation learning and policy training [53,92, 107-109]. They pretrain general model on easily-acquired viual or multimodal data (e.g., image captions) and fine-tune the policy for "downstream" navigation tasks. Though showing improved generalization and transfer abilities, the result is still collection of independent task-specific models rather than single agent instance.
VXN Dataset for Multitask Multimodal Embodied Navigation
3.1 Task Collection and Dataset Acquisition
VXN includes four famous navigation tasks, i.e., image-goal Rav. [5], audio-goal nav. [6], object-goal nav. [7], and vision-language nav. [8]. These tasks are with different input modalities/domains (i.e., visual, audio, semantic tag, and language); their original datasets adopt different world representations (i.e., graph based [8] vs discrete [6] vs continuous [15]), environment configurations (i.e., visually poor [22] vs perception rich [8] vs audiovisual realistic [6]), and success criteria (i.e., 3 m [8] vs m [5] vs m [7] vs m [100]). Hence, to study these four tasks in a single learning system, it is desired to build standardized dataset that initiates them with similar problem settings, e-g., dynamic transition, world representations, and audiovisual properties, instead of simply combining several single-task navigation datasets together. On the other hand, it is wise to maximize the reuse of existing datasets, ensuring continuity and compatibility w.r.t. former research, and reducing data annotation cost.
As many previous navigation datasets [6,8, (6,8,110) 110] are built upon Matterport3D (MP3D) [19] environments and Habitat [15] simulator, we derive unified, multitask navigation dataset- VXN - by converting previous task- mask-specific datasets to standardized, continuous, audiovisual- environments:
Our audio-god gool nav. is built upon SoundSpaces [6]. which offers audio renderings for MP3D and allows to navigate sounding targets, or conduct point navigation with extra aid of audio cues. Due to heavy acoustic simulation cost, [6] uses a grid-based world model: it samples room impulse res- ponse over a discrete, horizontal plane (1.5 m above the floor with 0.5 m x 0.5 grid size). devise an audio simulator to efficiently transfer grid-level audio renderings into continuous setting (cf. $3.2).
Our languag nav, is built upon R2R [8]. which labels MP3D with linguistic navigation ins- tructions. R2R is yet bounded to graph-based world representation each scene can be only observed from a few fixed points (~<17) and environment topologies are pre-given. We use [25] to convert R2R to the continuous setting, and then adopt [6] and our audio simulator for audio rendering.
Our image-goal nav. is built upon Habitat [15] ImageNav repository [111]. which is for photo target guided navigation in MP3D environments. Again, continuous audio rendering is made.
Our object-goal nav, is built upon Habitat2020 ObjectNav challenge [1] which requires an agent to navigate MP3D environment to find an instance of an object class. A total of 21 visually well defined object categories (e.g., chair) are considered and audio rendering is also made; but the GPS Compass sensor, used in [110], is not adopted in VXN, for formalizing different task settings.
3.2 Task Setting and Dataset Design
Panoramic Visual Simulator. With Habitat API, 360 egocentric RGBD view is rendered at 300 fps.
Audio Simulator. With [6], ambisonics are generated at locations sampled in MP3D scenes and converted to binaural audio [1 i2], agent emulates two human-like ears. To contioous auditory scenes, we use [113] for real-time binaural room impulse responses (BRIRs) interpolation. We adopt Dynamic Time Wrapping [1 [1]4] to temporally align left and right car BRIRs and then map the
continuous

---

warped interpolated vectors back into the "unwarped" time domain, to get BRIRs at arbitrary locations and directions. As in [100], the sounds of the 21 object categories [110] in object- goal nav. are used for audio rendering. Moreover, the sounds are associated with the objects of same semantic categories to ensure generating semantically meaningful and contextual audio [ [100]. For mmage-goat nav., object-guee -oaal nav, and assion-lnguuag nav., the audio is used as background sound, which can reveal the geometry of environment [6], complement the visual cues, and make the tasks closer to the real-world. For audio-gods nav, the navigation target is directly specified by the audio.
Episodes and Dataset Splits. In VXN, each episode is defined as a tuple: (scene, audio waveform, agent start location, agent start rotation, goal location, target description). ue use the standard 58/1 1/18 train/val/test split [115] of MP3D environments. Since previous navigation datasets [8, 15, 110] keep test annotations private, we only use train and val environments to create VXN (cf. Table 1).
Action Space. We adopt panoramic action space, which is widely used in recent embodied robotic tasks [3,43). Specifically, the panoramic view is horizontally discreted into a total of 12 sub-views. Agents can move towards a sub-view 0.25 m or stop.
Success Criterion. An episode is considered as successful the agent i) executes stop action, ii) within I m of the goal location, and iii) within time horizon of 500 actions (as in [15,92, 116)).
Dataset Features. As shown in Table 2, VXN poses greater challenges: the agent needs to master four navigation tasks with various input modalities in continuous, audiovisual complex environments, mine cross-task knowledge, and reason intelligently about all the senses available to it (RGB, depth, audio).
4 Our Approach
Problem Statement. In single-task navigation, an agent learns to reach a goal position. This is typically
formulated in RL framework that solves partially observable Markov decision process [1 (1117, a tuple (S,A,9, G, O,P. R. 7). where S, A. G are sets of states, actions and targets, o, = ((sz) denotes the
local observation at global state 8, E S at epoch (decision step) 1, P( 0x) is the transition prob- ability from 8 to 1++1 given action a E.A. R(s,, a) ER gives the reward, and 7 E (0, 1) discounts future rewards. The agent uses policy π (alo, o, g) produce its action o, conditioned its local observation o and target goal E G. and optimizes its accumulated discounted reward J "t "R((e, se, ag).
In our multitask navigation, a single agent needs to master K=4 tasks, i.e.. {audio- goal, object- goal, image-goal, (issonnlaangugee] } in VXN environments. We formalize this a MTRL problem: (S,,(SS,, A, Gk. O,P, Rg, } 1. where the agent concurretly learns K task-specific policies T that maximize the rewards J1.K. The single multitask agent is expected to exploit cross-task knowledge to achieve close or better navigation performance on the K tasks, compared with training K single-task agents individually.
Transformer Preliminary. The core of Transformer [16] is attention function (denoted as fATT), which takes a query sequence #e nxd and a context sequence y E IR as inputs, and outputs:
/ ar, y) = softmax (ax W / ()"").
where ÿ E is is with the same length and embedding dimension d as x, and Rd are learnable query, key, and value projection matrices, respectively. Note that Eq. 1 is applicable to both self-attention in Transformer encoder (i.e., x =))), and cross-attention in Transformer decoder (ie., x + y). Further, each Transformer layer block can be given as:
x' === + fMHA (x. y) "xd
z= fee'+s ER"xd,
(2)
where fMHA refers multi-head attention layer, derived by computing several SXIT in parallel, and fMLP is multi-layer perceptron. The layer normalization is omitted for brevity.
Core Idea. Built upon Transformer encoder-decoder architecture, our VIENNA unifies the four VXN tasks as an attention- based, porse-and--uers framework: the target description g E Gy is online parsed into a set of embeddings, which are used to "query" the encoded episode history; the retrieved suppor- tive cues are fed into the corresponding policy ग& for decision making. To better handle multiple tasks, VIENNA i) learns task-wis, wise context and involves all the task-specific embeddings into target parsing, ii) shares representations among tasks, iii) lets task-specific policies π T reuse knowledge, and iv) trains the polices via multitask version of Distributed Proximal Policy Optimization (DPPO) [1 11]].
VIENNA has three modules (cf. Fig. 2): i) an episodic encoder ($4.1 1) that fuses multisensory cues and encodes the full episode history of navigation; ii) a target parser ($4.2) that actively interprets the
5

---

Target Parser ($4.2)
Multitask Planner (84.3)
U
Javo
po Sartiae,
Fava
+0
panoramic image V.
FATT
0
pantorsttic desch
Jou
TACO
Episodic Encoder ($4.1)
Figure 2: Detailed network architecture of VIENNA, at epoch t in a vision-language nav. episode.
target specification into several embeddings; and ili) a multitask planner ($4.3) that uses the target embeddings query encoded episodic history and leverages the returned context for action prediction.
4.1 Episodic Encoder
At the start of each episode, VIENNA receives a target description g {goal image, target sound, target class, language instruction] and derives an embedding vector g E IR (detailed in $4.2). 3, At each epoch I, VIENNA has 360 egocentric audiovisual perception or, i.e., +GB+
of its surrounding.
Intra-Modal Encoders. A visual encoder fiNo maps perceived panoramic image V2 E R 12x224x224x3
into visual features (i- V12.;] E 12xd, where v IR is the feature vector of i-th sub-view in V, Similarly, a depth encoder fDur and an audio encoder fAUD map the perceived panoramic depth image D, R 12 256x256x1 and spectogram tensor of binaural sound (collected over 12 horizontal di- rections) H, E IR (2x41x44x2 into depth and audio features, i.e., D; = [d1,.." d12 E R 12xd and A,=(a1.. 1 E ERRER 12xd respectively.
Target-Guided Cross-Modal Encoder. With the target description vector E Rdd cross-attention fAm (ef. Eq. 1) separately applied over V., D., and H, to assemble target-related sensory information:
रे = (g. ER" f ER), f H.)
Then ü, de, and h, are concatenated for attention based multisensory information integration (MSI):
p.= d,, ) ER
(4)
where fMSi is achieved by stacking two self-attention based Transformer blocks Eq.
Episodic History Encoder. At epoch t. the multimodal observation embedding o, e 3xdd and latest navigation action embedding a;-1E E Rd are together projected into a compact "navigation token":
e, = (o,, a RR.
All the past navigation tokens, e summed with corresponding epoch embedding vectors, M4 E Rd are collected into sequence and fed into an episode history encoder (EHE) to get contextualized history representation:
@e ē.] = fene ([e) + +11, + pil),
(6)
where SEN is implemented as four self-attention based Transformer blocks (q. this way, VI- ENNA is able store and access its entire episode history of audiovisual observations and actions, lead- ing to persistent memorization and long-term reasoning. The attended history representation é1:c will serve as informative context for pre-
dicting the navigation action o at epoch (detailed in $4.3).
4.2 Target Parser
go the this corridor, turn left and enter laundry-room. go the end this corridor, turn left and enter laundry-room.
VIENNA is equipped with target parser that actively interprets the
Figure 3: Attention visualization of online target parsing (Eq. 8).

---

. to target (no matter it is specified as photo 9I, sound 9A, semantic tag 9r, or linguistic instruction 9r.) as a group of target embeddings, conditioned on the progress of the navigation episode. Guided by the online created target embeddings, valuable context are selected from episodic experiences e for flexible decision- cecision-making.... VIENNA thus formulates various navigation tasks in a unified scheme, allowing to exploit cross-task knowledge.
In image-goal nav, a goal image gr E12224x224x3 is given and embedded as gr= fiMo (gr) -Rxxd In Noxd audio-goal nav, the target is signaled by the binaural sound, i.e., / (1 and 9 H fA (H,) ) Naxd In object-goal nav. the target specified by a semantic tag gr (table, bed, .... }. and embedded into a class vector gr ER In dision-language nav., a language-based trajectory instruction St. is given and mapped into a sequence of word vectors 9 R. M2xd by a bi-LSTM. At the start of each episode, we first build augmented target description embedding GERl GERINexd,
G G-(gi+[T>)* ITAl 96 0 No, g (T.+(T2]>o))
where NG= max ( (Nr, 14 N.) TI.A.T IR are learnable task embedding vectors, and [-]No copies its input Ng times. Assuming VIENNA is in an image-goal nav. episode, we have gr= (gr) > R- and ga= [0] Naxd [o]=0] gr.=[0]Nk Nexd, We pad Ir...T.I to unified length Ng. by replication, so as to get J,,,,TT and make them contribute equally to G. We collect all the task-type embeddings
T....TLL and current target description E {91, 9A, {91,9A, 9r, } into G. In $4.3, we will show this strategy is essential for making use of cross-task knowledge. The target description vector e Rdd used in Eq. is given as: g= fava G). where fAva stands for the average pooling operation.
At epoch t, the target parser comprehends the augmented target description embedding G as a set of N compact embeddings on- thy-fly, conditioned on its episodic, contextualized history encoding ē1::
Q: = ( ... q = fMI (fave (ē1:)), E
(8)
where MMH is a Nehd attention layer (ef. Eq. 2), i..., explain G in different ways, with consideration of current episodic navigation progress. Each of the target embedding vectors q can be viewed as specific, time-varying goal, used to guide action selection a, at epoch t. As shown in Fig. 3, given a navigation instruction "go the end of this corridor, turn left and ..." the agent focuses more on "go to the end of this corridor" at the start of the navigation episode. After reaching the end of the corridor, the agent shifts its attention "furn left". Here a collection of N target embeddings q 1::Van generated at each epoch t, allowing the agent to capture different aspects of target-related information and making the time-varying goal well-planned. For instance, there may exist several essential landmarks in a goal image, or multiple discriminative audio clips in target-emitted sound; during navigation, the agent should be able to pay attention to all these informative clues simultaneously.
4.3 Multitask Planner
At epoch f, a multitask planner (MTP) uses the diversified target embeddings Q:=(gh, ... query the episodic history ē
C. C- .. [e,.-.., ē.)).
where faI is achieved by a four-layer Transformer decoder; the first two layers are shared among the four navigation tasks for capturing task-shared policies, while the last two layers are private for each task for task-specific policy learning. We empirically find such shared trunk based MTP design yields better performance than learning task-specific policies individually or just training one single "universal" policy (cf. $5.2).
The decision-making is conditioned on the retrieved context C, E Nxd and the presentations observations (ef. $3.2), including V [U,.. V12.]] E R 12xd D, = [di...... d12.;] IR 12xd, and A;= ". "., a12.r] E 12xd Specifically, at epoch t, VIENNA makes navigate decision by choosing between the 12 current sub-views
of current
well as extra STOP action. [ di,,, h;,,] as well as a STOP action embedding, i.e., b13,1 = Ö. represented as an all-zero vector, VIENNA predicts probability distribution P = [P..,. .P.3,]
Given 12 subview action embeddings (b,, E g-3d 1 i.e., o = o
Pi, softmax; ( WPb..) E 10.
where i E (1,...,111 ,13)
(10)
As the task embeddings TI,A,T,I are encoded into Qt. which used to find supportive cues from episodic observations ē1.; for long-ten term reasoning and recision-making TI,A,T,L L are essentially
7

---

trained as task-wise context they are sensitive to task-related cues. Thus collecting TI,A,T.L into G (cf. Eq. 7) enables clever use of cross-task knowledge. For instance, during image-goal nav. TΛ can help the agent notice some informative audio signals, Ty can alert the agent to visually essential semanties, while 7y can be activated by crucial landmarks. Related experiments can be found $5.2.
4.4 MTRL based Multitask Navigation Training
Reward Design. With standardized VXN environments, VIENNA adopts a same reward function for the four navigation tasks, i.e., R = R4. Concretely, R has four terms, i.e., a sparse success reward sscceess progress reward pprogress. a slack reward sslack, and exploration reward Texplore- I "succces=22.5 2.5 is only received at the end of successful episode, Y -ppo dist offers dense signals indicating the progress that an action contributes: Ageo dist gives the change in geodesic distance to the goal position by performing the action. Tslack =- -10-3 received at each epoch, penalizes redundant actions. I' Texplore [1 19] divides each environment into a voxel grid with 2.5m x2.5 x 2.5m voxels and rewards the agent for visiting each voxel. Texplore is defined as 0. 0.251,, where 17 = 0*/vis a coefficient that decays as episode epoch & and visited voxel number y increase, and 8 is a decay constant of 0.995.
episesde
Multitask Distributed Proximal Policy Optimization. We present a multitask distributed proximal policy optimization (MDPPO) algorithm, which utilizes the power of parallel processing to train MTRL agents in our continuous and large-scale environments, MDPPO is built upon (DPPO) [1 [118], a distributed version of proximal policy optimization (PPO) [ [120] that bounds parameter updates to a trust region to ensure stability. and distributes the computation over many parallel instances of agent and environment. Similarly, MDPPO has a server-client structure: each client worker has several agent copies that collect experiences from VXN environments, compute and send PPO's gradients to the server; the server worker averages the received gradients, updates the agent, and synchronizes the updated weights with the clients. For balanced multitask learning, i.e., training data in VXN are biased between aision-languag nav. and other navigation tasks: 10.8K vy 2.0~5.0M OM episodes (cf. Table 1), each client worker is required to build four agent copies corresponding to the four VXN tasks.
4.5 Implementation Detail
Network Architecture. The visual encoder fixe is made as an ImageNet [121|-pretrrinn] ResNet50 [122]. The CNN features are fed into linear layer for dimension compression and flattened into feature sequence. Similarly, the depth encoder fDu is a modified ResNet50 CNN. The audio encoder fAUD. following [6]. is a CNN of conv 8 8x8, conv 4 x 4, conv 3 x 3 and a linear layer, interleaved with ReLU. All the sensory features are combined with orientation embeddings. For the epoch embedding μ., use sinusoidal encoding. For the target parser, N 5 target embeddings generated at each epoch t. We set other dyper-parameters as: d=112, 16=16, N₁, = 1200 No=120.
Training and Test. VIENNA is trained on 32 RTX 2080 GPUs for 180 M frames, costing 4, 608 GPU hours. As in [25], we select the checkpoint for evaluation with the best SR on val unseen. For MDPPO, we use four client workers and set the discounted factor 7 as 0.99. We use Adam [123] optimizer with a learning rate of 205x100 Casual attention [16] is adopted to prevent the prediction at epoch from the influence of future tokens after t. Once trained, single instance of VIENNA can conduct the four navigation tasks. As normal, greedy prediction is adopted for action selection.
Experiment
In 15.1, we first report comparison results for the four VXN tasks. In $5.2, we conduct diagnostic studies to examine the efficacy of our core model design. More results are put in the supplementary.
oeveral Baseline. We test several open-source task-specific navigation methods 6,, Note that [2, are re- re-trained on VXN, since they use different training data [5], world representation [6] (discrete vs continuous), or object categories [2] (6 vs 21). For [25], we use its check-point but the success criteria are different (3 m vs m). Thus their scores on VXN are different from the original ones. In addition, we consider a Seq2Seq agent, which also serves as a standard baseline [8, 25]: an LSTM planner encodes the episode history and predicts navigation actions in sequential menner. For all the four tasks, we provide the performance of both the single-task and multitask versions of our VIENNA and Seq2Seq. Further, Random policy, i.e., choosing actions randomly, is included.
8

---

Table 3: Quantitative comparison results ($5.1) on VXN dataset (ST: Single-task; MT: Multitask).
Models
seen val
val
unseen
Models
val
seen
vel
unseep
SR
NE'
OR
SPL
SR'
NE'
OR
SPL
SR'
NE
OR
SPL
SR
NE'
OR
SPL
Bandon
1,9
2.2
1.4
Randon
0.0
17.15
16.84
0.0
0.0
Beq2Seqsr
15.1
10.44
19.1
12.6
9.3
12.02
13.9
7.4
Seg2Begst
17.4
10.11
19.0
15.8
11.0
10.83
13.3
8.8
Seq2Beqar
15.8
10.21
21.3
13.0
1022
10.22
15.4
8.5
Seg2Seequ::
18.1
9.69
20.3
16.0
11.8
10.76
14.1
9.3
Zhu [5]
22.0
13.1
16.6
8.9
Chen ol. [6]
8.84
13.1
9.26
15.7
10.4
VIENNAST
19.9
9.52
23.2
134
12.6
9.83
17.1
9.5
VIENNKAST
124
8.76
274
TW.
W22
16.5
18.6
VIENNASI
22.1
9.43
24.2
14.1
14,3
9.66
18.5
11.1
VIENNAMI
25.3
8.61
23.9
17.8
18.7
8.93
17.9
12.5
(a) image-goal nav. (IGN)
(b) audio-goal nav. (AGN)
Models
val
Models
val
val unseen
SR
NEF
OR
SPI!
SR
NE.
OR
SPL
SR
NE-
OR
SPI.
SR'
NE-
OR
SPL
Randon
7.67
L0
2.1
Randox
8.89
0.0
0.0
Sag2Segst
26.7
6.61
33.3
14.4
8.9
7.31
11.1
4.4
Beq2Beggr
13.2
7.54
17.7
12.1
5.2
8.49
9.7
4.6
Sea2Segeer
28.7
6.45
35.0
138
10.8
7.13
14.0
4.8
Beq2Bequr
17.6
7.29
22.8
15.4
7.6
8.21
13.4
6.5
Chaplos er al. [2]
31.3
6.15
35.2
16.7
17.6
7.08
21.3
7.5
Krantz al. [25]
23.7
7.22
25.9
21.
11.0
7.60
16.2
10.2
"" VIENNAST
38.2
6.11
3644
T71
183
639
22.
8T
VIENNAST
13.9
7.16
26.1
22.2
14.3
7.35
18:5
12.5
VIENNAME
5,92
37.8
19.4
25.1
10.7
VIENNASI
7.08
27.9
24.1
16.3
7.26
(c) object-goal nav. (OGN)
(d) vision-langueen nav. (VLN)
0.10
02
Sthtoe Bate
Bato
Vicssoun
0:0
0.0
253 SEST VISL JIOE 12NN ISIK 1OA
200 D80 VAL TISE 1351 TMOE 2004
200
YOS
JISI JESS SKON 1001
201 MAT
300 TAANS LONS SARSE
(a) image-goal nav.
(b) audio audio-goal nav.
(c) object-ggos goal nav.
(d) vision-language nav.
Figure 4: 4:Training curves of VIENNA agents compared to Seq2Seq agents on the four VXN tasks ($5.1).
Metric. Four widely-used metrics adopted for evaluation: i) Success Rate (SR); ii) Navigation Error (NE); iii) Oracle success Rate (OR); and iv) Success rate weighted by Path Length (SPL) [115].
5.1 Performance Benchmarking
Table 3 reports the comparison results on the four VXN tasks. Some key conclusions are list below:
VIENNA obtains impressive results, under val seen and unseen sets, across all the tasks and eval- uation metrics. This proves the versatility of VIENNA and the power of our parse-and-query regime. VIENNA consistently outperforms Seq2Seq. no matter they are trained on single tasks individually or multiple tasks jointly. Compared with other task-specific competitors 52,5,6, VIENNA gains comparable results on audio-goal nav. and object-goal nav., and performs better on image-god nav. and vision-language nav. tasks. These results verify the effectiveness of our model design. When considering the performance gain from the single-task setting to multitask, VIENNA yields more promising results, compared with Seq2Seq. For example, in Table 3a. VIENNAMI outper- forms VIENNAST by 2.2% SR and 1.7% SR, on val seen and unseen, respectively; however, in the same condition, Seq2SeqNT only provides 0.7% and 0.9% SR gains over Seq2SeqRT. These results demonstrates that VIENNA can make better use of cross-task knowledge.
When considering the performance gap between seen and unseen environments, VIRNNAMTiS is more favored than its single-task counterpart, VIENNAST: For instance, in Table 3c, VIENNAŞT suffers from relatively large performance drop, i.e., 33.2% -1.5% SR; however, VIENNAMT shows reduced degradation, i.e., 33.3% 19.4% SR. in unseen environments. This indicates that investigating inter-task relatedness may help to strengthen the generalizability of navigation agents.
The above results are particularly impressive considering the advantage of VIENNA in efficient parameter utilization, i.e., VIENNAME (31 M) ys VIENNASI X 4 (101 M) vs Seq2Seqиr (27 M) vs Seq2Seqst x (93 M) vs [5] + [6] + [2] + [25] (165M 40 M + 45 M + 38 M 42 M).
Fig. 4 plots the training curves of VIENNASTMT compared to Seq2SeqSTMT for the four VXN tasks in unseen envs. Aligning with the results in Table 3, VIENNA outperforms Seq2Seq. and benefits more from multiple task learning. This shows that VIENNA makes a better use of tross-task knowledge.
9

---

Table 4: Ablation studies ($5.2) with audio-goal nav. (AGN) and vision-language nav. (VLN) tasks.
Modality
(SR
VLN
G (Eq. 7)
AGN (SR')
VLN
Q.
AGN ASR ) VLN (SR)
(34.2)
saen/unsean
seen/unaeam
sean/unseen
seen/unseen
(Eq. 8)
seen/unseen seen/unsees
RGB
2.572.1
21.1711.
episodic larget only opisodic target 4
23.1/17.2
22.7/14.1
21.0/15.6
22.2/13.2
audio only
23.1715.9
0.370.2
X=
23.8/17.8
25.7715.5
RGBD only
2.4/2.3
22.9/
episodic task emhodding
24.2/17.9
/5.5
N=
25:771877
26.37163
RGBD audioo
25.3718.7
26377663
augmented farget embed.
25.3718.7
26.71663
24.9/18.
26.2716.2
(a) multisensory integration
(b) augmented target eebieeie
embedding
(c) diversified target parsing
Fxrr
AGN
VLN (SR')
AGN (SR')
(SR
AGIN (SR')
(LN(SR)
($4.4)
Task
(Eq. 9)
seen/unsoen sean/unseen
seen/unseen seen/unsecn
seea/uneeen
seen/uneeen
separate
25.07 27,2/
2.5
2.5/
single task
22.1/
23.8/ 14,1
1 shared
28.4718.1
25.8715.7
16.3
23.87147
AGN VLN
12.7/
24.47 14.
Shared
28:371877
263776"" 16.3
23.1 16.99
15.1
AGN + VLN IGN
24.1/17.3
25.17
julf- shardd
3-sbared
24.1717.6
27,8/17.9
25.67133
25.9/15.5
Trmgpren Tappiin
25.3 / 18.7
16.3
+ VLN+ IGN+ OGN
26.5 18.3
(d) multitask planner
(e) reward function
(f) multitask learning
5.2 Ablative Study
To thoroughly test the efficacy of crucial components of VIENNA, we conduct a series of diagnostic studies on vision-language nav. and audio-goal nav. tasks. The results are summarized in Table 4.
Multisensory Integration. Agents in VXN are equipped with a multimodal sensor so as to find the target by both seeing and hearing and make navigation setting closer the real-world. We first study the influence of different sensory signals (i.e., RGB, depth, audio) by training VIENNA with varying sensory modalities. As shown in Table 4a, fusing multimodal sensory cues (i.e., RGBD audio) more favored. For example, in VIN,, although considering audio alone brings poor performance, supplementing RGBD perception with audio yields notable improvements. This suggests audio is complementary to visual sensory in capturing physical and semantic properties of environments.
Augmented Target Description Embedding. To better master cross-task knowledge, VIENNA augments its episodic targets with all the four learnable task embeddings TI.A.I,L (cf. Eq. 7). We compare this design against two variants in Table 4b, and find such strategy is conducive to the performance. This is because, through end-to-ent training, TI.A.T,L carry task-specific knowledge, e.g.. TA is associated with some discriminative audio clips; Ty focuses essential visual landmarks. By taking TI,A,T,L together, VIENNA use key knowledge of different tasks in single task episodes.
Diversified Target Parsing, We online parse the augmented target description embedding G into N target embeddings Q.= .... g (ef Eq. 8), to achieve vivid and diversified interpretations of G. In Table 4c, we present evaluation scores with different numbers of generated target embeddings, i.e., N = 1, 3, 5, 7. As can be seen, diversified target parsing indeed boots navigation performance.
Multitask Planner. Several variants of multitask planner fMuP (ef. Eq. 9) are compared in Table 4d. The two-layer shared trunk design is adopted, due to its relatively better performance.
Reward Function. Next we examine the design of our reward function ($4.4). As seen in Table 4c, each reward term is indeed useful and combining all the four terms leads to the best performance.
Multitask Learning. Table 4f reveals the value of training VIENNA on multiple tasks: training with more navigation tasks improves both performance and generalizability. Compared to a composition of four single-task models, multi-task VIENNA also greatly reduces the model size: 10IM-31MM M.M..
6 Conclusion
In this work, we present VXN, large-scale 3D indoor dataset for multimodal, multitask navigation in continuous and audiovisual complex environments. Further, we devise VIENNA, a powerful agent that simultaneously learns four famous navigation tasks within single unified parsing-and-quer scheme. We empirically show that, through a fully attentive architecture, VIENNA is able to mine and utilize cross-task knowledge enhance the performance all the tasks. These efforts move us closer to a community goal of general-purpor robots capable of fulfilling a multitude of tasks.
Acknowledgement. This research is partially supported by China National Key R&D Program (2021 1000). Wenguan Wang acknowledges partial support from Australian Research Council (ARC), DECRA DE220101390.
10

---

References
[1] Dongsung Kim and Ramakant Nevatia. Symbolic navigation with a generic map. Autonomous Robots, 6(1):69-88, 1999. I
[2] Devendra Singh Chaplot, Dhiraj Gandhi, Abhinav Gupta, and Ruslan Salakhutdinov. Ohject goal navigation using goal-oriented semantic exploration. In NeurlPS, 2020. 1, 8, 9, 16
[3] Devendra Singh Chaplot, Ruslan Salakhutdinov, Abhinav Gupta, and Saurabh Gupta. Neural topological
SLAM for visual navigation. In CVPR, 2020. 1, 1,4,
[4] Kristina Striegnitz, Alexandre Denis, Andrew Gargett, Konstantina Garoufi, Alexander Koller, and Mariët Theune. Report the second challenge generating instructions in virtual environments (give-2.5). In European Workshop on Natural Language Generation, 2011.
[5] Yuke Zhu, Roozbeh Mottaghi. Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforeement learning. In ICRA, 2017. 1, 3,
4, 8,9
[6] Changan Chen, Unnat Jain, Carl Schissler, Sebastia Vicene Amengual Gari, Ziad Al-Halah, Vamsi Krishna Ithapu, Philip Robinson, and Kristen Grauman. Soundspaces: Audio-visual navigation in 3d environments. In ECCV, 2020. 1, 3, 4, 5, 8, 9, 16
[7] Wei Yang, Xiaolong Wang, Ali Farhadi, Abhinav Gupta, and Roozbeh Mottaghi. Visual semantic
navigation using scene priors. In ICLR, 2019. 1, 4
[8] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision-and-langunm navigation instructions in real environments. In CVPR, 2018. 1, 5,8,
navigation: Interpreting visually-grounded
[9] Yu Zhang and Qiang Yang. An overview multi-task learning. National Science Review, 5(1):30-43,
2018. 2
[10] M Alex Meredith and Barry Stein. Visual, auditory, and somatosensory convergence cells in superior colliculus results multisensory integration. Journal of neurophysiology, 56(3):640-662, 1986.
[11] Konrad P Körding, Ulikk Beierholm, Wei Ji Ma, Steven Quartz, Joshua B Tenenhaum, and Ladan Shams.
Causal inference multisensory perception. PLoS one, 2(9):e943, 2007. 2
[12] Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997. 2,
[13] Michael Crawshaw. Multi-task learning with deep neural networks: A survey. arXiv preprint arXiv:2009.09790 2020. 2,
[14] Yu Zhang and Qiang Yang. A survey multi-task learning. TKDE, 2021. 2
[15] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In ICCV, 2019. 2, 3, 4, 16
[16] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NewrIPS, 2017. 5,
[17] Georges Giralt, Ralph Sobek, and Raja Chatila. A multi-level planning and navigation system for mobile rohot: first approach to hilare. In IJCAI, 1979.
[18] Iro Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese.
3D semantic parsing of large-scale indoor spaces. In CVPR, 2016. 2
[19] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niebner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3D: Learning from rgb-d data in indoor environments. In 3DV, 2018. 2, 4, 16
[20] Yi Wu, Yuxin Wu, Georgia Gkioxari, and Yuandong Tian. Building generalizable agents with a realistic and rich 3d environment. arXiv preprint htt.:/801.0220 2018.
[21] Shuran Song, Fisher Yu, Andy Zeng, Angel X Chang, Manolis Savva, and Thomas Funkhouser. Semantic scene completion from a single depth image. CVPR, 2017.
[22] Fric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinay Gupta, and Ali Farhadi. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint 1rXiv:1712.05477 2017. 3,
[23] Fei Xia, Amir R Zamir. Zhiyang He. Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: Real-world pereeption for embodied agents. In CVPR, 2018.
[24] Matt Deitke, Winson Han, Alvaro Herrasti, Aniruddha Kembhavi, Eric Kolve, Roozbeh Mottgghi, Jordi Salvador, Dustin Schwenk, Eli VanderBilt, Matthew Wallingford, et al. Robothor: An open simulation to- real embodied ai platform. In CVPR, 2020. 3

---

[25] Jacob Krantz, Erik Wijmans, Arjun Majumdar. Dhruv Batra, and Stefan Lee. Beyond the nav-graph:
Vision-and-languaal navigation in continuous environments. In ECCV, 2020. 3, 4, 8, 9, 16
[26] Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied
question answering. In CVPR, 2018. 3,4
[27] Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. Wiioo-aadd-iialg navigation.
In CoRL, 2019. 3
[28] Khanh Nguyen, Debadeepta Dey. Chris Brockett, and Bill Dolan. Vision-based navigation with language-
based assistance via imitation learning with indirect intervention. In CVPR, 2018. 3
[29] Khanh Nguyen and Hal Daumé II1. Help, anna! visual navigation with natural multimodal assistance via retrospective curiosity-encenn encouraging imitation learning. In EMNLP-IJCNLP. 2019. 3
[30] Haiyang Wang. Wenguan Wang, Xizhou Zhu, Jifeng Dai, and Liwei Wang. Collaborative visual navigation.
arXiv preprint arXiv:2 151, 2021.
[31] Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, et al. Learning to navigate in complex environments. In JCLR, 2017.
[32] Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, and Jitendra Malik. Cognitive mapping and planning for visual navigation. In CVPR, 2017. 3
[33] Emilio Parisotto and Ruslan Salakhutdinov. Neural map: Structured memory for deep reinforcement learning. In ICLR, 3019.
[34] Jingwei Zhang, Lei Tai, Joschka Boedecker, Wolfram Burgard, and Ming Liu. Neural slam: Learning to explore with external memory. arXiv preprint (oxXix:770.0055500 2017.
[35] Yi Wu, Yuxin Wu, Aviv Tamar, Stuart Russell, Georgia Gkioxari, and Yuandong Tian. Bayesian relational memory for semantic visual navigation. In ICCV, 2019. 3
[36] Devendra Singh Chaplot, Emilio Parisotto, and Ruslan Salakhutdinov. Active neural localization. In ICLR, 2018.
[37] Nikolay Savinov. Alexey Dosovitskiy, and Vladlen Koltun. Cemi-parametric topological memory for navigation. In ICLR, 2018. 3
[38] Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhutdinov. Learning to explore using active neural slam. In ICLR, 2020. 3
[39] Hanqing Wang. Wenguan Wang, Wei Liang. Caiming Xiong, and Jianbing Shen. Structured scene memory for lision-lnguuae navigation. In CVPR, 2021. 3
[40] Kevin Chen, Junshen K. Chen, Jo Chuang. Marynel Vazquez, and Silvio Savarese. Topological planning with transformers for vision-and and-language navigation. In CVPR, 2021. 3
[41] Yusheng Zhao, Jinyu Chen, Chen Gao, Wenguan Wang, Lirong Yang, Haibing Ren, Huaxia Xia, and
Si Liu. Target-driven structured transformer planner for vision-language navigation. In ACMMM, 2022. 3
[42] Lisa Lee, Emilio Parisotto, Devendra Singh Chaplot, Eric Xing, and Ruslan Salakhutdinov. Gated path planning networks. In ICML, 2018. 3
[43] Zhiwei Deng, Karthik Narasimhan, and Olga Russakovsky. Evolving graphical planner: Contextual global planning for vision-and- and-language navigation. arXiv preprint arXiv:2007.05663 2020. 3,
[44] Hanqing Wang, Wenguan Wang, Tianmin Shu, Wei Liang, and Jianbing Shen. Active visual information gathering for vision-language navigation. In ECCV, 2020.
[45] Hanqing Wang, Wenguan Wang, Wei Liang, Steven CH Hoi, Jianbing Shen, and Lue Van Gool. Active perception for vasual-language navigation. International Journal of Computer Vision, pages 1-19, 2022.
3
[46] Ronghang Hu, Daniel Fried, Anna Rohrbach, Dan Klein, Trevor Darrell, and Kate Saenko. Are you looking? grounding to multiple modalities in vision-and- languuage navigation. In ACL, 2019. 3
[47] Yuankai Qi, Zizheng Pan, Shengping Zhang, Anton van den Hengel, and Qi Wu. Object-and-action aware model for visual language navigation. In ECCV, 2020. 3
[48] Xin Eric Wang. Vihan Jain, Eugene le, William Yang Wang, Zornitsa Kozareva, and Sujith Ravi. anvinoimeen--gnossicc multitask learning for natural language grounded navigation. In ECCV, 2020.
[49] Yicong Hong, Qi Wu, Yuankai Qi, Cristian Rodrriuezz-ppazo and Stephen Gould, VIn bert: A recurrent rision-and-lane
bert for navigation. In CVPR, 2021. 3
[50] Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis- Phill-phi Morency, Taylor Berg- Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrel. Speaker- Speaker-follower models for vision-and lan-uag navigation. In NewrIPS, 2018. 3
12

---

[51] Hao Tan, Licheng Yu, and Mohit Bansal. Learning to navigate unseen environments: Back translation with environmental dropout. In NAACL, 2019.
2019.
[52] Tsu-Jui Fu, Xin Wang, Matthew Peterson, Scott Grafton, Miguel Eckstein, and William Yang Wang. navigation via adversarial path sampling. In ECCV, 2020.
Counterfactual vision-and-
[53] Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh, and Dhruv Batra. Improv- ing vision-ande and language navigation with image-text pairs from the web. In ECCV, 2020. 3,4
[54] Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, and Jianfeng Gao. Towards learning generic agent for vision-and -ad---nd--anagg lan-uaga navigation via pre-training. In CVPR, 2020.
[55] Hanqing Wang. Wei Liang, Jianbing Shen, Luc Van Gool, and Wenguan Wang. Counterfactual cycle- consistent learning for instruction following and generation in vision-language navigation. In CVPR, 2022.
[56] Chen Gao, Jinyu Chen, Si Liu, Luting Wang, Qiong Zhang, and Qi Wu. Room-and-object aware knowledge reasoning for remote embodied referring expression. In CVPR, 2021. 3
[57] Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint
arXiv:1706.05098, 2017. 3
[58] Wenguan Wang. Yuanlu Xu, Jianbing Shen, and Song-Chun Zhu. Attentive fashion grammar network for fashion landmark detection and clothing category classification. In CVPR, 2018. 3
[59] Wenguan Wang. Shuyang Zhao, Jianbing Shen, Steven CH Hoi, and Ali Borji. Salient object detection with pyramid attention and salient edges. In CVPR, 2019. 3
[60] Jiasen Lu, Vodanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lec. 12-in-1: Multi-task vision and language representation learning. In CVPR, 2020. 3
[61] Ronan Collobert and Jason Weston, A unified architecture for natural language processing: Deep neural networks with multitask learning. In ICML, 2008. 3
[62] Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang. Facial landmark detection by deep multi-task learning. In ECCV, 2014. 3
[63] Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In CVPR, 2016. 3
[64] Dan Xu, Wanli Ouyang, Xiaogang Wang, and Nicu Sche. Pad-net: Multi-tasks guided prediction-and- distillation network for simultaneous depth estimation and scene parsing. In CVPR, 2018. 3
[65] Gjorgii Strezoski, Nanne van Noord, and Marcel Worring. Many task learning with task routing. In ICCV, 2019. 3
[66] Alex Kendall, Yarin Gal, and Roberto Cipolla, Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In CVPR, 2018. 3
[67] Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normal- ization for adaptive loss balancing in deep multitask networks. In ICML, 2018.
[68] Michelle Guo, Albert Haque, De-An Huang, Serena Yeung, and Li Fei-Fei. Dynamic task prioritization for multitask learning. In ECCV, 2018.
[69] Long Duong, Trevor Cohn, Steven Bird, and Paul Cook. Low resource dependency parsing: Cross-lingual parameter sharing in neural network parser. In IJCNLP, 2015. 3
[70] Victor Sanh, Thomas Wolf, and Sebastian Ruder. A hierarchical multi-task approach for learning embeddings from semantic tasks. In AAAI, 2019. 3
[71] Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. In NewrlPS, 2018. 3
[72) Joachim Bingel and Anders Søgaard. Identitying beneficial task relations for multi-task learning in deep neural networks. ACL, 2017. 3
[73] Amir R Zamir, Alexander Sax. William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In CVPR, 2018. 3
[74] Kshitij Dwivedi and Gemma Roig. Representation similarity analysis for efficient task taxonomy & transfer learning. In CVPR, 2019. 3
[75] Zhaoyang Yang. Kathryn E Merrick, Hussein A Abbass, and Lianwen Jin. Multi-task deep reinforcement learning for continuous action control. In ISCAI, 2017. 3
[76] Yee Whye Teh, Vietor Bapst, Wojeiech Marian Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: robust multitask reinforcement learning. In NewrIPS, 2017.
3
13

---

[77] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley. lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In ICML, 2018.
[78] Lerrel Pinto and Abhinav Gupta. Learning to push by grasping: Using multiple tasks for effective learning. In ICRA, 2017. 3
[79] Andy Zeng. Shuran Song, Stefan Welker, Johnny Lee, Alberto Rodriguez, and Thomas Funkhouser. Learning synergies between pushing and grasping with self-supervised deep reinforcement learning. In
IROS, 2018. 3
[80] Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojeiech Czarnecki, Simon Schmitt, and Hado van Hasselt. Multi-task deep reinforcement learning with popart. In AAAI, 2019. 3
[81] Carlo D'Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, and Jan Peters. Sharing knowledge in multi-tast task deep reinforcement learning. ICLR, 2020.
[82] Zhiyuan Xu. Kun Wu, Zhengping Che, Jian Tang. and Jieping Ye. Knowledge transfer in multi-task deep reinforcement learning for continuous control. arXiv preprint arXiv:2010.07494 2020. 3
[83] Nicolas Heess, Greg Wayne, Yuval Tassa, Timothy Lillicrap. Martin Riedmiller, and David Silver. Learning and transfer of modulated locomotor controllers. arXiv preprint 110iv:1610005182, 2016. 3
[84] Coline Devin, Abhishek Gupta, Trevor Darrell, Picter Abheel, and Sergey Levine. Learning modular neural network policies for multi-task and multi-robot transfer. In ICRA, 2017. 3
[85] Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distillation. In ICLR, 2015.
[86] Emilio Parisotto, Lei Jimmy Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning. In ICLR, 2016. 3
[87] Charles Beattic, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Victor Valdés, Amir Sadik, et al. Deepmind lab. arXiv preprint
16X2:1612.0380.. 2016. 3
[88] Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
3
[89] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Mela-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In CoRL, 2020.
3
[90] Devendra Singh Chaplot, Lisa Lee, Ruslan Salakhutdinov, Devi Parikh, and Dhruv Batra. Embodied multimodal multitask learning. arXiv preprint arXiv:1902.01385, 2019. 3
[91] Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, al. Learning to navigate in complex environments. In ICLR, 2017.
[92] Daniel Gordon, Abhishek Kadian, Devi Parikh, Judy Hoffman, and Dhruv Batra. Splitnet: Sim2sim and task2task transfer for embodied visual navigation. In ICCV, 2019. 5,5
[93] Deepak Pathak, Pulkit Agrawal, Alexci A Efros, and Trevor Darrell, Curiosity-driven exploration by self-supervised prediction. In ICML, 2017.
[94] Karol Gregor, Danilo Jimenez Rezende, Frederic Besse, Yan Wu, Hamza Merzie, and Aaron van den Oord, Shaping belicf states with generative environment models for rl. In NeurlPS, 2019. 3
[95] Joel Ye. Dhruv Batra, Erik Wijmans, and Abhishek Das. Auxiliary tasks speed up learning pointgoal navigation. arXiv preprint (XXX:22007.0456/1 2020.
[96] Chih-Yaa Ma, Jiasen Lu, Zuxuan Wu, Ghassan AlRegib, Zsolt Kira, Richard Socher, and Caiming Xiong. Self- melf-monittiing navigation agent via auxiliary progress estimation. In ICLR, 2019. 3
[97] Fengda Zhu, Zhu, Xiaojun Chang, and Xiaodan Liang. Vision-language navigation with self-supervised
auxiliary reasoning tasks. In CVPR, 2020. 3
[98] Kuan Fang, Alexander Toshev, Li Fei-Fei, and Silvio Savarese. Scene memory transformer for embodied agents long- horizon tasks. In CVPR, 2019.
[99] Federico Landi, Lorenzo Baraldi, Marcella Cornia, Massimiliano Corsini, and Rita Cucchiara. Perceive. transform, and act: Multi-modal attention networks for rision-and-language navigation. arXiv preprint 1x11771911.12377, 2020. 3
[100] Changan Chen, Ziad Al-Halah and Kristen Grauman. Semantic audio-visual navigation. In CVPR, 2021.
3, 4,5 5
14

---

[101] Heming Du Xin Yu, and Liang Zheng. Vinet: Visual transformer network for object goal navigation. arXiv preprint arXiv:2055.09447, 2021.
[102] Alexander Pashevich, Cordelia Schmid, and Chen Sun. Episodic transformer for vision-and- -nd-language navigation. arXiv preprini arXiv:2105.06455 2021.
[103] Lukasz Kaiser. Aidan N Gomez, Noam Shazeer, Ashish Vaswani, Niki Parmar, Llion Jones, and Jakob Uszkoreit. One model to learn them all. arXiv preprint arXiv:/706.051.77 2017. 3
[104] Jiasen Lu, Dhruv Batra. Devi Parikh, and Stefan Lee. Vilbert: Pretraining task agnostic visiolinguistic
representations for vision-and-lan and--auugge tasks. arXiv preprint arXiv:1908. 02265, 2019. 3
[105] Subhojeet Pramanik, Priyanka Agrawal, and Aman Hussain. Omninet: A unified architecture for
multi-moda multi bultiitaskk learning. arXiv preprint arXiv:1907 190.000407000 2019. 3
[106] Ronghang Hu and Amanpreet Singh. Unit: Multimodal multitask learning with a unified transformer. arXiv preprint arXiv:2102. 10772, 2021.
[107] Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, and Jianfeng Gao. Towards learning generic agent for vision-and -nd---ndd-aaaaa and-language navigation via pre-training. In CVPR, 2020.
[108] Juncheng Li, Xin Wang, Siliang Tang, Haizhou Shi, Fei Wu, Yueting Zhuang, and William Yang Wang. Unsupervised reinforcement learning of transferable meta-skills for embodied navigation. In CVPR, 2020.
[109] Pierre-Louis Guhur, Makarand Tapaswi. Shizhe Chen, Ivan Laptev, and Cordelia Schmid. Airbert: In-domain Pretraining for Vision -ndon-an---aa Language Navigation. ICCV, 2021.
[110] Dhruv Batra, Aaron Gokaslan, Aniruddha Kembhavi, Oleksandr Maksymets, Roozbeh Mottaghi, Manolis Savva, Alexander Toshev, and Erik Wijmans, ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects. arXiv preprint arXi::2000.111177 13171, 2020. 5,5
[111] Habitat image nav
[112] Z. Markus, S. Christian, and H. Robert. Binaural rendering of ambisonic signals by head-related impulse response time alignment and a diffuseness constraint. The Journal of the Acoustical Society of America, 143()) 2018.
[113] Gavin Kearney, Claire Masterson, Stephen Adams, and Frank Boland, Approximation of binaural room impulse responses. In ISSC, 2009. 4
[114] Hiroaki Sakoe and Seibi Chiba. Dynamic programming algorithm optimization for spoken word recogni- tion. IEEE Transactions Acoustics, Speech, and Signal processing, 26(1):43-49, 1978. 4
[115] Peter Anderson, Angel Chang, Devendra Singh Chaplot. Alexey Dosovitskiy. Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottghhi, Manolis Savva, et al. On evaluation of embodied navigation agents, arXiv preprint arXiv: 0076077777 2018. 5,9
[116] Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. Touchdown: Natural language navigation and spatial reasoning visual street environments. CVPR, 2019.
[117] Richard Bellman. A markovian decision process. Journal of mathematics and mechunics, 6(5))679-684,
1957. 5
[118] Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, SM Eslami, et al. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv: 1707.02286, 2017. 5,8 8
[119] Joel Ye, Dhruv Batra, Abhishek Das. and Erik Wijmans. Auxiliary tasks and exploration enable objectnav. arXiv preprint
2021.
[120] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707,06347, 2017. 7... 8
[121] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV, 115(3):211 2015.
[122] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[123] Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. In ICLR, 2018. 8
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the pa- per's contributions and scope? [Yes] Our main claims accurately reflect the paper's contributions and scope.
(b) Did you describe the limitations of your work? [Yes] The limitations are discussed in the supplemental material.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] The potential negative societal impacts are discussed in the supplemental material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We have already read the ethics review guidelines and ensured that our paper conforms to them.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] Our work does not involve any theoretical result.
(b) Did you include complete proofs of all thcoretical results? [N/A] Our work does not involve any theoretical result.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as a URL)? [Yes] The code and full dataset is available on the project page: https: / /ttpttttttttt//////////eeeeeeee. www.coo///aaaaae haw.g/anngggaiii///////a VXN.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? (Yes] We specify all the training details in $4.5 and ğ1 of supplementary.
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] We report the error bars in Fig. 4.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We report our compute hardware in $4.5.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We have cited all the used code and data.
(b) Did you mention the license of the assets? [Yes] Habitat Simulator [15] is ob- tained battts://githie www. cense. The algorithm of Chen et al. [6] is implemented based on the pub- licly released code (https://gith)) catuub.com/ff. wwwccom//aaaa/ee htmm/ffcbbookesearcchh aacch/soun-aee spacess is with CC-BY-4.0 license. The algorithm of Chaplot et al. [2] is implemented based on the publicly released code (attps://github com /de//ededdaaaaploo//// Object- Baact-Goa--Ba Goal- ieviiiaiion on) with MIT license. The algorithm of Krantz er al. [25] is implemented based on the publicly released code /attps://github com/ jacobkrantz/Vin /LN VVN--C) with MIT license.
with MIT li-
(c) Did you include any assets either in the supplemental material as URL? [N/A] We do not use any new assets excepting the list above.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes] We obtain Matterport3D [19] data by signing the terms of use luttp://kaldat vc. in. tum. de de/matteerport//e _TSS..... .dd).
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [Yes] The data used in our paper neither contain any personally identifiable information nor offensive content.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] Our work does not involve any crowdsourcing or conducted research with human subjects.
16

---

(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] Our work does not involve any crowdsourcing or conducted research with human subjects.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] Our work does not involve any crowdsoure- ing conducted research with human subjects.
17


Reviews:
[{"review_id": "ccs5Eihfwyj", "reviewer": "Reviewer_grF1", "summary": "This paper proposed a new visual navigation benchmark that combines containing information from different modalities. Then the authors propose a modular model for the proposed benchmark including Episodic Encoder to encode the history, Target Parser to interpret the target, and a planner to predict the navigation action. ", "questions": "See Weakness\n\n", "limitations": "Yes, they discussed that in the appendix.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The proposed task itself is a timely movement in navigation area. With the advancement in individual navigation task, it is natural to consider tasks with input of more modalities. \n\n2. Considering the complexity of the task, the proposed modular model is a great baseline. The modular structure is designed via analyzing the task component. The architecture itself also shows some intuition on the problem. For example, the author shows a good demo about how to handle input from different modalities.\n\n3. Both the paper and the appendix are detailed and well-written.\n\nWeakness: \n\n1. The main contribution is a proposed dataset, but it does not provide the full dataset or a sample in the supplemental material.\n\n2. The authors claim the task includes four navigation tasks. Actually, it is just a new task including the input types from the four tasks.\n\n3. One serious issue in navigation task is that sometimes utilizing more modality leads to a poor performance for some model. Therefore I think it is necessary to ablate different modality input and test the performance. For example, check the paper mentioned in Section 4.1.2 of [1]. There are limited ablation study in section 5.2. \n\n4. The paper should include more SOTA models on the four navigation tasks as baselines. The proposed model with more inputs should at least outperform these models with less input information.   \n\n\nSuggested citation:\n[1] A survey on visual navigation for artificial agents with deep reinforcement learning\n[2] Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions\n[3] Core Challenges of Social Robot Navigation: A Survey\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OjCy0h5dYLA", "reviewer": "Reviewer_DBvN", "summary": "This paper proposes VXN, a large-scale 3D indoor dataset for multimodal, multitask navigation in continuous and audiovisual complex environments. It can conduct four embodied navigation tasks, including image-goal navigation, audio-goal navigation, object-goal navigation, and vision-language navigation. Moreover, the authors devise a framework named VIENNA to simultaneously learn four navigation tasks within a single unified parsing-and-query scheme. Experiments show that the VIENNA outperforms the baselines among four navigation tasks.", "questions": "My main concerns are what kinds of knowledge can be reused and how to reuse this knowledge? Please explain more about this question.", "limitations": "Yes, the authors adequately addressed the limitations and potential negative social impact of their work.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n1. The authors are the first to innovatively combine four navigation tasks and propose a dataset that can simultaneously adopt them.\n2. The authors propose an agent that can solve the four navigation tasks using one single model without switching among different models and the model size is considerable.\n3. The proposed VIENNA agent leverages multisensory input for navigation. Input in different modalities provides supplementary information. This setting is more practical in the real world, where robots use multi-modal information for finishing their tasks.\n4. The proposed methods outperform the existing SOTA in every single task by leveraging multisensory input and multi-task learning. Also, the authors have conducted a thoughtful ablation study to evaluate the effectiveness of each component and claim.\n\n# Weaknesses\n1. My main concerns are what kinds of knowledge can be reused and how to reuse this knowledge? In Section 4.3, the authors claim that the knowledge re-usage is conducted by \\tau. However, in image-goal navigation, \\tau_A does not contain any information about what kind of images should be found. So how can \\tau_A help to retrieve target image-related audio for navigation? More explanations are needed.\n2. Some details are missing. Please explain the meaning and benefits of the vectors \\mu_{1:t} in section 4.1. \n3. VIENNA adopts the same reward function for the four navigation tasks. However, the reward function design will be different for different navigation tasks. For example, in object-goal navigation, a positive reward should be given when an object with goal-related semantics is found. It would be an interesting future research direction for considering different kinds of rewards for different tasks.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IvAE_8aqd3X", "reviewer": "Reviewer_kzm1", "summary": "The paper proposes a unified model for 4 popular navigation tasks (image-goal navigation, object-goal navigation, audio-goal navigation, vision-language navigation). A Transformer based architecture is utilized to combine these 4 tasks. The target signal (image, audio, etc.) is basically used as the query. The paper also proposes a dataset that supports all 4 tasks. The experiment section provides results for single-task and multi-task settings and provides comparisons with task-specific baselines. ", "questions": "- L226: what is \"epoch embedding\" ?\n\n\n- How is MDPPO different from DDPPO? Does it just balance the task samples? \n\n\n- What are Navigation Error (NE) and Oracle success Rate (OR) ?\n\n\n- The sequence length for the Transformer is 100. That means there is a huge memory consumption, and the sequence might not fit into a GPU. Is there any specific trick used ?", "limitations": "Yes, the limitations are discussed adequately. ", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n- The paper takes a step towards unified models for Embodied AI. This is an important direction since agents should be able to perform various tasks simultaneously.\n\n- The proposed dataset, which supports multiple types of navigation, will be useful for the community.\n\n**Weaknesses:**\n- The weakest part of the paper is its experiment section. The conclusion that are drawn from the experiments are not accurate.\n\n&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; (1) The difference between most of the single-task and multi-task results is within the range of training noise (8.3 vs 8.8 for object goal navigation or 10.6 vs 11.3 for audio goal navigation). The results of these types of training algorithms vary greatly depending on the initialization, sampling and various other factors. It is great that the multi-task model works as well as the single-task models. However, claims such as \"VIENNA also beats other task-specific competitors\" are not correct. \n\n&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; (2) Statements such as L345: \"VIENNA_ST suffers from relatively large performance\" are not accurate. If the training noise is considered the drop is the same for single-task and multi-task settings. \n\n&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; (3) Numbers in Tables 4b and 4d are within the range of noise. No conclusion can be drawn from these tables. Statements such as \"such strategy greatly boosts the performance\" are not accurate. \n\n&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; I will consider raising the score if these statements are corrected in the revision.\n\n- It is not clear how models are chosen for evaluation. Is it the model that achieves the best performance on the validation set, trained for a specific number of epoch, or other criteria? \n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept