AVLEN: Audio-Visual-Language Embodied Navigation in 3D Environments
Authors: Sudipta Paul
Amit Roy-Chowdhury
Anoop Cherian
Venue: NeurIPS 2022 Accept (2022)
Keywords: audio-visual navigation
hierarchical policy learning
vision and language navigation

Abstract: Recent years have seen embodied visual navigation advance in two distinct directions: (i) in equipping the AI agent to follow natural language instructions, and (ii) in making the navigable world multimodal, e.g., audio-visual navigation. However, the real world is not only multimodal, but also often complex, and thus in spite of these advances, agents still need to understand the uncertainty in their actions and seek instructions to navigate. To this end, we present AVLEN -- an interactive agent for Audio-Visual-Language Embodied Navigation. Similar to audio-visual navigation tasks, the goal of our embodied agent is to localize an audio event via navigating the 3D visual world; however, the agent may also seek help from a human (oracle), where the assistance is provided in free-form natural language. To realize these abilities, AVLEN uses a multimodal hierarchical reinforcement learning backbone that learns: (a) high-level policies to choose either audio-cues for navigation or to query the oracle, and (b) lower-level policies to select navigation actions based on its audio-visual and language inputs. The policies are trained via rewarding for the success on the navigation task while minimizing the number of queries to the oracle. To empirically evaluate AVLEN, we present experiments on the SoundSpaces framework for semantic audio-visual navigation tasks. Our results show that equipping the agent to ask for help leads to a clear improvement in performances, especially in challenging cases, e.g., when the sound is unheard during training or in the presence of distractor sounds.

AVLEN: Audio- Audio-Visual-Language Embodied Navigation in 3D Environments
Sudipta Paul gpaurr07@ucr edu University of California, Riverside
Amit Roy-Chowdhury' ami amitrcsece ucr edu Mitsubishii Electric Research Labs, Cambridge, MA
Anoop Cherian cherian@mer's com
Abstract
Recent years have seen embodied visual navigation advance in two distinct direc- tions: (i) in equipping the Al agent to follow natural language instructions, and (ii) in making the navigable world multimodal, e.g., audio-visual navigation. However, the real world is not only multimodal, but also often complex, and thus in spite of these advances, agents still need understand the uncertainty in their actions and seek instructions to navigate. To this end, we present AVLEN an interactive agent for Audio Visual- Lasuul-- Embodied Navigation. Similar audio- visual navigation tasks, the goal of our embodied agent is to localize an audio event via navigating the 3D visual world; however, the agent may also seek help from a human (oracle), where the assistance is provided in free-form natural language. To realize these abilities, AVLEN uses multimodal hierarchical reinforcement learning backbone that learns: (a) high-level policies to choose either audio-cues for navigation or to query the oracle, and (b) lower-level policies to select naviga- tion actions based on its audio-visual and language inputs. The policies are trained via rewarding for the success on the navigation task while minimizing the number of queries to the oracle. To empirically evaluate AVLEN, on the SoundSpaces framework for semantic audio-visuee navigation tasks. Our results show that equipping the agent to ask for help leads to clear improvement in performance, especially in challenging cases, e.g., when the sound is unheard during training in the presence of distractor sounds.
present experiments
Introduction
Building embodied robotic agents that can harmoniously co-habit and assist humans has been one of the early dreams of A recent incarnation this dream has been in designing agents that are capable of autonomously navigating realistic virtual worlds for solving pre-defined tasks. For example, vision-and-lane lan--uugg navigation (VLN) tasks [2]. the goal is for the Al agent to either navigate goal location following the instructions provided in natural language,
to explore the visual world seeking answers to a given natural language question 10 35 36). Typical VLN agents assumed deaf;; i.c., they cannot hear any audio events in the scene especially when the agent is expected to operate in the real world. To address this shortcoming, SoundSpaces [6] reformulated the navigation task with the goal of localizing audio source in the virtual scene; however without any language instructions for the agent to follow.
unnatural restriction,
Real-world navigation is not only audio-visual, but also is often complex and stochastic, the agent must inevitably seek a synergy between the audio, visual, and language modalities for successful navigation. Consider, for example, a robotic agent that needs to find where the "thud of a falling person" or the "intermittent dripping sound of water" is heard from. On the one hand, such sound may not last long and may not be continuously audible, and thus the agent must use semantic
Equal Contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

1
Start of Navigation, agent's Initial pose
Agent: Query Oracle: Go towards the dining table.
Agent queries oracle gives feedback natural language.
2
Assisted by oracle feedback, agent reaches sounding objects vicinity
1
3
3
Novigat Following
Figure 1: An illustration of our proposed AVLEN framework. The embodied agent starts navigating from location denoted ! guided by the audio-visual event at 3. At location 2 the learned policy for the agent decides to seek help from an oracle (e-g., because the audio stopped). The oracle provides short natural language instruction for the agent to follow. The agent translates this instruction to produce series of navigable steps to move towards the goal 3
knowledge of the audio-visual modality [5] reach the goal. On the other hand, such events need to be catered to timely and the agent should minimize the number of navigation mistakes it makes a situation that can be efficiently dealt with if the agent can seek human help when it is uncertain of its navigation actions. Motivated by this insight, we present AVLEN a first of its kind embodied navigation agent for localizing an audio source in realistic visual world. Our agent not only learns to use the audio-visual cues to navigate to the audio source, but also learns to implicitly model its uncertainty in deciding the navigation steps and seeks help from an oracle for navigation instructions, where the instructions are provided in short natural language sentences. Figure] | illustrates our task.
To implement AVLEN, we build on the realistic virtual navigation engine provided by the Matterport 3D simulator 22 and enriched with audio events via the SoundSpaces framework (6. A key challenge in our setup for the agent decide when to query the oracle, and when follow the audio-visual cues to reach the audio goal. Note that asking for help too many times may hurt agent autonomy (and is perhaps less preferred if the oracle is a human), while asking too few questions may make the agent explore the scene endlessly without reaching the goal. Further, note that we assume the navigation instruction provided to the agent is in natural language, and thus is often abstract and short (see Figure abbov)) making it difficult to be correctly translated to agent actions (as seen in VLN tasks (2)). Thus, the agent needs to learn the stochasticity involved in the guidance provided to it, as well as the uncertainty involved in the audio-visual cues, before selecting which modality to choose. To address these challenges, we propose a novel multimodal hierarchical options based deep reinforcement learning framework, consisting of learning high-level policy to select which modality to use for navigation, among (i) the audio-visual cues, or (ii) natural language cues, and two lower-level policies that learn (i) to sclect navigation actions using the audio-visual features, (ii) learns to transform natural language instructions to navigable actions conditioned on the audio-visual context. All the policies end-to-end trainable and is trained offline. During inference, the agent uses its current state, the audio-visuse cues, and the learned policies to decide if it needs oracle help or can continue navigation using the learned audio goal navigation policies.
38 propose tasks involving 38 model uncertainty is used to decide when to query the oracle, where the uncertainty is either quantified in terms of the gap between the action prediction probabilities [9] to be less than a hcuristically chosen threshold,
Closely related to our motivations, few recent works 9 26
interactions with an oracle for navigation. Specifically, in 9
use manually-
derived conditions to decide when an agent is lost in its navigation path [27]. In (26]. the future actions of the policy of interest are required to be fully observed to identify when the agent is making
mistakes and uses this information to decide when to query. Instead of resorting to heuristics, we propose a data-driven scheme to learn policies to decide when to query the oracle, these policies thus automatically learning the navigation uncertainty in the various modalities.
To empirically demonstrate the efficacy of our approach, we present extensive experiments on the aunguage-augmented semantic audio-visus visual navigation (SAVi) task within the SoundSpaces framework for three very challenging scenarios, i.c., when: (i) the sound source is sporadic, however

---

familiar to the agent, (ii) sporadic but unheard of during training, and (iii) unheard and ambiguous due to the presence of simultaneous distractor sounds. Our results show clear benefits when the agent knows when to query and how to use the received instruction for navigation, as substantiated by improvements in success rate by nearly 3% when using the language instructions directly and by more than 10% when using the ground truth navigation actions after the query, even when the agent is constrained to trigger help only to a maximum of three times in a long navigation episode.
Before proceeding to detail our framework, we summarize below the main contributions of this paper.
We are the first to unify and generalize audio-visual navigation with natural language instructions towards building a complete audio- cettoovvsull-aannn language embodied Al navigation interactive agent. We introduce a novel multimodal hierarchical reinforcement learning framework that jointly learns policies for the agent to decide: (i) when to query the oracle, (ii) how to navigate using audio-goal, and (iii) how use the provided natural language instructions.
Our approach shows state-of -ftte--rr- performances on the semantic audio-visual navigation dataset [5] with 85 large-scale real-world environments with a variety of semantic objects and their sounds, and under a variety of challenging acoustic settings.
2 Related Works
The SoundSpaces simulator, introduced in [6] to render realistic audio in 3D visual environments, pioneered research into the realm of audio-visual embodied navigation. The AudioGoal task in [6] consists of two sub-tasks, namely to localize an object: i) that sounds continuously throughout the navigation episode [6.7] and ii) that sounds sporadically or can go mute at any time [3]. In this work, we go beyond this audio-visual setting, proposing a new task that equips the agent with the ability use natural language- a setup that is realistic and practically useful.
Audio- Audio-Visual Visual Navigation.
CntrruttionFooloviiii Navigation. There are recent works that attempt to solve the problem of navigation following instructions (2) 16
17
21 25 The instruction can be of many forms; e.g..
structured commands (27] natural language sentences 2 goal images [22], or a combination of different modalities (26)). The task in vision-ane and---nd-land lan-uagg navigation (VLN) for example is to execute free-form natural language instructions to reach a target location. To embody this task, to render real or photo-realistic images and perform S or continuous environment [1]]. One important aspect of vision and language navigation is to learn the correspondence between visual and textual information. To achieve this, (34] uses cross- modal attention to focus on the relevant parts of both the modalities. In [23] and (24] additional module is used to estimate the progress, which is then used as a regularizer. In 13 and [1,, augmented instruction-traaectee
several simulators have been used 12 6 29]
agent navigation through a discrete graph
pairs is used to improve the VLN performance. In 37 long instructions learned to be decomposed into shorter ones, executing them sequentially (via e.g., navigation). Recently, there are works using Transformer-ban architectures for VLN [T6]] 25 In [16), the BERT [I] architecture is used in recurrent manner maintaining cross-modal state information. These works only consider the language- task. Different from these works, our proposed AVLEN framework solves vision-and- ladn-g-g navigation as a sub-task within the original semantic audio-visual navigation task 66.
navigation
Interactive Navigation. Recently, there have been works where agent is allowed to interact with an oracle or a different agent, receiving feedback, and utilizing this information for navigation The oracle instructions in these works are limited to either ground truth actions [9] or a direct mapping of a specific number of actions to consecutive phrases (7 is needed. Though 26 uses a fixed set of natural language instructions as the oracle feedback, it is coupled with the target image that the agent will face after completion of the sub-goal task. In Nguyen et al. [26), the agent needs reach a specific location query, which may be infeasible practically or sub-optimal if these locations are not chosen properly. Our approach differs fundamentally from these previous works in that we consider free-form natural language instructions and the agent can query the oracle from any navigable point in the environment, making our setup very natural and flexible.
9 26 27 [2)
3 Proposed Method
In this section, we will first formally define our task and our objective. This will be followed by details of our multimodal hierarchical reinforcement learning framework and our training setup.
3

---

Problem Setup. Consider an agent in a previously unseen 3D world navigable along a densely- sampled finite grid. At each vertex of this grid, the agent could potentially take one of a subset of actions from an action set A (stop, move_ wowwardarrd, Jurn_right turn_lefty Further, the agent assumed to be equipped with sensors for audio-visual perception via a binaural microphone and an ego-centric RGBD camera. The task of the agent in AVLEN is to navigate the grid from its starting location to find the location of an object that produces a sound (AudioGoal), where the sound is assumed to be produced by static object and is semantically unique, however can be unfamiliar, sporadic, or ambiguous (due to distractors). We assume the agent calls the stop action only at the AudioGoal that terminates the episode. In contrast to the task in SoundSpaces, an AVLEN agent is also equipped with a language interface to invoke a query to an oracle under budget (e.g., a limit on the maximum number of such queries). The oracle responds to the query of the agent via providing a natural language short navigation instruction; this instruction initial segment along the shortest path trajectory from the current location of the agent to the goal. For example, for a navigation trajectory given by the actions turn_riii rright, turn_rigg wrrght, move_ forwwrd, turn_left), the corresponding language instruction provided by the oracle to the agent could be "go around the sofa and turn to the door". As is clear, use this instruction to produce navigable actions, the agent must learn to associate the language constructs with objects in the scene and their spatial relations, as well as their connection with the nodes in the navigation grid. Further, given the limited budget to make queries, the agent must learn to balance between when to invoke the query and when to navigate using its audio-visue cues. In the following, we present a multimodal hicrarchical options approach to solve these challenges in a deep reinforcement learning framework.
describing (in natural language)
Problem Formulation. We formulate the AVLEN task as a partially-observalle Markov decision process (POMDP) characterized by the tuple A, T, R. O P, 7, where S represents the set of agent states, A A U {query) with the navigation actions A defined above combined with an action to query the oracle, T(s'l (s, a) is the transition probability for mapping a state-action pair (s, a) to a state s', R[s a) is the immediate reward for the state-action pair, O represents a set of environment observations o, P(ols', a) captures the probability of observing o E O in a new state s' after taking action a, and y E [0, 1] is the reward discount factor for long-howe horizoo trajectories. Our POMDP also incorporates a language vocabulary V consisting of a dictionary of words that the oracle uses to the agent
produce the natural language instruction. As our environment is only partially-observabble may not have information regarding its exact state, instead maintains a belief distribution b over S as an estimate of its current state. Using this belief distribution, the expected reward for taking an action a at belief state b can be written as R'(b, a) = Eees b(s) R(s, a). With this notation, the objective of the agent in this work is learn a policy π. [s| A + [0, 1] that maximizes the expected return defined by the value function V", while minimizing the number of queries made to the oracle; i.c.,
no
arg max VT where V+(b)= = E
Σ
r2 (I' (1)) - C(I + i)((ar= = =query) 1b, = b, म
(1)
where I denotes the indicator function, and the updated belief b,41 = update(0 be, ar) defined
for state s' as b (s") = n)P(o+1lss aa) E.es be (s) az) for a normalization factor 17 > O.
The function Ç: R. + I produces a score balancing between the frequency of queries and the expected return from navigation.
At any time step f. the agent (in belief state by) receives an observation o++1 E O from the en- vironment and selects an action a according to a learned policy π; this action transitioning the agent to the new belief state b as per the transition function (' 1 la,, b,) = Eoeo I(bb,+ = update(o, be, a²) P(o st, ar) while receiving an immediate reward R' (b,,, at). As the navigation state space S of our agent is enormous, keeping belief distribution on all states might be compu- tationally infeasible. Instead, similar to [5], we keep a history of past K observations in a memory module M, where an observation o at time step t is encoded via the tuple et = (FV, FA Pt) comprising neural embeddings of egocentric visual observation (RGB and depth) represented by FY, the binaural audio waveform of the AudioGoal heard by the agent represented as two channel spectrogram F.B, and the previous action taken F alongside the pose of the agent pt with respect to its starting pose (consisting of the 3 spatial coordinates and the yaw angle). The memory M initialized to an empty set at the beginning of an episode, and at a time step i, is updated as M = (ef i max(0, t K). K)....,t t). Apart from these embeddings, AVLEN also incorporates

---

Folly
Encaddr
Decader
Environment
Suppcce
Encoder
Encoder
10844
Osuervation
Spuad
State Encoder
Tranaformer
Embbadding
Figure 2: Architecture of AVLEN pipeline. We show the two-level hierarchical RL policies that the model learns (offline), as well as the various input modalities and the control flow.
goal estimation network fg characterized by a convolutional neural network that produces a step-wise estimate 9. = fF(P of the sounding AudioGoal; 9t consisting of: (i) the (x, y) goal location estimate L, from the current pose of the agent, and (ii) the goal category estimate c E IRC for C semantic sounding object classes. The agent updates the current goal estimate combining the previous estimates as 9t = Age + (1 ( Apt) where fp is a linear transformation of 9t-1 using the pose change Apt. We use À 0.5, unless the sound is inaudible in which case it is set to zero. We will use 9 E G RC+2 to denote the space of goal estimates.
Multimodal Hierarchical Deep Reinforcement Learning. As is clear, the diverse input modalities used in AVLEN have distinctly varied levels of semantic granularity, and thus a single monolithic end- to-end RL policy for navigation might be sub-optimal. For example, the natural language navigation instructions received from the oracle might comprise rich details for navigating the scene that the agent need not have to resort to any of the audio-visual inputs for say > 1 steps. However, there is a budget on such queries and thus the agent must know when the query needs to be initiated (e.g.. when the agent repeats sub- sub-trajectories) Further, from a practical sense, each modality might involve different neural network architectures for processing, can have their own strategies for (pre- )training, involve distinct inductive baises, incorporate heterogeneous navigation uncertainties.
All the above challenges naturally suggest to abstract the policy learning in the context of hierarchical options semi-Markov framework 33] 20 30 consisting of low-level options corresponding to the navigation using either the AudioGoal or the language model, and high-level policy select among the options. More formally, termination condition, and set of belief states in which the option is valid. In our context, we assume multi-time option policy for language-based navigation spanning v navigation steps and a primitive policy [30 for AudioGoal. We also assume these options may be invoked independent of the agent state. Suppose my R|S|X|M| G x [query) - [0. represent the high-level policy deciding whether to query the oracle or not, using the current belief, the history M, and the goal estimate Further, let the two lower-level policies be: (i) mg R|S]X|M| x G x A + [0, 1]. that tasked with choosing the navigation actions based on the mudio-visus features, and (ii) me R DA G instruction formed using words from the vocabulary V. assuming v steps are taken after each such query. Let R2 and H3 denote the rewards (as defined in (I)) corresponding to the T. and me options, respectively, where we have the multi-time discounted cumulative reward (with penalty () E 7d (b., aF) = .e. aj E A - <(t), while H2 is, being a primitive option, in except that the actions are constrained A. Then, have the Bellman equation for using the options given by:
option is triplet consisting of respective policy
00, 1], that navigates based on the received natural language
for (b,, a.)
VT (b) = (E
R
PPEE(ddD + -T (Eelb)
R'+ Σ P(d1b,E0V (2) d'EO
d'E0
"Otherwise terminated if the stop action is called before the option policy is completed.
5

---

where Eg and &e are shorthands for { = Ty and & = Te. respectively, and π = (Tg. Tgg me). Further, P' is the multi-time transition function given by: P'(''|b, -; E. Σ. ~ P(s', o', f/s E)b(s) is the probability to observe o' in steps using option& & [3). Our objective is to find the policies π that maximizes the value function in 2 for V (bo). Figure 2 shows block diagram of the interplay between the various policies and architectural components. Note that by using such a two-stage policy, we assume that the top-level policy T implicitly learns the uncertainty in the audio-visual and language inputs well as the predictive uncertainty in the respective low-level options Ty and πy for reaching the goal state. In the next subsections, we detail the neural architectures for each of these options policies.
where with slight abuse of notation, we assume
Navigation Using Audio Goal Policy, Tg. Our policy network for Tg follows an architecture similar to [5], consisting of Transformer encoder-decoder model [33] The encoder sub- mob-lodule takes in the embedded features E" from the current observation as well as such features from history stored in the memory M, while the decoder module takes in the output of the encoder concatenated with the goal descriptor g to produce a fixed dimensional feature vector, characterizing the current belief state b. An actor-critic network (consisting of linear layer) then predicts an action distribution xy (b,)).) and the value of this state. The agent then takes an action a ~ -b,,,,,, takes a step, and receives new observation. The goal descriptor network fa outputs the object category and the relative goal location estimation L. Following SAVi [5] we use off-policy category level predictions and an on-policy location estimator.
Navigation Using Language Policy, Tg. When an agent queries, it receives natural language instruc- tion instr E V from the oracle. Using instr and the current observation ef (FY F.P. F, 1. Pe). our language- based navigation policy performs a sequence of actions (at. ac+1, a(tv) as per Ty option, where each ag A. Specifically, for any step (l,...,A 1 (eG.gr) using a Transformer mncoder-decode network T the output of this Transformer is then concatenated with CLIP (28) embeddings of the instruction, and fused using fully-connected layer FC, The output of this layer is then concatenated with previous belief embeddings using a second multi-layer Transformer encoder-decoder T22 to produce the new belief state b,, i.e.
1), HI first encodes
6, = Ta (FC, (Ti gr), CLIP(instr)) (6, :t <<<''''' <T)) and Te(b,,.) = (P )).
(3)
Learning When-to-Query -uuryyyo-uuruuuuu Policy, Tg. As alluded to above, the Ty policy decides when to query, i.e., when to тe. Instead of directly utilizing model uncertainty [9]. we use the reinforcement learning framework to be train this policy in an end-to-end manner, guided by the rewards Ç.
Reward Design. For the Ty policy, we assign a reward of +1 to reduce the geodesic distance towards the goal, a +10 reward to complete an episode successfully, i.e., calling the stop action near the AudioGoal, and a penalty of -0.01 per time step encourage efficiency. As for the πe policy, we set a negative reward each time the agent queries the oracle, denoted Eg. as well as when the query is made within 7 steps from previous query, denoted Çz. If the ssoftly)-lllwwdd number of queries is K, then our combined negative reward function is given by Cy + {t,, where
k <<< K kx (r toxp(-u))
0 O<j<T otherwise,
(k(k)
and <r(s)
(4)
Fneg + exp(
k 2 K,
0
where Tneg is set to -1.2, and ry is set to -0.5. As a result, the agent learns when to interact with the oracle directly based on its current observation and history information. In the RL framework. the actor-critic model also predicts the value function of each state. Policy training is done using decentralized distributed proximal policy optimization (DD-PPO).
Policy Training. Learning Ty uses a two-stage training; in the first stage, the memory M not used, while in the second stage, observation encoders are frozen, and the policy network is trained using both the current observation and the history in M. The training loss consists (i) the falue-function loss, (ii) policy network loss to estimate the actions correctly, and (iii) an entropy loss to encourage exploration. Our language-basee navigation policy ne follows a two stage training as well. The first stage consists of off-policy training. We re-purpose the fine-grained instructions provided by [15) to learn the language-based navigation policy тy. The second stage consists of on-policy training. During roll outs in our hierarchical framework, the agent interacts with the oracle and
TTis is different Transformer from the one used for T. however taking 9r input the decoder.

---

Table 1: Comparison of performances against state of the art heard and unheard sound settings.
Heard Sound
Unheard Sound
Foodback
Success
SPL 1
S.A 1
DTG
SWS
Success f
SPL
SNA
DTG
SWS
1
Random Nav.
1.4
17.0
12
17.0
ObjectGool RL
X
1.5
0.8
0.6
16.7
1.1
1.5
0.8
0.6
16.7
1.1
al.
29.3
23.7
11.3
14.4
11.6
12.7
Chen el al 6
X
21.6
15.1
12.1
11.2
10.7
18.0
13.4
12.9
12.9
6.9
WaN
20,9
16.8
10.3
12.7
SMT +Audio
X
22.0
16.8
16.0
12.4
8.7
16.7
11.9
10.0
12.1
8.5
SAVi 5
33.9
24.0
21.5
13.2
9.9
AVLEN AVLEN
Language
36.1
24.6
19.7
8.5
23.1
26.2
17.6
14.2
9.2
1588
GT Actions
48.2
34.3
26.7
36.0
36.7
24.1
18.7
8.3
26.6
Table 2: Comparisons in heard and unheard sound settings against varied query-triggering methods.
Heard Sound
Unheard Sound
Feedback
Success
SPL
SNA 1
DIG 5
SWS
Success
SPL
SNA
DIG
SWS f
Random
Langunge
32.5
21.1
16.1
8.93
21.8
23.5
14.3
11.3
9.9
14.3
Uniform
Language
33.2
22.4
17.8
9.1
22.0
22.1
14.6
11.5
9.8
13.3
Model Uncertainty
Langunge
34.2
24.0
19.5
8.7
20.5
24.9
16.1
13.5
9.3
15.2
AVLEN
Language
36.1
24.6
19.7
23.1
26.2
17.6
14.2
9.2
15.8
receives language instructions, we use these instructions with the shortest path trajectory towards the goal to finetune Ty. In both cases, it is trained with an imitation learning objective. Specifically, we allow the agent navigate on the ground-trute trajectory by following teacher actions and calculate cross-entropy loss for each action in each step by, given by Σ. az- (og(Te((e,, 0z) where a" is the ground truth action and IT (eebb,, az) the action probability predicted by ITg.
Generating Oracle Navigation Instructions. The publicly available datasets for vision-and-languum tasks contain fixed number of route-and-instruction pairs at handpicked locations in the navigation grid. However, in our setup, navigating agent can query an oracle at any point in the grid. To this end, assume the oracle knows the shortest path trajectory s_path from the current agent location to the AudioGoal, and from which the oracle selects a segment consisting of n observation actioon pairs (we use n 4 in our experiments), i.e.. s path ((og. an), (0), a1) ............ (On-1, 1, an-1)). With this assumption, we propose to mimic the oracle by speaker model [13], which can generate distribution s__path"
| The observation and action pairs are sequentially encoded using an Encoders, (stpthsss)sh and decoded by another LSTM predicting the next word in the instruction by: wi = SpeakerDecoders ( (Fo , ris..... ...., F&) The instruction generation model is trained using the available (instruction, trajectory) pairs from the VLN dataset
of words ps (wrl 8
LSTM encoder, (F.S F ...., FR))
2] We use cross entropy loss and teacher foreing during training.
4 Experiments and Results
Dataset. We use the SoundSpaces platform (6] for simulating the world in which our AVLEN agent conducts the navigation tasks. Powered by Matterport3D scans [4], SoundSpaces facilitates a realistic simulation of potentially-comples 3D space navigable by the agent along densely sampled grid with Im grid-cell sides. The platform also provides access to panoramic ego-centric views of the scene in front of the agent both as RGB and depth images, while also allowing the agent to hear realistic binaural audio of acoustic events in the 3D space. To benchmark our experiments, we use the semantic audio-visual navigation dataset from Chen et al. 55] built over SoundSpaces. This dataset consists of sounds from 21 semantic categories of objects that are visually present in the Matterport3D seans. The object-specific sounds are generated the location of the Matterport3D objects. In each navigation episode, the duration of the sounds are variable and is -ormalldistributed with mean 15s and deviation 9s, clipped for a minimum 5s and maximum 500s (5). There are 0.5M/500/1000 episodes available in this dataset for train/val/test splits respectively from 85 Matterport3D scans.
Evaluation Metrics. Similar to [5]. we use the following standard metrics for evaluating our navigation performances on this dataset: i) success rate for reaching the AudioGoal, ii) success weighted by inverse puth length (SPL) [ iii) success weighted by inverse number of actions (SNA) 77. iv) average distance goal (DTG), and v) success when silent (SWS). SWS refers to the fraction of successful episodes when the agent reaches the goal after the end of the acoustic event.
's path is approximated in the discrete Room-te Moo--to-Re toom environment and then used to generate instruction.
7

---

Table Comparisons against varied query-triggering methods with ground truth action as feedback.
Heard Sound
Unheard Sound
Feedback
Success
SPI. |
SNA 1
DTG J
SWS 1
Success
SPL. 7
SNA 1
DTG 1
SWS 1
Random
GT Actions
39.8
30.0
29.6
22.1
16.3
Uniform
GT Actions
38.8
29.9
25.5
7.4
20.3
28.6
21.3
18.0
7.8
14,8
Model Uncertainty
GT Actions
41.3
30.6
24.8
7.3
26.3
31.4
22.7
18.4
8.2
19.3
AVLEN
GT Actions
48.2
34.3
26.7
7.5
36.0
36.7
24.1
18.7
8.3
26.6
Figure 3: (a) Comparison of AVLEN performances against baselines and when-to-query approaches in the presence of distractor sound, (b) Performance (SPL) comparison against varying the number of allowed queries, and (c) Distribution of queries triggered against the time steps in episodes.
Faciback
Succeas
Chea al AW-WUN tJ
4.0
2.4
2.0
2.3
3.0
2.0
1.6
SMITE
Muuioo
4.2
2.9
2.1
288
Kondom
Langrage
11.6
6.6
4.8
12.9
78
---
Uniforn
Langruge
11.5
6.8
5:1
13.5
7.7
Model Uzcertanoty
Latgrage
12.4
6.7
5.0
12.8
8.4
AVLEN
13.0
82
539
11.1
AVLEN
GT Actions
24.4
15.3
11.3
21.5
(a) Comparisons under distractors
Wxp
(b) Query sensitivity
(c) Query distribution
Implementation Details. Similar to prior works, we use RGB and depth images, center-cropped to 64 x 64. The agent receives binaural audio clip 65 x 26 spectrograms. The memory size for xg and q is 150 and for गy is 3. All the experiments consider maximum K = 3 allowed queries (unless 3 navigation steps in the environment
otherwise specified). For each query, the agent will take using the natural language instruction. We use a vocabulary with 1621 words. Training uses ADAM with learning rate 2.5 x 10 Refer to the Appendix for more details.
18
Experimental Results and Analysis. The main objective of our AVLEN agent in the semantic audio- visual navigation task is to navigate towards sounding object in an unmapped 3D environment when the sound is sporadic. Since we are the first to integrate oracle interactions (in natural language) within this problem setting, we compare with existing state-of-the-ar semantic audio-visual navigation approaches, namely Gan et al. [4], Chen et al. [6], AV-WaN 77]. SMT [12] + Audio, and SAVi (5). Following the protocol used in (5) and (6). we evaluate performance of the same trained model on two different sound settings: i) heard sound, in which the sounds used during test are heard by the agent during training. and ii) unheard sound, in which the train and test sets use distinct sounds. In both experimental settings, the test environments are unseen.
Table I provides the results of our experiments using heard and unheard sounds. The table shows that AVLEN (language)- - which is our full model based on language feedback shows a +2.2% and +1 +1.6% absolute gain in success rate and success-when-siles (SWS) respectively, compared to the best performing baseline SAVi (5] for heard sound. Moreover, we obtain 1.4% and 1. 1.1% absolute gain in success rate and SWS respectively for unheard sound compared to the next best method, SAVi. Our results clearly demonstrate that the agent is indeed able to use the short natural language instructions for improving the navigation.
natural question to ask in this setting is: Why are the improvements not so dramatic, given the agent is receiving guidance from an oracle? Generally, navigation based on language instructions challenging task in itself 22 255) since language incorporates strong inductive biases and usually spans large vocabularies; as result the action predictions can be extremely noisy, imprecise, and misguiding. However, the key for improved performance is to identify when to query. Our experiments show that AVLEN is able to identify when to query correctly (also see Table and thus improve performance. To further substantiate this insight, we designed an experiment in which the agent is provided the ground truth (GT) navigation actions as feedback (instead of providing the corresponding language instruction) whenever a query is triggered. The results of this experiment - marked AVLEN (GT Actions) in Table clearly show an improvement in success rate by nearly +15% for heard sounds and + +12% for unheard sounds, suggesting future work to consider improving language-basse based navigation policy ne.
Navigation Under Distractor Sounds. Next, we consider the presence of distractor sounds while navigating towards an "unheard" sound source provided in SAVi [5] In this setting, the agent must know which sound its target is. Thus, one hot encoding of the target is also provided as an input to the agent, if there are multiple sounds in the environment. The presence of distractor sounds
8

---

Feedback
Success
SPL
SNA
D
SWS 1.
Table 4: Comparisons in perfor-
x
33.9
mance for different architectural choices for language- based policy лe in heard sound setting.
(Glove (RL)
Language
36.1
24.6
8.5
(Clove Tranffemee)
Language
36.2
8.4
23.8
AYLEN CLIP-TranSoomee AVLEN (CLIP- à GRU)
Language
36.1
25.6
19.7
8.5
23.1
Lae Language
37.7
25.5
19.9
8.5
25.1
Unade Foodbed: Wulk lowed lamiingt
3
Torget oeenst lnatioa
Drucks Focebctk: Walh doar talbway
Targee abject (eitund)
Sllctied
Despleddara
Figure 4: Two qualitative results from AVLEN's ' navigation trajectories. We show egocentric views and top down maps for three different viewpoints in agent's trajectory. The agent starts from D, receives oracle help in (2, navigates to the goal in 3. In the top episode, agent receives directional information ("Walk forward and turn right'), whereas in the bottom episode, agent receives language instruction more grounded on the scene ('Walk down the hallway').
makes the navigation task even more challenging, and intuitively, the agent's ability to interact with the oracle could come useful. This insight is clearly reflected in Figure 3al where AVLEN shows 2.2% and 2.7% higher success rate and SWS respectively compared to baseline approaches. Figure 3a shows that AVLEN outperforms other query procedures as well and the performance difference is more sigmificant compared to when there is no distractor sound.
Analyzing When-to-Query To evaluate if AVLEN is able to query at the appropriate moments, we designed potential basclines that one could consider for deciding when-to-query and compare their performances in Table Specifically, the considered baselines are: i) Uniform: queries after every 15 steps, ii) Random: queries randomly within the first 50 steps of each episode, and iii) Model Uncertainty (MU) based on [9] queries when the softmax action probabilities of the top two action predictions of Tg have an absolute difference are less than a predefined threshold (5 0.1). Table 2 shows our results. Unsurprisingly, we see that Random and Uniform perform poorly, however using MU happen to be strong baseline. However, MU a computationally expensive baseline as requires a forward pass on the T. model to decide when to query. Even so, observe that compared to all the baselines, AVLEN shows better performance in all metrics. To further understand this, in Figure 3c we plot the distribution of the episode time step when the query is triggered for the unheard sound setting. As is clear, MU is more likely to query in the early stages of the episode, exhausting the budget, while AVLEN learns to distribute the queries throughout the steps, suggesting our hierarchical policy ny is learning to be conservative, and judicial in its task. Interestingly we also find reasonable overlap between the peaks of the two curves, suggesting that n. is considering the predictive uncertainty of ng implicitly.
Is the AVLEN reward strategy effective to train the model to query appropriately? To answer this question, we analyzed the effectiveness of our training reward strategy (i.c., to make the agent learn when to query) by comparing the performances of AVLEN with the other baseline querying methods (i.e., Random, Uniform, and Model Uncertainty) when the oracle feedback provides the ground truth navigation actions, instead of language instructions. The results are provided in Table 3 and clearly show the superior performances of AVLEN against the alternatives suggesting that the proposed reward signal is effective for appropriately biasing the query selection policy.
Are the improvements in performance from better Transformer and CLIP models? To understand the architectural design choices (e.g., modules used in mz) versus learning the appropriate policy to query (learning of nq), we conducted an ablation study replacing the sub-modules in my neural network with close alternatives. Specifically, we consider four architectural variants for ze for comparisons,

---

namely: (i) AVLEN (Glove GRU), (ii) AVLEN (Glove + Transformer), (iii) AVLEN (CLIP + GRU), and (iv) AVLEN (CLIP- + Transformer). We compare the performances in Table Our results show that the improvements when using AVLEN is not due to CL.PP context or Transformer representations alone, instead it from our querying framework, that consistently performs better than the baseline SAVi model, irrespective of the various ablations.
Sensitivity Allowed Number of Queries, v. To check the sensitivity AVLEN for different number of allowed queries (and thus the number of natural language instructions received), we consider E {0. 1. 2, 3, 4, 5} and evaluate the performances. Figure 3b shows the SPL scores this experiment in the unheard sound setting. As is expected, increasing the number of queries leads to an increase in SPL for AVLEN, while alternatives e.g., Random drops quickly; the surprising behaviour of the latter is perhaps a mix of querying at times when the xg model is confident and the T. instructions being noisy.
Qualitative Results. Figure provides snapshots from two example episodes of semantic audio- visual navigation using AVLEN. The sounding object is 'cushion' in the first episode (first row) and a 'picture' in the second episode (second row). 2 of both episodes shows the viewpoint where agent queries and receive natural language instructions. In the first row, agent receives directional information: 'Walk forward and turn right', while in the second row episode, the agent receives language instruction ground in the scene: 'Walk down the hallway'. In both cases, the agent uses the instruction to assist its navigation task and reach around the vicinity of target object a Please refer to the supplementary materials for more qualitative results.
Conclusions
The ability interact with oracle/human using natural language instructions to solve difficult tasks is of great importance from a human-machine interaction standpoint. In this paper, we considered such a task in the context of saafio-visual-langgaa embodied navigation in realistic virtual world, enabled by the SoundSpaces simulator. The agent, visualy navigating the scene to localize an audio goal, is also equipped with the possibility of asking an oracle for help. We modeled the problem as one of learning a multimodal hierarchical reinforcement learning policy, with a two-level policy model: higher-level policy to decide when ask questions, and lower-level policies to either navigate using the audio-gon goalo or follow the oracle instructions. We presented experiments using our proposed framework; our results show that using the proposed policy allows the agent achieve higher success rates on the semantic audio-visual navigation task, especially in cases when the navigation task is more difficult in presence of distractor sounds.
Limitations and Societal Impacts
Limitations. There are two key ingredients in our design that the performance of AVLEN is dependent on, namely (i) the need for a strong language- based navigation policy (re) to assist the semantic audio-visual navigation task, and (ii) pre-trained Speaker model to generate language instructions so that AVLEN can cater to a query at any location in the 3D navigation grid. These modules bring in strong priors (from their pre-traine pre-training) that in some of the cases the generated language instructions can be irrelevant, ambiguous, or incorrect, hampering the performance of the entire system. Further, this work uses only English language datasets and the MatterPort 3D dataset includes scans that mainly resemble North American houses, potentially causing an unintended model bias.
Societal Impacts. Our proposed AVLEN framework is step towards building human-robot cohesive ecosystem synergizing each other towards solving complex real-world problems. While, full autonomy of the robotic agents could be a important requirement for several tasks, we may not want an agent to take actions that it is unsure of, and our framework naturally enables such a setting. However, as noted in our experimental results, there seems to be gap in how the language is interpreted by the agent when translating to its actions, and this could be a point of societal concern, even when there is the possibility of interactions with the humans.
Acknowledgement, SP worked on this problem as part of a MERL internship. SP and AR are partially supported by the US Office of Naval Research grant N000141912264 and the UC Multi- Campus Research Program through award #A21-0101- (UW) and Dr. Moitreya Chatterjee (MERL) for useful discussions.
The authors thank Mr. Xiulong Liu
10

---

References
[1] Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, et al. On evaluation of embodied navigation agents. arXiv preprint /rXiv:/807.0000 /807.06757 2018.
[2] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, lan Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and- languugge navigation: Interpreting visually-groundee navigation instructions in real environments. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3674-3683, 2018.
[3] Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
[4] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint arXiv: (tt9:1770.06110 58,,, 2017.
[5] Changan Chen, Ziad Al-Halah, and Kristen Grauman. Semantic audio-visual navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15516-15525, 2021.
[6] Changan Chen, Unnat Jain, Carl Schissler, Sebastia Vicene Amengual Gari, Ziad Al-Halah, Vamsi Krishna Ithapu, Philip Robinson, and Kristen Grauman. Soundspaces: Audio-visual navigation in 3d environments. In European Conference on Computer Vision, pages 17-36. Springer, 2020.
[7] Changan Chen, Sagnik Majumder, Ziad AI-Halah, Ruohan Gao, Santhosh Kumar Ramakrishnan, and Kristen Grauman. Learning to set waypoints for audio-visual navigation. In International Conference Learning Representations, 2021.
[8] Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. Touchdown: Natural language navigation and spatial reasoning in visual street environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12538-12547, 2019.
[9] Ta-Chung Chi, Minmin Shen, Mihail Eric, Scokhwan Kim, and Dilek Hakkani-tur. Just ask: An interactive learning framework for vision and language navigation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 2459-24666 2020.
[10] Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1-10, 2018
[11] Jacob Devlin, Ming-Was Weig Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04880 04805, 2018.
[12] Kuan Fang, Alexander Toshev, Li Fei-Fei. and Silvio Savarese. Scene memory transformer for embodied agents in long-hore long-horizon tasks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 538-547, 2011
[13] Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg-Kirkpatriee Kate Saenko, Dan Klein, and Trevor Darrell. Speaker- follower models for vision-and lnn-lag navigation. Advances in Neural Information Process- ing Systems, 31, 2018
[14] Chuang Gan, Yiwei Zhang, Jiajun Wu, Boqing Gong, and Joshua B Tenenbaum. Look, listen, and act: Towards audio-visual embodied navigation. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9701-9707 IEEE, 2020.
[15] Yicong Hong, Cristian Rodriguez-Opazo, Qi Wu, and Stephen Gould. Sub-instruction aware daioo-and-language navigation. arXiv preprint arXiv:2004.02700) 2020.

---

[16] Yicong Hong. Qi Wu, Yuankai Qi, Cristian vodriguez-Opazer and Stephen Gould. VIn bert: A recurrent vision-and- -anggagg bert for navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1643-1653, June 2021.
[17] Liyiming Ke, Xiujun Li, Yonatan Bisk, Ari Holtzman, Zhe Gan, Jingjing Liu, Jianfeng Gao, Yejin Choi, and Siddhartha Srinivasa. Tactical rewind: Self-correction via backtracking in sision-and-anguuee navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 6741-6749, 2019.
[18] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 2014.
[19] Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lee. Beyond the nav- graph: Aision-and-language navigation in continuous environments, In European Conference on Computer Vision, pages 104-120. Springer, 2020.
[20] Tuyen P Le, Ngo Anh Vien, and TaeChoong Chung. A deep hierarchical reinforcement learning algorithm in partially observable markov decision processes. leee Access, 6:49089-49102, 2018.
[21] Chong Liu, Fengda Zhu, Xiaojun Chang, Xiaodan Liang, Zongyuan Ge, and Yi-Dong Shen. Vision-language navigation with random environmental mixup. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1644-1654, 2021.
[22] Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference robot learning, pages 1113-1132. PMLR, 2020.
[23] Chih- Ma, Jiasen Lu, Zuxuan Wu, Ghassan AlRegib, Zsolt Kira, Richard Socher, and Caiming Xiong. Self-monitoring navigation agent via auxiliary progress estimation. arXiv preprint arXiv: 0t////911000000 100035, 2019.
[24] Chih Yao Ma. Zuxuan Wu, Ghassan AlRegib, Caiming Xiong, and Zsolt Kira. The regretful agent: Heuristic-aided navigation through progress estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6732-6740. 2019.
[25] Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh, and Dhruv Batra. Improving vision-and-languan navigation with image-text pairs from the web. In European Conference on Computer Vision, pages 259-274. Springer, 2020.
[26] Khanh Nguyen and Hal Daumé III. Help, anna! visual navigation with natural multimodal assistance via retrospective curiosity-encourage imitation learning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCH) Kong, China, November 2019. Association for Computational Linguistics.
pages 684 694, Hong
[27] Khanh Nguyen, Debadeepta Dey, Chris Brockett, and Bill Dolan. Vision-based navigation with language-based assistance via imitation learning with indirect intervention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12527-12537, 2019.
(28) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 7748-8763. PMLR, 2021.
[29] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform for Embodied AI Research. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), 2019.
[30] Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 1 112(1- 2):181-211, 1999.
12

---

[31] Hao Tan, Licheng Yu, and Mohit Bansal. Learning to navigate unseen environments: Back translation with environmental dropout. arXiv preprint 0404:999440049555, 2019.
[32] Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. Vision-and-dialog navigation. In Conference Robot Learning, pages 394-406, PMLR, 2020.
[33] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in newral information processing systems, 30, 2017.
[34] Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, William Yang Wang, and Lei Zhang. Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6629 66399,668,,11 2019.
Wang.
[35] Erik Wijmans, Samyak Datta, Oleksandr Maksymets, Abhishek Das, Georgia Gkioxari, Stefan Lee, Irfan Essa, Devi Parikh, and Dhruv Batra. Embodied question answering in photorealistic environments with point cloud perception. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6659-6668, 2019.
[36] Licheng Yu, Xinlei Chen, Georgia Gkioxari, Mohit Bansal, Tamara L Berg, and Dhruv Batra. Multi-target embodied question answering. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9309-6318, 2019.
[37] Wang Zhu, Hexiang Hu, Jiacheng Chen, Zhiwei Deng, Vihan Jain, Eugene le, and Fei Sha. Babywalk: Going farther in ssion-and-languaal navigation by taking baby steps. arXiv preprint 2rXiv:2005.04625, 2020.
[38] Yi Zhu, Yue Weng, Fengda Zhu, Xiaodan Liang, Qixiang Ye, Yutong Lu, and Jianbin Jiao. Self-motivated communication agent for real-world vision-dialog navigation. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1594-1603, 2021.
13

---

NeurIPS Paper Checklist
the main claims made in the abstract and introduction accurately reflect paper's contribu- tions and scope? (Yes)
Have you read the ethics review guidelines and ensured that your paper conforms to them? (Yes)
Did you discuss any potential negative societal impacts of your work? (Yes)
Did you describe the limitations of your work? (Yes)
Did you include the code, data, and instructions needed reproduce the main experimental results (either in the supplemental material or as a URL)? (Instructions are provided in supplementary material)
Did you specify all the training details (e.g., data splits, hyperparameters, how they chosen)? (All details are provided in supplementary material)
Did you report error bars (e.g.. with respect the random seed after running experiments multiple times)? (Provided in the supplementary materials.)
Did you include the amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? (Provided in supplementary material)
If your work uses existing assets, did you cite the creators? (Yes)
Did you mention the license of the assets? (Not required, it's public dataset)
Did you include any new assets either in the supplemental material or as a URL? (No)
Did you discuss whether and how consent was obtained from people whose data you' rr us- ing/curating? (Not applicable)
Did you discuss whether the data you are using/curating contains personally identifiable informa- tion offensive content? (Not applicable)
14


Reviews:
[{"review_id": "wAorAgVrNCw", "reviewer": "Reviewer_FS8x", "summary": "The manuscript modifies the semantic audio-visual navigation task to incorporate oracle feedback, intermittently, throughout episodes. The approach leverages an options framework, in order to encourage the policy to learn when to query the oracle.", "questions": "Why is assuming access to an oracle considered a reasonable problem setting?\n\nIn order to more flexibly accommodate the penalties, why not formulate the problem as a constrained POMDP, with, among other signals, the queries as costs? Indeed, it would provide more control over the query budget. Can you discuss the tradeoff(s) in doing so?", "limitations": "See above — i.e., reward design, problem formulation, (lack of) ablations", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nThe manuscript provides an options framework and extends the SAVi baseline to incorporate natural language. \n\nThe manuscript is reasonably well-written.\n\nWeaknesses\n\nSection 2: Consider other interaction paradigms — see some of the new embodied AI surveys, e.g., https://www.jair.org/index.php/jair/article/download/13646/26807/\n\nSection 3: More analysis necessary: I would have liked to see a discussion that justifies that the reward signal is sufficiently expressive for appropriately biasing the high-level policy.\n\nSection 3: Whereas predicating the reward design on distance-to-goal may have been essential to get started on applying RL frameworks to these embodied tasks, it is now important to move beyond this. Methods should pursue self-supervisory alternatives as reward signals, which would narrow the gap to practical real-world deployment.\n\nSection 4: The manuscript is missing sufficient ablations, to confirm that the performance improvements are indeed resulting from the querying framework, as opposed to the more-expressive CLIP context, transformer representations, and VLN pre-training tasks; these are not present in the previous baselines. Despite the addition of those components, with unsupported contribution, the improvements are admittedly not so dramatic, as the manuscript self-admits.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HdMuxi0mnp", "reviewer": "Reviewer_ob8D", "summary": "This work proposes a new audio-visual-language embodied navigation task, where the agent not only hears the audio from the sounding object but also is able to query the environment of the goal location. To deal with the multimodal navigation task, this paper propses a multimodal hierarchical policy that consists a high-level pocliy that decides which low-level policy and two low-level policies that either navigates with audio-visual cues or navigates with queried instructions. In the experiment, authors compare with the SOTA model in semantic auido-visual navigation and show that the proposed model outperforms the SOTA model.", "questions": "The trained oracle speaker seems to be useful for reproduce the results. Will you release the model and code?\n\nIn the supp video, it looks like a couple successful cases were because of the query of stop. Is it possible most the performance improvement comes from the stop query?\n\nDo authors have an intution of why the performance improvement is not very big? It seems like the language instructions provide pretty strong guidance for navigation.\n", "limitations": "Both the limitations and societal impact were discussed in the conclusion.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed task is interesting and adds one more modality to the audio-visual navigation task, although the ability to query is not very realistic in the real world.\n\nThe multimodal hierarchical model is also pretty novel in that the hierarchy is for in modality selection rather than the common goal prediction/actuation command hierarchy.\n\nUsing RxR dataset to train a speaker model as the oracle is inspiring and this show a promising way to bridge the gap between the audio-visual navigation task and the vision-language navigation task.\n\nThe paper is clearly written and easy to understand. The experiments and analysis are also pretty sufficient.\n\nI expected the success rate and SPL improvement be higher than just a couple percents. I wonder if authors have answer for that.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "CLaEB-Al3dE", "reviewer": "Reviewer_kLxQ", "summary": "This paper introduces AVLEN, Audio-Visual-Language Embodied Navigation. In this setup, an embodied agent aims to find the location of an audio event by navigating in a simulated 3D visual world by interacting with the environment and an oracle. The oracle provides free-form natural language utterances to guide the agent. To address this task, a hierarchical RL system is proposed. Query policy \\pi_q learns whether to use audio-goal policy \\pi_g or query the oracle and use language-feedback policy \\pi_l. Experiments show that the proposed method surpasses audio-only methods and successfully learns when to query the oracle.  ", "questions": "\n* L87 we --> our\n\n* L193 please eliminate epsilon notation for policies. \n\n* L243 First, please clarify the intuition behind this formulation for the reward function. Second please add equation number. Also, please clarify how you choose hyperparameters r_neg and r_f\n\n* L251 Please explain the entropy loss. It is not clear to me.\n\n* Table1: Please add AVLEN only \\pi_l where budget is infinity. also add a column with checkmarks for language input available or not. AFAIK none of the models you compare have access to the oracle.\n\n* L278 in front --> in front\n\n* L281-282 How do you ensure that the sound comes from a unique object? There are several objects of the same type in the environment. \n\n* Figure 3a: please report  \\pi_g, \\pi_l only models here as well. \n\n* Figure 3b: please report K=0,1,2 as well\n\n* L328 it's --> its\n", "limitations": "Please discuss limitations in terms of language i.e. you only worked with English annotations also visual diversity i.e. MatterPort images come from North American houses. ", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "* S1 paper contributes to the literature by showing that audio-only methods can be improved with interactive agents when they are provided with natural language instructions\n\n* S2 paper shows that rather than using the model's uncertainty as the query policy, we can train the system to learn when to query.\n\n* W1 While reporting results, the paper misses some essential information or experiments which could be misleading or hard to evaluate fairly the merit of the work. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "10YnPRXuld9", "reviewer": "Reviewer_zxx9", "summary": "In many scenarios, to effectively navigate in the real-world, agents need to combine a variety of sensory information including audio, visual, language, etc. VL agents navigate based on only vision and language input. More recently, SoundSpaces introduced the task of navigating conditioned on an audio source but without language guidance. This paper presents AVLEN, an embodied agent that leverages audio-visual cues in addition to natural language instructions from an oracle for navigation. The agent is trained using hierarchical RL and consists of one high-level policy for deciding when to query the oracle and two low-level navigation policies conditioned on either audio-visual cues or language instructions. The paper demonstrates that the proposed multimodal agent performs better across several evaluation metrics compared to SOTA models on several challenging sound settings in the AudioGoal task. Several ablations are conducted to analysis the usefulness of querying for language guidance from an oracle, using RL for learning when to query, and robustness of the navigation agent to distracting sounds. ", "questions": "- The paper claims that AVLEN solves VL navigation as a subtask of the SAVi task. It'd make a stronger argument if the paper also included experiments to support this claim by investigating the performance of the AVLEN agent in traditional VL benchmarks like RxR, CVDN, etc.\n- Is there an explanation as to why the random model's performance decreased with a larger budget for number of queries? Also the performance gain with more queries seems a bit negligible. It seems that if an agent had access to more oracle guidance, the performance should increase more linearly. \n- Is there a breakdown of SR for different goal categories? Wondering if the SR is pretty similar across all categories of sounds or if there are certain categories that are better than others. ", "limitations": "The authors point out several limitations of their work. Specifically they mention the assumptions that the agent is able to query anywhere in the environment and a pretrained speaker model is used for generating instructions. In terms of societal impact, the authors acknowledge that language-guided / instruction following agents that interface with humans are increasingly relevant and will soon permeate our everyday lives. They state that such agents facilitate automation of challenging human tasks, but neglect to address the negative aspects of such a trend, e.g. putting people out of work. The authors briefly mention that perhaps the simulated environment the agents are trained in, which are hand selected by humans, may be biasing the agent's behavior. It may also be interesting to (which may be quite far-fetched for the current state of the field) consider the negative consequences of introducing sound as an additional modality and equipping robots with audio-capturing sensors which may be used for nefarious purposes. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n- First to combine audio-visual navigation with natural language instruction guidance from oracle. Introduces a navigational agent that can leverage full range of modalities, and not just a subset.\n- Paper is well motivated both in terms of the task setting and technical approach. \n- The authors report SOTA metrics on the benchmark task across all metrics and compare their method against several strong baselines. \n- Good set of ablations, demonstrating that the performance gain is bottlenecked by the speaker model and also how the agent learns to distribute the queries throughout the trajectory unlike the MU baseline\n\nWeaknesses:\n- There have already been many prior works focused on solving the individual subproblems of AV navigation and separately agents that can interface with an oracle to ask for help. \n- Could use an explanation for why n=4 is used for the number of segments used for producing the oracle instruction. Was it chosen arbitrarily? \n- Could also compare against a non-hierarchical baseline that uses a single network fusing together the different modalities and producing action to show that the hierarchical abstraction is indeed helpful\n- The ablation with distractor sounds makes sense, but I'd imagine that the language guidance should help to disambiguate the distracting sounds a bit more. It might be challenging with only audio-visual, but language instructions should provide a stronger discerning signal. \n\nMinor:\n- L330 - \"the agents interaction ability is would come useful\". wording\n- Typo in equation after L234\n- For Figure 3c is this a distribution of queries over multiple rollouts? ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept