Knowledge-driven Scene Priors for Semantic Audio-Visual Embodied Navigation
Authors: Gyan Tatiya
Jonathan Francis
Luca Bondi
Ingrid Navarro
Eric Nyberg
Jivko Sinapov
Jean Oh
Venue: Submitted to ICLR 2023 (2023)
Keywords: Audio-Visual Navigation
Scene Priors
Object Relations
Embodied AI

Abstract: Generalisation to unseen contexts remains a challenge for embodied navigation agents. In the context of semantic audio-visual navigation (SAVi) tasks, generalisation includes both generalising to unseen indoor visual scenes as well as generalising to unheard sounding objects. Previous SAVi task definitions do not include evaluation conditions on truly novel sounding objects, resorting instead to evaluating agents on unheard sound clips of known objects; meanwhile, previous SAVi methods do not include explicit mechanisms for incorporating domain knowledge about object and region semantics. These weaknesses limit the development and assessment of models' abilities to generalise their learned experience. In this work, we introduce the use of knowledge-driven scene priors in the semantic audio-visual embodied navigation task: we combine semantic information from our novel knowledge graph that encodes object-region relations, spatial knowledge from dual Graph Encoder Networks, and background knowledge from a series of pre-training tasks---all within a reinforcement learning framework for audio-visual navigation. We define a new audio-visual navigation sub-task, where agents are evaluated on novel sounding objects, as opposed to unheard clips of known objects. We show improvements over strong baselines in generalisation to unseen regions and novel sounding objects, within the Habitat-Matterport3D simulation environment, under the SoundSpaces task. We release code, knowledge graphs, and dataset generation details in the supplementary material.

Under review as a conference paper at ICLR 2023
KNOWLEDGE-DRIVEN SCENE PRIORS FOR SEMANTIC AUDIO- VISUAL EMBODIED NAVIGATION
Anonymous authors Paper under double-blind review
ABSTRACT
Generalisation unseen contexts remains challenge for embodied navigation agents. In the context of semantic audio-visual navigation (SAVi) tasks, the no- tion of generalisation should include both generalising to unseen indoor visual scenes as well as generalising to unheard sounding objects. However, previous SAVi task definitions do not include evaluation conditions on truly novel sounding objects, resorting instead to evaluating agents on unheard sound clips of known objects; meanwhile, previous SAVi methods do not include explicit mechanisms for incorporating domain knowledge about object and region semantics. These weaknesses limit the development and assessment of models' abilities to generalise their learned experience. In this work, we introduce the use of knowledge-driven scene priors in the semantic audio-visual embodied navigation task: we combine semantic information from our novel knowledge graph that encodes object-region relations, spatial knowledge from dual Graph Encoder Networks, and background knowledge from series of pre-training tasks-all all within reinforcement learn- ing framework for audio-visual navigation. We also define a new audio-visual navigation sub-task, where agents evaluated novel sounding objects, opposed to unheard clips of known objects. We show improvements over strong baselines generalisation to unseen regions and novel sounding objects, within the Habitat-Maan Matterport3D simulation environment, under the SoundSpaces task. We will release all code, knowledge graphs, and pre-training datasets upon acceptance.
INTRODUCTION
Humans able background experience, when navigating unseen or partially-observal environments. Prior experience informs their world model of the semantic relationships between objects commonly found in an indoor scene, the likely object placements, and the properties of the sounds those objects emit throughout their object-object and object-scene interactions. Artificial embodied agents, constructed perform goal-directed behaviour in indoor scenes, should be endowed with similar capabilities; indeed, as autonomous agents enter our homes, they will need intuitive understanding about how objects are placed in different regions of houses, for better interaction with the environment. Whereas external (domain) knowledge can yield improvements in agent comple-efficienge while learning, generalisability to unseen environments during inference, and overall interpretability in its decision-making, the goal of finding generalisable solutions by injecting knowledge in embodied agents remains elusive (Oltramari et al. 2020) Francis et al. 2022).
The task of semantic audio-visual navigation (shown in Fig. lends itself especially well to the use of domain knowledge, e.g., in the form human-inspired background experience (encapsulated as prior over regions and semantically-relate associated with particular places, e.g., smoke alarm is more likely to originate in the kitchen. To infer such semantic information from sounds in an environment, we propose the idea of knowledge- enhanced prior.
objects contained therein). Certain sounds can be
By using a prior enriched with general experiences, we hypothesise that the learned model would generalise to novel sound sources. We adopt modular training paradigm, which has been shown to lead to improvements in cross-domain generalisability and more tractable optimisation (Chen et al.) 2021b [Chpllot et al. 2020b] Francis et al. (2022). To verify our hypotheses, we evaluate the agent's s performance on a set novel sounding objects that were not introduced during training.

---

Under review as a conference paper at ICLR 2023
Contributions. First, introduce the use of knowledge-drive scene priors in the semantic audio-viss visual embodied navigation task: we com- bine semantic information from our novel knowl- edge graph that encodes object-region relations, spatial knowledge from dual Graph Encoder Net- works, and background knowledge from a series of pre-training taskslall within a reinforcement learn- ing (RL) framework. Second, we define a knowl- edge graph that encodes object-object, object- region, and region-region relations in house en- vironments. Next, we curate a multimodal dataset for pre-training a visual encoder, in order to en- courage object-awareness in visual scene under- standing. Finally, we define new task of semantic audio-visud navigation, wherein assess agent performance on the basis of their generalisation to truly novel sounding objects. We offer experi- mental results against strong baselines, and show improvements over these models on various perfor- mance metrics in unseen contexts. We will provide all code, dataset-generation utilities, and knowl- edge graphs upon acceptance of the manuscript.
x
x
X
Figure 1: Illustration of the proposed semantic audio- visual navigation task. The agent initialised at a random location in a 3D environment and tasked to navigate to the sounding object based on audio and visual signals. The sound signal may stop while the agent navigating (c.g., sound produced by wash- ing machine stops). Thus, the agent is encouraged to understand the sound and visual semantics to reason about where to search for the sounding object. For example, in the image above, the agent hears the wash- ing machine sound and decides to navigate near the bathroom to search for the washing machine.
2 RELATED WORK
Modularity in goal-driven robot navigation. Goal-oriented navigation tasks have long been
topic of research in robotics (Kavraki et al.) [996; Lavalle et al., 2000; Canny 1988. Koenig & Likhachev] 2006). Classical approaches generally tackle such tasks through non-learning techniques
for searching and planning. e.g., heuristic-based search (Koenig & Likhachev 2006) and probabilistic planning (Kavraki et al. 1996). Although classical approaches might offer better generalisation and optimality guarantees in low-dimensional settings, they often assume accurate state estimation and cannot operate on high dimensional raw sensor inputs (Gordon al. 2019). More recently, researchers have pursued data-driven techniques, e.g., deep reinforcement learning (Wijmans et al. 2020; Batra et al. 2020; Chaplot et al., (2020a) Yang al. 2019; Chen et al., 202 a Gan et al. 2020) and imitation learning (Irshad et al. 2021 Krantz et al. 2020). to design goal-driven navigation policies. End-to-end mechanisms have proven to be powerful tools for extracting meaningful features from raw sensor data, and thus, are often favoured for the setting where agents are tasked with learning to navigate toward goals in unknown environments using mainly raw sensory inputs. However, as task complexity increases, these types of systems generally exhibit significant performance drops, especially in unseen scenarios and in long-horizon tasks (Gordon et al.] 2019; Saha et al. (02T). To address the aforementioned limitations, modular decomposition has been explored in recent embodied tasks. Chaplot et al. (2020c) design a modular approach for visual navigation, consisting of mapping module, global policy, and local policy, which, respectively, builds and updates map of the environment, predict the next sub-goal using the map, and predicts low-level actions to reach the sub-goal IIshad et al. (2021) also define a hierarchical setup for Vision-Language Navigation (VLN) (Anderson et al. 2018). where a global policy performs waypoint-prediction. given the observations, and a local policy performs low-level motion control. Gordon et al. (2019) design hierarchical controller that invokes different low-level controllers in charge of different tasks such as planning, exploration, and perception. Similarly, /Shaa et al. (202T) design a modular mechanism for mobile manipulation that decomposes the task into: mapping, Language-understan grounding, and planning. Aforementioned modular designs have shown to increase task performance and generalisability, especially in unexplored scenarios, compared to their end-to-end counterparts. Motivated by these, we develop a modular framework for semantic audio-visual navigation, which includes pre-trained and knowledge-enhancee scene priors, enabling improved unseen generalisation.
modality
Knowledge graphs in visual navigation. Combining prior knowledge with machine learning systems remains a ieeetiiiyettigttedd topic in various research fields, such as natural language processing (Ma et al., 2202]; 2019); |rancis et al.] (002).. due to the improvements in generalis- ability and sample-efficiency that symbolic representation promises for learning-based approaches.
2

---

Under review as a conference paper at ICLR 2023
Historically, integrating symbolic knowledge with, e.g., navigation agents has proven non-trivial, yielding a collection of research areas focusing on smaller components of the problem- suchem-suchhae as finding the appropriate representation of the knowledge (e.g.. logical formalism, knowledge graphs, probabilistic graphical models), the appropriate type of knowledge that should be encoded (e.g.. spatial commonsense, declarative facts, etc.), and the best knowledge-injection mechanism (e-g., graph convolutional networks, grounded natural language, etc.) (Ma et al. 2019). Knowledge graphs have gained popularity due to their interpretability and general availability as existing large-scale resources, such as ConceptNet (Speer et al. 2017) and VisualGenome (Krishna et al. 2016). Fortu- itously, graph processing of structured data has experienced a surge of popularity in recent years, leading to renewed interest in this neuro-symbolism (Oltramari et al.] 2020: Wu
deep learning
jet al. (021). Some visual navigation works exploit knowledge graphs in the pursuit generalisation (Moghaddam et al., 2020, Yang et al.) 2019 Lv et al. 2020 Du et al. 2020; Vijay et al. (019). Yang ct al. (2019) create knowledge graphs based on Visual (Krishna et al. 2016) and inject Teatures extracted from the graph as prior knowledge in visual navigation. In similar fashion. Qiu et al.
(220)) provide agents with knowledge of object relational semantics. However, the priors provided by these works only leverage object-object connections. Lv et al.] (2020) show improvements in goal-directed visual navigation by injecting 3D spatial knowledge into learning-baseer agents. Inspired by these works, we construct knowledge graph that includes all object-object, object-region, and region-region declarative semantics, which enables the more complex reasoning path, sound + object + region, in audio-visur navigation. Therefore, to our best knowledge, we become the first to study knowledge-driven scene priors for the audio-visual navigation task family.
Generalization to unseen contexts. Chen et al. 22200 2021b a) leverage the SoundSpaces (Chen
al. 2020) simulation environment and dataset to design and assess Audio- Navigation policies.
The dataset is based on photorealistic indoor environments from the Matterport3D (Chang et al. 2017) and Replica (Straub et al. 2019) datasets, to which 102 sound sources commonly found in indoor environments (e.g.. household appliances, musical instruments, telephones, etc.) were incorporated. The SoundSpaces dataset is split, such that indoor scenes encountered during testing are not found in the episodes used during the training stage. However, sounds of objects encountered during training may also appear during testing. Gan et al. (2020) also explore Audio Visual Navigation, but using the simulation platform A12-THOR (Kolve et al. 2017). which contains computer-generated graphical imagery. The authors introduce the Visual-Audi Room (VAR) benchmark consisting of seven different indoor environments-two of which were used for training and five for testing. The VAR benchmark incorporates three different audio categories: ring tone, alert alarm, and clocks. Similar to the AVN task introduced before, the same sound sources are found both in the training scenes, as well as and the testing scenes. In this paper, we argue that in the context of Audio Visual Navigation tasks, generalisation to unseen environments pertains to both generalising to unseen visual scenes, as well as to unheard sounds. Current Audio Visual benchmarks not take into consideration the latter. Thus, there is no direct assessment of generalisation performance to unheard sounds. To tackle this limitation, we propose a curated version of the SoundSpaces dataset where we evaluate our agent in four conditions: (1) seen houses and heard sounds, (2) seen houses and unheard sounds, (3) unseen houses and heard sounds, and (4) unseen houses and unheard sounds.
3 PROBLEM DEFINITION
We first consider the semantic audio-visual navigation (SAVi) task (Chen et al.) 2021a): an agent is initialised at a random location of an unmapped 3D house environment, which contains a sounding object (e.g.. piano). The agent must reach the sounding object, using its sensory inputs, consisting of vision and audio. Two assumptions are made in this task: firstly, the target sound has variable length in an episode and may not be available at every time step: the sound (c.g., a telephone ringing) may saaatiicallymmainngtte
stop during navigation; secondly. the sounding object has a physical and embodiment in the scene (e.g., the sound of a telephone ring associated with a physical manifestation of a telephone, as opposed to the sound of an airplane passing overhead being associated with the center of the living room). These assumptions are realistic because sound events have variable length in the real world and are based on the semantics of the corresponding sounding objects. Due to the variable-length nature of the sound, the agent cannot rely exclusively on the audio signal to reach the sounding object: instead, the agent must use the audio signal to both predict the sounding object's location as well as understand the object's semantics. Moreover, the agent needs to associate its visual cues with the sound and reason about object and region relationships, in order navigate effectively.
3

---

Under review as a conference paper at ICLR 2023
To study these phenomena, we extend the SAVi task by evaluating agents on completely unheard sounding objects. In the original task (Chen et al. (021a), agents were evaluated on unheard clips of known sounding objects, whereas in our task, agents are evaluated on completely unknown sounding objects. More formally, we consider a set of sounding objects O (e.g., shower, TV monitor, etc.), set of indoor regions R (e.g., bathroom, living room), and set of houses H. A particular house h; E H has set of regions {ri T......, Taa and a set of objects {0a1, 022,... ........ Ock J, where there are k objects placed in j regions of the house h4. Note that there are multiple instances of each sounding object o E O and region r E R across all houses H. We divide the total set of possible houses H into two mutually exclusive subsets: Hseen and Hunseen- Similarly, we divide sounding objects O into Ohaad and Ounheard- The houses in Haren and the sounding objects in Oheard are only experienced by agents during training; agents are evaluated on unheard sounding objects Ounheard- To solve this task, agents must learn to reason about the novel sounds based on prior knowledge; our work aims to enable agents to reach sounding objects they have never experienced before.
4 KNOWLEDGGE-DRIVEN SCENE PRIORS FOR VISUAL NAVIGATION
We introduce a knowledge-driven approach for semantic audio-visual embodied navigation (K- SA-EVE,), which incorporates scene priors in knowledge graph form and extracts relational features using Graph Encoder Networks (GEN) (Kipf & Welling 2017) for audio and visual modalities. GENs provide agents with reasoning capability, using prior knowledge, and dynamically update their beliefs according to new observations. Our model also incorporates Scene Memory Transformer (SMT) (Fang et al. 2019) that captures long-term dependencies by recording visual features in memory and locating the goal by attending to acoustic features. We compute visual features by combining a vision-based semantic knowledge vector with visual encoder representations. Similarly, we use audio observations to compute acoustic features, combining audio-based semantic knowledge vector, features encoded from the audio encoder, and location prediction. Thus, the prior knowledge-drive reasoning capability using GENs with the memory- -based attention mechanism using SMT allows the agent to generalise to novel houses and sounding objects, exploit spatio-temporal dependencies, and efficiently navigate to goal. The 6 modules of K SAVEN are summarised in Fig. 2 1) Pre- trained models that, given the audio and visual observations from the environment, predict objects and regions; 2) Graph Encoder Networks that compute audio-semantic and visual-semanti feature embeddings; 3) Vision Encoder that projects the visual observations at each step to embedding space; 4) Audio Encoder that projects the audio observations at each step to an embedding space; 5) Location Predictor that, given the acoustic signal from the sounding object, predicts its relative distance and direction from the agent: 6) Scene Memory Transformer that uses an attention- based policy network, which computes distribution over actions, given the encoded observations in scene memory and the acoustic observation that captures goal information. We detail each module below.
Modular Pre-training. In our task, the agent relies on audio observations to its goal and uses visual observations to navigate to that goal; the agent must detect objects and regions in given observation. To this end, we trained audio classification model få to predict a score for each object E O, as likelihood that o produced the acoustic observation, and vision classification model f to predict a score for each object o E O and region r E R as likelihood that the observation corresponds to region r. The acoustic event has variable length and may not be present at each time step, so the agent cannot rely on the current audio observation alone as a persistent signal. Thus, our model aggregates the current prediction èt with the previous prediction di- df fa(c, df_1) = (1 5)24 + 5ck_ where 8 is the weighting factor set to 0.5. When the acoustic event stops (i.e., zero sound intensity), the agent uses its latest estimate c..
Knowledge graph construction. Our knowledge graph captures spatial relationships between object-to-object object-to-region, and region-to-region. This prior knowledge about how objects are placed in regions of houses enables the agent to reason about where to find novel-sounding objects for efficient navigation; more precisely, this prior knowledge enables the reasoning path, Sound -> Object + Region, which is crucial to the task of audio-conditioned visual navigation. For example, suppose the squeaky sound produced by a chair is novel to the agent, and it knows that chairs are usually kept close to tables cushions and found in living rooms, offices. In that case, it may decide to navigate to regions that usually have chairs and objects usually placed close to chairs, which would lead to finding the chair faster than not knowing such spatial and semantic relationships between objects and regions. Our knowledge graph is denoted by an undirected graph G = (V, E). where V and F denote vertices and edges, respectively. Each vertex denotes an object or region, and
4

---

Under review as a conference paper at ICLR 2023
Vikin Nebworka
Pallyy Network
Causiootion
GEN!
Endedärge
Nox baror
Audio Networks
Drean apdtrogana
x?
iP
Adion
Lant
OEN
Veite
Graph
Spunskard
Festures
Bndedding
(2)
Figure 2: (a) K-SAVEN's system overview. Visual observation Ut is fed to two modules: vision encoder fe which encodes the visual observation, and pre-trained vision model fä. which, given the visual observation, predicts classification scores ci for objects and regions. These scores are used by the vision- graph encoder network GEN" to compute visual-semantic feature embeddings. Binaural audio ohservation b, is fed to three models: audio encoder f (encodes the audio observation), location predictor f.c (predicts distance and direction 1, of the sounding object from the agent, and direectto-reveebberatt-aatt 6,), and pre-trained audio model f² (given the audio observation. predicts classification scores cj for objects). These scores are used by the audio-based graph encoder network GEN to compute audio-semantic feature embeddings. The outputs of f². GEN" and f are stored in memory M, along with the agent's pose pt and previous action 0r-1- The sncettion-based policy network conditions the encoded visual information Mr on the acoustic information, enabling the agent to associate visual cues with acoustic events and predict the state representation &;, which contains spatial and semantic cues helpful reach the goal faster. The actor-critic network, given the state 8t, predicts the next action dr. (b) Graph Encoder Networks. Each vertex denotes an object or region category. The initial vertex features fed into the GEN" are initialised with the joint embedding obtained by concatenating word embeddings of object or region names and classification scores of objects and regions based on the current observation. GEN" performs information propagation through the three layers, and the output of the GEN" is spatial and semantic aware embeddings. The audio-base GEN uses fe and GEN instead of fe and GEN".
each edge denotes the relationship between pair of vertices. To compute these relationships, we use Matterport3D dataset (MP3D: Chang et al. (2077) as it contains semantic labels of 42 objects and 30 regions for 90 houses. We only use 21 objects and 24 regions ((V) 45), which were used in the original SAVi task (Chen et al. Ta) to build the knowledge graph (more details in Section (5). More specifically, two objects are connected with an edge if they are found in the same region, and their frequency of occurrence is above a threshold. We compute this frequency with respect to the most frequent object of that region and set the threshold the maximum value that connects each object with at least one other object. An object and region are connected if the region contains other objects, which connected with the object based object-to-object relations. Finally, two regions are connected if their frequency of containing connected objects. based on object-to-objec relations, above threshold. We set the threshold to the maximum value connecting each region with at least one other region. Further knowledge graph construction and representation details are in Appendix B
Location Prediction and Sireet-to-Reverberete Ratio Estimation. The audio observation contains information about the relative distance and direction from the agent to the sounding object. Thus, we jointly trained location predictor Mic to predict a location i, = (Ax, Ay). relative to the current pose Pt. of the agent, and the direct-to-reverberad between the sounding source and the agent. Similar to the pre-trained audio model, our location prediction also aggregates the current estimate 1, with the previous 4 4 fa 4-1, Ap;, 6,4, + (1- (1
ratio (DRR) E [0, 1] of the impulse response
Apt). where f-(-) transforms the previous location prediction 4-1 based on the last pose change Ap,. Here, 8, is either fixed to 0.5 vt (cxponential average) or assigned the value of the estimated DRR (dynamic average). The agent uses its latest estimate = (4,4.11 1, Apt) when the acoustic event stops. Note that DRR prediction also serves as an auxiliary task, as it will help the agent better estimate the directness and location of the sounding object. In fact, DRR provides an indirect measure of the acoustic distance between the source and the agent, independent of the sound level of the source. At training time, we build the ground truth for ò, from the room impulse response (RIR) between the source and the agent the ratio between the energy of the RIR in the first 10ms after the peak and the overall energy of the RIR. Thus, 6, measures how direct the acoustic propagation between the sounding object and the agent is: when the agent is far from the source, de
5

---

Under review as a conference paper at ICLR 2023
tends towards 0; the agent gets closer to the source, & increases. When the source is silent, d, equals O; thus, 51 predicts trustworthiness of location prediction, based on the binaural sound itself.
Encoder Networks. The goal of GEN" and GEN are to extract semantic knowledge vector using the graph G = (V, E). As shown in Fig. 2 the input to each vertex e is feature vector ze. which is a concatenated representation of both semantic cues (i.e., language embeddings) and the visual or acoustic cues (i.e., the classification score for objects and regions based on the current visual image or sound signal). The language embeddings are generated by Glo (Pennington et al. 2014) (Jénc) and the classification score is generated by pre-trained vision (J²)) or audio (J2) models (see Section
The knowledge graph represented as a binary adjacency matrix A. Similar to Yang et al. (2019) : Kipf & Welling (017)) we perform normalisation on A obtain A. Let vvvl V
R be the inputs of all vertices and Z [z1,... 2|v1] RIVIX be the output of the GENs, where D and F denote the dimension of the input and output feature. Our GENs perform the following layer-wise information propagation rule: H1++1) = ((H( Here, H = X, H)LL = Z, wi() the parameter for the 1-th layer, L is the number of GEN layers, and o denotes an activation function. We initialise each vertex based on current observation then perform cntemaation-rrppagatie
X 11 ................................
to compute audio-based and vision-based semantic knowledge vectors. The vision- based knowledge vector is stored in memory M, and the audio-basee knowledge vector is used to attend to the encoded memory Me. The output is a graph embedding which serves as a spatial- and semantic-aware representation for policy optimisation. Our vision encoder f" encodes the visual observations, consisting RGB and depth images from the agent's perspective. Our audio encoder f encodes the binaural audio observations heard by the agent into a two-channel log-mel spectrogram, with third channel encoding the generalised cross-correlation with phase transform Knapp & Carter 1976) between the two channels.
Policy Network. We use transformer-basee architecture for our RL policy network, which stores observations in memory M. At each time step, our model encodes each visual observation, ex = (e and et -e = GEN" (f" (v:)) to save in the memory. Our model also stores in memory the agent's pose p, defined by its location and orientation (z, y. #) with respect its starting pose Po in the current episode, and a;-1, the previously executed action. Thus, the encoded observation stored in memory is et = [e", 6, g--gee___ Pt, at-1] The model stores these observation encodings up to time / in memory: M (er
mar I SM); )....,)). where SM is the memory size. The transformer uses the memory M stored so far in the episode and encodes these visual observation embeddings with a self-attention mechanism to compute the encoded memory M.- = E'noder (M). Then, using the audio observation embeddings, a decoder network attends to all cells in Me to calculate the state representation = Decoder( (ecoder(Me,, h h-ggee gen .1)), where e, b-gen GEN Using this attention mechanism, the agent captures long-term spatio-temporee associations between the crousti-drrven goal prediction and the visual observations. Moreover, model preserves the most relevant information to reach the goal by conditioning visual-semantic embeddings stored in Mc on audio-semantic embeddings computed using current audio observation. The actor-critic network uses 8 to predict the value of the state and action distribution. Finally, the action sampler takes next-action a, from this action distribution.
Learning and Optimisation. To train the vision classification model fe" we collect a dataset using 85 MP3D houses, consisting of 82,828 images, each corresponding location and rotation angle in the SoundSpaces simulator (see Section (5). Each image has 128 x 128 resolution and 4 modalities: RGB image, depth image, object semantic image, and region semantic image. We use the binary cross-entropy loss for optimising the vision classification model and train it as a standard multi-label classifier. To train the audio classification model f% we use the SoundSpaces simulator to generate 1.5M spectrograms using different source and receiver positions, each corresponding to sounding object in one of the 85 MP3D houses. We treat detecting sounding objects as a multi-class classification problem and optimise the audio classification model using cross-entropy loss. Our vision classification model takes an RGB image as input, and the audio classification model takes I second sound clip represented as two 65 x 26 binaural spectrograms as input. We trained both vision and audio classification models using a ResNet-18 (He et al. 2015) architecture, pre-trained on ImageNet. The vision classification model predicts a score for 21 objects and 24 regions, and the audio classification model predicts a score for 21 objects (see Section 5). These models are pre-trained before and are frozen during policy optimisation. While we use MP3D, in this paper, for training these classification models, we assert that our modules may also be trained on other house

---

Under review as a conference paper at ICLR 2023
Table I: Results of baseline models and our proposed approach.
Seas Blurons, Ilrann Sonvas
Saar Ron sas, Uvan arn Survane
Metked
SR:11)
SPL.17
SSAets
DTGGi SWS(t)
5R.(1)
SPiL
SSArty
pro
swacth
31.3
332
21.3
78
98
154
18.8
119
003
14.
795
57
196
122
16.
11.
97
536
213
157
63
RM
52A
1.36
310
251
53
UNSKEN Borses, HI SOURIN
UNARE Hlccas, Usars Sox Now
Methed
SPLth
Randen:
622
07
122
5.6
0.1
148
58
Ankotin cis p107
15.
149
183
168
42
163
133
104
128
35
313
101
108
100
353
264
222
84
186
344
254
213
66
163
Table 2: Results of various ablative model configurations, across all dataset splits.
Sias Borros, BRAAR SHIINIS
Saar Bon 925, UNRO AmD STrINI
Methel
Sk:(1)
SPC. (1)
S3A(17
SWS11)
SR:11)
SPP,171
SNA
DTG
SWSSTT
53.5
11.2
Keaaoko vely G.AA"
73.2
58.7
61.1
1.5
16.4
34.3
31.8
3317
6.3
14.9
4-10oro ruck G2EAA
734
58.6
1,3
39-4
22.6
31.0
16:11
71.3
$2.6
65.9
1.78
31.0
37.8
21.1
35.5
5.3
17.8
UNSEEN HOUSES,
UNSEEN HOUSES, UNITLARO
Methed
SR(1))
(13
SNA-T)
DIGGL
SW5(T)
38((1)
SPL(1)
SSA11)
DTG
SW3(T)
32-8
21.3
18.5
10.1
18.0
15.5
19.8
8.6
10.0
5.1
31.3
31.3
9.8
15.1
36.1
9.5
1030
GR
E-5tomm Auch GENA
31.8
31.7
201
100
15.0
23.0
15.
13.7
6/3
10.1
8-SAVi -ach GENA
196
13.0
17.
8.5
1022
environments that provide semantic labels of objects and regions in houses. For location predictor floc. we use simplified ResNet- 18 architecture and train it jointly with the policy, using the same experience. We optimise the location predictor using the mean-squared error loss and update with the same frequency as the policy network. We train the policy network using the decentralised distributed proximal policy optimisation (DD-PPO) (Wijmans et al. 2020), which consists of a value network loss, policy network loss, and entropy loss to encourage exploration (Schulman et al. 2017). We adapt the two-stage training procedure proposed by Fang et al. (2019) for effectively training the vision networks (f², GFN In the first stage, the SMT policy is trained without attention by setting the memory size 8a = 1 and storing the latest observation embeddings. In the second stage, the memory size is set sM 150, and the parameters of the vision networks are frozen. The input to the vision encoder f is 64 x 64 RGB, and depth images cropped from the center. We optimise our model using Adam (Kingma & Ba 2015) with a learning rate of 2.5 x 10-4 for the policy network and 1 x 10-3 for the pre-trained audio and vision networks using Py PyTorch (Paszke et al., 2019). We train our method and the baselines for 300M steps and roll out policies for 150 steps. See Appendix C for more details.
5 EXPERIMENTS
Simulator and semantic sounds. We use SoundSpaces (Chen et al. 2020) to simulate an agent navigating in visually- and acoustically-realisiic 3D house environments. While, SoundSpaces supports two real-world environment scans (Replica (Straub et al., 2019) and Matterport3I) (MP3D) (Chang et al. (007)), we used MP3D as it provides larger number of houses and object-region semantics therein. We use the same 21 object categories as [Chen et al. (2021a) for MP3D; these object categories are visually present in the 24 regions of the 85 MP3D houses. We use the publicly- available sound clips from the experiment performed by [Chnn et al. (2021a), in which audio clips from freesound. org database were used. We generate sound by rendering the specific sound that semantically matches the object at the locations in MP3D houses. For example, the water-dropping sound will be associated with the sink in the kitchen. See Appendix A and F for more information about object/region categories and episode specification.
Rewards. The agent receives a sparse reward of +10 when it reaches the goal, a dense reward of +1 for reducing the geodesic distance to goal, and an equivalent negative reward for increasing it. To encourage trajectory efficiency, we also assign a reward of -0.01 per time step. To avoid simpler episodes, wherein is easy to reach goal (c.g., straight paths short distance), we used 2 conditions while sampling episodes: 1) the ratio of geodesic distance to euclidean distance must be greater than 1.1; 2) the geodesic distance from the start location to the goal location must be greater than 4 meters.
7

---

Under review as a conference paper at ICLR 2023
Baseline models. We compare model against several baselines: Random walk is baseline which uniformly samples one of the three navigation actions with probability of 0.33, or Stop with probability 0.01. Stop is also executed automatically by the simulator when the agent's 's location is within Im radius of the target sounding object, or if more than 500 steps are taken by the agent. AudioGoal (Chen et al. (020) is an end-to-end RL policy based on the PointGoal task (Wijmans) itt al. (020) based on a Seq2Seq mechanism which uses a GRU state encoder that leverages colour and depth images to navigate the unknown environments. In contrast to PointGoal, which uses GPS sensing to guide the agent toward its goal, this baseline uses audio spectrograms, AudioObjectGoal a Seq2Seq mechanism similar to AudioGoal, but the agent is also provided with the semantic label of the target object. SAVi (Chen et al. 2021a) is transformer-baset model that uses a goal descriptor network to predict both spatial and semantic properties of the target sounding object. is the state-of-the-are deep RI. model for the semantic audio-visual navigation task; like K-SAVEN, it uses SMT and a pre-trained audio classification model.
Evaluation metries. We follow Chen et al. (2221bl)) in reporting agent performance against the following metrics: 1) success rate (SR): 2) success rate weighted by path length (SPL); 3) success rate weighted by number of actions (SNA); 4) average distance to goal (DTG) on episode success/termination; and 5) success when silent (SWS). We assess model generalisation by evaluating our method on unheard sounding objects, across the following settings: 1) seen houses and heard sounds; 2) seen houses and unheard sounds; 3) unseen houses and heard sounds; and 4) unseen houses and unheard sounds. We randomly split the houses and sounding objects for training and testing. We use 68 seen houses, 17 unseen houses, 16 heard sounding objects, and 5 unheard sounding objects; we average the results over 1,000 episodes for each setting.
6 RESULTS
Quantitative results discussion. The performance comparison between the aforementioned baseline agents-across across Seen-Housell House/Heard-Soume Sounds (SH/HS), Seen- Heen-HouseUUnhaard Sounds (SH/US), Unseen- Hooss//Heard-Sounds (UH/HS), and Unseen- HousenheHuss/UUnhaard doundds (UH/US) conditions is sum- marised in [ Overall, in all cases except Seen-HousesHHeard Sounds, our approach outperforms all baseline methods across all metrics. More specifically, in the there is an improvement of 20.4%, 20.1%, and 16.1% in SR, in the Hinseen-HousesHHHeee -oundds, there is improvement of 19.6%, 20.4%, and 3.3% in SR, and in the Sounds case, there is an improvement of 17.9%, 20.1%, and 19.1% in SR as compared to AudioGoal, A AudioObjectGoal, and SAVi, respectively. These results indicate that our agent could leverage the reasoning capability using GENs with the memory-based attention mechanism using SMT and generalise to the novel sounding objects. In the Heee--Hoses/HHaard-SSee has experienced the sounding objects during training, and it is more critical to reason about the visual cues than the sound semantics to succeed, our approach performs comparable to SAVi. We emphasize that SAVi also has a vision encoder and scene memory to store encoded vision observations like our approach resulting in comparable performance the Been-Houees/Headd Sounds case with our approach and making it challenging improve SAVi with significant margins. Additionally, due to fair comparison, we strictly trained ours and SAVi's models for 300M steps, for both stages.
case,
case, where the agent
A blations. We provide ablation results in Table 2 to evaluate our system's 's key components. Overall, all ablative configurations of our approach perform better than SAVi in all metrics. We note that SAVi also leverages the audio classification model and SMT policy with scene memory, similar to our (DRR) modules helps to improve the agent's 's performance further. full model performs best in all metrics, except for the Howses/Heard-Sounne
approach, and as shown in Table 2] adding GEN and direct-to-reverbbet
(SH/HS) case. These results indicate that our agent can indeed associate visual cues with sound semantics and use the prior knowledge-driven reasoning capability from both GENs to generalise to novel sounds and novel environments to navigate efficiently. Moreover, in the case, only-GEN' outperforms other models in most metrics indicating that GEN" has a comparatively more sigmificant impact on our model's performance. However, relying exclusively on only-GEN"" would not enable the agent to navigate the novel sounds effectively. We evaluate the impact of using the estimated DRR as weight for location belief update, by comparing K-SAVEN- -full model to K-SAVEN -both GENs + 8,, the former using 5, = 0.5, Vt (exponential average) and the latter using the estimated DRR as 8, (dynamic average). The fill model caunicall--sstiaate weighting
Senn-Huuss/Heardd-Sooee
achieves better performance by a margin compared to the use of a factor dg Our intuition is that DRR-estimation as auxiliary task for the eecation-predictor induces better 1, estimation, DRR acts a proxy to the estimation of the distance to the source. However,
8

---

Under review as a conference paper at ICLR 2023
Hpre
MIGHIINN SPENIA, 58H10
Visualisation of navigation trajectories. Row-wise, show trajectories and egocentric views for K- K-SAVEN and SAVi two episodes from the UH/US (a), (d): beginning of the episode, with the starting pose and view the agent and the target sounding object. (b), (e): K-SAVEN's visual results along with SPL/SR metrics. (c), (f): SAVI's visual results along with SPL/SR.
Figure 3:
the estimated 8, is not reliable enough to provide consistent weighting scheme across the episode, thus an exponential average with 6, = 0.5 provides better overall performance.
In heard sounds cases, the agent is familiar with sounds, so vision reasoning is more important. Both only-GENN and both-GENs have GEN"; thus, they both perform better than only-G with only-GEN" performing slightly better than both-GENNs,
only-GEN" forces the agent to reason
only based on vision. For example, in the SH/HS case, the success rate (SR) of only-GEN' is 64.4, and the SR of only-GEN and both-GEN is 73.2 and 73.0, respectively. In the UH/HS case, the SR of only-GEN" is 31.1, and the SR of oNly-GENN and both-GEN is 32.8 and 31.9, respectively. Similarly, in unheard sounds cases, the agent is unfamiliar with sounds, so audio reasoning is more important. Both only-GF.N' and both-GF Ns have only-GEN thus they both perform better than only-GFNNy with only-GFEN performing slightly better than both-GF.Ns as only-G forces the agent to reason only based on audio. For example, in the SH/US case, the SR of only-GEN is 29.7, and the SR of only-GEN and both-GENs is 31.7 and 30.5, respectively. In the UH/US case, the SR of only-GEN is 21.2, and the SR of only-GE and both-GEN is 23.3 and 22.9, respectively. Furthermore, it is crucial to effectively combine the reasoning capabilities introduced by GENs, s, location prediction, and classification models. Our full model performs better in most ablative cases, indicating that our agent could leverage the reasoning capability using GENs with the memory-based attention mechanism from SMT and generalise to heard and unheard sounds.
Qualitative results discussion. We illustrate how our approach qualitatively improves navigation performance, in Fig. 3 we compare K- SAVEN and SAVi trajectories on the same episodes, each episode shown row-wise, alongside the episode's ' corresponding expert trajectory, shown in green. The episodes were obtained from the UH/US set. From these examples, we observe that K-SAVEN reaches the goal location in fewer steps, whereas SAVi tends to take more steps and roam throughout the episodes. The latter supported by the SNA and SPL metrics Table [[] where K-SAVEN achieves higher success in terms of path length (SPL.) and number of actions (SNA).
DISCUSSION AND CONCLUSION
We introduce a framework for leveraging cnowledge-enhaneeee scene priors, in the form of object and region semanties, for the semantic audio-visual navigation task. Notably, we show performance improvements over strong baselines in multiple unseen contexts, particularly in conditions where the agent needed to find novel sounding objects. We also provide knowledge graph for training models, a curated visual dataset, and a new task definition-all guided towards developing and assessing model generalisation performance in unseen environments. We recognise future improvements of our work. e.g., in the selection of the knowledge resource used for encouraging scene priors in the semantic audio-visus navigation task. We would consider constructing knowledge resource that characterises sound-object relations (i.e., with descriptions of the sound that is generated by various objects), more befitting of pre-training the acoustic GEN stream. Furthermore, we can consider using scene priors as additive modules on frameworks in other tasks, particularly within the family of embodied multimodal planning. Finally, sounds are not merely a product of individual objects, but of different types of actions and interactions (e.g., sitting, dropping, playing music) that often involve multiple agents and/or objects. Therefore, in future work, we plan to incorporate such semantic knowledge about sounds, objects, actions, and interactions in our knowledge graphs further improve performance.

---

Under review as a conference paper at ICLR 2023
8 ETHICS STATEMENT
Enabling agents to leverage previous experience and background knowledge, through scene priors, and to better identify goal locations through importance in human-machhe machine interaction and robot task task-following scenarios. Indeed, systems that are endowed with these capabiliitiss are better- for such applications as environmental health monitoring, acoustic anomaly detection, navigation with resource constraints, and others. However, training scene priors using data that is not general enough for the deployment scenarios could bring bias to the agent's predictions.
Ratio estimation are of paramount
9 REPRODUCIBILITY STATEMENT
We encourage reproducibility and are committed to enabling broad scientific use of our work, upon acceptance. We will provide code, demonstration videos, our novel knowledge graphs, and our dataset generation scripts-al at an anonymised hyperlink, directed to the Reviewers and Areas Chairs, once the submission forums are opened for all submitted papers (as suggested the ICLR 2023 Author Guide). We provide model implementation details in Section [ particularly in the paragraph on Learning and Optimisation. In Appendix C and Table[3] we further detail the important hyperparameters used in our approach and outline the computing hardware that we used for training and evaluation.
REFERENCES
Peter Anderson, Qi Wu, Damien Teney, Jake Brucc, Mark Johnson, Niko Sünderhauf, Ian D. Reid, Stephen Gould, and Anton van den Hengel. Vision-and-language navigation: Interpreting visually- grounded navigation instructions in real environments. In 2018 IEEE Conference on Computer Vi- sion and Puttern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 3674- 3683. Computer Vision Foundation IEEE Computer Society, 2018. doi: 10.1109/CVPR.201 00387. URL http:://peeacceess. thectf wwweeccwccoo/cooooooooeooeeoeeoe content cvpr 2018 2018//tmll /Anderson Vision- and www.-anggggggeaiiga.....o Btt.apaignggaaairaiooooooooeaaae CVPR wat__paperrrree
Dhruv Batra, Aaron Gokaslan, Aniruddha Kembhavi, Oleksandr Maksymets, Roozbeh Mottaghi, Manolis Savva, Alexander Toshev, and Erik Wijmans. ObjectNav Revisited: On Evaluation of Emhodied Agents Navigating Objects. In arXiv:2006. 13 13171, 2020.
John F. Canny. The Complexity of Robot Motion Planning. MIT Press, Cambridge, MA, USA, 1988. ISBN 0262031361.
Angel Chang, Angela Dai, Thomas A. Funkhouser, Maciej Halber, Matthias Nießner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from RGB-D data indoor environments. In
2017 International Conference on 3D Vision, 3DV 2017, Qingdao, China, October 10-12, 2017, pp. 667-676. URL https::///aat org wwwp////..0......... 1109/
IEEE Computer Society, 2017. doi: 00.1109/3Y.2011..000
3DV 2017.00081 00081
Devendra Singh Chaplot, Dhiraj Gandhi, Abhinav Gupta, and Russ R. Salakhutdinov. Object goal naviga- tion using goal-oriented semantic exploration. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Had- sell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Sys- tems 33: Annual Conference Neural Information Processing Systems 2020, NewrIPS 2020, Decem- ber 6-1 2020, virtual, 2020a. URL /tt.pppp//prccecingg. wwwp:eeewiinnn...nnnneeeeeeeeeee www.nnuuriiic.. wwwp:rpppp.eppppe//e/e.. 20tpappe//222/hshhh 22/5c126811
-wwp:a222-bbbttttt Abst httm
Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhutdinov. Learning explore using active neural slam. arXiv preprint arXiv:2004.05555 2020b.
Devendra Singh Chaplot, Ruslan Salakhutdinov, Abhinav Gupta, and Saurabh Gupta. Neural topological SLAM for visual navigation. In 2020 IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition. CVPR 2020, Seattle WA, USA, June 13-19, 2020, pp. 12872- 12881. Computer Vision Foundation IEEE, 2020c.
doi: 10.11099CVP44260.22200000 poppenaccess.c. thecff com/ www/ content, CVPR 2t20/hhmm//hpppll/ccccm Nte Topologi looooogica ca laa_aasAAA LLL.__AA__._______ Laan_foo_Viauuu Navi gat onnciin__ccce ww__oop_______________________e_ [a____CV__2000022 20a00_00aerrr_____________a_ paper.pppp....... ht.l
URL. httpttss://paaacccesssss
Changan Chen, Unnat Jain, Carl Schissler, Sebastia Vicene Amengual Gari, Ziad AI-Halah, Vamsi Krishna Ithapu, Philip Robinson, and Kristen Grauman. Soundspaces: Audio-visual navigation in 3d environments. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan Michael Frahm (eds.), Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 28, 2020, Proceedings, Part VI, volume 12351 of Lecture Notes in Computer Science, pp. 17-36. Springer, 2020. doi: 10. 11100009778--000-5585399 2. URL 030
https: //dei wwgg 10. 100 / 030- 58539-6
10

---

Under review as a conference paper at ICLR 2023
Changan Chen, Ziad Al-Halah, and Kristen Grauman. Semantic audio-visual navigation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 15525, Computer Vision Foundation / IEEE, 2021a. URL
theevf
Com httprccnc/cve/////mmm////mm Bat.aaaooo..aaaa........ooooeae
Audio-Visua ood.ooo
2021 wwpprappppp..... html
Changan Chen, Sagnik Majumder, Ziad Al-Halah, Ruohan Gao, Santhosh Kumar Ramakrishnan, and Kristen Grauman. Learning to set waypoints for audio-visual navigation. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021b. URL
https://cpenrewiice net/ www.oooddoooelli... PRPPAPd
Heming Du, Xin Yu, and L. Zheng. Learning object relation graph and tentative policy for visual navigation.
ArXiv, abs/2007.11018, 2020.
Kuan Fang, Alexander Toshev, Li Fei-Fei, and Silvio Savarese. Scene memory transformer for embodied
agents in long-horizon tasks. In IEFF Conference on Computer Vision and Pottern Recognition, CVPR 20/9, Long Reach, CA, USA, June 16-20, 2019, pp. 538-547. Computer Vision Foundation / TEEF, 2019. doi:
10.1 10.1109/CVPR.200 2019P.0.009 URL pttp::/pppaaccessssss.s. http://pppccccccceeeeeeeeeee.e.. thecv. com/ content. CVPR 2019/
/
httm//Fan Scene Memory Transformer for Embodled Agents in Long-Horize
Tasks CVPR 2019 paper.html
Jonathan Francis, Nariaki Kitamura, Felix Labelle, Xiaopeng Lu, Ingrid Navarro, and Jean Oh. Core challenges in embodied vision-language planning. Journal of Artificial Intelligence Research, 74:459-515, 2022.
Chuang Gan, Yiwei Zhang, Jiajun Wu, Boqing Gong, and Joshua B Tenenbaum. Look, listen, and act: Towards audio-visual embodied navigation. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp.
IEEE, 2020.
Daniel Gordon, Dieter Fox, and Ali Farhadi. What should now? marrying reinforcement learning and symbolic planning. CoRR, htss/190..01499 2019. URL httt::://axxiii org. abs/ 1wtp//9911..1922
Kaiming He, Xiangyu Zhang. Shaoqing Ren, Jian Sun. Deep residual learning for image recognition. CoRR. abs/1551.033385 2015. URI. htthhttpp//axxiis wwg./bs/ wtt.a///1111.3333
Muhammad Zubair Irshad, Chi Ma, and Zsolt Kira. Hierarchical cross-mmoll agent for robotics vision-and- 10674
language navigation. CoRR, 0ahh2104410674 2021.
Lydia E. Kavraki, Petr Svestka, -lan-Claude Latombe, and Mark H. Overmars. Probabilistic roadmaps for path planning nigh-dimensional configuration spaces. IEEE Trans. Robotics Autom., 12(4):566-58), 1996. doi: 10.1109/70.50841
http:/ttss///oo.orrr/oooo 1twp://10..11110.0001000 508439
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. UBLL/tttpp//aaxxi..or/////
6980
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In
International Conference on Learning Representationss ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https:/ http://ppppeeeeeeeeeeeeeeeeoeeep net Forum?
httpps://ppeneeriew..tt
Ed=SJU4ayygi
C. Knapp and G. Carter. The generalized correlation method for estimation of time delay. IEEE Transactions on Acoustics, Speech, and Signal Processing, 24(4):320-327, 1976. doi: 10. 1091////SS..1997. 1162199666666600..11 166830..
Sven Koenig and Maxim Likhachev. Real-time adaptive a*. In Hidcyuki Nakashima, Michael Wellman, Gerhard Weiss, and Peter Stone (cds.), 5th International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS 2006), Hakodate, Japan, May 8-12, 2006, pp. 281 288, ACM, 2006. doi: 10.1145/1 160633. 160682.
wrw.:/i..rrrgg/// 10. 1145/1 60633. 60682
Eric Kolve, Roozbeh Mottaghi, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. AI2-THOR: an interactive 3d environment for visual AL. CoRR, abs/1712.055774, 2017. URL http: /ttp:::///xxiii org/ abs/
[712.05474
Jacob Krantz, Erik Wijmans, Arjun Majundar, Dhruv Batra, and Stefan Lee. Beyond the nav-graph: Vision and language navigation in continuous environments. In European Conference Computer Vision (ECCV), 2020.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Fei-Fei Li. Visual genome: Connecting language and vision using crowdsourced dense image annotations, 2016.
11

---

Under review as a conference paper at ICLR 2023
Steven M. Lavalle, James J. Kuffnner and Jr. Rapidly-exploring random trees: Progress and prospects. In Algorithmic and Computational Robotics: New Directions, pp. 293-308, 2000.
Yunlian Lv, Ning Xie. Yimin Shi, Zijiao Wang, and Heng Tao Shen. Improving target-driven visual navigation with attention on 3d spatial relationships, 2020.
Kaixin Ma, Jonathan Francis. Quanyang Lu, Eric Nyberg. and Alessandro Oltramari. Towards generalizable neuro-symbolic systems for commonsensee question answering. Proceedings of the First Workshop on Commonsense Inference in Natural Lunguage Processing, pp. 22-32. 2019.
Kaixin Ma, Filip Ilievski, Jonathan Francis, Yonatan Bisk, Eric Nyberg, and Alessandro Oltramari. Knowledge- driven data construction for zero-shot evaluation in commonsense question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 13507-13515, 2021.
Mahdi Kazemi Moghaddam, Qi Wu, Ehsan Abbasnejad, and Javen Qinfeng Shi. Optimistic agent: Accurate graph-based value estimation for more successful visual navigation, 2020.
Alessandro Oltramari, Jonathan Francis, Cory Henson, Kaixin Ma, and Ruwan Wickramarachchi. Neuro- symbolic architectures for context understanding, Knowledge Graphs for eXplainable Artificial Intelligence: Foundations, Applications and Challenges, pp. 143-160. IOS Press, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf. Edward Yang. Zach De Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style. perg--erformmncee deep learning library. CoRR, abs/1912.01703, 2019.
org abs / 1912.11111 01703
Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods Natural Language Processing (EMNLP), pp. 1532-1543, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.3115/vl D14-1162. URL /tttps://aclrrtn
orq/D14-1162
Yiding Qiu. Anwesan Pal, and Henrik I. Christensen. Learning hierarchical relationships for object-goal navigation, 2020.
Homagni Saha, Fateme Fotouhif, Qisai Liu, and Soumik Sarkar. A modular vision language navigation and manipulation framework for long horizon compositional tasks in indoor environment. CoRR. abs/2101.07891, 2021. httpt:
org www.//////////o.ooe//oooeeeeeeee / CC881
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017. URL /att:://azxix org/ abs abs/1707.0634
Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proc. of AAAL, AAAI' 17. pp. 4444 4451, 2017.
Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, Anton Clarkson, Mingfei Yan, Brian Budge, Yajie Yan, Xiaqing Pan, June Yon. Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales. Tyler Gillingham, Elias Mueggler, Luis Pesqueira, Manolis Savva, Dhruv Batra, Hauke M. Strasdat, Renzo De Nardi, Michael Goesele, Steven Lovegrove, and Richard A. Newcombe. The replica dataset: A digital replica of indoor spaces. CORR, abs/1906.05797, 2019. URL http:htt:///aat /ttp:////axvv..// wtw.g//////////////////////a wttpb/////9...00 05797
Joshua B Tenenbaum, Vin De Silva, and John C Langford. A global geometric framework for nonlinear dimensionality reduction. science, 23015500):221--2500 2323, 2000.
Varun Kumar Vijay, Abhinav Ganesh, Hanlin Tang. and Arjun Bansal, Generalization to novel objects using prior relational knowledge, 2019.
Erik Wijmans. Abhishek Kadian, Ari Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. DD-PPO: learning near perrect pointgoal navigators from 2.5 billion frames. In 8th International Con- ference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. 2020. httpst wwwp:://eeeeeeeeeeeeeeeeee
d= QX8C4YPT
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chenggi Zhang, and Philip S. Yu. A comprehensive survey on graph neural networks. IEEE Transactions Neural Networks and Learning Systems, 32(1):4 24, Jan 2021. ISSN 2162-2388. doi: 10.1 20.100/m...2222229733335
TNNIS. 2,000. 2 2978386
Wei Yang, X. Wang. Ali Farhadi. Abhinav Gupta, and Roozbeh Mottaghi. Visual semantic navigation using scene priors. In International Conference on Learning Representations (ICLR), New Orleans, LAA, USA, 2019. OpenReview.net. URL httppsss//opeeeeviiieeeee wwt. wttp://iwwwweeeeeeoueeuueeeeeeee
12

---

Under review as a conference paper at ICLR 2023
A ADDITIONAL DETAILS: SIMULATOR, OBJECTS, AND REGIONS
We use SoundSpaces (Chen 2020) simulate an agent navigating in visually- and acoustically-raalin 3D house environments. The simulator renders sounds at any pair of source (sounding object) and receiver (agent) locations on a uniform grid of nodes spaced by 1 meter. While, SoundSpaces supports two real-world environment scans (Replica (Straub et al. 2019) and Matterport3D (Chang et al. 2017 we used Matterport3D as it provides larger number of houses and object-region semantics therein. We use the same 21 object categories as Chen et al. (221aa) for MatterportäD: chair, table, picture, cabinet, cushion, sofa, bed, chest-of- drawers, plant, sink, toilet, stool, towel, tv monitor, shower, bathrub, counter, fireplace, gym equipment, seating. and clothes. These object categories are visually present in the 24 regions (balcony, bathroom, bedroom, closet, dining room, entryway/foyer/lobby, yumilyrom//ounge, hallway, junk, kitchen, laundryroon/mudronm living office, other room, porch/terrae/ddek, rec/game, spa/sauna, toilet, unllityooo//oolrrom and wwrkouu/gym/exerctiin of the 85 Matterport3D houses. use the publicly available sound clips from the experiment performed by Chen et al. (2021a), in which audio clips from Free s sound. org database were used. We generate sound by rendering the specific sound that semantically matches the object at the locations in Matterport3D houses. For example, the dropp-g sound will be associated with the sink in the kitchen.
room, lounge, meetinggoom/coffrreeeeee
B ADDITIONAL DETAILS: KNOWLEDGE GRAPH
Knowledge graph construction. Our knowledge graph captures object-to-object, object-to-region, and region- to-region relations. To compute these relations, we use the semantic labels of objects and regions in Matterpont3.. The heuristic we use to find these relations is frequency- based: the main idea is to connect an object with another object if they frequently exist across various regions. Similarly, they both have similar objects placed them. The resultant knowledge graphs are provided Tables Jadd 4 which can be represented as adjacency matrices, with an indicator of to characterise a co-occurrence edge between objects, other objects, regions, and other regions. For example, 2 objects (chair and chest-of theestof-ddawers are connected because they frequently found in the bedroom region. These 2 objects are also frequently found in other regions such living room and office. Thus, we make region-to-region connections by connecting the bedroom to the living room and the office because these regions also frequently contain the same connected objects (chair and chest-of- -o-dwwers) as the bedroom region.
connect region with another region if
Knowledge graph representation. F illustrates the GloVe embedding space and figure 4 represents the object-region adjacency matrix, both as two-dimensional projections. We reduced the dimension of the Glo |SOMAP/ennnbaun embeddings. for each object and region, into by using ISOMAP Tenenbaum et al. (2000) (shown in Figure Fb..
We also reduced the dimension of vector in the adjacency matrix that encodes the relationship of each object and region with other objects and regions (shown in Figure )). As shown in Figure regions and objects are clustered together, and objects found together in houses, such as tables and chairs, are close together.
Alternatielly these graphs can be represented in the same format as existing large-scale commonsense knowledge resources, such as ConceptNet (Speer et al. (2017): : i.e., collection of head h / relation r tail triples of the form (h, r. t), with the ConcepiNet LocatedNes Locat in relation for each (h, (=(object, region) instance pair, and with the LocatedNess relation for each (h, ()errgion, region) instance pair pair-with saliency weights, based on frequency. Some instances can be further expanded with additional relations, such UsedFor: or, derived from activity annotations in the region labels. The following examples are taken from the first and tenth rows of Table 3
relation for each (h, t)=(object, object) instance pair, the
(bathtub, (bathtub, LocatedNear, sink) (bathtub, At loocctii i on, bathroom)
towel)
(gym equipment, UsedFor, workou() (gym equipment, At Location, gym) (gym_ cerm_eeeuimettt..m UsedFor, exercise)
C ADDITIONAL DETAILS: MODEL. IMPLEMENTATION
Hyperparameters. For all experiments, implemented models using the Torch deep learning library, version 1.1..0. Shows the output size different modules in SAVi et al. (021a) and different configurations used in the ablation studies of K- SAVEN. In Table "M "M Size" refers to the size of the bision-based knowledge vector stored in memory M. and "Belief Size" refers to the size of the audio based knowledge vector used to attend to the encoded memory M.. We use ReI.U as the activation function in both GENss In our experiments, we used D 300 and F = 64, respectively, as the input and output feature dimensions in the encoder networks.
13

---

Under review as a conference paper at ICLR 2023
(4) Gle emibedlings ID
(b) Adjacency maiii in 2D
Figure 4: Lower-dimensional projections. illustrating object-region similarity. (a) GloVe embeddings for each object and region into 2D space. (b) Adjacency matrix that encodes the relationship between objects and regions in 2D space. Best viewed in magnification.
Action encoder takes action represented in onc-hot vector as input and projects into embedding of size 16. Note that Action encoder is omitted from Fig. (a) for simplicity. The K-SAVEN -only GEN", K-SAVEN -only same as the S-SAVEN -full model except the location rovveebbaa ratio (DRR) 6, The R-SAVEN -both GENs + is also same as the K- R-SAVEN -full model except the ó; is not fixed to 0.5 and used to estimate the sounding object's location. More specifically, the location is estimate by 4 = (L,, 4-1, Apr, 6.) = 6,4, + (1 8.) Sp:). where 6, is dynamically updated by the location predictor.
GEN" and K- S-SAVEN -both GENs s contigurations predictor not trained to predict
Computing hardware. For rendering the simulator and performing local agent verification and analysis, we used single GPU machine, with the following CPU specifications: Intel(R) Core(TM) i5-4690K CPU @ 3.50GHz2; 1 CPU, 4 physical cores per CPU, total of 4 logical CPU units. The machine includes a single GeForce GTX TITAN X GPU, with 12.2GB GPU memory. For generating multi-instance experimental results, we used a dual-GPU machine, with the following CPU specifications Intel(R) Core(TM) 19-9920X CPU @ 3,50GHz; I CPU, 12 physical cores per CPU, total 24 logical CPU units. The machine includes two NVIDIA Titan RTX GPUs, each with 24GB GPU memory.
D ADDITIONAL DETAILS: VISION DATASET
To train the vision classification model fc. which given an RGB image predicts score for objects and regions. we collect vision dataset using the SoundSpaces simulator as described in Section 4 Initially, we collected 82,828 images across 85 Matterport3D houses, which is the maximum number of images possible there total of 20,707 nodes and rotation angles in SoundSpaces. However, we faced the following challenges with the scans and semantic labelling in the Matterport3D: 1) objects are not clearly visible because of glitches in scans (see RGB image in Figure 5p) 2) object and region semantic labels are improper (see object and region semantic images in Figure and [ 3) the way some objects are placed is not common due the luxurious nature some houses Matterport3 (e.g.. scene ID aay Hof, there is big garage, which has a car, a fridge, a table, and chairs; moreover, there is big pool table in the game room, which is not commonly found in houses), and some objects are not semantically placed (see Figure (
To address these challenges, we filtered some images and only used 45,233 images to train our vision classification model fe We use the following filtration criteria: Filter are black (zero value): 2) There are 42 objects in Matterport3D, and we are interested in only 21 objects in our experiments, so we filter out an image if it does not contain any of those 21 objects; 3) Filter out an image if the most frequent object is taking less than 3% of the total pixels in the image; 4) Filter some of the semantic labels of an image based on a threshold (8.18 for object and (l2 for region). First, for each semantic label in the image, we compute the ratio of its proportion of the pixels to the proportion of the most frequent semantic label in that
image in which 75% of the pixels
14

---

Under review as a conference paper at ICLR 2023
Table 3: Relational knowledge graph for spatial object-object interactions
Sounding objects (21)
Objects (21)
Regions (22)
pictone, cabinen counter table, plani
bethnoom
clvis pictom, table, sink, vrating andtian, cabinet, cheat of drnwers shower. plant, counter, IN v__mootorr towel
qpakaums, jonk, holnoom
cabinet
clottes, chair, towel, seating, slower, solet, picture, tbll, sink, custion.
spalauna, bethroom, familymonn/honge. Dring room. entry (oyns//lly, kiiches, offixt allllynammhssnem ofher roum, hall- way, mendtnnentonio clost
plani, sufa, coumerr bed, ches of slasees builruh, fy.memice slarl,
Eaqplase
shair
gyn zyncegpigment picture seating, cudimss table phant, cubinet, sink, chest drawers, couneer sofa, montee, stoel,
spaiauns, fanilyycenohonge living jan, entywayllioyrllde reclgime
onlittyommmyslleeen bedmam, bakeny, lmange ponishemottleck hillway. dining
Emephase
drnwwerss
chair, picuree custion, table, bed, ty my_monnirr, cabinet
odhee. bedroome
clothes
cabinst, picture
cliset
cluin, triks, picture, euchion. bed. Isble, batloch phat,
Iehhnom, junk, kitckes, silltgeomohbsioomm, bandggneermmuurrmmm
cushion
chair. pictue. seating, table, sink, plant, cabizet. shower.
spaisuun, familyyoomobongg, living room, Jank, entro anrrowyytoyerlebty
chest of drovens bed, soto, coumer, towel, by. monnor, stool, ferplace
office. unlingreemecolrom bedmomm other room, recigame, balcony. lougg, paciitermae/eck
Gieplace
caalion, lable, chair, pictuer, sufa, plant stndl cabinet
living noon
p3m equipment
pictire,
pocture
cloches, gym equipment, roter, chair seating. shower, cushion, towel, cabinet, tblk, sisk, chest di_deneess bed, counter, plani soGa bubtud. bv tv_mooite, stoel, Areplace
spalauna, bathroom, familrrem/loungg, living mom, junk, entry- way vaySoyeeralbbs, kitcher, offic, utlityyonthosioon bedroms, other room. reolgame. lourge, halltay, lunddyyom/mmiion closet, diting room, pertiggoombboeeeeeeeeeo m. tolle, wodionalgmmeeeece
piant
chair, picture, sink, tneel tahic cudtion slunee tuils, eastngg calinet, sifa, countes bal, buthtud, musiioo, soail, lixelaxx
spulonnna lalizeem, familyrenddomng, living mem. junk, looyeshbbyy redgamme halemy, purtikemakkiak
seating
chiir. sink. picture, plant, cabinee stower. bed, coshion,
spolsuuaa eerry corywarroyeerkoooy... other room
stower
chair. sink, towel, table. tollet, scoting. cabitet, picture, counter. bed. plan. battmat cestion
sposauna, bothroom
kisk
cabinet, chair, towel, shower, willet. scuting, picture, table, counter. caslion, bed, i mumite, platt, buhhus, stud
spabatuna ballewom, jusk, kitchen, oiiliyroomhhodoomm. luunlyy- mon/aandoon
chair. pictur table, plane,
montor, fieplace
familyyoom living mom, recigame. balcory. pondilemscriaac fumilyromsshuungs living room, kichten
lounge
cushion, shalr, pictue, tuble, cubinet, counter, sofa, plant sink. by ty_mooin,, fireplace
chair, soeel, pictone seating chowee teelst, castion, sink, cabind plant, bai, chesl chawere counerr sola, buchtuss by munitur, sindl, fine- place
spulonuss belhnon. Gamilymonshanng: living roum, maylkyenhubbby, kitchen olles, atliiyoom Iairrum, other mum, roc/game bulony, lounge, puctviemacslledd hallway, dining mocinnnemioonfeeeooowe
sunwer,
picture, connerr plane
bathnom,
lowel
soilet, chair, sink, ubik, shower seating, cubinet, picture, counter, bed.
spobauss, bathoom, toilet
plan. bulbuus, cashion
Iv. moniter
clair, tible, ouhion, sink, plant, eofa, cobinet, comnter bul. choal of deseers slnd
6mmigrommhonng,, jank, efficr
image. Then, semantic labels with ratios less than the threshold filtered out. 5) For pre-training the vision module, we selected a class cutoff threshold of 0.5, based on grid search for this hyperparameter, shown in Table
E PRE-TRAINED MODELS' PERFORMANCE
Audio classification model f:: Given audio signal, our audio classification model. få, predicts the sounding object. We generated 1.5M spectrograms using different source and receiver positions, cach corresponding to a sounding object in one of the 85 Matterport3D houses. We used 1,201,147 spectrograms for training and 300.317 spectrograms for testing the audio classification model. We accuracy A sornect predictione % as the metric evaluate the audio classification performance. We receive 97.3% accuracy when evaluate the audio classification model on the test set.
total prodictions
Vision classification model f": Given an image, our vision classification model f" predicts objects and regions in that image. Out of 45,233 images collected across 85 Matterport3D houses (see Appendix DD. we used 36,153 images for training and 9,080 images for testing the vision classification model. We used two metrics to evaluate the performance of the vision classification model. First, we consider exact match ratio, EMR = [T = أ()) where n is number of examples, y and g(s) are the true and predicted labels of the vth example, respectively. The EMR calculates the ratio of examples for which the prediction is identical to its ground truth class labels, over all examples. The EMR is always in the range of 0.0-1.0. high value of the EMR indicates high classification performance. The second metric is the hamming loss
HT + Σ. E [/,) # "),") where n is number of examples, L is number of classes, and y
15

---

Under review as a conference paper at ICLR 2023
Table 4: Relational knowledge graph for spatial region-region interactions
Regions (22)
Objects (21)
Other regions (22)
baloony
clsir, plant, cushion,
Familyroom''ung lungs, melgame pechhummoltick
battnoom
towel, sink, shower, pictore, cabiner toilet, countrr bottrah, tahin plant
quhauns
bodroous
custion, peture, chest of Jrowers, bed, chaic table
spaisausa offic
pictum
hallway, enteyonpliieehhbyy living family-
membbangs kitcenn handkymnnohmndrenn spetsuus, yillemnanllleennem
dizting zoom
chair. picture, table
bedroom. halbaay, earyway-tyeeteeey lring mom. family
room/lounge othice kitchen lounge re/ggmet spa/samm,, other room. in mnlinmmembooirrm meoetgroommonneeeeeee
picturr, claiir tabie. plant, cabinet, cuslion, sealing
spaisune
caslion pictuse, ubli, plun, sofa, cabines, Wv_nooinor, picture, cabinet, cizir, table
living
halbwy
enysayyhyyereeeh livin room, fomitooorsloungs office, kitcher,
quhaurs, other nom, miliynommhhoooen
junk
picturr, chair, sink, cusiiin, counter, plant, bed, ty_moniirr
speisuut
kinchen
cahinet, chair, counter stool, pictier,
wiliymemmooiremm
bandgroommmdeee
calinet, counter, pectur, eink
bothroom kitchen unlityroemhueerom
living room
castion, tbble chair, picturs, sufa, plun, stool, Ereplace, cabinet
fumilyeonm//ounge
lounge
cliir, picture tabie, cuibiou,
Gamilyyonn roolame
meeting merragoemneoorreemmmmii pictume, tahie
bedroam, hallway, dining room, entryayyleymlette living rom, fimllconddungg office, kitchen loings molgane, quaisum, other mom, miliymorniieemmon
office
Chair, table. pictare. tv monioor chest, of drawers, cabinet, cushion
fumilyromm/bunge
spring, clein, picture calisus
amyyoopfnyartkbhyy spa/sunta
cheir, plant, tabl,, cualion, sola
Ioloany living romm fanilymonthangg, roofgame
reclgame
Chiir, table, custion, picture, sufa, plant
living romm fanilyynom'/ougee
salie. chall, sink, cabitee picture, bed, towel. coslion
buthroom, cutyeay/oyenteebyy
Iolks
willi, pictur, lose
beahnocn
cabinet, chair, picture table, couner, custion, sink
kichen, spatsuna
wortsoutigrmieeeniee
aym equipmenr, picture, chair
bedroom. hallsay, dizing eutrywaoooyyesooet living
fumilycondduungg. offics, kitcen, spaisuus,
willeyoasiilllnonn metinggomm aectiggoombbassteccioo
Table 5: The output size of different modules in SAVi Chen et al. (2021)) and different configuration
used in the ablation studies of -SSVVEN.
Method
Vision Encoder (22
GAWN" Audio Encoder (12) je
Action Encoder
Size
Location C
Sizee
SAM
2021a
210
210
BAVAE -an'y
21
2-BANEN
274
21
R-SAVEN
274
21
K-SAVEN mode!
274
21
and bi are the true and predicted labels the ith example and jih class, respectively. The HL measures the average number of false positives and false negatives. For given class, a low value of the hamming loss indicates that the class is casy to recognize, while high value shows the opposite. We receive EMR of 0.48 for objects and 0.68 for regions when we evaluate the vision classification model on the test set. It is challenging get high score on EMR because it does not account for partially correct labels. Comparatively, classifying regions is easicr than classifying objects, as indicated by higher EMR. Table 7 shows the hamming loss results. The average HL for all objects is 0.011089785 and for all regions is 0.01 76211 16. Lower average HI. value for regions compared to that of objects indicate classifying regions is easier than objects, as indicated by EMR results.
F ADDITIONAL DETAILS: EXPERIMENTS
Episode specification and success criteria. An episode of semantic audio-visual embodied navigation task defined by a house, start location, and rotation angle of the agent, goal location, sounding object, and duration of the audio event. In each episode, the start location and rotation of the agent randomly selected. For selecting the sounding object, an instance of an object category in the house is also chosen randomly. We define a set of viewpoints within I meter of the object's boundary from where the object is visible to the agent. When the agent executes the Stop action at any of these viewpoints, the episode will be successfully completed. We sample 367,155 episodes for training and 1000 episodes for each of the testing settings. To select the duration of
16

---

Under review as a conference paper at ICLR 2023
RGB
Otgest
(c) Regicu vemualie seagee
Dasclouee office
chaz
rendow
celling
40: mec
well
Domol
sculing
eljects
Figure 5: Examples of issues with the scans and semantic labeling in the Matterport3D. The images in the first correspond to the ID aayBAfsNo d and node: (93, 270), images in the second correspond scene ID 2nGKARJN3HM
node: (8, 180), and images the third row correspond (X and node: (12, 90). (a) shows RGB image examples in which the objects are not clearly visible due to glitches in the scan. (b) shows the semantic labels, and (c) shows the semantic labels; however, these objects and regions not clearly visible in the corresponding RGB image.
scene ID
583 dr awables
culled
Pinhole canera
59
duration
GPU duration ion: 387
Ver tex fetch rati e: sa
Primi Lives el clppped
Figure 6: Examples of unusual semantically-plant objects in scene ID 2n9KARJN of Matterport3D. In the image show, bathtub placed the living room, and chairs are kept top table, which is unusual placement these objects.
the audio event, first, we sample a value from normal distribution with a mean of 15 and a standard deviation of 9, and then we clip this value to limit the duration between 5 and 500 seconds.
Action space and sensors. There 4 actions the agent's action space: MoveForward, TurnLeft, TurnRight, Stop. MoveForward changes the agent's current location to the node front only that node reachable without collision. Stop can be used by the agent to report sounding objects and terminal the episode. The TurnLeft, TurnRight, and Stop actions can always be executed successfully. There are 4 sensory inputs: egocentric binaural sound (two-channel audio waveforms), RGB image, depth image, and the agent's current pose relative the starting pose of the episode. The resolutions of the RGB and depth images are 128 128.
17

---

Under review as a conference paper at ICLR 2023
Table 6: Class threshold hyperparameter search, for visual module pre-training. "Shuffle" refers shuffling the observations, during training time; "EMR" refers to exact match ratio. For "Frozen Pre-trained Params" asterisks (+) refer to the practice of freezing all but the final layer.
Threshold
Normalised
Dimeusiio
Frozen Pre-trained Params
Single/ Multiple GPUs
Shuffle
Train Time
EMR:
EMK: Test
0.713216
1.595627
Makiiple
128x128
Single
128x
Single
No
951m
0.863587
1,559282
0.6
Yes
128 128
No
Single
No
957m 16x
0.686674
1.5619799
0.6
Yes
128
Yes No
Single
No
1257m 6s
0.154515
0.162003
0.8
Yes
128x128
Single
No
1240mn 58s
0.6491 19
1,548759
Table 7: Hamming loss for each object class computed by vision classification model f-
Sounding objects
Hamming less
Sounding objects
Humming loss
bathnah
0.012776331
plant
0.041079298
bed
0,032709252
scating
0.0191
cabinct
0.089008814
shower
0.010462300
chair
D.121365644
sink
0,010162535
chest of drawers
0.0331449777
sofa
0.061233-983
clothes
0.003414407
stool
0.024339208
counter
0.035132159
table
0.141519830
cushion
0.048237886
toilet
0.0039644758
fireplace
0.034691632
towel
0.006497799
eqqupimmtt
0.003193833
ty_monicor
0.022356829
0.058127753
Agest
Stant lsation
Gool locatin
Shorted path
Agent
urea
Ocousiedare
Target: C101
Deginning eplonde
Agent encounters target object
(CT Agent encounters object the ENITH chess Innget
Figure 7: Visualisation of failure case where agent encounters multiple objects with the same class as the target object. (a): beginning episode, with starting pose view the agent and the target sounding object. (b): K- SAVEN agent encounters the target object, but continues navigating. (c): K-SAVEN agent finds a different object with the same class as the target object.
At timestep t, the agent must select and execute an action er E A. The goal is to learn parameterised mapping (e.g.. policy), such that given sequence of observations [Oo, O1, Ot...., Or), agent that begins at an initial location in house h, E H. can navigate sounding object
Evaluating the memory in SMT. To evaluate effectiveness of memory used in Scene Memory Trans- former (SMT), we evaluate our model's 's performance after the first training stage, in which the memory size ($жr) is one, and the agent uses only the current observations. 9holesssows the results of K-SAVEN after stage I (*M 1) and stage (*M 150) training. As shown in Table the agent performs consistently better acmoss all metrics in all test after stage 2 training, indicating that adding memory helps navigate efficiently.
18

---

Under review as a conference paper at ICLR 2023
Table 8: Hamming loss for each region class computed by vision classification model f".
Regions
Hamming loss
Regions
Hamming loss
balcony
0.004955947
lounge
0.022026433
bathroom
0.034471367
bedroom
0.055506609
office
0.017621147
closet
0.007819383
other
0.008590309
dining
0.031-497799
outdoor
0.002973568
entryway/foyer/llbhy
0.013325991
porch/terrace/deck
0.016189428
familyroom/ounge
0.035022028
rec/game
0.011
hallway
0.047136564
spa/sauna
0.004405287
junk
stairs
0,006277533
kitchen
0.046035245
toilet
0.000440529
laundryroom/mudrromm
0.003634361
uillttyroom/toorroo
0,002533040
living room
0.043171808
workoutfgym//eerrise
0.0040074890
Table Results of SAVi Chen et al. all datasct splits.
2021a and K K-SAVEN after stage and stage training of SMT, across
HOUSES,
HOUSEN, UNHEARD
Method
(R
SPL(1)
SNA
DTG
SWS (1)
SR (f)
SPL SNA()) DTG
SWS(1>
(stage (stge
22.2
DAVEN (slage22 2)
UNSEEN HOUSES, HEARD SOUNDS
UNSEEN HOUSES, UNITARD SOUNDS
Method
SR.(1)
SPL(1)
DTG
SPL
SNA(T)
DTG SWS(())
2021at
(stage
13.3
7.3
SAM
(5tge
32.0
21.2
18.5
10.1
18.0
10.3
8.8
S0 De Aarierang Cange
15.00
11.1
10.0
12.3
7.0
12.5
8.6
7.3
11.7
4.3
X-BAVEN (stage 2)
38.3
24.4
22.2
8.4
18.6
34.4
23,
21.7
6.6
14,3
G ADDITIONAL DETAILS: VISUALISATION OF NAVIGATION RESULTS
X-SAVEN'S Failure Cases. In Figure we depict one of the common failure scenarios observed within K-SAVEN's 's navigation results, This is related to cases where multiple objects with the same class as the target robot are in the vicinity of the target sounding object. In many of these episodes, the agent wanders around these objects for extended period, often encountering the target object, although stopping the wrong location. Figure X shows another failure scenario where the agent gets stuck with surrounding objects
environment.
Typical instances of this failure mode include; the agent getting stuck in narrow halls or colliding with objects within outside its field-of -iew. In these situations, the agent frequently keeps predicting the Forward action for long time before predicting Stop. Conversely. it spins around the region where it gets stuck.
19

---

Under review as a conference paper at ICLR 2023
Agrnz
loatton
Agent path
area
Occupted
Torpet char
Target: Ster
Target cubext
(x) Cullision duw narrow palh
Cu La visible aigett
(c) Colfeiam duw ubslacie the spent
Visualisation of failure case where the agent gets stuck due collisions with its surroundings. (a): collision with narrow path. (b): collision with an object the field-of-vies the agent, (c): collision with an object outside the field-of-vis oi-view the agent.
Figure 8:
20


Reviews:
[{"review_id": "YHdY1hBIsZ", "reviewer": "Reviewer_djj6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new kind of audio-visual navigation task which requires agent to generalize to novel audio sources in 3D indoor environments from the Matterport dataset. The proposed approach leverages multiple sources of information of which objects the source is likely coming from, where they might be found, and where the source might be present in the house to tackle the task. In experiments, they show that the proposed approach significantly outperforms existing methods on unseen houses, unheard sounds. \n", "summary_of_the_review": "The task is interesting, and the approach is intuitive. But, I think the paper is also missing certain analyses (classification model performance, ablations by replacing with oracle modules) that will further strengthen the paper. \n\nUpdate after rebuttal: I thank the authors for providing responses to all my concerns and the experimental results for Oracle modules. I am tending towards acceptance for the paper. I am hesitating to give a strong accept is because I would have liked to see experiments where *not all sounds* are heard during the pre-training stage. Since they are building a new benchmark, it'd be nice to push the limits a little bit more and cover the case where a truly novel sound source is present during test stage. For that reason, I am only updating the score to 6.", "strength_and_weaknesses": "Strengths: \n- The task definition is interesting. Generalizing to unheard sounds is an interesting split to look at, and I am glad that this is being looked at. \n\n- The proposed approach is somewhat intuitive and seems to work well, specially on unheard sounds. \n\nWeaknesses: \n- It's not clear if the pre-training happened on all sources of sounds (seen and unseen). The formulation seems to suggest that the audio classification model was trained on all sources (seen and unseen). I think this is a bit problematic, since it makes the assumption that at test time, no novel sound source will exist. \n\n- If the pre-training involved both seen and unseen sources, then comparison with SAVI is unfair because SAVI presumably didn't have access to these sounds. \n\n- It'd be nice to show results for classification models apart from navigation results. These will help the readers understand how classification accuracy correlates with navigation accuracy. \n\n- I also think it'd be nice to replace certain parts of the model with \"oracle\" modules. For instance, if the audio classification model, or the location prediction model was perfect, how will the performance get affected. Doing these sort of analyses helps build a much stronger intuition for the task.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SUZVKukYIy", "reviewer": "Reviewer_8Qq6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper aims to propose two architectural contributions and a new task for semantic audio-visual embodied navigation. \n\nThe paper first employs a knowledge graph for encoding object-object, object-region, and region-region relations. Then, the paper introduces multiple auxiliary models to facilitate audio-visual embodied navigation. \nFurthermore, the paper proposes a new task of semantic audio-visual embodied navigation. In the new task, the agent is designed to steer toward an object without unheard sound.\n\nThe paper also demonstrates performance improvements in all permutation of settings, ie., seen/unseen houses with heard/unheard sounds.\n", "summary_of_the_review": "Overall, the paper raises a novel audio-visual navigation task with unheard sounds. However, the other three contributions need more explanations to justify their novelty.\n\nI lean toward rejecting the current version. However, if the author persuades me that all these three contributions are meaningful, I would like to modify my recommendation to be above the acceptance threshold.\n\nBesides the review for this paper, I believe that this kind of open-set setting is interesting and will definitely benefit our community. However, the goal of raising a new task is not because it is a new task. **In my opinion, the goal should be to provide more insights or chances for the community members.** To achieve that, I recommend the author to think questions about some specific problems in open-set tasks and designing methods following corresponding motivations. At final, I hope to see the edited paper in the future.", "strength_and_weaknesses": "Pros: The paper demonstrates the overall architecture clearly in Figure 2.\n\nCons: \n- It is hard for readers to catch up on the key points of the paper. Since almost every subsection only contains one long paragraph, readers might fail to understand the meaning.\n- The paper claims to introduce pre-defined semantic information to encode object-object relationships. However, in ICLR 2019, [1] proposed a scene prior graph learned from Visual Genome dataset. It would be helpful if the author could provide a comparison between the proposed semantic information with the semantic knowledge learned in ScenePriors [1].\n- The second contribution of the paper is to establish object-object, object-region, and region-region relationships via a knowledge graph. However, in ICLR 2021, [2] proposed a transformer-based architecture to encode similar relationships in visual navigation. The author first misses this reference and second lacks the difference with VTNet [2] in building the aforementioned relationships. \n- The paper does not provide convincing analysis for some ablation experiments. For example, in both SH/HS and UH/HS, introducing $GEN^b$ alongside $GEN^v$ will lead to a performance drop compared with only adopting $GEN^v$. Could the author provide some extra explanation for these results?\n- Furthermore, it is hard for me to understand why the performance of GEN^s is lower than that of $GEN^b$ in both SH/US and UH/US settings. Could the author please provide more experiments for readers to understand these experiment results?  \n\n\n\n[1] Yang, W., Wang, X., Farhadi, A., Gupta, A., & Mottaghi, R. (2018). Visual semantic navigation using scene priors. arXiv preprint arXiv:1810.06543.\n\n[2] Du, H., Yu, X., & Zheng, L. (2020, September). VTNet: Visual Transformer Network for Object Goal Navigation. In International Conference on Learning Representations.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "HrPZzrbdkVJ", "reviewer": "Reviewer_tJCa", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the Semantic Audio-Visual Navigation (SAVi) and proposes the use of Knowledge-driven scene priors for encoding the object-region relations, spatial knowledge, and background knowledge to address the embodied navigation problem. The method named K-SAVEN incorporates graph encoder networks to model audio and visual inputs, as well as applies scene memory transformer to capture long-term visual-audio dependencies. Numbers of pre-training tasks are defined, and a multimodal dataset is curated to support the learning. Moreover, a new SAVi task is defined to evaluate the agent performance on novel objects and sounds. ", "summary_of_the_review": "This is the second time that I review this paper. Comparing to its previous ICLR and CVPR versions, it is great to see manuscript has been significantly improved and most of my previous concerns are nicely addressed, especially the new experiments in Table 1 (top), Table 2, and Table 7 are added to justify the method and designs, as well as the insightful analysis section.", "strength_and_weaknesses": "Strengths:\n\n1.\tThe paper proposes novel and effective methods and defined a valuable task to address SAVi, which will be highly beneficial to future research in relevant directions. The use of knowledge graph to connect sounds, objects and regions could be an important step towards generalizable audio-visual navigation in novel environments. Results show significant improvement over previous approaches. Moreover, the new task offers a setting for evaluating the agent’s generalization ability on novel objects (and sounds).\n\n2.\tIt is great to see the comprehensive details presented in the method section and the Appendix, e.g., the knowledge graph construction and representation, including the resultant graphs in Table 3 and Table 4.\n\n3.\tOverall, the paper is clearly presented.\n\nWeaknesses:\n\n1.\tIt is wonderful to see the new experiments added to this paper, but they can go deeper. Since knowledge graph is the key novelty, I expect more experiments on using the knowledge and the graph construction (e.g., the source and type of knowledge, the connection between objects and regions, etc.). Although this work is built based on SAVi and several methods are inherited, it will be valuable to see the influence of the pre-training and classification tasks in the proposed architecture. Moreover, it would greatly strengthen the paper if the method is evaluated on more dataset such as Replica and compare with the baselines.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject