Structure from Silence: Learning Scene Structure from Ambient Sound
Authors: Ziyang Chen
Xixi Hu
Andrew Owens
Venue: CoRL2021 Oral (2021)
Keywords: audio perception
multi-modal learning
self-supervision
navigation

Abstract: From whirling ceiling fans to ticking clocks, the sounds that we hear subtly vary as we move through a scene. We ask whether these ambient sounds convey information about 3D scene structure and, if so, whether they provide a useful learning signal for multimodal models. To study this, we collect a dataset of paired audio and RGB-D recordings from a variety of quiet indoor scenes. We then train models that estimate the distance to nearby walls, given only audio as input. We also use these recordings to learn multimodal representations through self-supervision, by training a network to associate images with their corresponding sounds. These results suggest that ambient sound conveys a surprising amount of information about scene structure, and that it is a useful signal for learning multimodal features.

Structure from Silence: Learning Scene Structure from Ambient Sound
Ziyang Chen', Xixi Hu', Andrew Owens
University of Michigan
https://ificl gg thub. frructure-fron 1ence
LG
(a) Quier Campus dataset
(b) Depth estimation
(c) Multimodal self-supervision
Figure 1: What can ambient sound tell about 3D scene structure? (a) We collect "in-the-wild" dataset of paired audio and RGB-D recordings from quiet indoor scenes. (b) Given audio from scene, estimate distance to wall. (c) We use this ambient sound learn audio-visual representations through self-supervision.
Abstract: From whirling ceiling fans ticking clocks, the sounds that we hear subtly vary as we move through scene. We ask whether these ambient sounds convey information about 3D scene structure and, if so, whether they provide a useful learning signal for multimodal models. To study this, we collect a dataset paired audio and RGB-D recordings from a variety of quiet indoor scenes. We then train models that estimate the distance to nearby walls, given only audio input. We also use these recordings to learn multimodal representations through seff-supervision. by training network to associate images with their corresponding sounds. These results suggest that ambient sound conveys a surprising amount of information about multimodal features.
structure, and that it is useful signal for learning
Keywords: audio perception, multi-modal learning, self- self-supervision, navigation
Introduction
Humans make extensive use of sound for understanding 3D scene structure. Recent work has sought reproduce these capabilities machines, often in simulations with loud, distinctive sounds, such as buzzing alarm clocks and footsteps. Yet it is unclear how often sounds like these are available to robots practice. Real-world scenes tend be quiet, and the sounds that we do hear within them, such ventilation noise, tend to be ambiguous, and propagate through scene in complex ways.
Human hearing, by contrast, is capable of estimating scene structure from subtle ambient sound cues. Perhaps the most familiar these are contextual associations, such as knowing that noisy vents and sound-leaking windows tend to be attached to walls. Work in psychology, however, has also proposed that humans use simple low-level cues. These include using the accumulation of low frequencies to tell whether they are near walls [1], using sound shadows to estimate the shape of obstructions [2]. and using reverberation interpret scene geometry [3]. One feature of these cues is that they are available solely through passive observation. This is in contrast to echolocation, which requires the observers to actively produce sounds.
Indicates equal contribution.
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

These sounds also provide a "free" learning signal for vision [4]. Learning to successfully predict audio from images (or vice versa) requires understanding how the two modalities vary with pose and scene structure. For example, as one moves toward a sound-making object, it will simultaneously get louder and increase its size within the visual field. Rotating the camera, on the other hand, drastically alters the visual signal without significantly changing the sound. A model that can predict one modality from the other must learn both sources of variation, such as by learning a visual representation that is somewhat invariant to rotations but sensitive to distances to sound sources.
In this paper, we ask whether ambient sounds that occur in real-world scenes convey 3D structure, and whether they can be used for multimodal self-supervised learning. To study this, we collected a dataset of "in-the-will"" audio recordings from quiet, indoor scenes typical of what a robot would encounter when solving navigation tasks. Each sound in our dataset is paired with a corresponding recording from an RGB-D sensor, which provides a visual signal and pseudo ground-truth depth. The resulting dataset, which we call the Quier Campus dataset (Fig. Ia), covers a variety of room shapes, background sounds, and materials.
Using this dataset, we conduct experimental study of depth estimation from audio. First, we show that audio can be used to estimate the distance to nearby walls in a variety of scenarios, including predictions of the relative depth between two recordings. We demonstrate that the model can be used as part of a very simple robotic navigation system, in which a wheeled robot moves along a wall using ambient audio cues (Fig. lb). Next, we ask whether ambient sound can be used learning signal, without explicit use of depth. We study several multimodal self-supervised learning formulations, each of which requires a model to associate subtle changes in sound with a visual signal (Fig. 1c). We show that the resulting model learns a feature set that can be used to solve downstream distance-to- wall estimation tasks.
We make the following contributions: i) dataset of paired audio and RGB-D data from real-world scenes, ii) an experimental study of audio-based depth estimation, iii) showing that audio-visual recordings from these scenes can provide useful sele-super for depth estimation tasks. Our results suggest that ambient sound conveys a surprising amount of information about structure, and that it provides useful training signal for multimodal learning.
2 Related Work
Human auditory perception. The field of psychoacoustics has studied what humans can infer from sounds, such size, material, shape, and depth. One well-known cue for depth estimation is echolocation, whereby humans estimate distance by generating sounds and analyzing their echoes [5, 6]. However, humans may also infer a great deal from passive observation. Our work is inspired by Ashmead el 's studies of the navigation abilities of children with visual impairments [1, 7, 8]. They propose that visually impaired people use accumulations of low-frequency sounds to detect obstacles, and propose physical models of wave interference that would lead to changes in ambient sound fields near obstacles. This effect is similar to how a seashell held to one's ear amplifies ambient sounds to create ocean-like noises [9]. Traer and McDermott [3] show that humans can use phase changes from natural reverberation to distinguish different spaces. While these cues have been studied extensively in psychoacoustics, they are not widely used in robotic perception and self-superviser learning.
Audio-based depth estimation. We take inspiration from work in ocean acoustics that estimates the structure of the ocean from ambient sound using microphone arrays [10, 11]. In classic work, Kac [12] related the shape of a drum to its sound. Recently, Purushwalkam et al. [13] reconstructed floor plans from audio-visuu visual signals, using a simulated environment in which sound effects from FreeSound [14] were synthetically inserted into the scenes (e.g.. beeping alarm clocks, flushing toilets, and dishwashers). However, is unclear how often distinctive sounds like these occur in practice, and whether the simplified models of sound propagation are sufficiently accurate. By contrast, we experimentally study aadio-based depth estimation in real-world scenes. Other work uses multiple microphones to localize sound sources [15, 16, 17, 18, 19]. Our method, by contrast, requires only monaural audio. Recent work also uses self- eeofuproduced echolocation sounds produced by onboard speakers. Christensen el al. [20] predict depth maps from real-world scenes using echo responses. Gao el al. [21] learns visual representations by echolocation in a simulated environment [22]. In contrast, we learn through passive observation, rather than active sensing.
Audio visual learning. Many recent works have proposed to use paired audio-visual data for representation learning. In seminal work, de Sa [23] proposed multimodal self-supervised learning as alternative to single-modal models. Later, Ngiam et al. [24] learned a multimodal Boltzmann
2

---

Motion
Figure 2: The Quiet Campus Dataset. We collected dataset of paired audio and RGB-D recordings from a variety of quiet indoor scenes. We show selected images from the static and motion subsets, which contain stationary and moving microphones respectively. Please refer the project webpage for audio-visual examples.
machine. Owens et al. [25] learned self self-supervised visual representations from impact sounds, and used ambient sound to learn visual features [4]. In the latter, the sounds are taken from internet video and thus contain much wider range of auditory events than what we consider in this work. Later work simultaneously learned audio and visual representations [26, 27, 28, 29, 30, 31]. Other work has learned cross-modal distillation [32], sound source localization [33, 34, 35, 27, 36, 37, 38, 39, 40], active speaker detection [41, 42, 43], source separation [44, 45, 46, 47]. We take inspiration from this work and show that quiet indoor sounds also provide a useful self-supervised learning signal.
Visual depth prediction. Recent work has learned to recover 3D structure from RGB images reconstructions [48, 49, 50, 51, 52, 53, 54], via voxel grids, point clouds, and meshes [55, 56, 57, 58). Other work has made significant progress predicting relative camera pose [59, 60, 61, 62, 63J. We take inspiration from the techniques proposed in these models, though we predict depth from audio instead of from images.
Robotic navigation. Robots often use visual signals to navigate in novel environments [64, 65, 66, 67J. While vision is often reliable cue for depth estimation, there are many situations where it is unavailable (e.g. due to sensor failures or poor lighting), necessitating "backup" modalities. Recent works have proposed methods that use sound for robot navigation. Those robotic systems are designed to localize sound sources and to navigate to audio goals in indoor environments [68, 69, 22, 70, 71, 72]. Unlike these methods, which largely use distinctive sound sources, use ambient sounds collected in real-world scenes.
3 The Quiet Campus Dataset
To study the aructure-from-att problem, we collected a dataset containing a number of "in-the-wild" indoor ambient audio recordings, paired with concurrently recorded RGB-D data. We collected our data from a large number of classrooms and hallways of a college campus. To ensure that the sounds within the recordings were subtle ambient sounds (e.g. ventilation noise), rather than distinctive audio events (e.g. speech), we collected the data at times of the day when there were relatively few people occupying the buildings. Please refer to our webpage for a random sample of sounds.
Depch Camera
Microphone
We collected two types of data: static recordings in which the microphone and camera are stationary, and motion recordings where human collector moves slowly toward or away from wall. We show some visual examples from our collected dataset Fig. 2. our experiments, we split the data so that there is no physical overlap, such that the rooms in the training and tests sets are disjoint.
Static recordings. We collected data in 46 classrooms from 12 buildings on The University of Michigan's is campus, amounting to approximately 200 minutes audio. Inside each classroom, we selected 16 30 positions and recorded 10 secs. of audio at each one. Since we are interested in predicting distance to walls, we point the camera and microphone toward the nearest wall when recording, that the distance is well-defined.
Figure 3: Captur- ing equipment.
Static recordings with dense coverage. In order to evaluate generalization to different orientations, we use four classrooms from static recordings, divided each one into 20 1.5 1.5m² depending angles (rather than always facing a wall). We call this subset the static-dense recordings. These two datasets were recorded at different times of the year. The static recordings were made in spring 2021, when buildings were at reduced occupancy due to the COVID-19 pandemic, while static-dense were recorded in the winter of 2020, when buildings were at normal occupancy.
grid cells of size their sizes, and collect 10 hours of audio densely with wide range of
Motion recordings. To help understand whether small motion (and the subtle changes in audio over space) provides useful information about structures, we collected approximately 90 minutes of videos in motion (222 videos total). To ensure that there no unintended obstructions nearby, we collect
3

---

Audio Branch
Viaua Branch
Concat
Audio
Visual
Fusion
Fuslon
MLP
Audio
Match Order
Audio
Image
Figure 4: Self-supervised audio-visual model. We encode pair of visual and pair of audio examples using CNNs, and fuse each pair independently using concatenation. The two modalities are then combined and classified using small multi-layer perceptron, which predicts whether the two pairs in the same order. Please the supplementary material for details.
this data hallways. During recording, the microphone and RGB-D camera move toward or away from wall (Fig. 2). As in the static setting, point the microphone toward the wall. To reduce possible sources of sound, the human operator moves slowly and does not take footsteps. We split this dataset based on building, i.e., the videos in the training and test set are hallways recorded in different buildings.
Estimating wall distance. For experiments that require distance to the wall, we obtain this by cropping the center 320 x 240 region from the depth image, and averaging the depth values. Since the camera is oriented toward the wall, this generally an accurate estimate depth.
Hardware. We use a ZOOM Hin Handy Recorder for collecting audio since it a simple, inexpen- sive, and widely-used stereo audio recorder. For the RGB-D data, we use Intel Real-Sense Depth Camera D415 (Fig. 3).
4 Predicting 3D Structure from Ambient Sound
We design models and experiments that will experimentally evaluate the ability model to learn to predict depth from audio in natural scenes.
4.1 Depth estimation tasks
Inspired by work in human obstacle detection with sound [6], we study the ability of machine hearing system to predict the distance to a wall ahead them. We pose this problem in multiple ways, which has applications to variety of robotic navigation applications.
Obstacle detection. First, pose the problem of estimating whether the microphone is within small distance of a wall (we use 0.5 meters), a binary classification problem.
Relative depth order. many cases, a robotic system needs only to move toward or away from an obstacle, and hence only needs to know which direction is closer to it. We also hypothesize that relative depth estimation is easier task than absolute estimation, as is visual depth perception [73, 48). Given two audio clips, we train network to predict which one is closer to wall.
Relative depth estimation. Given two audio clips, we train network to predict the difference between their distances to the wall. We also consider predicting the log of their difference, following the work in visual depth estimation [50, 49], and as multi-way classification task, after clustering depth differences the training set with k-mmans.
Absolute depth estimation. Finally, we train model to regress the distance to wall, rather than a thresholded "near-or-far" prediction.
4.2 Self-supervised audio-visual learning
Having evaluated our model's ability to estimate depth directly from audio, we now ask whether we can learn about structure through audio-visual supe-vupevniion. We hypothesize that, through the task of associating raw (RGB) visual signal with audio, we will learn a representation with features that will be helpful for downstream depth estimation tasks.
One simple model, considered prior work, is to solve audio-visual synchronization task [41, 27, 37]: given the video, distinguish whether audio and visual stream have been misaligned through random shift. We call this model the AV-Syne model. Finally, we pose new audio-visse visooovisallll learning problem that requires model to learn relative depth between examples. We train model to distinguish between ordered audio-visual pairs, which we call the AV-Order model. Given a pair of audio tracks and pair of images (which correspond to the two audio tracks), we train network to determine whether the two pairs are in the same (or mismatched) order, which we frame classification problem.

---

Prob:
0.85
Prot: 0.90
0.34
Figure 5: Obstacle detection. After training model, walk through and show the predicted probability being near wall. The frames with blue border are correct predictions while frames with red border are failure cases. Please refer the webpage for video results (with audio).
4.3 Audio-based robotic navigation
To demonstrate the effectiveness of our depth estimation model, we use it to guide wheeled robot (TurtleBot) to follow wall. The robot uses the obstacle detection model to predict whether there is wall near the left/right of the robot, and moves accordingly. We use a simple policy, based on that of Gandhi et al. [74] (please see the supplement for full details). The robot moves forward in sequence of steps. For each step, it rotates to an orientation that is derived from the obstacle detection model. It senses the audio on its left side, then rotates 180 and senses audio to its right. We run the obstacle detection model on both audio recordings, and choose an orientation such that the robot moves in the direction that has lower probability of being near a wall.
4.4 Models
Network backbone. We represent audio using the VGGish network [75]. For our experiments. we either replace the final layer with a new regression or classification head, or we use the (128- dimensional) penultimate layer. The network takes 0.96 sec of audio as input, in the form of a log-me spectrogram. Due to the importance of low-frequency sound in physical models of wall-sound interaction [8], we do not remove the very lowest frequencies of the audio, in contrast to common practice in speech and audio event recognition [75]. We analyze the importance of this decision through experiments. We convert the stereo audio recordings to mono by averaging the two channels.
For models that use visual data, we use ResNet-18 [76). with 224 x 224 images, and modify the last linear layer to output the feature vectors with the same dimension as audio features. We also consider a model whose weights are finetuned from ImageNet [77].
Learning details. For fully supervised tasks, we use SGD [78] optimizer with a learning rate 10-3 momentum 0.9, and weight decay 5 x 10 For the self- self-supervised learning experiments, we use Adam [79] with learning rate of 10-4 4 for representation learning. For linear probing experiments, we use SGD with learning rate of 10-2, We schedule learning rates via cosine decay and pick the model with best validation performance.
Audio processing. Our audio preprocessing resembles that of [75]. We resample the audio to 16 kHz and extract segments of 0.96 secs. We then convert them to spectrograms by applying a Short-time Fourier transform a stride of 10 ms and 25 ms, and convert them into 96 64 log-mel spectrograms. For augmentation, we randomly crop the waveform in time and multiply the amplitudes by random factor uniformly sampled from [0.5, 1.5).
Depth estimation networks. For the regression and binary classification tasks, we add small multi-layer perceptron (MLP) on top of the VGGish network. For the relative estimation tasks, our network uses two audio inputs. For these tasks, we use Siamese network that fuses the extracted feature vectors via concatenation prior to the MLP. For regression, we normalize depths to be in the range 0-0.5, 0.5] during training.
Network for self-supervised learning tasks. For AV-Sync task, we concatenate image and audio features before passing through the multi-layer perceptron. For the AV-Order model, we fuse Siamese networks from both modalities via concatenation and add multi-layer perceptron (Fig. 4). In the downstream task, we take the output features from the last convolution layer after global average pooling and freeze them. We pass our 512-d learned features to a linear layer, which performs classification (and which is the only part of the model optimized during training).
5 Experiments
Having collected a dataset of in-the-wild audio and models for inferring structure from it, we conduct an experimental study of what can be inferred from ambient sound in real-world scenes. We report our results along with 95% confidence intervals.
5.1 An experimental study of audio-based depth estimation
Obstacle detection. First, we evaluate our model's 's ability to solve the obstacle detection problem (Tab. 1). We split the data such that the test rooms were not heard during training. We report accuracy
5

---

1.0
0.91
0:78
0.818
0.76
0.
Table I: Obstacle detection and relative depth order. We evalu- ale our model's ability to determine whether microphone within 0.5 meters of a wall and identify which sound has a smaller distance to the wall. Pre refers to pretraining.
0.97
0.4
0.4.
0.57
0,86
0.89
0.35
0.26
0.35
0:71
Obstacle detection
Relative order
Model
Pre.
Task
AP(%)
Acc(%)
AP(
Acc(
Audio
static
68.3
(+1.))
60.0
85.5 (+.m)
77.2 =0000
Image
static
99.2
95.5
L..5
94.6
86.4
0.92
0.97
0.96
0.92
0.87
-03
Image
static
99.5 (+0.1)
98.4 (00.0
97.7 (+0.2)
(e0.5
-22
Chance
static
:6.4
50.0 L.1
47.2 (i
50.0
Figure 6: Wall detection in a single room. We divide room into grid, and record audio within each cell. We show the average obstacle detection probability in each one.
Audio
motion
65. :+1.4)
64.5 c-
87.1 (+1.0)
81.3 -00.0
Image
motion
73.4 (
68.2
87.9 (40.))
81.2
Image
motion
78.5
(_...)
97.1 (10.0)
90.6.20.0)
Chance
motion
50.4 :0.4:21.4)
50.0 (0.0m
50.5 (21.4)
:0.0::1.0
on this binary classification task, as well as average precision (to account for possible dataset shift from using held-out rooms). We evaluated our model in both the static and motion settings. To help interpret the effectiveness of audio the context of other modalities, we also included simple visual models, with both random initialization and ImageNet pretraining' Interestingly, we see that the audio-based model performs better than chance in both cases, suggesting that ambient sound, indeed, conveys information about scene structure.
We also evaluated our model's 's ability to generalize across orientations and recording times. We found that a model trained on static recordings could achieve 86. 1% AP and 72.9% accuracy on static-dense recordings, indicating strong generalization performance.
We qualitatively demonstrate the obstacle detection results by training a version of the model on the static-dense data, since it contains a large number of viewpoint variations and angles (see supplement for quantitative results). We apply the learned obstacle detection model to every frame of video in which a person walks through a room (Fig. 5). In Fig. 6, we also apply this model densely within a room, divided according to its spatial grid. We averaged the probability of the model for each grid cell. We see that successfully distinguishes between the boundaries of the room and the center.
Relative depth order. In Tab. 1, we evaluate our model's 's ability to predict which of two audio clips is closer to a wall. We find that the model obtains re- sults that are significantly bet- ter than chance, and that relative prediction is significantly easier than obstacle detection. We also asked whether our model might use binaural cues (even though the stereo channels are averaged).
Table 2: Relative depth ratio. We evaluate our model's ability of pre- dieting relative depth ratio from two ambient sounds, for the motion recordings.
Regression
Regression by Classiffiaation
Model
MAE
Med. 4
R 1
Top-1 1
Top-3
Avg. Dist 1
Audio
0.55
0.44
0.48 m0.0
22.8 (1
80.7 team
1.66 cuuss)
Image
0.34 12 m
0.42 a m
0.49 is mm
26.6
83.6
1.47 c2:40
Image (Pre.)
0.39
0.29
0.72
34.2
:0 m
Chunce
0.89
0.00
0.79 (29...00
0.00 00.00
9.45 (10.0)
$2.6
183.0)
2.79...0
No input
0.82 1z
0.75 (:1
0.00
(3.7.
51.6:21.0
4,50
Using only the left channel, we obtain nearly identical performance: 87.2% AP and 81.2% accuracy. Finally, we analyzed the effects of mixing in spurious distractor sounds (see supplement).
Relative depth ratio. Since the network able to tell relative depth order well, we explore rel- ative depth setup further by pre- dicting depth difference of audio pairs (log depth ratio). We use re- gression predict the log-ratio from ambient sounds with an L1 loss and via classification. We report the performance of mean absolute error, median error, and R2. For classification, we report top-1 and top-5 accuracy, and report average distance from the ground truth bin using the distance between centers of paired bins. We report the performance of
Table 3: Absolute depth estimation. We evaluate model's ability of predicting absolute distance the wall for the motion recordings.
Regression
Regression-by- Clyy-CCssficattion
Model
MAR J
Med.
x²
Top-1 1
Top-5
Avg. Dist.
Single
Audio
0.28
0.25 um
-0.34...0
30.8 10.
Image
0.51 :2:
0.27
-D.67 om
95.9
1.05
cm
Image (Pre.)
0.26
0.21
-0.24:1.0
50.8.11.
99.2.10a:
0.62
No input
0.27
-0.19.1.0
21.3
88.3
Cooditional
Audio
0.21 m
0.17
am
36.9 11:10
90.0 .4:41
Image
0.22
0,18 4.40
0.12 am
38.2 (102
95.5 c4
0.93
(Pre.)
0.18
0.14
0.39
51.7
99.8
0.59
No input
0.01
26.4 4000
95.9
Chance
0.78 :1
0.84:1.0
-3.38 i..
23.3
56.9.1
2,83 cL.
We do not use pretrained VGGish audio model since existing models remove low frequencies and are not compatible without significant changes.

---

6.5 I
Drstinatio
Table 4: Linear probing experiments. We evaluate our self- supervised feature set for obstacle detection and relative depth order, for the motion recordings. Here, Audio means taking audio only as inputs. Visual means taking images only as inputs. Both means taking both audio and image as inputs.
Track
3
Obstacle detection
Relative order
Model
Pre.
AP(%)
Aoo(S)
AP(S))
Acc(S))
Start
Scratch
(1.9
73. aa.
VGGish [75]
58.2
61.1 (211
61. (1..
AV-Sync
64.0
ag...
74. 1a
9.43
AV. Order
63.4
61.5
84,2 (3:
79.4
VGGish [75]
.9.0
56.7
1.
67.7 (23--
(15.0)
AV AV-Sync
V
62.8
(20
82.1 (2)
76.4. (...
AV-Onder
(11.))
64.5
(:0.9)
85.5 (.3.))
(1..9)
Scraich
70.1
64.0
79.7
71.5 (3..0)
Visaal
AV-Sync
(7.1
69.2
(:09)
85.3 120..
AV-Order
a6.8
68.8 mo
87.4
79.1 (2.
ImageNet 177, 761
80.4.11a
74.5 m00.0
(..0
85.8 (10.7)
AV-Sync
89.0
75.6 mo
92.8 (2.8:00.00
85,4
AV-Order
86.5 aa
76.3
95.8 cs0 41
88.9 2200
Figure 7: Classroom floor plan and track setting.
AV-Order
77.1 17.1
69.
89.0
80.5 8
AV. Order
88.1 ca0.:;
76.9
95.8
(1:0 40
88.9 9 (....o)
zeroing out two imputs as a baseline. As shown in Tab. 2. we can see that our method outperforms chance and, interestingly, has relatively small gap between the vision-bass approach.
Absolute depth estimation. We evaluated our model's ability to predict depth from a single audio example, using similar experiments as the relative depth ratio prediction (Tab. 3). Our model is more accurate than chance (which we obtain for all metries by simply ablating the input and training). Likewise, the
model improves over chance performance.
We also consider another model that estimates depth for given audio input after conditioning on both reference sound recorded within the same scene and its depth (see the supplementary material for details). We call this the conditional absolute depth model. This model obtains better performance than our unconditional model, suggesting that the reference audio provides useful information.
5.2
Audio-visual representation learning
Finally, we investigate whether Table 5: Linear probing experiments. We evaluate our learned representation for relative depth ratio for the motion recordings.
learn useful representation with AV- Order and AV-Syne models. In the AV-Order binary classification pretext task, our model obtains 85.8% AP and 75.8% accuracy (80.3% AP and 70.7%
Model
Pre. Top-1 (9)
Top-5 (4)
Avg. Dist
Scratch
19.2
12.8
2.33 cn
VGGish [75]
11.4
53.9 o1
3.78 (10.co
Audio
AV-Sync.
19.2 (tom
72.7 s:
2.07 deoucs)
AV-Order
22.2 1
79.6
1.86
accuracy with random initialization) on the test set where the random chance is 50%. Given its performance on the pretext task, we test whether the learned audio and visual represen- tations are useful for the downstream depth- eesttmesstmationn tasks: (1) obstacle detection and (2) relative depth esti- mation. To evaluate the quality of our spatial representation, we freeze the learned features and train a linear clas- sifier on the depth estimation tasks (Section 4.1). In the downstream tasks, evaluate image and audio features individually and compare them with random features, pre-trained ImageNet fea- tures [77, 76), pre-trained AudioSet features [75] and AV-Syne features. Since AudioSet pre-trained weights require a special audio pre-processing that differs from ours, we provide another random feature baseline based on the same processing as [75].
VGGish [75]
54.0 c4.0
3,59 cHouuD)
AV-Syne.
20.7 (Lous)
75.1 (20.9)
1.99 cpoucc)
AV-Order
23.6 oe
80.5
1.75 reouus)
Scratch
18.5 (a00
70.8
2.66
AV-Sync.
22.2(10.:)
76.8
1.85 (LOVCO)
Visual
AV-Order
24.7 cto.s)
80.2 (t0.s)
1.71 (eo.cc)
ImageNNer (77, 76]
27.4
87.1 (10 m
1.60 (1ov0ca
AV-Syne.
85,2 (:2coon))
1.53
AV-Order
28.9 (b
88.6 c
1.40 ceoouss
Bohh
AV-Order
23. (1:0:)
81.5 (10.7)
1.59 (10.c0)
AV-Order
30.0 dr
89.3
1.31 (eovus))
Relative depth. We report results for relative depth order and relative depth ratio in Tab. 4 and Tab. 5 respectively. Our method outperforms baselines in both audio and visual modalities suggesting that we learn useful depth representations. Moreover,
results are often close to supervised features.
Obstacle detection. As the results shown in Tab. 4, AV-Order model outperforms Scratch and AudioSet pretraining, while loses to AV-Syne model. The AV-Order obtains worse performance since AV-Order, perhaps because the model deals with two inputs during the pretext task. We provide
7

---

an additional experiment by tine-tuning our learned AV-Order feature and we compare with the fully supervised method in Tab. 1. This obtains a 4% AP boost (70.0% v.s. 65.6%), suggesting that our learned representation is helpful.
Interestingly, the multimodal models that use both audio and visual features consistently outperform the single-modality models, suggesting the audio useful in conjunction with visual signal. We conduct few-shot learning experiments to show audio- visual self-supervision improves fine-tuning performance (please see the supplementary for the details).
Audio cues. To help understand what information the model is using, we retrained our model with access to only part of the frequency spectrum. We zeroed out different frequency ranges in the Mel spectrogram, then re-trained the relative depth order model. As
90
Trequercy- AP
Full frequancy ACCgro.5
Reletive Depth Grder- AP
Reletivv Depth Order- ACC
Metrics (50
75 20
can be seen in Fig. 8, the accuracy of the model with low frequencies (0- (0-1000Hz) comparable to the model with the full frequency range, suggesting that our model mainly use those frequencies infer the structure information. This may be because there is relatively little high frequency information in the dataset. We also compare the data distribution of our dataset with (non-ambient) music sounds from the VGGSound dataset [80] (see supp. for details).
60
55
0-18
2k-
36
68
5k 5t 6%
Te Mr.
Frequency Range (tte)
Figure 8: Performance vs. different frequency.
5.3 Robotic navigation with ambient sound
We ask whether our depth estimation model can be used to guide a robot to follow a wall, solely using ambient sound. For this, used model trained and evaluate on a classroom that was not included in the training set. The track is 8m long, and forces the robot to turn a corner (Fig. 7). The robot starts navigating from one of several unknown locations (40em, 80em, 120cm from the wall) and orientations (30' left or right, or facing forward). We repeat the experiment 18 times and measure the average distance along the track it attained before colliding with a boundary. Quantitative results are shown in Fig. 9. We compare with a random policy, a "straight line" policy that always drives forward, and policy based on the vision model. For the random policy, the setting is the same as our method, except that we use an obstacle detection network with random weights. The straight line policy is effective when the robot starts with a front-facing orientation. also has the advantage that it does not rotate to sense audio, and hence avoids accumulating errors?. As expected, the visual model significantly outperforms the audio model. We see that our method performs better than random and straight- line policies, and successfully guides the robot past the corner in nearly half of the trials. We repeat experiments in the different rooms (please see the supplementary for additional results).
Random Straight Audio volon.
static recordings,
Average distance without collision (m)
Figure 9: Robot navigation results.
Conclusion
We see this work as opening two directions. The first is in creating robotic systems that exploit real- world, passively obtained audio signals for depth estimation, especially as an inexpensive "backup" signal that can be used when other modalities fail. Second, we see our work as a step toward using inexpensive and plentiful audio signals to help train other modalities, such as vision.
Limitations. While ambient sound provides useful information about scene structure, it is generally less effective than vision. We also lack full understanding of what cues the model is using, and what can (and cannot) be perceived from it, such as whether there are fundamental limitations on resolution, material, or angle. Additionally, our experiments were conducted on single college campus, and hence may not be representative of all indoor spaces.
'Thse errors could be reduced by using multi multi-microppone array, though for simplicity we use a single microphone.
8

---

Acknowledgments
We thank James Traer for his invaluable help explaining work on human perception of ambient sound. We would also like thank Linyi Jin and Shengyi Qian for help with visual experiment setups and their valuable suggestions. We also thank David Fouhey for his comments and feedback, Peter Gaskell for his help on robot equipment, and Yichen Yang for the help on data collection. We thank the valuable suggestions and feedbacks from David Harwath, Kristen Grauman and UT-AAutin Computer Vision Group. This work was funded in part by DARPA Semafor and Cisco Systems. The views, opinions and/or findings expressed are those of the authors and should not be interpreted as representing the official views or policies of the Department of Defense or the U.S. Government.
References
[1] D. H. Ashmead, R. S. Wall, S. B. Eaton, K. A. Ebinger, M.-M. Snook-Hill, D. A. Guth, and X. Yang. Echolocation reconsidered: Using spatial variations the ambient sound field to guide locomotion. Journal of Visual Impairment & Blindness, 92(9):615-66 1998.
[2] L. D. Rosenblum and R. L. Robart. Hearing silent shapes: Identifying the shape of sound- obstructing surface. Ecological Psychology, 19(4):351-366, 2007.
[3] J. Traer and J. H. McDermott. Statistics of natural reverberation enable perceptual separation of sound and space. Proceedings of the National Academy of Sciences, 1 113(48) 2016.
[4] A. Owens, J. Wu, J. H. McDermott, W. T. Freeman, and A. Torralba. Ambient sound provides supervision for visual learning. In ECCV. Springer, 2016.
[5] M. R. Rosenzweig, D. A. Riley, and D. Krech. Evidence for echolocation in the rat. Science, 1955.
[6] D. H. Ashmead, E. W. Hill, and C. R. Talor. Obstacle perception by ongenitally blind children. Perception & psychophysics, 46(5):425 433, 1989.
[7] D. H. Ashmead and R. S. Wall. Auditory perception walls via spectral variations in the ambient sound field, Journal of rehabilitation research and development, 36(4):313-322, October 1999. ISSN 0748-7711.
[8] D. H. Ashmead and R. S. Wall. Low frequency sound as navigational tool for people with visual impairments. Journal of Low Frequency Noise, Vibrution and Active Control, 21(4): 199-205, 2002. doi: 012111222000026202222222836
[9] E. J. Chamberlain, L. D. Rosenblum, and R. L. Robart. She hears seashells: Detection of small resonant cavities via ambient sound. The Journal of the Acoustical Society of America, 2006.
[10] K. G. Sabra, P. Gerstoft, P. Roux, W. Kuperman, and M. C. Fehler. Extracting time-domain green's 's function estimates from ambient seismic noise. Geophysical Research Letters, 32(3), 2005.
[11] Traer, Gerstoft, and W. Hodgkiss. Ocean bottom profiling with ambient noise: model for the passive fathometer. The Journal of the Acoustical Society of America, 18955):8225-1888 2011.
[12] M. Kac. Can one hear the shape of drum? The american mathematical monthly, 73(4P2): 1 1-23, 1966.
[13] S. Purushwalkam, S. V. Gari, V. K. Ithapu, C. Schissler, P. Robinson, Gupta, and K. Grauman. Audio-visual floorplan reconstruction. arXiv preprint arXiv:2012. 15470, 2020.
[14] F. Font, G. Roma, and X. Serra. Freesound technical demo. In ACM International Conference on Multimedia (MM'13) pages 411-412 Barcelona, Spain, 21/10/2013 2013. ACM, ACM. ISBN 978- 178-1-4503-2404-5 doi: 1e0.11145/25000 1.252550
[15] S. Thrun. Affine structure from sound. Advances in Neural Information Processing Systems, 18:1353-1366 1360, 2005.

---

[16] C. Gan, H. Zhao. P. Chen, D. Cox, and A. Torralba. Self-supervised moving vehicle tracking with stereo sound. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7053-7062, 2019.
[17] K. Yang, B. Russell, and J. Salamon. Telling left from right: Learning spatial correspondence of sight and sound. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9932-9941, 2020.
[18] Morgado, N. Vasconcelos. T. Langlois, and O. Wang. Self-supervised generation of spatial audio for 360 video. arXiv preprint arXiv:1809.088 028880900277,, 2018.
[19] B. Chen, M. Chiquier, H. Lipson, and C. Vondrick. The boombox: Visual reconstruction from acoustic vibrations. arXiv preprint arXiv: 10.:21005.00022. 2021.
[20] J. H. Christensen, S. Hormauer, and X. Y. Stella. Batvision: Learning to see 3d spatial layout with two ears. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 1581-1587. IEEE, 2020.
[21] R. Gao, C. Chen, Z. Al-Halah, Schissler, and K. Grauman. Visualechoes: Spatial image representation learning through echolocation. In European Conference on Computer Vision, pages
Springer, 2020.
[22] C. Chen, U. Jain, C. Schissler, S. V. A. Gari, Z. Al-Halah, V. K. Ithapu, P. Robinson, and K. Grauman. Soundspaces: Audio-visual navigation in 3d environments. In Proceedings of the European Conference on Computer Vision (ECCV). Springer, 2020.
[23] V. R. de Sa. Learning classification with unlabeled data. In Advances in neural information processing systems, pages 112-119. Citeseer, 1994.
[24] J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A. Y. Ng. Multimodal deep learning. In ICML., 2011.
[25] A. Owens, P. Isola, J. McDermott, A. Torralba, E. H. Adelson, and W. T. Freeman. Visually indicated sounds. In CVPR, 2016.
[26] R. Arandjelovic and A. Zisserman. Look, listen and learn. In Proceedings of the IEEE International Conference on Computer Vision, pages 609-617, 2017.
[27] A. Owens and A. A. Efros. Audio-visual scene analysis with self-supervised multisensory features. In ECCV, 2018.
[28] B. Korbar, D. Tran, and L. Torresani. Cooperative learning of audio and video models from self-supervised synchronization. arXiv preprint arXiv:18807.002330 2018.
[29] F. Xiao, Y. Lee, K. Grauman, J. Malik, and C. Feichtenhofer. Audiovisual slowfast networks for video recogmition. arXiv preprint arXiv:2001.087400, 2020.
[30] P. Morgado, N. Vasconcelos, and I. Misra. Audio-visual instance discrimination with cross- modal agreement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12475-12486, 2021.
[31] Y. M. Asano, M. Patrick, C. Rupprecht, and A. Vedaldi. Labelling unlabelled videos from scratch with multi-modal self-supervision. arXiv preprint arXiv:2006. 13662, 2020.
[32] Y. Aytar, C. Vondrick, and A. Torralba. Soundnet: Learning sound representations from unlabeled video. In Advances neural information processing systems, pages 892-900, 2016.
[33] A. Senocak, T.-H. Oh, J. Kim, M.-H. Yang, and I. So Kweon. Learning to localize sound source in visual scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4358-4366, 2018.
[34] R. Arandjelovic and A. Zisserman. Objects that sound. In Proceedings of the European Conference on Computer Vision (ECCV), pages 435-451, 2018
[35] D. Harwath, A. Recasens, D. Surís, G. Chuang, A. Torralba, and J. Glass, Jointly discovering visual objects and spoken words from raw sensory input. In Proceedings of the European conference computer vision (ECCV), pages 649-665, 2018.
10

---

[36] Y. Tian, J. Shi, B. Li, Z. Duan, and C. Xu. Audio-visual event localization in unconstrained videos. In Proceedings of the European Conference on Computer Vision (ECCV), pages 247-263, 2018.
[37] T. Afouras, A. Owens, S. Chung, and A. Zisserman. Self-supervised learning of audio-visual objects from video. arXiv preprint arXiv:2008. 04237, 2020.
[38] R. Qian, D. Hu, H. Dinkel, M. Wu, N. Xu, and W. Lin. Multiple sound sources localization from coarse to fine. arXiv preprint arXiv:2007.066355, 2020.
[39] D. Hu, R. Qian, M. Jiang, X. Tan, S. Wen, E. Ding, W. Lin, and D. Dou. Discriminative sounding objects localization via self-supervised audiovisual matching. Advances in Neural Information Processing Systems, 33, 2020.
[40] H. Chen, W. Xie, T. Afouras, A. Nagrani, A. Vedaldi, and A. Zisserman. Localizing visual sounds the hard way. arXiv preprint arXiv:2104.022 026911 2021.
[41] S. Chung and Zisserman. Out of time: automated lip syne in the wild. In Asian conference on computer vision, pages 25 1-263. Springer, 2016.
[42] S.-W. Chung, Chung, and H.-G. Kang. Perfect match: Improved cross-modal embeddings for audio-visual synchronisation. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3965-3969. IEEE, 2019.
[43] J. Roth, S. Chaudhuri, O. Klejch, R. Marvin, A. Gallagher, L.. Kaver, S. Ramaswamy, A. Stopezynski, C. Schmid, Z. Xi, et al. Ava active speaker: An audio-visual dataset for active speaker detection. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4492-4496. IEEE, 2020.
[44] R. Gao, R. Feris, and K. Grauman. Learning to separate object sounds by watching unlabeled video. In ECCV, 2018.
[45] H. Zhao, C. Gan, A. Rouditchenko, C. Vondrick, J. McDermott, and A. Torralba. The sound of pixels. In ECCV, 2018.
[46] H. Zhao, C. Gan, W-C. Ma, and A. Torralba. The sound of motions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1735-1744, 2019.
[47] R. Gao and K. Grauman. Visualvoice: Audio-visual speech separation with cross-modal consistency, arXiv preprint uxXiv:1001.33149 2021.
[48] D. Hoiem, A. A. Efros, and M. Hebert. Recovering surface layout from an image. International Journal of Computer Vision, 75(1):151-17, 2007.
[49] A. Saxena, M. Sun, and A. Ng. Make3d: Learning 3d scene structure from a single still image. IEEE transactions on pattern analysis and machine intelligence, (1(5) 2008.
[50] D. Eigen and R. Fergus. Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In ICCV, 2015.
[51] W. Chen, Z. Fu, D. Yang, and J. Deng. Single-image depth perception in the wild. In Advances in neural information processing systems, pages 730-738, 2016.
[52] W. Chen, S. Qian, and J. Deng. Learning single-image depth from videos using quality assessment networks. In CVPR, 2019.
[53] Z. Li and N. Snavely. Megadepth: Learning single-view depth prediction from internet photos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2041-2050, 2018
[54] C. Godard, O. Mac Aodha, M. Firman, and G. J. Brostow. Digging into self-supervised monocular depth estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3838-3838, 2019.
[55] S. Tulsiani, S. Gupta, D. F. Foubey, A. A. Efros, and J. Malik. Factoring shape, pose, and layout from the 2d image of a 3d scene. In CVPR, 2018
[56] G. Gkioxari, J. Malik, and J. Johnson. Mesh r-cnn. In ICCV, 2019.
11

---

[57] R. Girdhar, D. F. Foubey, M. Rodriguez, and A. Gupta. Learning a predictable and generative vector representation for objects. In ECCV. Springer, 2016.
[58] W. Yin, J. Zhang, O. Wang, S. Niklaus, L. Mai, S. Chen, and C. Shen. Learning to recover 3d scene shape from a single image. arXiv preprint arXiv:2012.0990 093665, 2020.
[59] R. Hartley and A. Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, ISBN: 0521540518, second edition, 2004.
[60] R. I. Hartley. In defense of the eight-poing algorithm. IEEE Transactions on pattern analysis and machine intelligence, 19(6):580-555 593, 1997.
[61] C. Wu, B. Clipp. X. Li, J.-M. Frahm, and M. Pollefeys. 3d model matching with viewpoint- invariant patches (vip). In 2008 IEEE Conference Computer Vision and Pattern Recognition, pages 1-8. JEEE, 2008.
[62] S. Qian, L. Jin, and D. F. Fouhey, Associative3d: Volumetric reconstruction from sparse views. In European Conference on Computer Vision, pages 140-157. Springer, 2020.
[63] R. Cai, B. Hariharan, N. Snavely, and H. Averbuch-Elor. Extreme rotation estimation using dense correlation volumes. arXiv preprint arXiv:2 13530, 2021.
[64] Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi. Target- driven visual navigation in indoor scenes using deep reinforcement learning. In 2017 IEEE international conference on robotics and automation (ICRA), pages 3357-3364. IEEE, 2017.
[65] Mirowski, R. Pascanu, F. Viola, H. Soyer. A. Ballard, A. Banino, M. Denil, Goroshin, L. Sifre, K. Kavukcuoglu, et al. Learning to navigate in complex environments. arXiv preprint arXiv: 161 1.03673, 2016.
[66] M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun,
J. Malik, et al. Habitat: A platform for embodied ai research. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 9339 9049999374444 2019.
[67] E. Wijmans, A. Kadian, A. Morcos, S. Lee, I. Essa D. Parikh, M. Savva, and D. Batra. Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames. arXiv preprint arXiv: 1xXiv:1911.0057 003557, 2019.
[68] C. Rascon and Meza. Localization of sound sources in robotics: A review. Robotics and Autonomous Systems, 96:184-210, 2017.
[69] I. An, M. Son, D. Manocha, and S.-E. Yoon. Reflection-aware sound source localization. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 66-73. IEEE, 2018.
[70] C. Gan. Y. Zhang, J. Wu, B. Gong, and J. B. Tenenbaum. Look, listen, and act: Towards audio-visual embodied navigation. arXiv preprint 1xXiv:192..1168. 2019.
[71] C. Chen, S. Majumder, Z. Al-Halah, R. Gao, S. K. Ramakrishnan, and K. Grauman. Learning to set waypoints for audio-visual navigation. arXiv preprint arXiv:2008.0999 09622, 1(2):, 2020.
[72] C. Chen, Z. AI-Halah, and K. Grauman. Semantic audio-visual navigation. arXiv preprint arXiv:2012.1 150.20022.//83,, 2020.
[73] S. E. Palmer. Vision science: Photons to phenomenology. MIT press, 1999.
[74] D. Gandhi, L. Pinto, and A. Gupta. Learning to fly by crashing. In 2017 IEEE/RSJ International IEEE, 2017.
Conference Intelligent Robots and Systems (IROS), pages
[75] S. Hershey, S. Chaudhuri, D. P. Ellis, J. F. Gemmeke, A. Jansen, R. C. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seybold, et al. Cnn architectures for large-scale audio classification. In 2017 ieee international conference on acoustics, speech and signal processing (icassp), pages 131-135. IEEE, 2017.
[76] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
12

---

[77] J. Deng, W. Dong, R. Socher, Li, Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 148-255. leee, 2009.
[78] Sutskever, J. Martens, Dahl, and G. Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pages 1139- 1147. PMLR, 2013.
[79] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. 2015.
[80] H. Chen, W. Xie, A. Vedaldi, and A. Zisserman. Vggsound: A large-scale audio-visua dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 721-725. IEEE, 2020.
[81] E. Fonseca, J. Pons Puig, X. Favory, F. Font Corbera, D. Bogdanov, A. Ferraro, S. Oramas, A. Porter, and Serra. Freesound datasets: platform for the creation of open audio datasets. In Hu X, Cunningham SJ, Turnbull D. Duan Z, editors. Proceedings of the 18th ISMIR Confer- ence; 2017 oct 23-27; Suzhou, Chiina/Canada); International Society for Music Information Retrieval; 2017. p. 486-93. International Society for Music Information Retrieval (ISMIR), 2017.
[82] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, t.. Kaiser, and I. Polosukhin. Attention is you need, In Advances neural information processing systems, pages 5998-6008, 2017.
13


Reviews:
[{"review_id": "owP3XOUEUp", "reviewer": "Reviewer_8xhP", "summary": "This work aims to detect obstacles (and walls) from ambient sounds in the environment.  A static dataset is collected for training under two conditions (heating and cooling) and predictions are made both based on human and robotic movements.  Models are evaluated both on absolute and relative depth predictions.  Audio based models are then compared against image based approaches (with and without pretraining) and jointly with them.  Representations are also tested via probe.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. Interesting application of acoustic information\n2. Construction of novel resource\n3. Demonstration of technique with both human and robotic agent\n4. Analysis of approach and comparison to and with other modalities\n\n**Weaknesses**\n1. Throughout the writing it's unclear where and why a human vs a robot is used.  Specifically, motor noise from the robot itself is not discussed and how this affects the predictions (and/or should be accounted for during training)\n2. As best I can tell, all experiments are done within rectangular classroom settings with nearly identical properties/furniture.  This is a worrying simplification.  The walking video does indicate that bags and other random objects may have been present but there's no discussion of their effects.\n3. There's a passing note about the sounds of others in the building but no analysis of their effect\n4. There's no intuition provided for the shape of the curve in Fig 8.\n5. Why in the video (and Fig 6) are predictions never as close to 0 as they are to 1? The gap to zero is (fig 6) 0.18 but the gap to 1 is 0.03.  This appears to be even more dramatic in the supplemental video.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This is a very nice (though simple) demonstration of the role of audio.  It does not show a very general formulation nor does it live up to the claims (e.g. refrigerator example) in the introduction. The results are however tested live by both a human and the robotic platform.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Expanded explanation and discussion of points raised in weaknesses.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {}}, {"review_id": "o2aPRJ3t_Gp", "reviewer": "Reviewer_13Bf", "summary": "* This paper shows that ambient sound conveys a surprising amount of information about scene structure. This information can be successfully exploited by learning systems.\n* Introduces the quiet campus dataset: \"in the wild\" audio recordings paired with RGB-D data from quiet indoor scenes.\n* Experimental study for depth estimation from audio.\n* Results showing that ambient sound can be used to infer depth-aware visual representations.\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n* Interesting motivation: humans use subtle spatial variations in the ambient sound spectrum to better perceive 3D scene structure. An important open question is whether this same information can be used by machines for perception or as a labeling source for self-supervised cross-modal learning.\n* Data collection. good variation in data collected that represent realistic scenarios. Static recordings, recordings under motion. Running experiments on cheap commodity hardware will help reproducibility.\n* Diverse set of depth estimation tasks with different motivations that are relevant to multiple robotic learning problems.\n* Good details on training visual-audio self-supervised representations that will be useful beyond this work.\n\n# Weaknesses\n* Claims of significance without providing statistically significant evidence. All experiments are presented as point estimates without standard deviation over multiple training runs. Are these numbers the maximum over a large hyperparameter sweep of a single model or the average expected performance? Without this information, it is difficult to substantiate whether the main claims of the paper are indeed true or if we are simply observing sampling error. Examples:\nL226: \"Interestingly, we see that the audio-based model performs significantly better than chance in both cases, suggesting that ambient sound, indeed, conveys information about scene structure\". \nL241: \"Once again, we find that the model obtains significantly better results than chance\"\nL262: \"As shown in Tab. 2, we can see that our method significantly outperforms chance and, interestingly, has a relatively small gap between the vision-based approach\" Hard to say \"significantly\" or \"relatively small gap\" without confidence intervals.\nL266\nL294: \"Our method outperforms baselines significantly in both audio and visual modalities suggesting that we learn useful depth representations. Moreover, our results are often close to supervised features\". This is a very strong statement to make on pretty weak evidence. For example, the point estimate success between pretrained Imagenet is 27.4 and your AV-Order success is 28.9. Is a 1.5% gap actually significant? It's difficult to say without confidence intervals. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This work motivates ambient sound as a rich source for self-supervised cross-modal learning. It presents early experiments that show this new kind of data can be exploited to help learn rich capabilities like depth-aware visual embeddings. Update: my early concerns have been addressed and I would like to raise my score.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Update: my early concerns have been addressed and I would like to raise my score.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "V0A03CF-JmS", "reviewer": "Reviewer_Vy16", "summary": "* The quiet campus dataset. Large number of in-the-wild indoor ambient audio recordings, paired with concurrently recorded RGB-D data. They collected the data in three modes: static recordings, static with dense coverage and motion recordings. \n* Demonstrated audio signals provide a better information for depth estimation over chance.\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Novel topic.\n* Well written and comprehensive examination of related work.\n* Thorough experimentation.\n* Well presented results. Particularly liked Figure 6.\n* The ablation study demonstrating that low frequency audio signals are the most useful for depth estimation is very interesting.\n\nWeaknesses:\n* The dataset was collected with the microphone and camera directed towards the closest wall. Since this is the input that is being used to train the models. I worry this is somehow leaking information into the model. This also means the models trained are wouldn't be able to identify depth for any audio signal in the wild, only the ones that were collected in this way.\n* The audio-visual representation learning section trains AV-Order and AV-Sync models by using audio + visual tasks. But then, it uses either audio or visual features for training a regression model for obstacle detection and relative order tasks. The audio only models have low performance when compared to the visual ones. The visual models do outperform the pretrained features from Imagenet but a baseline trained on the quiet dataset images only is missing to truly understand the utility of the audio signals.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The use of audio in robotics for the purpose of depth estimation and obstacle detection is an appealing idea. It could potentially lead to better ways to avoid collisions during training and deploying navigation policies. However, the paper doesn't argue a clear advantage of using audio signals over visual signals or even in addition to visual signals. Audio signals alone, although better than a random policy, fail to produce performing models (<65% accuracy). The paper doesn't clearly demonstrate the usefulness of using audio signals in addition to visual ones. It is difficult to asses the impact of learning scene structure through audio given that most robotics systems are equipped with a camera these days.\n\n========================\nUpdate after rebuttal:\nThe experiments show a clear advantage of using audio signals in addition to visual signals. The community would benefit from these results.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "* It is not clear whether the audio signal is useful in addition to the visual signal. Training a linear classifier on the depth estimation tasks using both visual and audio features would be a good experiment to evaluate this. \n\n* Train a model with the quiet campus dataset images only and depth estimation as a baseline. This will help discard differences in how the features were trained in ImageNet and also help understand the utility of audio signals.\n\n* It would be interesting to train the robotic navigation agent in the motion recordings. This would be more similar to the kind of data that can be more commonly collected in the wild.\n\n* Testing on a different distribution than training, i.e. training the robotic navigation in the motion recordings and evaluating in a static environment or viceversa.\n\n========================\nUpdate after rebuttal:\nAll issues have been addressed by the authors.\n"}}, {"review_id": "MwJiYCpNQam", "reviewer": "Reviewer_jRS5", "summary": "This paper explores the problem of deploying a system able to estimate depth from ambient sounds. The paper also shows how to combine ambient sound with image in a self-supervised learning task.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n*** Strengths:\n+ The idea seems original\n\n+ Experimental results are thorough\n\n+ A good dataset is provided.\n\n\n** Weaknesses: See Issues", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The idea seems original and the paper provides numerous experiments together with a dataset that can be useful to other researchers. Therefore, I initially suggest acceptance.\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {"issues": "- The paper contains too much information and details and sometimes it is difficult to follow. The template for this conference does not provide much space so too many small subsections are not a good idea because the information is scattered. Several sections can be merged, for example, subsection \"Evaluating the representation.\" into \"Experiments,\" section \"4.1\" into \"Experiments\", etc.\n\n- Experimental results are sometimes unclear with respect to the input: audio, image, both? For example, Tables 4 and 5 separate the audio from the video but what I understood from the paper is that both are combined in the input as shown in Figure 4. Moreover, the link/relation between section 4.2 and section 5.2 is somehow unclear to me.\n\n- When estimating the depth in Tables 1 and 2, it makes sense to compare with random but I do not see the point of comparing with images alone. The paper is about sound. Is vision the baseline?\n\n- In section 5.4, authors can add a comparison to a robot that uses images to estimate depth.  \n\nTwo final thoughts: \n- I like the paper as a proof of concept, and I am curious to find some potential direct application of this system. Any comments on this?\n- What happens when the environment contains other non-ambient sounds? \n"}}]

Author rebuttal:

Meta-review: The reviewers generally agree, that the use of acoustics for robot navigation is an interesting concept.
However, the experimental evaluation of the approach is rudimentary and arguably only suffice as a proof of concept.

The reviewers voiced concerns about both, the experimental setup itself and the claims based on the results.
E.g.:
- microphone and camera directed towards the closest wall
- low performance of audio only models
- qualitative statements without reported confidence intervals
- high similarity between spatial layouts of the experiments

The paper should address the concerns of the reviewers wrt the experimental evaluation and add clarifications in the experimental sections.
Furthermore the paper should discuss potential limitations of the work.

## Post-Rebuttal Update:
The authors did a great job clarifying and improving the paper through the rebuttal process.
All major concerns have been addressed.
Decision: Accept (Oral)