Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: * The quiet campus dataset. Large number of in-the-wild indoor ambient audio recordings, paired with concurrently recorded RGB-D data. They collected the data in three modes: static recordings, static with dense coverage and motion recordings. 
* Demonstrated audio signals provide a better information for depth estimation over chance.

strengths_and_weaknesses: Strengths:
* Novel topic.
* Well written and comprehensive examination of related work.
* Thorough experimentation.
* Well presented results. Particularly liked Figure 6.
* The ablation study demonstrating that low frequency audio signals are the most useful for depth estimation is very interesting.

Weaknesses:
* The dataset was collected with the microphone and camera directed towards the closest wall. Since this is the input that is being used to train the models. I worry this is somehow leaking information into the model. This also means the models trained are wouldn't be able to identify depth for any audio signal in the wild, only the ones that were collected in this way.
* The audio-visual representation learning section trains AV-Order and AV-Sync models by using audio + visual tasks. But then, it uses either audio or visual features for training a regression model for obstacle detection and relative order tasks. The audio only models have low performance when compared to the visual ones. The visual models do outperform the pretrained features from Imagenet but a baseline trained on the quiet dataset images only is missing to truly understand the utility of the audio signals.

summary_of_recommendation: The use of audio in robotics for the purpose of depth estimation and obstacle detection is an appealing idea. It could potentially lead to better ways to avoid collisions during training and deploying navigation policies. However, the paper doesn't argue a clear advantage of using audio signals over visual signals or even in addition to visual signals. Audio signals alone, although better than a random policy, fail to produce performing models (<65% accuracy). The paper doesn't clearly demonstrate the usefulness of using audio signals in addition to visual ones. It is difficult to asses the impact of learning scene structure through audio given that most robotics systems are equipped with a camera these days.

========================
Update after rebuttal:
The experiments show a clear advantage of using audio signals in addition to visual signals. The community would benefit from these results.
issues: * It is not clear whether the audio signal is useful in addition to the visual signal. Training a linear classifier on the depth estimation tasks using both visual and audio features would be a good experiment to evaluate this. 

* Train a model with the quiet campus dataset images only and depth estimation as a baseline. This will help discard differences in how the features were trained in ImageNet and also help understand the utility of audio signals.

* It would be interesting to train the robotic navigation agent in the motion recordings. This would be more similar to the kind of data that can be more commonly collected in the wild.

* Testing on a different distribution than training, i.e. training the robotic navigation in the motion recordings and evaluating in a static environment or viceversa.

========================
Update after rebuttal:
All issues have been addressed by the authors.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work aims to detect obstacles (and walls) from ambient sounds in the environment.  A static dataset is collected for training under two conditions (heating and cooling) and predictions are made both based on human and robotic movements.  Models are evaluated both on absolute and relative depth predictions.  Audio based models are then compared against image based approaches (with and without pretraining) and jointly with them.  Representations are also tested via probe.
strengths_and_weaknesses: **Strengths**
1. Interesting application of acoustic information
2. Construction of novel resource
3. Demonstration of technique with both human and robotic agent
4. Analysis of approach and comparison to and with other modalities

**Weaknesses**
1. Throughout the writing it's unclear where and why a human vs a robot is used.  Specifically, motor noise from the robot itself is not discussed and how this affects the predictions (and/or should be accounted for during training)
2. As best I can tell, all experiments are done within rectangular classroom settings with nearly identical properties/furniture.  This is a worrying simplification.  The walking video does indicate that bags and other random objects may have been present but there's no discussion of their effects.
3. There's a passing note about the sounds of others in the building but no analysis of their effect
4. There's no intuition provided for the shape of the curve in Fig 8.
5. Why in the video (and Fig 6) are predictions never as close to 0 as they are to 1? The gap to zero is (fig 6) 0.18 but the gap to 1 is 0.03.  This appears to be even more dramatic in the supplemental video.
summary_of_recommendation: This is a very nice (though simple) demonstration of the role of audio.  It does not show a very general formulation nor does it live up to the claims (e.g. refrigerator example) in the introduction. The results are however tested live by both a human and the robotic platform.
issues: Expanded explanation and discussion of points raised in weaknesses.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: * This paper shows that ambient sound conveys a surprising amount of information about scene structure. This information can be successfully exploited by learning systems.
* Introduces the quiet campus dataset: "in the wild" audio recordings paired with RGB-D data from quiet indoor scenes.
* Experimental study for depth estimation from audio.
* Results showing that ambient sound can be used to infer depth-aware visual representations.

strengths_and_weaknesses: # Strengths
* Interesting motivation: humans use subtle spatial variations in the ambient sound spectrum to better perceive 3D scene structure. An important open question is whether this same information can be used by machines for perception or as a labeling source for self-supervised cross-modal learning.
* Data collection. good variation in data collected that represent realistic scenarios. Static recordings, recordings under motion. Running experiments on cheap commodity hardware will help reproducibility.
* Diverse set of depth estimation tasks with different motivations that are relevant to multiple robotic learning problems.
* Good details on training visual-audio self-supervised representations that will be useful beyond this work.

# Weaknesses
* Claims of significance without providing statistically significant evidence. All experiments are presented as point estimates without standard deviation over multiple training runs. Are these numbers the maximum over a large hyperparameter sweep of a single model or the average expected performance? Without this information, it is difficult to substantiate whether the main claims of the paper are indeed true or if we are simply observing sampling error. Examples:
L226: "Interestingly, we see that the audio-based model performs significantly better than chance in both cases, suggesting that ambient sound, indeed, conveys information about scene structure". 
L241: "Once again, we find that the model obtains significantly better results than chance"
L262: "As shown in Tab. 2, we can see that our method significantly outperforms chance and, interestingly, has a relatively small gap between the vision-based approach" Hard to say "significantly" or "relatively small gap" without confidence intervals.
L266
L294: "Our method outperforms baselines significantly in both audio and visual modalities suggesting that we learn useful depth representations. Moreover, our results are often close to supervised features". This is a very strong statement to make on pretty weak evidence. For example, the point estimate success between pretrained Imagenet is 27.4 and your AV-Order success is 28.9. Is a 1.5% gap actually significant? It's difficult to say without confidence intervals. 

summary_of_recommendation: This work motivates ambient sound as a rich source for self-supervised cross-modal learning. It presents early experiments that show this new kind of data can be exploited to help learn rich capabilities like depth-aware visual embeddings. Update: my early concerns have been addressed and I would like to raise my score.
issues: Update: my early concerns have been addressed and I would like to raise my score.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper explores the problem of deploying a system able to estimate depth from ambient sounds. The paper also shows how to combine ambient sound with image in a self-supervised learning task.
strengths_and_weaknesses: 
*** Strengths:
+ The idea seems original

+ Experimental results are thorough

+ A good dataset is provided.


** Weaknesses: See Issues
summary_of_recommendation: The idea seems original and the paper provides numerous experiments together with a dataset that can be useful to other researchers. Therefore, I initially suggest acceptance.

issues: - The paper contains too much information and details and sometimes it is difficult to follow. The template for this conference does not provide much space so too many small subsections are not a good idea because the information is scattered. Several sections can be merged, for example, subsection "Evaluating the representation." into "Experiments," section "4.1" into "Experiments", etc.

- Experimental results are sometimes unclear with respect to the input: audio, image, both? For example, Tables 4 and 5 separate the audio from the video but what I understood from the paper is that both are combined in the input as shown in Figure 4. Moreover, the link/relation between section 4.2 and section 5.2 is somehow unclear to me.

- When estimating the depth in Tables 1 and 2, it makes sense to compare with random but I do not see the point of comparing with images alone. The paper is about sound. Is vision the baseline?

- In section 5.4, authors can add a comparison to a robot that uses images to estimate depth.  

Two final thoughts: 
- I like the paper as a proof of concept, and I am curious to find some potential direct application of this system. Any comments on this?
- What happens when the environment contains other non-ambient sounds?
review_title: null

Meta-review: The reviewers generally agree, that the use of acoustics for robot navigation is an interesting concept.
However, the experimental evaluation of the approach is rudimentary and arguably only suffice as a proof of concept.

The reviewers voiced concerns about both, the experimental setup itself and the claims based on the results.
E.g.:
- microphone and camera directed towards the closest wall
- low performance of audio only models
- qualitative statements without reported confidence intervals
- high similarity between spatial layouts of the experiments

The paper should address the concerns of the reviewers wrt the experimental evaluation and add clarifications in the experimental sections.
Furthermore the paper should discuss potential limitations of the work.

## Post-Rebuttal Update:
The authors did a great job clarifying and improving the paper through the rebuttal process.
All major concerns have been addressed.
Decision: Accept (Oral)