Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper predicts the rgb and depth of an object in a bin given the audio signals from contact microphones attached to the bin. The audio signals capture the sounds/vibrations caused by the object when it’s dropped into the bin.
strengths_and_weaknesses: The paper’s motivation is very promising and could lead to better use of audio for robotic manipulation. However, there are several concerns with the paper which are listed below:

1. The Localization error metric only counts the number of times the two bounding boxes overlap by a certain distance. Adding an additional metric that computes the distance between the two bounding boxes will be useful. Specifically capturing  both position accuracy and orientation accuracy. Without a measure of accuracy in meters/radians, it’s not clear how accurate the method will be for use in robot manipulation. 

2. How will the predictions scale to more objects? E.g., Can the method differentiate between a ball and a cube where the ball could move around a lot before settling? Can the method differentiate between sharp edged objects (star?) and soft edged objects?

3. It’s unclear if the accuracy of the predictions are sufficient for manipulation tasks. Using the prediction in a downstream task and evaluating task success across the different methods is needed.

4. Can these predictions be generated online? This is especially important for feedback control.

summary_of_recommendation: Mentioned weaknesses need to be addressed to show the benefit of the proposed approach.
issues: To reiterate, some things to be done to improve the paper:
1. Needs experiments across more objects. 
2. Show application in a downstream task such as moving the object to the center of the bin.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents a fun idea: predicting the content of a container attached from 4 low-cost microphones. The vibrations from dropping objects into a container are fed to a convolutional network that predicts the visual reconstruction of the scene (RGB-D output). There's nothings exciting going on in the method, but the application domain is very interesting and seems very much unexplored in the deep-learning robotics era. 
strengths_and_weaknesses: **Strengths:**
- The paper is well written and the problem statement is made clear.
- The method is simple and elegant, and deserves to be built upon.
- The ablation and feature visualisation is interesting and informative.

**Weaknesses**
- The key for an application heavy paper like this, is a solid set of results that proves the claim that this is useful in the wider robotics community for a variety of applications. However, the paper only presents results on 3 wooden blocks. This makes it very difficult to assess how well this method would generalise to a larger number of objects, or if the method would have the capability to generalise to unseen test data given a large diverse training set of objects. 
- Section 5.3 is perhaps a little misleading: initially it seems that the paper is presenting results to suggest that they are evaluating on an unknown shape, which (perhaps subjectively) would indicate that the object was not in the training set, however, what they simply mean is that the model is train simultaneously on the 3 shapes, rather than individually training a network for each shape (as in Section 5.4: Reconstruction with known shape).
- The lack of thorough experimentation suggests to me that either the paper did not have enough time to run additional experiments (due to conference deadline) or that additional, challenging setups were run but the method was not successful. Hopefully the former, and I hope the authors use the time after CoRL to continue additional experiments. There are several un answered questions: What about multiple objects in the bin? What if there is already and object in the bin? What is the generalisation capability across unknown objects? What about across containers?
summary_of_recommendation: In summary, I very much like the paper, and the idea is promising, however the combination of poor technical novelty and poor experimental analysis makes this paper difficult to accept in its current form. I urge the authors to continue working on this line of work and resubmit to a new venue after evaluating the method on a larger dataset (of objects and containers) and assessing generalisation ability atleast across many objects, but also, if possible, across containers. 
issues: The method is great, but it is severally lacking additional experiments; see 'weaknesses' section for suggested next steps.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper introduces a hardware setup called Boombox and an algorithm that can predict visual signals from acoustic vibrations.
Specifically, the box is equipped with four contact microphones, one on each side, to provide audio signals. The learning algorithm is trained to predict the RGB or depth image of the final scene of the inside of the box after a wooden object gets dropped into it.
This system is shown to work more reliably than the alternative methods considered, qualitatively and quantitatively under the two metrics IOU and localization. The three alternative methods are time difference of arrival (TDoA) using Generalized Cross Correlation with Phase Transform(GCC-PHAT), random sampling from the training set, average bounding box from the training set.
strengths_and_weaknesses: Main strengths:
1. The paper is well written
2. The authors built a system that predicts the 3D visual scene from audio signals and showed that it worked well under the setting described 
3. Ablation studies were thorough: flipped microphones, removed amplitude, temporal shift. The learned features were also shown to have learned spatial information.

Weaknesses / Suggestions:
1. It’s not clear to me that this type of approach would scale under the situation of object collisions, more realistic object types:
a. The objects that were included in the experiments only differed in shape. They shared the same color and materials. 
b. The paper only considered dropping one object at a time into an empty box. It did not discuss whether this approach can be adapted to handle more complex settings when there're collisions between objects.
While the above two points might be too much to ask in a pilot study, I was hoping that they would be discussed in the paper. If the authors consider that to be future works, I would suggest discussing it in the paper as part of the limitation of the method.

2. I feel that there may be another baseline that this paper did not experiment with. In the baseline method “random sampling”, one image from the training set is chosen as the prediction of the test audio sample. One could imagine a nearest neighbor baseline that finds the nearest neighbor match from the training set of the test audio. This new baseline would indicate the similarity between the train and test distributions.

3. Background subtraction seems to be an important step in computing both evaluation metrics since the error in the background subtraction output would affect the quality of downstream tasks (during test time). Yet, neither the method of background subtraction nor its impact was described.

summary_of_recommendation: The task of using sound to help to estimate object state is interesting. 
The paper describes a system that works in relatively simple settings where a single object is dropped into an empty bin. 
It's well written and the method shows some real-world success. 
Hence my overall impression of the paper is positive. But I hold some concerns about the practicality of the approach. In real-world tasks where there are a larger variety of objects and with the addition of object-object collision, using acoustic vibration alone to estimate the object state seems difficult.  
issues: Please refer to the three items discussed in the weaknesses section.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper proposes a boombox, a container that can estimate the location of objects thrown into the box from an array of microphones. The audio signal is processed by a convolutional network to reconstruct the RGB and depth image of the objects from a bird's eye view. The approach is validated over multiple setups and shown to outperform traditional audio processing methods.
strengths_and_weaknesses: Overall, I find this paper to be very interesting. The idea of reconstructing the visual signal from only audio is very intriguing and novel (to the best of my knowledge), and I am surprised that it works so well. Much related work on audio-visual processing exists (as pointed out in the related work), but I think this work stands out from existing methods from both the problem setup and the results. Along these lines, it would have been great to read a discussion in related work about other methods of non-line-of-sight shape reconstruction, e.g. [1] or [2]. 

The approach is clearly presented and well-executed, with in-depth experimental analysis to understand how the approach compares to traditional methods and what features in the audio signal are used for reconstruction.

However, there are still some issues and points to clarify to make the contribution even stronger. 
From a methodological point of view, it is unclear why learning an RGB image from audio. Clearly, no color information is available in the audio, and the results are in "color" just because of the overfit to the datasets' color. What can be done to solve this issue (and possibly favor generalization to other boxes/objects)? I would assume that it would be much easier to learn an "intensity-free" image (e.g., depth) or some low-level feature (e.g., the x-y coordinate of the object or the object segmentation) instead of an RGB or grayscale image. Along these lines, I am very surprised that reconstructing a depth works consistently worse than the RGB. Why is this the case? What are the challenges that are present for depth but absent in RGB? Therefore, I would be very interested in seeing a more detailed discussion about the topic and possibly seeing some quantitative results of directly regressing the object segmentation and bounding box from the audio signal. In addition, it would be nice to see how other state-of-the-art architectures for audio processing, e.g., wavenet and its more recent successors, perform compared to the simple proposed encoder-decoder architecture.

Finally, I think that the presentation could be improved in some places. More specifically:
1. The discussion in the intro on why visual sensors may fail reads a bit weak. I believe that "occlusions" in the box could also be a problem for the proposed method (since a completely different audio signal would be generated). Also, motion blur does not seem relevant to the task since one would always wait for the objects to land in a stable position. Maybe some stronger motivation can be found (e.g., redundancy with a vision sensor?)
2. Some figures (e.g., Fig 6 and 10) are tiny and difficult to read. 
3. Paragraph 5.3 (reconstruction with unknown shape) is a bit deceiving. The experiment does not show what happens with shapes that are outside the training dataset. I would therefore recommend changing the title of the section to "Reconstruction with mixed shapes" or something along these lines. That being said, it would actually be nice to see some experiments where reconstruction of previously unseen shapes is studied (I guess it would not work, but to which extent?)
4. The "Random" baseline is very weak and uninformative. I would substitute it with a stronger one that returns the label associated to the nearest neighbor (according to some simple metric) of the training dataset.
5. There is no discussion about the limitations of the approach. I think this is fundamental to enable researchers to follow up the work.



[1] A Theory of Fermat Paths for Non-Line-of-Sight Shape Reconstruction, Xin et al.
[2] Acoustic Non-Line-of-Sight Imaging, Lindell, et al.
summary_of_recommendation: I mainly base my recommendation on the novelty of the idea and the interesting results.
issues: I think that during the rebuttal period some simple experiments could be done to make the contribution stronger. Specifically:
1. Predict directly the object location or segmentation instead of the depth or rgb image.
2. Include the nearest neighbor baseline explained above.
In addition, it would be nice to see the comments on the presentation to be included.
review_title: null

Meta-review: This paper presents a new hardware setup for estimating the shapes and poses of objects inside a bin using only auditory signals. The objects inside the bin are bounced and the sounds resulting from their collisions are captured by four microphones. The signals from the microphones are used as inputs to a convolutional neural network that returns an RGB image of the objects inside the bin. The proposed method is evaluated on scenes containing single wooden cubic objects. The experimental results clearly indicate that the proposed method is able to reconstruct the interior of the bin. 
The reviewers agree that the proposed method is interesting and novel. The reviewers find that the main contribution of this work is the application, rather than the technical approach. Therefore, the paper should include a richer and more exhaustive set of experiments. The current ones are limited to single objects that have simple regular shapes. The reviewers are not sure why the paper does not include experiments on realistic objects, or multiple objects in the bin. The evaluation metrics can also be improved. Finally, stronger baselines should be considered. For example, the authors can try simple nearest-neighbors methods, or traditional shallow learning techniques. 

In their rebuttal, the authors added new experiments and addressed several concerns of the reviewers and the AC. From the discussion, it is clear that the reviewers appreciate the novelty of this application. However, concerns regarding the generalization and practicality of the proposed technique remain. Therefore, the reviewers are leaning toward a weak accept. In conclusion, this is a good paper that deals with a challenging problem where it is difficult to collect more data. The area chair believes that this is a promising research direction, and this paper may open the door to more elaborate works in this new domain. 
Decision: Accept (Poster)