Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper presents audio-visual object classification and motion prediction work on a novel dataset of 60 different objects rolling around in a bin tilted to and fro by a robot, with video and 4-channel audio recordings of the object impacts.   The data is rather novel, is large enough to do ML (around 17 hours of eventful audio/video) and is to be publicly released.  The model architectures  are not of theoretical novelty.   However, the experiments are somewhat interesting.  It was found that the audio contains significant object classification information.  The audio was also good for predicting the trajectory of the object.  This might not be surprising since the microphones are geometrically arranged and may contain directional information along with information about velocity and/or distance traveled.   Overall the experiments are rather thin with only a few experimental results.   A more thorough undertaking might be expected for ICLR papers, with more novel theoretical development and more extensive experiments.  
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper studies the role of audio in object and action perception, as well as how auditory information can help learning forward and inverse dynamics models. To do this, the authors built a 'tilt-bot', which tilts a box and the object within to collect data (sound & vision) of object interactions. The authors then tested how audio embeddings help object recognition and forward model prediction. 

The idea presented in this paper is quite interesting. However, there are no significant technical innovations, the experimental evaluations are quite limited, and the writing can be improved. My overall recommendation is weak reject.

The problem of integrating audio for perception is interesting and has been quite widely explored; however, this paper extends the setup to also explore the effect of audios on dynamics modeling. This is relatively new and may lead to many potential future developments in this direction.

Technically, however, this paper mostly builds on existing technicals on learning forward and inverse models, except that the input is now audio in addition to video. The experimental results are also very limited. They are restricted to a single domain, a fixed collection of objects, and there are no comparisons with published, SOTA algorithms. There are also no results on downstream tasks such as robot manipulation.

I also wonder how the authors think of the related work from Zhang et al: http://sound.csail.mit.edu/ , as they've also studied the effect of auditory and visual data in shape and material recognition.

The writing can be improved. Currently, the model and results are in the same section and mixed together. It'd prefer to separate them. There are a number of typos (incorrect spacing, etc.), especially in Section 3.4. Please double check.

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper explores the interesting connections between action and sound, by building a sound-action-vision dataset with a tilt-bot. This is a good paper overall, I appreciate the efforts on the dataset, and this direction of research is worth pursuing. 

Regarding experiments, I like the way it is set up, especially the four microphones, and the action space of the robot. I

A couple of questions:
(1) In the inverse model learning, Fig 3(a) bottom, why are images used as input as well? Don't we want to predict action purely from sound?
(2) In forward model prediction, how are the ground truth locations defined and labeled? Is it the center of mass, and annotated by humans? More details on this experiment will help.

review_title: Official Blind Review #2

Meta-review: This paper investigates using sound to improve classification, motion prediction, and representation learning all from data generated by a real robot.

All the reviewers were intrigued by the work. The paper provides experiments on real robots (never a small task), and a data-set for the community, and a sequence of illustrative experiments. Because the paper combines existing techniques, its main contribution is the empirical demonstrations of the utility of using sound. Overall, it was not quite enough for the reviewers. The main issues were: (1) motion prediction is perhaps expected given the physical setup, (2) lack of comparison with other approaches, (3) lack of diversity in the demonstrations (10 objects, one domain).

The authors added two new experiments with a different setup, further demonstrating their claims. In addition the authors highlighted that the novelty of this task means there are no clear baselines (to which r3 agreed). The new experiments are briefly described in the response (and visuals on a website), but the authors did not update the paper. The new experiments could potentially significantly strength the paper. However, the terse description in the response and the supplied visuals made it difficult for the reviewers to judge their contribution.

Overall, this is certainly a very interesting direction. The results on real world data demonstrate promise, even if they are not the benchmarking style the community is used too.   
Decision: Reject