The Boombox: Visual Reconstruction from Acoustic Vibrations
Authors: Boyuan Chen
Mia Chiquier
Hod Lipson
Carl Vondrick
Venue: CoRL2021 Poster (2021)
Keywords: Multimodal Perception
Object State Estimation
Audio

Abstract: Interacting with bins and containers is a fundamental task in robotics, making state estimation of the objects inside the bin critical. 
While robots often use cameras for state estimation, the visual modality is not always ideal due to occlusions and poor illumination. We introduce The Boombox, a container that uses sound to estimate the state of the contents inside a box. Based on the observation that the collision between objects and its containers will cause an acoustic vibration, we present a convolutional network for learning to reconstruct visual scenes. Although we use low-cost and low-power contact microphones to detect the vibrations, our results show that learning from multimodal data enables state estimation from affordable audio sensors. Due to the many ways that robots use containers, we believe the box will have a number of applications in robotics.

The Boombox: Visual Reconstruction from Acoustic Vibrations
Boyuan Chen, Mia Chiquier, Hod Lipson, Carl Vondrick Columbia University boombox. cs. cs.cooubiaa edu
Abstract: Interacting with bins and containers is fundamental task in robotics, making state estimation of the objects inside the bin critical. While robots often use cameras for state estimation, the visual modality is not always ideal due to occlusions and poor illumination. We introduce The Boombox, container that uses sound to estimate the state of the contents inside a box. Based on the obser- vation that the collision between objects and its containers will cause acoustic vibration, we present convolutional network for learning to reconstruct visual scenes. Although we use low-cost and low-power contact microphones to detect the vibrations, our results show that learning from multimodal data enables state estimation from affordable audio sensors. Due to the many ways that robots use containers, we believe the box will have a number of applications in robotics.
Keywords: Multimodal Perception, Object State Estimation, Audio
Introduction
In order for robots to robustly grasp and pick up objects inside containers, they need to accurately lo- calize and estimate the state of the objects inside the container. Vision-based perception has enabled numerous advances for state estimation [1, 2, 3] in object manipulation and grasping, such as pack- age bin-picking and object retrieval in household settings. Due to the importance of this problem, pose and state estimation from vision has been long studied in robot vision [4, 5, 6, 7, 8, 9].
However, environmental conditions are not always ideal for cameras. Although robots frequently need to interact with containers, objects inside containers often become occluded from cameras, making state estimation from vision impractical. In unconstrained settings, environments can also lack ideal lighting conditions, with limited visual signals to indicate the location and state of ob- jects. Moreover, camera systems require extensive calibration for 3D state estimation, which often fragile during contact and collisions.
In this paper, we propose robotics. Unlike vision, sound remains robust during occlusions or poor illuminations. For example, when an object is dropped inside a container, it may not be visible to any camera, but the collision between the object and the bin will cause sound that can be easily picked up by a contact microphone. The exact incidental vibration will depend on the location and pose of the objects inside the bin. We demonstrate how to use this audio signal to reconstruct the objects' states inside containers.
sound another sensing modality for object state estimation in
We introduce The Boombox, smart container that uses the vibration of itself reconstruct an image of its contents. The box is no larger than a cubic square foot. Unlike most containers, the box contact microphones to detect its vibration. Exploiting the link between acoustic and visual structure, we show that a convolutional network can use these vibrations to predict the visual scene inside the container within centimeters, even under total occlusion and poor illumination. Figure illustrates our box and one reconstruction from the sound.
The main contribution of this paper is an integrated hardware and software platform for using sound to reconstruct the visual structure inside containers. The remainder of this paper will describe The Boombox in detail. In section 2, first review background this topic. In section 3, introduce our perception hardware, and in section 4, we describe our learning model. Finally, in section 5, we
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

Figure The Boombox. We introduce "smart" container that is able to reconstruct an image of its inside contents. The approach works even when the camera cannot see into the container. The box has four contact mi- crophones each face. When objects interact with the box, they cause incidental acoustic vi- brations. From these vibrations, we learn to predict the visual scene inside the box.
Mic
Mic
3D Reconstruction from Vibration
Mic
Mic 4
quantitatively and qualitatively analyze the performance and capabilities of our approach. We will open-source all hardware designs, software, models, and data.
2
Related Works
Vision and Sound. Our paper builds on work that integrates sound and visual perception. Given visual observations, there has been extensive studies enhance sounds [10, 11], fill in missing sounds [12], and generate sounds entirely from video [13, 14]. Further, there have been recent works in integrating vision and sound to improve recognition of environmental properties [15, 16] and object properties, such as geometry and materials [17, 18]. Audiovisual data have also been studied for representation learning [19, 20]. Lastly, there has been work for generating face given voice [21] and scene from ambient sound [22]. In contrast, our work uses sound to predict the 3D visual structure inside a container.
Sound in Robotics, Most related research to our work is on audiovisual object understanding for robot perception and control. Gandhi et al. [23] investigates vision and sound in a robot setting to predict which robot actions caused the given sound. Matl et al. [24] leverages audio signals from ball bouncing motions to calibrate the stochastic dynamical events for "sim2real" tasks. Bu and Huang [25] predicts the trajectory of a falling cube for robot object retrieval outside visible region. Instead of predicting the 2D trajectory for single cube, this paper learns predict the entire visual scene under fully occluded conditions by outputting RGB and depth images three different blocks.
Audio Analysis. The primary features in audio that used for sound localization [26] time difference arrival and level (amplitude) difference. Specifically calculating these exact features is non-trivial, especially in situations where the signal is not broad-band and in motion [27, 28, 29, 30]. Furthermore, these rough approximations can only be used to localize the object, whereas our goal only localize objects, but also predict the 3D structure, which includes shape and orientation of the object. As such, we develop model that learns the necessary features for reconstruction.
3 The Boombox
In this section, we present the The Boombox hardware. We also discuss the characteristics of the acoustic signals captured by it.
3.1 Detecting Vibrations
The Boombox, shown in Figure 2A, is plastic storage container that is 15.5cm 26cm x 13cm (width x length x height) with an open top. The box is a standard object that one can buy at any local hardware store. When object collides with the box, small acoustic vibration will be produced in both the air and the solid box itself. We have attached contact microphones on each wall of the plastic cuboid storage bin in order to detect this vibration. Unlike air microphones, contact microphones are insensitive to the vibrations in the air (which human ears hear as sound). Instead, they detect the vibration of solid objects.
The microphones are attached on the outer side of the walls, resulting in four audio channels. We arrange the microphones roughly at the horizontal center of each wall and close to the bottom. As

---

Acoustic (lnpul))
Stable (Oulput)
Top cown cament
Mix1
Mice
Mics
Miol
RGB
Dupdi
Sarble
Contact Micrpphnnee (Fach bace)
Sarele 2
Sampls 3
(8) Objects:
(C) samples
Figure 2: The Boombox Overview. (A) The Boombox can sense the object through four contact microphones each side of a storage container. A top-down RGB-D camera is used to collect the final stabilized scene after the object movements. (B) We drop three wooden objects with different shapes. (C) Input and output data visualizations.
our approach will not require calibration, the microphone displacements can be approximate, We used TraderPlus piezo contact microphones, which are very affordable (no more than $5 each).
3.2 Vibration Characteristics
When objects collide with the box, the contact microphones will capture the resulting vibrations. Figure 3 shows an example of the vibration captured from two of the microphones. We aim to recover the visual structure from this signal. As these vibrations are independent of the visual conditions, they allow perception despite occlusion and poor illumination. There is rich structure in the raw acoustic signal. For example, the human auditory system uses inter-aural timing difference (ITD), which is the time difference of arrival between both ears. Humans also locate sounds with inter-aural level difference (ILD), which is the amplitude level difference between both ears [31].
However, in our settings, extracting these characteristics is challenging. In practice, objects will bounce around in the container before arriving at their stable position, as shown in Figure 4. Each bounce will produce another, potentially interfering vibration. In our attempts to analytically use this signal, we found that the third bounce has the best signal for the time difference of arrival, but as can be seen from Figure 3, even on the third bounce the time difference of arrival is unclear in the actual waveform. There are a multitude of factors that make analytical approaches not robust to our real-world signals. Firstly, we are working with a moving signal, whereas time difference of arrival calculations work best on stationary signals due the fact that it compares the time taken for signal to travel from a fixed location. This makes it very difficult to analytically segment the signal into chunks of roughly the same location. Secondly, there are echos that make non-learning based methods difficult to identify phase shifts as the environment is a small container. Finally, the fact that the microphones are close together means that the time difference of arrival is encompassed in few samples, thus making it susceptible to noise.
Instead of hand crafting features, we will train a model identify the fraction of the signal that is most robust for final localization. Our model will learn to identify the useful features from the signals reconstruct a 3D scene, which includes the shape, orientation, and position of the contents.
3.3 Multimodal Training Dataset
Since vision and sound are naturally synchronized, we will use vision as self-supervision to learn robust characteristic features of the acoustic signal. We collected a multimodal training dataset by dropping objects into the box and capturing resulting images and vibrations. We position an Intel RealSense D435i camera that looks inside the bin to capture both RGB and depth images, which
We found that these microphones gave sufficiently clear signals while being more affordable than avail- able directional microphone arrays. Each microphone was connected to a laptop through audio jack to USB converter. We use GarageBand software record all four microphones together synchronize the recordings.

---

Thire
Figure 3: Vibration Characteristics. We visu- alize vibration captured by two microphones in the box. Several distinctive characteristics need to be combined over time in order ac- curately reconstruct an image with the right po- sition, orientation, and shape of objects.
Figure 4: Chaotic Trajectories. We show the chaotic trajectories of objects they bounce around the box until becoming stable. The mov- ing sound source and multiple bounces also cre- ate potentially interfering vibrations, complicat- ing the real-world audio signal.
only use during training.² 2 We use three wooden blocks with different shapes to create our dataset. The blocks have the same color and materials, and show these objects in Figure 2B. We hold the object above the bin, and freely drop it. After dropping, the objects bounce around in the box a few times before settling into resting position. We record the full process from all the microphones and the top-down camera. Overall, our collection process results in diverse falling trajectories across all shapes with a total of 1,575 sequences. Figure 2C shows an overview of the dataset. After learning, our approach will be able to reconstruct the 3D visual scene from the box's ' vibration alone.
4 Predicting the Visual Scene from Vibration
To estimate the state inside the container, we will learn to reconstruct the visual modality from the audio modality. We present convolutional network that translates vibrations into images.
4.1
Model
We will a model that reconstructs the visual contents from the vibrations. Let A; be a spectrogram of the vibration captured by microphone such that i E f1,2,3,45 4). Our model will predict the image XRGB fRGB (A; 0) where f is a neural network parameterized by 0. The network will learn to predict the image of a top-down view into the container. We additionally have a corresponding
network to produce a depth image Xaeph = Saepti(;; (A;
Reconstructing pixel requires the model to have access to the full spectrogram. However, we also want to take advantage of the spatio-temporal structure of the signal. We therefore use a fully con- volutional encoder and decoder architecture. The network transforms spectrogram representation (time frequency) into an C-channel embedding with width and height being 1, such that the receptive field of every dimension reaches every magnitude in the input and every pixel in the out- put. Unlike image-to-image translation problems [32, 33, 34], our task requires translation across modalities.
We use multi-scall decoder network [35, 36, 37]. Specifically, each decoder layer consists of two branches. One branch is transposed convolutional layer to up-sample the intermediate feature. The other branch passes the input feature first to a convolutional layer and then a transposed convolution so that the output for the second branch matches the size of the first branch. We then concatenate the output from these two branches along the feature dimension as the input feature for the next decoder layer. We perform the same operation for each decoder layer except the last layer where only one transposed convolution layer is needed to predict the final output image.
We use spectrogram to represent audio signals. We apply Fourier Transform before converting the generated spectrogram Mel scale. Since we have four microphones. audio clips are concate- nated together along a third dimension in addition to the original time and frequency dimension.
4.2 Learning
In most cases, we care about predicting the resting position of the object. We therefore train the network f to predict the final stable image. For RGB image predictions, we train the network to
The camera is 42cm away from the bottom of the bin to capture clear top-down images.

---

minimize the expected squared error between the image XRGB and the predictions from audio A:
LRGB = EA.x [R] A: 0) XRGB l
(1)
In order to reconstruct shape, we also train the network to predict depth image from the audio input. We train the model to minimize the expected L1 distance:
Ldepth EA.x Ils (A: Φ) Xaspth [I1]
(2)
Since ground truth depth often has outliers and substantial noise, we use an LI loss [38]. We use stochastic gradient descent estimate the network parameters 0 and Φ. After learning, the model
predicts both the RGB image and the depth image from just the vibration. The visual modality is only supervising representations for the audio modality, allowing reconstructions when cameras not viable, such during occlusions or low illumination.
4.3 Implementation Details
Our network takes in the input size of 128 x 128 x where the last dimension denotes the number of microphones. The output is a 128 x 128 x RGB image or a 128 x 128 x 1 depth image. We use the same network architecture for both the RGB and depth output representations except the feature dimension in the last layer for different modalities. All network details are listed in the Appendix. Our networks networks for 500 epochs with Adam [41] optimizer and batch size of 32 on a single NVIDIA RTX 2080 Ti GPU, The learning rates starts from 0.001 and decrease by 50% at epoch 20, 50, and 100.
configured in PyTorch [39] and PyTorch-Liighniin
[40]. We optimized all the
5 Experiments
Our experiments analyze the hardware and software at reconstructing an image of the contents from audio. In this section, we first quantitatively evaluate the performance. We then show qualitative results for the reconstructions. Finally, we visualize the learned representations.
Since the physics behind our dataset is chaotic, everytime object is dropped into the container, we obtain unique example with different resting position and orientation. We randomly split the dataset into a training set (80%), validation set (10%), and testing set (10%). All of our results are evaluated with three random seeds for training and evaluation with various splits of the dataset. We report the mean and the standard error of the mean for all outcomes.
5.1 Evaluation Metrics
We use two evaluation metrics for our final scene reconstruction that focus on the object state.
IoU measures how well the model reconstructs both shape and location. Since the model predicts an image, we subtract the background to convert the predicted image into a segmentation mask. Similarly, we performed the same operation on the ground-truth image. IoU metric then computes intersection over union with the two binary masks.
Localization score evaluates whether the model produces an image with the block in the right spatial position. This metric is especially useful for object picking tasks with suction gripper where the spatial location of the block matters. With the binary masks obtained in the above process, we can fit a bounding box with minimum area around the object region. We denote the distance between the center of the predicted bounding box and the center of the ground-truth bounding box as d, and the length of the diagonal line of ground-trute truth box as l. We report the fraction of times the predicted location is less than half the diagonal: N Σn [d< S 1/2).
5.2 Baselines
Time Difference of Arrival (TDoA), We compare against an analytical prediction of the location. standard practice is to localize sound sources by estimating the time difference of arrival across an array of microphones. In our case, the microphones surround the sound source. There are several ways to estimate the time difference of arrival, and we use the Generalized Cross Correlation with Phase Transform (GCC-PHAT), which is established, textbook approach [42]. Once have
5

---

Audio Input
Prediction
Ground Truth
Audio Input
Prediction
Ground Truth
Figure 5: Model Predictions with Mixed Shapes. From left right on each column, we visualize the audio input, the predicted scene, and the ground-truth images. Our model can produce accurate predictions for object shape. position and orientation.
time difference of arrival estimate, find the location in the box that would yield time difference of arrival that is closest to our estimate.
Random Sampling. To evaluate the learned models simply memorize the training data, we com- pared our method against a random sampling procedure. This baseline makes prediction by ran- domly sampling image from the training set and using it as the prediction.
Average Bounding Box. The average bounding box baseline measures to what extent the model learns the dataset bias. We extracted object bounding boxes from all the training data through background subtraction and rectangle fitting to obtain the average center location, box sizes and box orientation. This baseline uses the average bounding box as the prediction for all the test samples.
Nearest Neighbor. To evaluate the generalization performance from training data distribution, construct a nearest neighbor baseline. For each test input audio, we use the resulted image from the training data with most similar audio as the prediction. The similarity measured by L.2 distance.
5.3 Reconstruction with Mixed Shapes
Mull -ahu-ataeee The Jol Soons)
Natt-aaape Thaking Lucc
We first analyze how well The Boombox recon- structs its contents when the shape is not known priori. We train single model with all the ob- ject shapes. The training data for each shape are simply combined together so that the training, validation and testing data are naturally well- balanced with respect to the shapes. This set- ting is challenging because the model needs to learn audio features for multiple shapes at once.
Figure 6: Reconstruction with Mixed Shapes. Our model outperforms baseline methods at local- ization and shape prediction.
Figure 6 shows the convolutional networks are able to learn robust features to localize both the position and orientation, even when shapes are mixed. Our method outperforms TDoA often by significant margins, suggesting that our learning-based model is learning robust acoustic features for localization. Due to the realistic complexity of the audio signal, the hand-crafted features are hard to reliably estimate. Our model outperforms both the random sampling and average bounding box baseline, indicating that our model learns the natural correspondence between acoustic signals and visual scene rather than memorizing the training data distribution. We show qualitative predictions for both RGB and depth images Figure 5.
5.4 Reconstruction with Known Shape
We next analyze how well the model performs when the object shape is known, but the position and orientation is not. We train separate models for each shape of the object independently. Figure 7 shows The Boombox is able reconstruct both the position and orientation of the shapes. The convolutional network obtains the best performance for most shapes on both evaluation metrics.

---

Ilu Score)
(OU
Score)
Bansom Sampling
6008
Average Bounding Box
8888
B00
(000
TDsA
866
(dooth outour
Boombex (HG22 sutcrr
0.1
0.8
0.4
0.016
0.03
0.04G
0.08
0.00
0.12
0.18
0.84
Cuba (Localizttion)
Stick (1 (Localzatior)
Block (Localization))
Random Bampping
18.99%
88.63%
Boombex (deoth outout
75 15%
6000
C5 45-445%
Boumbbo (1222
BR
0%
2596
50%
7556
100%
0%
2099
50.
7595
100%6 0%6
25%
50%
75%
100%
Figure 7: Reconstruction with Known Shape. We show the performance of each individual model trained with of the three objects. We report both the mean and the standard error of the mean from three random seeds. Our approach enables robust features to be learned to predict the location and shape of the dropped objects.
Mi (Beloro, Aftet
Mic (BBeoore Alwr)
(Before Aftr)
Mic (Before, Aber)
(A) Remove amplitude
M2
Ground Truth Bafora Fllpping
Prodict on Batara Filop ng
Predict on Alaz Flipping
(8) Fllp microphones Mol
Mic!
Mic (Pellor,, Strit (.0na)
Mic (Retzee, Stift 0.016)
Mic (Relloe, Shaft D.01k)
Mic 4 (Relooe, Shitt D.018)
(C) Temporal Shift
300
Temporal Shitt
Figure 8: Visualization of Ablation Studies. We visualize the impact of different ablations on the model. A) By thresholding the spectrograms, remove the amplitude from the input. B) We experimented with flipping the microphones only at testing time. The model's predictions show corresponding flip as well the predicted images. C) We also experimented with shifting the relative time difference between the microphones, introducing an artificial delay in the microphones only at testing time. shift in time causes a shift in space in the model's predictions. The corruptions are consistent with a block falling in that location.
These results highlight the relative difficulty at reconstructing different shapes from sound. By comparing the model performance across various shapes, the model trained on cubes achieves the best performance while the model trained on blocks performs slightly worse. The most difficult shape the stick.
ROR Ourput Roow
Fon acalostion
When the training data combines all shapes, the model should share features between shapes,
0/209
thus improving performance. To validate this, compare performance
ruens
7009
the multi-shape versus models trained with single known shape. Figure 9 shows that the performance on the block and stick shapes are improved by large margin. We notice that the performance of the cube drops due to the confusion between shapes. When the cube confuses with the stick or the block, because of the smaller surface area of these two shapes, the cube performance slightly degrades.
Book
0221
7204
0..0%
Singlwdtape tmaniing
Kiu Eisthape tmaining
Figure 9: Shape Transfer. Performance improves by training with multiple shapes.
7

---

5.5 Ablations and Analysis
To better understand what features the model has learned specifically, we perform several ablation studies, shown in Figure 8 and Figure 10.
Flip microphones. The microphones' layout should matter for our learned model to localize the objects. When we flipped the microphone lo- cation, due to the symmetric nature of the hard-
aso sutpur geil Scort)
FGB cupput ( (Locabator
11:00
tenpooi ani 220
ware setup, the predictions should also be flipped accordingly. To study this, we flipped the input of Micl and Mic4 as well as the input of Mic2 and Mic3 in the testing set, shown in Figure 8. Our results Figure 8B shows that our model in-
0479
6.15
0.055
25%
50%
Depth but (S0 Soard
Dushi oupu Local eator)
O1ginal
Rlpedd nach
9.11
Tampost pit 230
109
0.59
0.18%
deed produces a flipped scene. The performance in Figure 10 nearly drops to zero, suggesting that the model implicitly learned the relative micro- phone locations to assist its final prediction.
Figure 10: Quantitative ablation studies. We experiment with different perturbations our input data to understand the model predictions.
Remove amplitude. The relative amplitude between microphones can indicate the relative position of the sound source to different microphones. We removed the amplitude information by thresh- olding the spectrograms, shown in Figure 8. We retrained the network due to potential distribution shift. As expected, even though the time and frequency information are preserved, the model per- forms much worse (Figure 10), suggesting that our model additionally learns to use amplitude for the predictions.
Temporal shift. We are interested to see if our model learns to capture features about the time difference of arrival between microphones. If so, when we shift the audio temporally, the prediction should also shift spatially. We experimented with various degrees of temporal shifts on the original spectrograms. For example, shifting 500 samples corresponds shifting about 0.01s (500/ 44,000). By shifting the Micl's 's spectrogram forward and Mic4's spectrogram backward with zero padding to maintain the same amount of time, and preforming similar operation on Mic2 and Mic3 respectively, we should expect that the predicted object position shifts towards the left- direction. In Figure 8, we can clearly observe this trend as temporal shift increases. Shifting the signal in time decreases the model's 's performance, demonstrating that the model has picked up on the time difference of arrival.
Feature Visualization: We finally visualize the latent features in between our encoder and decoder network with t-SNE[43), shown in Figure 11. We colorize the points based on ground truth position and orientation. The magnitude distance from the center of the im- age is represented by saturation, and the angle from the horizontal axis is represented by hue. We find that there is often clear clus- tering of the embeddings by their position and orientation, showing that the model is robustly discriminating the location of the impact from sound alone. Moreover, the gradual transitions between colors suggest the features are able to smoothly interpolate spatially.
Angle Peo
Figure 11: Low-dimensional embedding. We visualize the learned features in the en- coder with t-SNE.
6 Conclusion
This paper introduces The Boombox, low-cost container integrated with a convolutional network that uses sound to reconstruct an image of the contents inside it. Containers ubiquitous in ev- eryday situations, and this paper shows that we can equip them with sound perception to estimate the state inside the container. When cameras not available, such during occlusions or poor illumination, our results suggest that sound is a promising sensing modality.
In the future, one exciting direction is to leverage the acoustic signatures to not only infer the 3D geometries of the object, but also the materials of various objects. In multi-object interaction setting such as collision, audio can also be an important sensing signal to understand their dynamics. We also aim to scale our current object collections to include diverse objects.
8

---

Acknowledgments
We thank Philippe Wyder, Dídac Surís, Basile Van Hoorick and Robert Kwiatkowski for helpful feedback. This research based on work partially supported by NSF NRI Award #1925157, NSF CAREER Award #2046910, DARPA MTO grant L2M Program HR001 1-18-2-0020, and Ama- zon Research Award. MC supported by a CAIT Amazon PhD fellowship. We thank NVidia for GPU donations. The views and conclusions contained herein are those of the authors and should be interpreted as necessarily representing the official policies, either expressed or implied, of the sponsors.
References
[1] A. Zeng, S. Song, K.-T. Yu, E. Donlon, F. R. Hogan, M. Bauza, D. Ma, O. Taylor, M. Liu, E. Romo, al. Robotic pick-and-place of novel objects clutter with multi-affordance grasp- ing and cross-domain image matching. In 2018 IEEE international conference on robotics and automation (ICRA), pages 3750-3757. IEEE, 2018.
[2] S. Levine, Pastor, A. Krizhevsky. J. Ibarz, and D. Quillen. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. The International Journal of Robotics Research, 37(4-5):4214436 2018.
[3] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, al. Qt-opt: Scalable deep reinforcement learning for vision- based robotic manipulation. arXiv preprint 18Xiv: 10293, 2018.
[4] J. Besl and N. D. McKay. Method for registration of 3-d shapes. In Sensor fusion IV: control paradigms and data structures, volume 1611, I, pages 386-606. International Society for Optics and Photonics, 1992.
[5] S. Hinterstoisser, S. Holzer, C. Cagniart, S. Ilie, K. Konolige, N. Navab, and V. Lepetit. Mul- timodal templates for real-time detection of texture-less objects in heavily cluttered scenes. In 2011 international conference computer vision, pages 858 865. IEEE, 2011.
[6] E. Malis. Survey of vision-based robot control. ENSIETA European Naval Ship Design Short Course, Brest, France, 41:46, 2002.
[7] F. Chaumette and S. Hutchinson. Visual servoing and visual tracking, 2008
[8] A. Singh, J. Sha, K. S. Narayan, T. Achim, and P. Abbeel. Bigbird: A large-scale 3d database of object instances. 2014 IEEE international conference on robotics and automation (ICRA), pages 509-516. IEEE, 2014.
[9] A. Zeng, S. Song, M. Nießner, M. Fisher, J. Xiao, and T. Funkhouser. 3dmatch: Learning the matching of local 3d geometry in range scans. In CVPR, volume 1, page 4, 2017.
[10] A. A. Nair, A. Reiter, C. Zheng, and S. Nayar. Audiovisual zooming: what you see is what you hear. In Proceedings of the 27th ACM International Conference on Multimedia, pages 1107-1118, 2019.
[11] R. Gao and K. Grauman. 2.5 visual sound. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 324-333, 2019.
[12] H. Zhou, Z. Liu, X. Xu, P. Luo, and X. Wang. Vision-infused deep audio inpainting. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 283-292, 2019.
[13] A. Owens, P. Isola, J. McDermott, A. Torralba, E. H. Adelson, and W. T. Freeman. Visually indicated sounds. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2405-2413, 2016.
[14] Y. Zhou, Z. Wang. C. Fang, T. Bui, and T. L. Berg. Visual to sound: Generating natural sound for videos in the wild. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3550-3555 2018

---

[15] R. Arandjelovic and A. Zisserman. Look, listen and learn. In Proceedings of the IEEE Inter- national Conference on Computer Vision, pages 609-617 617, 2017.
[16] E. Kazakos, A. Nagrani, A. Zisserman, and D. Damen. Epic-fusion: Audio-visual tempo- ral binding for egocentric action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5492-5501, 2019.
[17] Z. Zhang, Wu, Q. Li, Z. Huang. Traer, H. McDermott, J. B. Tenenbaum, and W. T. Freeman. Generative modeling of audible shapes for object perception. In Proceedings of the IEEE International Conference on Computer Vision, pages 1251-1260, 2017.
[18] Z. Zhang, Q. Li, Z. Huang, Wu, J. B. Tenenbaum, and W. T. Freeman. Shape and material from sound. 2017.
[19] A. Owens, J. Wu, J. H. McDermott, W. T. Freeman, and A. Torralba. Ambient sound provides supervision for visual learning. European conference on computer vision, pages 801-816. Springer, 2016.
[20] Y. Aytar, C. Vondrick, and A. Torralba. Soundnet: Learning sound representations from unla- beled video. arXiv preprint arXiv:1610000000 2016.
[21] T.-H. Oh, T. Dekel, C. Kim, I. Mosseri, W. T. Freeman, M. Rubinstein, and W. Matusik. Speech2face: Learning the face behind voice. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7539-7548. 2019.
[22] C.-H. Wan, S.-P. Chuang, and H.-Y. Lee. Towards audio scene image synthesis using gener- ative adversarial network. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 496-500. IEEE, 2019.
[23] D. Gandhi, A. Gupta, and Pinto. Swoosh! rattle! thump!-actions that sound. arXiv preprint arXiv:2007.0185/ 2020.
[24] C. Matl, Y. Narang, D. Fox, R. Bajesy, and F. Ramos. Stressd: Sim-to-real from sound for stochastic dynamics, arXiv preprint arXiv:2011.03136, 2020.
[25] Bu and C.-M. Huang. Object permanence through audio-visual representations. arXiv preprint arXiv:2010.09998, 2020.
[26] J. C. Middlebrooks. Sound localization. Handbook of clinical neurology, 129:99-116, 2015.
[27] Z. Liang, X. Ma, and X. Dai. Robust tracking of moving sound source using multiple model kalman filter. Applied acoustics, 1350:))::3001333 2008
[28] M. Zhu, Yao, Wu, Z. Lu, X. Zhu, and Q Huang. Gaussian filter for tdoa based sound source localization in multimedia surveillance. Multimedia Tools and Applications, 77(3): 3369-3385 2018.
[29] I. An, M. Son, D. Manocha, and S.-E. Yoon. Reflection- aware sound source localization. In 2018 IEFE International Conference on Robotics and Automation (ICRA), pages 66-73. IEEE, 2018.
[30] Bestagini, M. Compagnoni, F. Antonacci, A. Sarti, and S. Tubaro. Tasa-based acoustic source localization in the space-range reference frame. Multidimensionall Systems and Signal Processing, 25(2):337-359, 2014.
[31] W. Strutt. On our perception of sound direction. Philosophical Magazine, 13(74))214-32, 1907.
[32] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle- consistent adversarial networks. In Proceedings of the IEEE international conference on com- puter vision, pages 2223-2233 2017.
[33] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125-1134, 2017.
10

---

[34] Y. Choi, M. Choi, M. Kim. J.-W. Ha, S. Kim, and J. Choo. Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8789-8797, 2018.
[35] M. Mathieu, C. Couprie, and Y. LeCun. Deep multi-scale video prediction beyond mean square error. arXiv preprint ArXiv: 1.04400 2015.
[36] E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2462-2470, 2017.
[37] B. Chen, C. Vondrick, and H. Lipson. Visual behavior modelling for robotic theory of mind. Scientific Reports, 11(1):1-14, 2021.
[38] Ma and S. Karaman. Sparse-to-dense: Depth prediction from sparse depth samples and single image. In 20/8 IEEE International Conference on Robotics and Automation (ICRA), pages 4796 1796-4803. IEEE, 2018
[39] A. Paszke, Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang. J. Bai, and S. Chin- tala. Pytorch: An imperative style, high-performance deep learning library.
In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc. E. Fox, and R. Gar- nett, editors, Advances in Neural Information Processing Systems 32, pages 8024 8035. Curran Associates, Inc., 2019. URL http://papers. neurips. newrips.ccpppper. 9015-pytorch- an- pdf.
[40] W. Falcon and al. https://ithub..cm//// hww..///woooooo.ooooooooooororoe
Pytorch lightning.
GitHub.
Note:
3, 2019.
[41] D. P. Kingma and Ba. Adam: method for stochastic optimization. arXiv preprint
arXiv:1412.6980, 2014.
[42] C. Knapp and G. Carter. The generalized correlation method for estimation of time delay. IEEE transactions on acoustics, speech, and signal processing, 24(4):320-327 1976.
[43] L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(1 1), 2008


Reviews:
[{"review_id": "ujs_-xuNmbj", "reviewer": "Reviewer_Ezuu", "summary": "The paper predicts the rgb and depth of an object in a bin given the audio signals from contact microphones attached to the bin. The audio signals capture the sounds/vibrations caused by the object when it’s dropped into the bin.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper’s motivation is very promising and could lead to better use of audio for robotic manipulation. However, there are several concerns with the paper which are listed below:\n\n1. The Localization error metric only counts the number of times the two bounding boxes overlap by a certain distance. Adding an additional metric that computes the distance between the two bounding boxes will be useful. Specifically capturing  both position accuracy and orientation accuracy. Without a measure of accuracy in meters/radians, it’s not clear how accurate the method will be for use in robot manipulation. \n\n2. How will the predictions scale to more objects? E.g., Can the method differentiate between a ball and a cube where the ball could move around a lot before settling? Can the method differentiate between sharp edged objects (star?) and soft edged objects?\n\n3. It’s unclear if the accuracy of the predictions are sufficient for manipulation tasks. Using the prediction in a downstream task and evaluating task success across the different methods is needed.\n\n4. Can these predictions be generated online? This is especially important for feedback control.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Mentioned weaknesses need to be addressed to show the benefit of the proposed approach.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "To reiterate, some things to be done to improve the paper:\n1. Needs experiments across more objects. \n2. Show application in a downstream task such as moving the object to the center of the bin.\n", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "oM3WhiUymSI", "reviewer": "Reviewer_aA4p", "summary": "This paper proposes a boombox, a container that can estimate the location of objects thrown into the box from an array of microphones. The audio signal is processed by a convolutional network to reconstruct the RGB and depth image of the objects from a bird's eye view. The approach is validated over multiple setups and shown to outperform traditional audio processing methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, I find this paper to be very interesting. The idea of reconstructing the visual signal from only audio is very intriguing and novel (to the best of my knowledge), and I am surprised that it works so well. Much related work on audio-visual processing exists (as pointed out in the related work), but I think this work stands out from existing methods from both the problem setup and the results. Along these lines, it would have been great to read a discussion in related work about other methods of non-line-of-sight shape reconstruction, e.g. [1] or [2]. \n\nThe approach is clearly presented and well-executed, with in-depth experimental analysis to understand how the approach compares to traditional methods and what features in the audio signal are used for reconstruction.\n\nHowever, there are still some issues and points to clarify to make the contribution even stronger. \nFrom a methodological point of view, it is unclear why learning an RGB image from audio. Clearly, no color information is available in the audio, and the results are in \"color\" just because of the overfit to the datasets' color. What can be done to solve this issue (and possibly favor generalization to other boxes/objects)? I would assume that it would be much easier to learn an \"intensity-free\" image (e.g., depth) or some low-level feature (e.g., the x-y coordinate of the object or the object segmentation) instead of an RGB or grayscale image. Along these lines, I am very surprised that reconstructing a depth works consistently worse than the RGB. Why is this the case? What are the challenges that are present for depth but absent in RGB? Therefore, I would be very interested in seeing a more detailed discussion about the topic and possibly seeing some quantitative results of directly regressing the object segmentation and bounding box from the audio signal. In addition, it would be nice to see how other state-of-the-art architectures for audio processing, e.g., wavenet and its more recent successors, perform compared to the simple proposed encoder-decoder architecture.\n\nFinally, I think that the presentation could be improved in some places. More specifically:\n1. The discussion in the intro on why visual sensors may fail reads a bit weak. I believe that \"occlusions\" in the box could also be a problem for the proposed method (since a completely different audio signal would be generated). Also, motion blur does not seem relevant to the task since one would always wait for the objects to land in a stable position. Maybe some stronger motivation can be found (e.g., redundancy with a vision sensor?)\n2. Some figures (e.g., Fig 6 and 10) are tiny and difficult to read. \n3. Paragraph 5.3 (reconstruction with unknown shape) is a bit deceiving. The experiment does not show what happens with shapes that are outside the training dataset. I would therefore recommend changing the title of the section to \"Reconstruction with mixed shapes\" or something along these lines. That being said, it would actually be nice to see some experiments where reconstruction of previously unseen shapes is studied (I guess it would not work, but to which extent?)\n4. The \"Random\" baseline is very weak and uninformative. I would substitute it with a stronger one that returns the label associated to the nearest neighbor (according to some simple metric) of the training dataset.\n5. There is no discussion about the limitations of the approach. I think this is fundamental to enable researchers to follow up the work.\n\n\n\n[1] A Theory of Fermat Paths for Non-Line-of-Sight Shape Reconstruction, Xin et al.\n[2] Acoustic Non-Line-of-Sight Imaging, Lindell, et al.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I mainly base my recommendation on the novelty of the idea and the interesting results.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "I think that during the rebuttal period some simple experiments could be done to make the contribution stronger. Specifically:\n1. Predict directly the object location or segmentation instead of the depth or rgb image.\n2. Include the nearest neighbor baseline explained above.\nIn addition, it would be nice to see the comments on the presentation to be included."}}, {"review_id": "nww6zYjx7DH", "reviewer": "Reviewer_5sJt", "summary": "This paper introduces a hardware setup called Boombox and an algorithm that can predict visual signals from acoustic vibrations.\nSpecifically, the box is equipped with four contact microphones, one on each side, to provide audio signals. The learning algorithm is trained to predict the RGB or depth image of the final scene of the inside of the box after a wooden object gets dropped into it.\nThis system is shown to work more reliably than the alternative methods considered, qualitatively and quantitatively under the two metrics IOU and localization. The three alternative methods are time difference of arrival (TDoA) using Generalized Cross Correlation with Phase Transform(GCC-PHAT), random sampling from the training set, average bounding box from the training set.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Main strengths:\n1. The paper is well written\n2. The authors built a system that predicts the 3D visual scene from audio signals and showed that it worked well under the setting described \n3. Ablation studies were thorough: flipped microphones, removed amplitude, temporal shift. The learned features were also shown to have learned spatial information.\n\nWeaknesses / Suggestions:\n1. It’s not clear to me that this type of approach would scale under the situation of object collisions, more realistic object types:\na. The objects that were included in the experiments only differed in shape. They shared the same color and materials. \nb. The paper only considered dropping one object at a time into an empty box. It did not discuss whether this approach can be adapted to handle more complex settings when there're collisions between objects.\nWhile the above two points might be too much to ask in a pilot study, I was hoping that they would be discussed in the paper. If the authors consider that to be future works, I would suggest discussing it in the paper as part of the limitation of the method.\n\n2. I feel that there may be another baseline that this paper did not experiment with. In the baseline method “random sampling”, one image from the training set is chosen as the prediction of the test audio sample. One could imagine a nearest neighbor baseline that finds the nearest neighbor match from the training set of the test audio. This new baseline would indicate the similarity between the train and test distributions.\n\n3. Background subtraction seems to be an important step in computing both evaluation metrics since the error in the background subtraction output would affect the quality of downstream tasks (during test time). Yet, neither the method of background subtraction nor its impact was described.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The task of using sound to help to estimate object state is interesting. \nThe paper describes a system that works in relatively simple settings where a single object is dropped into an empty bin. \nIt's well written and the method shows some real-world success. \nHence my overall impression of the paper is positive. But I hold some concerns about the practicality of the approach. In real-world tasks where there are a larger variety of objects and with the addition of object-object collision, using acoustic vibration alone to estimate the object state seems difficult.  ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Please refer to the three items discussed in the weaknesses section.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {}}, {"review_id": "GAV6zGzIA3o", "reviewer": "Reviewer_XY9t", "summary": "The paper presents a fun idea: predicting the content of a container attached from 4 low-cost microphones. The vibrations from dropping objects into a container are fed to a convolutional network that predicts the visual reconstruction of the scene (RGB-D output). There's nothings exciting going on in the method, but the application domain is very interesting and seems very much unexplored in the deep-learning robotics era. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n- The paper is well written and the problem statement is made clear.\n- The method is simple and elegant, and deserves to be built upon.\n- The ablation and feature visualisation is interesting and informative.\n\n**Weaknesses**\n- The key for an application heavy paper like this, is a solid set of results that proves the claim that this is useful in the wider robotics community for a variety of applications. However, the paper only presents results on 3 wooden blocks. This makes it very difficult to assess how well this method would generalise to a larger number of objects, or if the method would have the capability to generalise to unseen test data given a large diverse training set of objects. \n- Section 5.3 is perhaps a little misleading: initially it seems that the paper is presenting results to suggest that they are evaluating on an unknown shape, which (perhaps subjectively) would indicate that the object was not in the training set, however, what they simply mean is that the model is train simultaneously on the 3 shapes, rather than individually training a network for each shape (as in Section 5.4: Reconstruction with known shape).\n- The lack of thorough experimentation suggests to me that either the paper did not have enough time to run additional experiments (due to conference deadline) or that additional, challenging setups were run but the method was not successful. Hopefully the former, and I hope the authors use the time after CoRL to continue additional experiments. There are several un answered questions: What about multiple objects in the bin? What if there is already and object in the bin? What is the generalisation capability across unknown objects? What about across containers?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "In summary, I very much like the paper, and the idea is promising, however the combination of poor technical novelty and poor experimental analysis makes this paper difficult to accept in its current form. I urge the authors to continue working on this line of work and resubmit to a new venue after evaluating the method on a larger dataset (of objects and containers) and assessing generalisation ability atleast across many objects, but also, if possible, across containers. ", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "The method is great, but it is severally lacking additional experiments; see 'weaknesses' section for suggested next steps.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new hardware setup for estimating the shapes and poses of objects inside a bin using only auditory signals. The objects inside the bin are bounced and the sounds resulting from their collisions are captured by four microphones. The signals from the microphones are used as inputs to a convolutional neural network that returns an RGB image of the objects inside the bin. The proposed method is evaluated on scenes containing single wooden cubic objects. The experimental results clearly indicate that the proposed method is able to reconstruct the interior of the bin. 
The reviewers agree that the proposed method is interesting and novel. The reviewers find that the main contribution of this work is the application, rather than the technical approach. Therefore, the paper should include a richer and more exhaustive set of experiments. The current ones are limited to single objects that have simple regular shapes. The reviewers are not sure why the paper does not include experiments on realistic objects, or multiple objects in the bin. The evaluation metrics can also be improved. Finally, stronger baselines should be considered. For example, the authors can try simple nearest-neighbors methods, or traditional shallow learning techniques. 

In their rebuttal, the authors added new experiments and addressed several concerns of the reviewers and the AC. From the discussion, it is clear that the reviewers appreciate the novelty of this application. However, concerns regarding the generalization and practicality of the proposed technique remain. Therefore, the reviewers are leaning toward a weak accept. In conclusion, this is a good paper that deals with a challenging problem where it is difficult to collect more data. The area chair believes that this is a promising research direction, and this paper may open the door to more elaborate works in this new domain. 
Decision: Accept (Poster)