SonicSense: Object Perception from In-Hand Acoustic Vibration

Authors: jiaxun  liu
boyuan  chen
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-05-28 16:42:39

Abstract: We introduce SonicSense, a holistic design of hardware and software to enable rich robot object perception through in-hand acoustic vibration sensing. While previous studies have shown promising results with acoustic sensing for object perception, current solutions are constrained to a handful of objects with simple geometries and homogeneous materials, single-finger sensing, and mixing training and testing on the same objects. SonicSense enables container inventory status differentiation, heterogeneous material prediction, 3D shape reconstruction, and object re-identification from a diverse set of 83 real-world objects. Our system employs a simple but effective heuristic exploration policy to interact with the objects as well as end-to-end learning-based algorithms to fuse vibration signals to infer object properties. Our framework underscores the significance of in-hand acoustic vibration sensing in advancing robot tactile perception.

Main paper: 
 introduction: 
 Fig. 1 : SonicSense enables container inventory status differentiation, heterogeneous material prediction, 3D shape reconstruction, and object reidentification on a diverse set of 83 realworld objects.By shaking a container, we can tell its inventory status from the generated acoustic vibrations, such as the quantity and geometry of the objects inside. Similarly, we can identify the material and geometry of the entire object through multiple tappings. Human hands are equipped with high-frequency skin vibrations to help capture such complex object properties [1] . However, despite the significance of acoustic vibrations for tactile perception, equipping robot manipulators with acoustic vibration sensing capability for rich object perception remains difficult [2, 3, 4, 5, 6] .Though previous research has explored placing air microphones near robot platforms to estimate liquid height [7] and pouring amounts [8] , classify object materials [9] and categories [10, 11, 12] , air microphones mainly capture sound waves transmitted through air, leading to noisy signals with ambient noises. On the other hand, contact microphones only sense the acoustic vibrations caused by physical contact. Past work has studied contact microphones for estimating the amount and flow of granular material [13] , object position and category [14] , and collectively performing object spatial reasoning for visual reconstruction [15] .Several major challenges remain to advance acoustic vibration sensing for robot object perception. Most current solutions focus on constrained settings with a small number (N < 5) of primitive objects [7, 8, 9, 13, 15, 16] , homogeneous material composition for each object [9, 13, 16, 17] , single-finger testing [16, 17] , and training and testing on different contacts but same objects [16, 17] . However, it is not clear whether such testing results can work with noisy and less controlled conditions. In addition, previous computational algorithms mainly utilize small machine learning models [10, 12, 16, 17] with a limited amount of data, which could be difficult to generalize. Moreover, the interaction mechanisms to collect acoustic data with objects rely on human manual movements [15, 16] or replaying pre-defined fixed robot poses [7, 8, 9, 10, 11, 13, 14, 16] , making it difficult to scale to a large number of objects.We present SonicSense (Fig. 1 ), a holistic design on both hardware and algorithm advancements for object perception through in-hand acoustic vibration sensing. Our design enables effective object perception abilities that are difficult for previous approaches to achieve altogether on 83 diverse realworld objects, including objects with complex geometry and heterogeneous materials. Our robot is capable of differentiating the inventory status of an occluded container through interactions. In more challenging tasks, through a naive but effective heuristic exploration policy to autonomously collect acoustic vibration characteristics, we can successfully infer material compositions, reconstruct the complete 3D object shape through sparse tapping, and re-identify previous objects. Additionally, we propose a set of end-to-end learning-based models by leveraging our large-scale dataset. Moreover, our design is cost-effective ($215.26) and easy to build with off-the-shelf components and 3D printing. Our experiments on large-scale real-world datasets demonstrate strong quantitative and qualitative performances of our approach. Overall, our method presents unique contributions and opens up new opportunities for robot tactile perception. 
related works: 
 Tactile Perception Tactile sensing [18, 19, 20] perceives object properties through contact. Visionbased tactile sensors [21, 22, 23, 24, 25] employ RGB cameras and light sources to detect robot skin deformations under contact. Barometric pressure tactile sensors [26, 27, 28, 29] perceive contact information by detecting the air or liquid pressure change between the soft elastomer and the sensor. Capacitive [30] and piezoresistive [31] sensors can monitor both static and dynamic pressure. Magnetic-based tactile sensors [32, 33, 34] achieve force detection by the movement of the magnetic field. Thermal conductivity and temperature tactile sensors [35] can identify the material and contact pressure by sensing the temperature change.Our work explores the use of acoustic vibration as another modality for tactile perception. Acoustic vibration offers the advantage of detecting high-frequency vibration characteristics of objects using low power. Our design also features an intuitive mechanical and electrical design that is both affordable and robust. Furthermore, we will fully open-source our hardware, software, models, and dataset. Since it remains an open topic on which one type or combinations of certain types of tactile sensors serve as the optimal solution for tactile perception, we believe that it is valuable to explore different types of modalities and leave the investigation of combining them for future work.Acoustic Sensing for Object Perception Extensive literature has investigated the leverage of acoustic sensing to estimate object states such as liquid amount [7, 8] , object materials [9] , object classes [10, 11, 12, 36] , and object shapes and poses [15] . However, existing studies mostly focus on one or a few aspects of object states and have not been shown to be directly applicable to various aspects of object perception. In contrast, our method aims to generalize on multiple challenging object perception tasks ranging from solid and liquid object inventory status estimation to material classification, 3D shape reconstruction, and object re-identification.Most recently, robot hand designs equipped with acoustic sensors have been demonstrated with strong performances on object material classification [16, 17] , contact detection [16, 17, 37, 38] , and object manipulations [39, 40, 41] . While recent efforts have achieved very promising results, current methods have focused on constrained settings with a small number of primitive objects [7, 8, 9, 13, 15, 16] , homogeneous material composition [16, 9, 13] , small machine learning models trained on a limited amount of data [16, 10, 12] , and human manual data collection process by either moving the hand directly or replaying pre-defined poses with fixed contact point and forces [7, 8, 9, 10, 11, 13, 14, 15, 16] . Our work moves one step towards larger and more diverse set of real-world objects with complex geometry and heterogeneous materials. With our simple but effective heuristic exploration policy, we can autonomously collect rich acoustic vibration responses of objects for training our learning-based computational models.Acoustic Sensing Dataset To leverage the advancement of deep learning, there have been large efforts to create synthetic audio datasets [9, 42, 43] . Due to the intrinsic complexity of physical object properties, it remains challenging to close the simulation-to-real gap. For existing datasets created on real-world objects, the interaction is either performed by a human [3, 6, 43, 44, 45] or by an impact hammer [46] in a noise-controlled environment. Therefore, it is difficult to leverage such datasets for object perception in real-world robotics, whereas our work directly investigates acoustic vibration perception in real-world robotic systems.3 The SonicSense Our robot hand (Fig. 2 ) has four fingers and each finger has one joint with one degree of freedom. Our intuitive mechanical design enables a range of interactive motion primitives for object perception including tapping, grasping, and shaking motions.At each fingertip, a piezoelectric contact microphone is embedded inside the plastic shell while a round counterweight is mounted on the outer shell surface to increase the momentum of the finger motion. We found that the counterweight plays an important role in enabling large striking vibrations during tapping motion. Our contact microphones are synchronized and can pick up the acoustic vibrations at a frequency of 44, 100 Hz. Our design is highly affordable with a total estimated cost of $215.26 including all electronics components, motors, sensors, and 3D printing materials. See detailed splits and hardware specifications in Supp.A. We have developed a dataset with 83 diverse real-world objects shown in Fig. 3 . Our dataset covers nine material categories (i.e., plastic, glass, wood, metal, ceramic, paper, rubber, foam, and fabric) which includes challenging materials such as foam and fabric, and 22.9% of the objects include more than one material. We provide highquality 3D scanned meshes and point cloud models for all objects, along with fine-grained per-point annotations of material categories on the point clouds. Our objects cover a variety of geometries, from simple primitives to complex shapes and from smaller objects to larger or longer objects. We also include objects that are traditionally difficult for vision sensors to perceive such as objects with transparent or reflective surfaces. 
heuristic-based interaction policy: 
 Unlike previous work that uses humans to manually hold the robot's hand to interact with objects or design fixed interaction poses and forces for replay, we derive a simple but effective heuristic-based interaction policy to collect the acoustic vibration response of objects. Our policy works well for all our realworld objects covering variable sizes and geometries. First, due to the unknown shape of the object, the policy will attempt to make contact with the object from both top-down and side directions, from high to low heights, with a fixed step size, until the first contact event is detected from each direction. Since our motor provides voltage resistance feedback, detecting contact events can be achieved through voltage feedback or acoustic signal change. We used the voltage feedback due to its robustness and easy access. Second, from such initial exploration, we can estimate the height of the object (from the top-down direction) and its radius (from the side direction). Finally, with the estimated height and radius, the robot will use a grid sampling schedule to make sparse contact with the objects to collect acoustic responses. We provide a detailed description of our policy in Supp.B.We make several assumptions when designing our interaction policy. First, we assume the maximum height of all objects. Second, we assume the size of the object can be held by the robot hand during tapping. Third, we fix the object on the table surface by following previous studies [47, 48, 49, 50] . Relaxing these assumptions may involve a learning-based exploration policy or dynamic tracking algorithm developments with tactile information which we leave as future work. 
object perception from in-hand acoustic vibration 4.1 material classification model and training: 
 We aim to leverage in-hand acoustic vibration sensing on the task of material classification for unknown objects. Our material classification model takes in the Mel-spectrogram of the acoustic vibration signal from the impact sound and learns to predict the material label for that contact location. The network shown in Fig. 4 (A) takes the form of three Convolutional Neural Network (CNN) layers followed by two Multilayer Perceptron (MLP) layers. Given a Mel-spectrogram A i of the i th sample in our interaction data, we train the material classification model f mc , parameterized by θ mc , to output the corresponding material label category mi . We optimize the following loss function L mc with the cross-entropy loss:EQUATIONOur shape reconstruction model takes the sparse contact points to generate a dense and complete 3D shape of the object. However, the contact point locations only serve as a rough estimation due to real-world interaction noises and ambiguity around the fingertips. Therefore, it is essential to leverage learning-based methods to capture the prior knowledge of 3D shapes of objects from training data. As shown in Fig. 4 (B), We design the shape reconstruction model by referring to the Point Completion Network (PCN) [51] . We stack two PointNet [52] layers to encode the input contact points to a global feature vector with the dimension of 1 × 1024. We then feed the global feature vector into a decoder network with fully-connected layers to produce the final point cloud with 2, 024 points. Unlike the original PCN, we do not use the folding-based decoder.Given a contact point cloud C i of the i th object collected through our robot hand interaction, our objective is to train the shape reconstruction model g sr , parameterized by θ sr , to generate a dense and complete point cloud of the corresponding object Pi . We optimize the objective function in Eq. 2 with the Chamfer Distance (CD) loss in Eq. 3:EQUATIONFig. 5 : We conducted synthetic data collection of contact points on a large number of 3D objects in the simulation for data augmentation.Because of the challenging nature of this task and the limited amount of real interaction data, we constructed a simulation environment for such contact interaction to augment the dataset as shown in Fig. 5 . We first pre-train the network with only our synthetic dataset to capture necessary prior knowledge and then gradually reduce the percentage of synthetic data and increase the percentage of real-world data during the training process. See Supp.E for the training schedules.When an object has been interacted with by the robot with its acoustic vibration responses, we aim to have our robot re-identify the object through a set of new tapping interactions. To make the problem harder, we restrict the new tapping interactions to happen on different locations on the object. In our object re-identification model, we input both the collection of Mel-spectrograms and their associated contact points to the network to predict the label of this object. As shown in Fig. Given a set of spectrograms A i and contact point cloud C i of the i th object in our interaction data, we train the shape reconstruction model h or , parameterized by θ or , to predict the corresponding object label ôi . We optimize the following loss function L or with the cross-entropy loss:EQUATIONAll models are trained on a single NVIDIA GeForce RTX 3090 GPU. All training lasts from less than an hour to a few hours depending on the tasks. We used grid search during training and selected the best set of hyperparameters based on the validation performance. We present the dataset details, model architecture, training hyperparameters, and detailed results in Supp.D-F. We will also opensource the hardware design, the software, the models, and the dataset.5 Experimental Results 
characterizing basic sensing capabilities of sonicsense: 
 Resistance Against Ambient Noise To test whether SonicSense is robust against ambient noise, we compared the amplitude of signal recordings from SonicSense's fourth finger with the recording from an external air microphone. The overall setup is shown in Fig. 6 and the detailed information is in Supp.G. Starting from the natural environmental noise in a typical lab, we played Gaussian white noise through a speaker and gradually increased the noise decibels. As shown in Fig. 6 , though the amplitude recorded from the air microphone increased by a few thousand units, the amplitude from our robot fingertip barely changed by a few units. This comparison suggests that our hardware design has a strong resistance against ambient noises and only focuses on the vibration signals through physical contact. We first placed different numbers of dice and then a series of dice with various shapes inside a plastic container. We had the robot hold the container and rotate it forward and backward by 180°around the wrist. In the second experiment, the robot held a bottle with three different initial amounts of water (e.i. 0ml, 100ml, and 200ml). We then poured 100 mL water three separate times into the bottle with a constant flow using a dispenser bottle. Next, the robot held the bottle and performed a horizontal shaking motion with three amounts of water (i.e., 100 mL, 200 mL, and 300 mL).From the visualization of the vibration signals captured by our robot in Fig. 7 , we can tell that the signals, though subtle, indeed reflect the spatial and temporal features of different inventory statuses. Quantitatively, we derived twelve interpretable features based on traditional acoustic signal processing, including the root mean square of the signal, spectral centroid, bandwidth, contrast, flatness, roll-off, zero crossing rate, tempogram, poly features, Mel-frequency cepstral coefficients, chroma, and tonnetz, all averaged across time. We then performed an unsupervised nonlinear dimensionality reduction with t-SNE [53] on this 12-dimensional feature vector for all our experiments as shown in Fig. 7 . The clear clusters indicate that SonicSense is able to provide informative cues to distinguish not only the numbers and geometries of solid objects but also the continuous and subtle liquid states in a small container. Unlike previous studies, in the material classification task, we split the data by objects to avoid overlapping acoustic events. We use a split of 60, 11, and 11 for training, validation, and testing. Since the testing dataset is unbalanced caused by different contact points for each object, we report the average F1 score as our evaluation metric. All our experiments are conducted with three different splits to obtain the mean and standard deviation.As shown in Fig. 8 (A), the initial result of our method leads to a 0.523 F1 score. However, many errors stem from outlier-like predictions, even though most predictions in the surrounding object regions remain accurate. Therefore, we propose an iterative refinement procedure assuming that materials are relatively uniform and smooth around local regions. Our iterative algorithm works as follows: for each object, we first filter out the predictions with low occurrence with a threshold M and reassign their labels with the highest occurrence label. For each point, we assign the label based on a majority vote among all its K nearest neighbors. We then repeat this step for N steps. The values of M , K, and N are selected based on the best validation performance. We provide the full results in Supp.D. With this refinement algorithm, our final average F1 score reaches to 0.763.We compared our results with a random search baseline and a nearest neighbor baseline. For the nearest neighbor baseline, we compared each audio spectrogram in the test dataset with all spectrograms in the training dataset and selected the label based on the lowest mean square distance. Our method outperforms both baselines, suggesting that our algorithm generalizes beyond the training set and provides accurate material label prediction on unseen objects.The confusion matrix in Fig. 8 (B) indicates that our method can produce highly accurate predictions even on challenging soft surface materials such as foam and fabric, which typically do not create obvious striking signals.This result implies that our method can capture subtle but critical signals through in-hand acoustic vibrations caused by the slight differences in the stiffness of these materials. Ceramics is usually confused with glass due to their similar properties. Predicting plastic material is challenging because the plastic materials in our dataset vary greatly in their thickness and stiffness. Moreover, we experimented with pre-training our model with the recent audio-material dataset [6, 45, 54] . However, we found that this pre-training scheme hurts the performance (Fig. 8 ). The acoustic signals in these datasets were collected with air microphones and noise-controlled experiments. Hence, a large domain gap exists. As shown in Fig. 9 , through sparse contact tapping points, we obtained an average of 0.00876m Chamfer-L1 distance score. Fig. 10 shows examples of our shape reconstruction results on our testing dataset. We compared our results with a random search baseline and a nearest neighbor baseline. For the nearest neighbor baseline, we compared our test tapping points with all tapping points in our training dataset and selected the corresponding groundtruth point cloud with the lowest Chamfer-L1 distance as the prediction. The Chamfer-L1 distances for these baselines are 0.03653m and 0.01347m respectively. Our model greatly outperforms both baselines and shows strong generalization abilities on unknown shape reconstructions.The prediction on objects with primitive shapes generally has near perfect performance. Additionally, our method exhibits the capability to reconstruct objects with concave geometries. It is also worth noting that there is only one wine glass, as shown in the middle of column three, in our real object dataset. Therefore, this result is likely achieved through our augmentation dataset in the simulation. Some failure prediction examples include the nozzle of the spray and the cap of the bottle, due to the limited number of spray objects in our dataset and its complex shape within a small region. Fig. 11 : SonicSense can accurately reidentify the objects that it has interacted with while sensing their acoustic vibration responses.We evaluated our method with fifteen channels of spectrograms from fifteen random contact locations, which is analogous to our robot random tapping objects a few times. Our model can accurately re-identify the objects with a 92.52% test accuracy while the random baseline and nearest neighbor baseline only gave 1.33% and 43.45% respectively, as shown in Fig. 11 . By looking into the detailed confusion matrix of our testing results as provided in the supplementary files, we can see that smaller objects are more difficult due to the limited number of interaction data. We can also observe that the model performs worse when the objects are relatively small, and the materials have similar acoustic properties, such as ceramic and glass.Additionally, in order to verify the importance of both the shape and material information for this object re-identification task, we conducted two ablation studies to either remove the acoustic vibration input or the tapping point input. The acoustic vibration-only network reaches an accuracy of 84.11%, and the tapping point-only network reaches an accuracy of 48.89%. Therefore, acoustic information provides a more informative representation of the object and the performance will be further improved by incorporating an additional modality of rough contact positions for object re-identification. 
conclusion: 
 We have introduced SonicSense, an integrated hardware and software solution to enable rich object perception capabilities with in-hand acoustic vibration for a multi-finger robot hand. Our experimental results demonstrate the versatility and efficacy of our design on varieties of object perception tasks including solid and liquid object inventory status estimation within containers, material classification, 3D shape reconstruction, and object re-recognition. Unlike previous approaches, our study involves a significantly larger number of real-world objects with complex geometry and heterogeneous materials. Our testing is also conducted on unseen objects instead of different contact events on the same objects. Our simple but effective exploration policy avoids previous manual interaction or fixed pose replay. Our investigations outline the challenges and necessities of considering real-world noises and robot-specific interactions for robot object perception. Overall, our method presents unique contributions to tactile perception with acoustic vibrations and opens up new opportunities for future robot designs to build a more robust, complete, versatile, and holistic perceptual model of the world. 
 
 Appendix: We see a number of opportunities to improve our current approach in future research. One immediate assumption to relax is to avoid fixing the objects on the table. Existing previous work has not been able to relax such assumptions. However, recent efforts on object tracking with tactile information [55] are promising. Future work can consider adapting such algorithms as an online object estimation and tracking within the interaction policy. Our work does not consider multiple objects. However, in the real world, acoustic vibrations can also be helpful for object perception in cluttered environments. Moreover, our current hardware design does not consider complex robot manipulation skills such as dexterous manipulations. With a more anthropomorphic and higher degree-of-freedom hand design, embedding our findings of in-hand acoustic vibration sensing can enable future exploration of dexterous interactions with everyday objects. Furthermore, though our focus in this paper is acoustic vibration sensing for object perception, future work shall involve integrating multiple sensing modalities to obtain more complementary information for robots to work in complex and unstructured environments.Our hardware is built by 3D printing with Polylactic Acid (PLA) filament. We use the LX-224 servo motor to actuate the finger which provides 20kg•cm torque and accurate position and voltage feedback. The contact microphones we used are commercially available on Amazon and the two counter weights on each fingertip is 40g. We place the controller of the motors as well as the audio jack inside the palm. The cost split of building our acoustic robot hand is shown in Tab. We will introduce the calculation of the approximated contact point location in this section. We set the fingertip position at the center of the fingerprint. Once the binary contact event is detected, we will record the angle of each finger joint. Based on our CAD model, we can also obtain other kinematic parameters of the hand such as the length of each link. Fig. 12 shows the sketch of our hardware design and its coordinate system. We use the center of the palm as the origin of the hand. The y-axis is parallel to the four finger joints and the z-axis is perpendicular to the palm, facing towards the finger. The coordinate system of the finger joints has the same orientation as the above central coordinate system of the hand. The center of the finger joint coordinate is located at the center of each motor. We denote the position of joint 1 asp 1 = x 1 y 1 z 1, where x 1 = 7.845cm, y 1 = 3.429cm, z 1 = 13.691cm. Considering that the four fingers are symmetrical, the positions of the rest of the fingers can be represented asEQUATIONWe use P e i to represent the fingertip position under its joint coordinate. Since the length between the origin of joint coordinate and the fingertip is l = 7.6cm, the fingertip position of finger i under the robot hand coordinate, denoted as H P e i , can be calculated byEQUATIONHere, P e i = l 0 0. For finger 1 and finger 2, R can be represented as:EQUATIONThe θ i here represents the joint angle of finger i. For finger 3 and finger 4, R can be represented as:EQUATIONWe attach the robot palm on the end effector of the Franka Emika Panda arm and align the end effector coordinate with the robot hand coordinate to ensure H P e i = E P e i . Having the transformation matrix of the end effector coordinate to the robot arm R E T , we can then calculate the contact position under the robot arm coordinate, represented as R P e i , by:EQUATIONWe will introduce the interaction policy for the robot to conduct real-world data collection in this section. Since vision is not used, the dimension of the object is unknown. Therefore, the first challenge of the interaction policy is to estimate the dimension of the object.We fix the object at the center of a black base on top of a wooden board as shown in Fig. 13A . The robot will first perform two tapping motions from the side with two different heights (Fig. 13B and 13C ) and save the valid contact points. The tapping motion is able to make contact with objects since the objects' body is across the centerline of the black base. Based on the highest detected contact point location, the robot will have a rough estimation of the height of the object. To obtain a more accurate estimation of height and radius of the object, the robot will reset to its initial home position to lift its two fingers in front, and use its third finger to explore the highest edge on top of the object. Specifically, starting from a suitable height away from the objects, the third finger of the hand moves towards the centerline of the black base (Fig. 14A ) and reaches another side of the centerline (Fig. 14B ). The robot keeps monitoring the acoustic signal from the third finger during this process. Once a contact event is detected, the robot will record the contact position through the kinematic model and stop exploring. If the third finger does not make contact with the object and there is no contact event being detected, the robot hand will move back to the initial position, and move down a little bit as shown in Fig. 14C and keep executing the same edge exploring motion until a contact sound is detected as shown in Fig. 14D . We use the distance between the contact point and the centerline as the estimated radius of the object and the distance between the contact point and the black base plane as the final estimated height of the object.Following the estimation, the robot will interact with the object through tapping motions on three sides of the object in sequences including the top side, the left side, and the backside of the object.During each tapping sequence, the robot hand will approach and tap the object step by step, ensuring continuous tapping while gradually transitioning from top to bottom (or left to right for top tapping) to cover the entire surface of the object. A detailed illustration of the tapping motion is provided in Fig. 16 . The robot relies on the estimated radius to determine whether to collect tapping data on top of the object since the useful information is limited when the surface area on top of the object is too small. We set the radius threshold to be 2cm in this case. Additionally, depending on the estimated height of the objects, the robot decides to start from a higher position or a lower position for two different scales of data collection during the left-side tapping and back-side tapping as illustrated in Fig. 15 . We set the height threshold to be 20cm. For a more comprehensive visual demonstration, please refer to our Supplementary Movies. When our robot hand taps objects, we can determine such contact events from the voltage feedback from the motor in each finger. We put a 10Ω resistor in series with the servo motor power. When the motor encounters external force, it receives an increase of current supply and, as a result, a decrease of voltage from the motor voltage sensor readings. Though we also experimented with detecting salient acoustic signals to detect the binary contact event, we found that the natural motor voltage feedback embedded by the motors provides more reliable and accurate signals. This is because the voltage feedback can reflect subtle resistance encountered by the robot finger, where acoustic vibrations from the contact microphone also include motor noises. After the contact event is detected, we can localize the contact points in 3D space with respect to the robot with the forward kinematics of our robot hand and arm.We conducted this real-world data collection on the 83 real-world objects mentioned in our main texts and used the tapping data only with valid contact events for our material classification, shape reconstruction, and object re-recognition tasks. The tapping data are saved in txt file format, including position in the x-axis (x), position in the y-axis (y), position in the z-axis (z), binary tapping detection from acoustic vibration signal (a), binary tapping detection from voltage signal (v), finger index (f), and index of tapping (i): {x, y, z, a, v, f, i}Each tapping data is also accompanied by a five-second audio recording, which is guaranteed to cover the acoustic signals of the impact sound. We will present the audio processing procedure in the next section.To obtain more useful information, we first extracted the striking vibration signal from the 5-second recording. Typically, the anticipated tapping vibration signal exhibits a noticeable characteristic, often resembling a peak-shaped pattern. However, real-world acoustic vibration signals may contain random noises, so simply extracting the signal around maximum amplitude will not work. To extract informative acoustic vibration signals, we used a window of 1000 units of acoustic waveform to monitor the average absolute amplitude of the acoustic signals. If the average amplitude in the current window is larger than both the average amplitude in the previous window and the next window, 20000 units of the acoustic signals (20000/44100Hz = 0.45351474s = 453.5147ms) will be extracted around the beginning timestamp of the current window. We applied the same extracting mechanism to the recording with no obvious striking signal, for example, foam objects. For those objects, most extracted signals are motor noises. However, we consider this as a feature of the soft materials perceived by our robot hand. Finally, we followed previous work to convert the raw audio signals to Mel spectrogram representations with a dimension of 64 × 64 resolution. The length of the Fast Fourier transform (FFT) window is set to 2048 and the number of the Mel bands is set to 64. The highest frequency is restricted to 8192Hz. To ensure the correct dimension of the Mel spectrogram along the temporal dimension, the number of samples between consecutive frames is rounded to the nearest value obtained by dividing the total number of audio samples by 64.In this section, we explain how we obtained the material label for each of the contact positions around the object. During tapping data collection, we used two depth cameras to capture two point clouds of the object after fixing the object on the platform. We fuse those two point clouds and use the annotated point cloud model to align with the object. We then searched the nearest point of the annotated point cloud for each of the contact points and assigned the annotated material label to that contact point based on the label of the nearest point. The material label serves as the ground truth for training our material classification model.Tapping data from 82 objects is used for the material classification task. We removed the acoustic data collected from a disinfecting wipe because the paper material is too soft and there is a thin plastic film attached to the paper, so the signal is too unique and out of distribution from all other paper materials. We conducted separate training on three different random splits of the dataset to evaluate our method. Note that our splits are based on objects to avoid strong overlapping between training, validation, and testing data. We show the detailed architecture design of our material classification network in Tab. 3.We list all the hyperparameter selections and experimental results across different random data splits in Tab. 4. We select the model based on the best validation accuracy. Due to the testing dataset being Tab. 4: Material classification hyperparameters and results on three random data splits. We leverage the contact positions collected from the 83 objects for our shape reconstruction task. The tapping interaction results in highly sparse contact points. The maximum number of contact points is 300 for each object. We divided the dataset based on 15 shape categories(i.e., bottle, can cup, hammer, mug, wine glass, cube, cube(concave), cylinder, cylinder(concave), cone, quadrangular pyramid, triangular pyramid, prism, irregular) and split the dataset for training, validation, and testing according to these categories. To balance the training dataset for learning, we randomly duplicated the data up to 100 for each category. We then randomly sampled 80% to 90% points and augmented the dataset to 1500 for each shape to augment the training dataset.To further boost the training dataset, we leveraged the PyBullet simulator to collect synthetic tapping points. We collected 629 synthetic 3D models under the same categories of real-world objects. In the simulation, We implemented the same interaction policy and data augmentation techniques as in our real-world data construction. The ground truth model consists of a point cloud model comprising 5000 points for both the real-world objects and the synthetic objects. Our experiments include three different splits of data with different random seeds where both the validation objects and testing objects only include real-world objects. For real-world objects, we split them into 61, 11, and 11 objects for training, validation, and testing. In order to leverage the simulation data to improve our model on our real-world objects, initially, we use all the synthetic datasets to pre-train our model. Gradually, we blended in more real-world data and reduced the percentage of synthetic data in our training set. We show the blending schedule in Tab. 5. 8th Conference on Robot Learning (CoRL 2024), Munich, Germany. 


Decision: The authors present a good hardware platform for using audio feedback to perceive objects. The task and the setting are interesting. The reviewers have questions about the experiment setting, like how the fixture will influence the audio response-- in other words, more discussion of the generalizability of the method is needed. Also, the authors are encouraged to demonstrate the contribution of the method further compared to existing audio-based object perception.

----

The authors addressed most of the reviewers' questions well. While the reviewers still think the use of contact microphone in object detection is not a new concept, they believe the new dataset and results are beneficial to the community.

Reviews: review: title: title: Multimodal sensing using tactile and audio; State Estimation;
paper_summary: paper_summary: The paper presents SonicSense, an integrated hardware and software system designed to enhance robot object perception through in-hand acoustic vibration sensing. SonicSense employs a heuristic exploration policy to interact with objects and uses end-to-end learning algorithms to fuse vibration signals for inferring object properties. The paper demonstrates the effectiveness of SonicSense using a diverse set of 83 real-world objects with significant advancements over existing acoustic sensing methods.
main_review: main_review: The setup of a multi-finger robot hand equipped with contact microphones is quite interesting. The dataset, collected using this system from real-world objects, has significant potential for contribution if made open-source. The tasks tested with this dataset, including container inventory status differentiation, material prediction, 3D shape reconstruction, and object re-identification. Additionally, the paper is well-organized and clearly written and easy to read.

Strengths:
- A novel method to tactile perception using in-hand acoustic vibrations
- A diverse dataset of 83 real-world objects are collected and methods are tested on this dataset to demonstrate effectiveness
- The system setup is not expensive and easy to reproduce

Weaknesses:
- Objects need to be fixed on the table, while in many real world scenarios, the interaction are much more complicated.
- Though using a multi-finger robot hand but no complex manipulation skills are involved
- More modalities can be integrated with audio
questions: questions: - Another two related works in robot manipulation with audio signals [1, 2]
- As mentioned in last section, the interaction is more complicated in the real world scenarios. This might be a open research question, but could you elaborate more on the current role and future work of audio datasets in robot learning community? Robotic audio datasets (maybe not only audio but among vision, audio, and tactile, audio have the least amounts) nowadays still limited in real world applications. Robot audio data often cannot generalize to more realistic scenarios.

[1] Li, Hao, et al. "See, hear, and feel: Smart sensory fusion for robotic manipulation." arXiv preprint arXiv:2212.03858 (2022).
[2] Du, Maximilian, et al. "Play it by ear: Learning skills amidst occlusion through audio-visual imitation learning." arXiv preprint arXiv:2205.14850 (2022).
limitations: limitations: 3
review_summary: review_summary: The setup and dataset of this paper could make a potential contribution to the community, therefore I recommend accept this paper.
title: title: Interesting paper with a relevant contribution
paper_summary: paper_summary: This paper presents an integrated hardware and software solution to recognize object material, shape, and instance. The hardware consists of contact microphones attached to four single-jointed fingers. These are used to explore objects with a simple explorative policy. Contact points, computed based on voltage feedback from the actuated joints, are used to estimate object shape. A feature vector based on audio is used to estimate object material. Audio and contact points are jointly used to recognize object identity. The paper reports good results in each task.
main_review: main_review: The paper investigates a relevant an active research topic: acoustic sensing for manipulation. It is well written and comprehensible. The video and appendix are useful and support the paper well. It is great that the authors jointly tackle the hardware and software side of the problem and the engineering choices are well motivated and comprehensible. The paper makes a solid contribution.

The only critique I have, is that the paper paints itself as focusing on sound, but the shape reconstruction is not sound based. I don't think this is a major problem, as the paper also argues it presents an integrated hardware and software solution, and it shows what is possible given the presented system. However, it would be nice to see if using sound could improve the shape estimation results. E.g. contact points would likely not be useful to estimate if something is solid or hollow, but I could imagine that this is possible using sound. Also if the authors tried this and shape estimation did not benefit from sound, this would also be an interesting result.
questions: questions: - L330: "Unlike previous approaches, our study involves several orders of magnitude more number of real-world objects with complex geometry and heterogeneous materials." -- it is not several orders of magnitude. please downsize this claim...
limitations: limitations: 3
review_summary: review_summary: The paper tackles a relevant problem and makes a solid technical contribution. It could be stronger if they acutally used sound for the shape reconstruction.
title: title: SonicSense: Object Perception from In-Hand Acoustic Vibration
paper_summary: paper_summary: The authors introduce a robot hand with embedded contact microphones and use it for object perception: including object identification, 3d shape reconstruction, and container volume differentiation. They collect example data through a exploration policy outlined here, and finally conclude with a learned method to fuse their system data to infer object properties.
main_review: main_review: The authors introduce a robot hand with embedded contact microphones and use it for object perception: including object identification, 3d shape reconstruction, and container volume differentiation. They collect example data through a exploration policy outlined here, and finally conclude with a learned method to fuse their system data to infer object properties.

The authors aim to expand on previous work by increasing the number of object classes to identify to 83 unique items, covering a range of materials and shapes from real-world and 3d printed objects. They present three different networks for using the audio data for (1) material classification out of 9 classes, (2) 3d pose reconstruction from sparse contact points, and (3) object classification out of 83 (unclear) classes. They also extract 12 manually defined features for ~3 class container volume classification. 

The paper is mostly clear and well-written, I can follow the experimental design and results with just a few questions. The video supplement is a useful addition, covering related works and limitations. Overall, this work appears to be several classifications tasks given audio input, which is technically sound but not a huge step beyond previous work. They claim advancement through the combination of multiple perception capabilities, but as each appears to be an individual network, I am not convinced of the contribution.
questions: questions: The authors fix the objects to the table surface before collecting data samples. Does this affect the 'realism' of the audio signal collected for more real-world scenarios? To confirm, I understand and support this choice for data collection, but I am curious to characterize what artifacts this may add to your samples beyond discussion from limitation. Have the authors been able to compare the fixed vs not-fixed audio samples?

The authors are particular about the object train/test splits not containing any overlap - I agree. To support that claim, the authors should outline their defined objects train/test/validation splits across the objects in the Appendix at the very least. I think it will help readers understand how much variety there was between the splits. 

The robustness to ambient noise section and Figure 6 do not need to be included in the main text, it is a well-known property for contact microphones and not a core focus or contribution of the work. 

The authors use both mel-spectrograms and sparse contact points to identify the objects. Can you ablate these inputs to understand how much of the information can be accounted per input?
limitations: limitations: 3
review_summary: review_summary: This paper appears to combine various concepts for audio-based classifiers that have been previously demonstrated in other works. The combination of multiple tasks/networks may not be sufficient to warrant interest at the conference.