See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation
Authors: Hao Li
Yizhi Zhang
Junzhe Zhu
Shaoxiong Wang
Michelle A Lee
Huazhe Xu
Edward Adelson
Li Fei-Fei
Ruohan Gao
Jiajun Wu
Venue: CoRL 2022 Poster (2022)
Keywords: Multisensory Perception
Robotic Manipulation
Robot Learning

Abstract: Humans use all of their senses to accomplish different tasks in everyday activities. In contrast, existing work on robotic manipulation mostly relies on one, or occasionally two modalities, such as vision and touch. In this work, we systematically study how visual, auditory, and tactile perception can jointly help robots to solve complex manipulation tasks. We build a robot system that can see with a camera, hear with a contact microphone, and feel with a vision-based tactile sensor, with all three sensory modalities fused with a self-attention model. Results on two challenging tasks, dense packing and pouring, demonstrate the necessity and power of multisensory perception for robotic manipulation: vision displays the global status of the robot but can often suffer from occlusion, audio provides immediate feedback of key moments that are even invisible, and touch offers precise local geometry for decision making. Leveraging all three modalities, our robotic system significantly outperforms prior methods.

See, Hear, and Feel:
Smart Sensory Fusion for Robotic Manipulation
Hao Life
Yizhi Zhang! Junzhe Zhu! Shaoxiong Wang*
Michelle A Leel Ruohan Gao Jiajun Wu
Huazhe Xu
Edward Adelson² Li Fei-Fei"
Stanford University
"Massachusetts Institute of Technology
Equal contribution.
*Equal advising.
Abstract: Humans all of their senses to accomplish different tasks in everyday activities. In contrast, existing work on robotic manipulation mostly relies on one, or occasionally two modalities, such as vision and touch. In this work, we systematically study how visual, auditory, and tactile perception can jointly help robots solve complex manipulation tasks. We build robot system that can see with camera, hear with contact microphone, and feel with a vision-basse tactile sensor, with all three sensory modalities fused with self-attention model. Results two challenging tasks, dense packing and pouring, demonstrate the necessity and power of multisensory perception for robotic manipulation: vision displays the global status of the robot but can often suffer from occlusion, audio provides immediate feedback of key moments that are even invisible, and touch offers precise local geometry for decision making. Leveraging all three modalities, our robotic system significantly outperforms prior methods.
Keywords: Multisensory Perception, Robotic Manipulation, Robot Learning
Introduction
Imagine you are savoring tea in a peaceful Zen garden: a robot sees your empty cup and starts pouring. hears the increase of the sound pitch as the water level rises in the cup, and feels with its fingers around the handle of the teapot to tell how much tea is left and control the pouring speed. For both humans and robots, multisensory perception with vision, audio, and touch plays a crucial role everyday tasks: vision reliably captures the global setup, audio sends immediate alerts even for occluded events, and touch provides precise local geometry of objects that reveal their status.
Though exciting progress has been made on teaching robots tackle various tasks [1, 2, 3, 4, 5], limited prior work has combined multiple sensory modalities for robot learning. There have been some recent attempts that use audio [6, 8, 9] or touch [10, 11, 12. 13, 14] in conjunction with vision for robot perception, but no prior work has simultaneously incorporated visual, acoustic, and tactile signals three principal sensory modalities, and study their respective roles on challenging multisensory robotic manipulation tasks.
We aim demonstrate the benefit of fusing multiple sensory modalities for solving complex robotic manipulation tasks, and to provide an in-depth study of the characteristics of each modality and how they complement each other. Towards this end, we equip Franka Emika Panda robot arm with a camera, a contact microphone, and a vision-based tactile sensor to receive visual, acoustic, and tactile signals, respectively. We tackle two challenging robotic manipulation tasks: 1) dense packing, where the goal is to insert a target object into a densely cluttered box (see Fig. 1), and 2) pouring, where the goal is to pour tiny beads from one cylinder container to the other and reach the target level. We choose these two tasks as our case studies due to their broad real-worke applications, complexities, and dependence on multisensory perception.
We propose a new multisensory self-attention model for fusing visual, acoustic, and tactile sensory data, and use imitation learning for action prediction. Our model uses self-attention mechanism to attend both across modalities and across time, outperforming prior fusion methods and its ablated versions without certain modalities by a large margin. Through series of quantitative and quali-
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Figure 1:
Illustration of the real-world dense packing task that we tackle, where the robot needs to insert a glass into the densely cluttered box by leveraging multisessory feedback.
tative analyses, we uncover interesting findings about the characteristics of the three modalities in robotic tasks that correlate human behaviors and agree with intuition.
Our main contributions are threefold. First, we present the first in-depth study that simultancously leverages vision (from camera), audio (from a contact microphone), and touch (from tactile sensor) for robotic manipulation tasks. Second, we introduce a multisensory self-attention model for fusing all three sensory modalities, and strongly outperforms existing approaches. Finally, show its applications on two challenging robotic tasks: dense packing and pouring, and demonstrate the benefit of multisensory perception in robot learning.
2 Related Work
Deep Multimodal Learning. The progress of deep learning algorithms [15, 16, 17, 18, 19] in the last decade has contributed to the the growing potential of learning from heterogeneous sources of data streams, including visual, textural, acoustic, motion, and temporal data. Many real-world problems are inheriently multimodal. Research different fields have combined multiple modalities for various tasks, including representation learning [20, 21, 22, 23), embodied learning [24, 25, 26, 27], video understanding [28, 29, 30, 31, 32], and cross-modality prediction [33, 34, 35, 36, 37]. Different from all the work above, we focus on three important sensory modalities--istte audio, and touch, and investigate their different roles in robotic manipulation tasks.
Robot Learning for Manipulation. Robotic manipulation has been mangsttading challenge [38, 39, 40]. To tackle these challenges, robot learning methods has been widely used in various robotic manipulation tasks [1, 2, 41, 3, 4, 5, 42]. The tasks in this paper are closely related to the classical peg-in-hole insertion task [43, 44] and the pouring task [42, 45]. contrast to the previous work, we leverage multiple sensory modalities to solve complex tasks such as dense packing and pouring to a target amount. In order to accomplish these two tasks, the robot needs to focus on different modalities at different stages of the tasks.
Multisensory Perception for Robotics. Recent inspiring work uses audio touch to enhance robot's perception in a series of interesing tasks. Audio has been shown to be useful for classifying object instances [46] terrain types [47], modeling object dynamics [8, 7], estimating amount and flow of granular material [48], learning food embeddings [9], separating impact sounds [49], and mitigating visual occlusion [6]. Tactile sensing is used for shape reconstruction [14, 50, 51]. robotic grasping [10, 11, 12], peg insertion [13, 52, 53), dense box- nox-packing [54], planar pushing and door
opening [55], and object pose estimation [56].
Despite the encouraging progress, no prior work has simultaneously leveraged visual, acoustic, and tactile sensory data to tackle challenging robotic manipulation tasks. We present the first in-depth

---

history
ResNet
Tactile data
Multiseesonn sel F-aatentioon
ResNet
Dense packing action
MLP
Visual data
ResNet
Acoustic data
Pouring action
(a) Multisensory data capture
(b) Our model
(c) Tasks
Figure 2: Our multisensory robot learning framework. The visual, acoustic, and tactile data from the corresponding sensors (left) processed and fused by multisensory self-attention model (middle) to predict a task-specific action for the dense packing task and the pouring task (right).
study on fusing all three sensory modalities, and systematically explore their respective roles in the context of robot learning. The work most related ours is ObjectFolder [57, 58], where they also use all three sensory modalities. However, while their goal is to create dataset of neural objects, and use simulated multisensory data for some simple tasks, we collect data from a real robot system and study two complex real-world robot manipulation tasks.
3 Multisensory Robot Learning
We aim demonstrate how multisensory perception can help solve complex robotic manipulation tasks. In the following, we first introduce how we capture the visual, acoustic, and tactile data and their respective characteristics (Sec. 3. Fig. 2a). Then, we introduce our multisensory self-attention model for fusing multiple modalities (Sec. 3.2, Fig. 2b). Finally, we show how we leverage multi- sensory data for two challenging robotic tasks, i.e., dense packing and pouring (Sec. 3.3, Fig. 2c).
3.1 Multisensory Data Capture
At each time step, we record a tuple of observations from the visual (V), acoustic (A), and tac- tile (I)) sensors, respectively. Fig. 2a shows our data capture setup and examples of multisensory observations. Below introduce the details of how we enable the robot see, hear, and feel:
See: We capture the visual data using third-person view camera placed facing the robot, which provides global view of the robot setup and its end-effector position.
Hear: We attach piezo contact microphone to the object of interest, and receive acoustic signals through vibration. The microphone is sensitive any contact with the surface that the microphone is attached to and captures minimal environmental noise.
Feel: We attach GelSight sensors [59, 60] to the fingertips of the robot gripper. GelSight sensors are vision-basee tactile sensors that interact with the object through an elastomer and measure the high- nigh-resolution contact geometry and forces with embedded camera.
During data collection, the robot control input is given by the human expert policy, which is selected from task-dependent discrete action space and commanded real-time through the key- board, which we will discuss in 3.3.
3

---

3.2 Multisensory Self-Attention Model
Next, we introduce our Multisensory Self-Attention (MULSA) model as shown in Fig. 2b.
which correspond to the visual, acoustic, and tactile sensory data captured by the camera, contact microphone, and GelSight sensor, respectively. For audio, we use the mel spectrogram representation to encode the acoustic signal. For each modality, we append a brief history of observations from previous timesteps, making Xv E IR x x XA E IR MxxxN, and XT IR x W x3x3xN where N is the number of total observations used, Hv, Wv v are the height and width of the visual images, M. L denote the length of the frequency and the time dimensions of the mel spectrogram, and Hr, Wr are the dimensions of the tactile images.
Model Input. At each timestep, the model takes input
Sensory Feature Extraction. We use ResNet-18 [16] network to extract feature vectors of di- mension D from the observations of all three sensory modalities. We modify the input channels of the first convolution layer of ResNet-18 accordingly. Therefore, for each timestep. we have set of feature embeddings {el, e...... } from each modality, where e E {v, a, t and v, a, t denote the features extracted from the visual, acoustic, tactile data, respectively.
Temporal Feature Fusion. We use a self-attention mechanism to fuse the set of features (Vi,..., VN,, ar...... aN, b,....,tyy L, from the three modalities. We use the standard multi-head self- attention [18], and concatenate the output feature vec- tors from the self-attention layer to obtain the fused multisensory feature m = [o,|02| ... [oaN]. where denotes the concatenation operation. As shown in Fig. 3, there are three types of attention in this process: 1) cross-modality aftention, which allows the features of one modality e, to be compensated by the feature ē; of another different modality at the same time step, 2) cross-lime attention, which allows a feature e; to attend to feature ej a different time step within the same modality to capture the temporal change, and 3) cross- modality and cross-time attention, which combines the above two types. Finally, the multisensory feature m is fed to a Multi-Layer Perceptron (MLP), which outputs the prediction ú.
Vi
aj
t
aj
tj
Cross-modality --> Cross-time
Cross-modality and cross-time
Figure 3: Multisensory self-attention.
Model Training. We train the model following the imitation learning pipeline. The model takes as input the time-aligned multisensory data and the human demonstration action u to learn a policy ú no Xx. All actions are picked from discrete set defined for both manipulation tasks that we tackle (discussed below). We use the cross-entropy loss between the ground truth action and predicted action ú for training the network.
3.3 Task Setup
Fig. 2c illustrates the settings of dense packing and pouring, which we introduce below.
Dense Packing. The goal of this task is to fit an object into small space in an almost full box. To successfully place the object, the robot needs to first move the object from the starting position to location above the box, then avoid obstacles and locate the empty space in the box, and finally correctly insert the object. We perform two versions of dense packing: one with 3D printed objects that allow us to do a more controlled study on the role of each modality for this task, and one with diverse real-world objects.
For the first case, we 3D print cylinder peg as the in-hand object and base that is covered with two walls as substitution for a real box. Similar to a real cardboard box, the base has walls on top of it. As shown in Fig. 4a, we design four types of bumps with different configurations inside the base: 1) hard slanted bump, which has a hard 45 slanted surface, 2) soft slanted bump, which has a 45" slanted surface padded with a layer of soft sponge, 3) left flat bump, which is flat bump located on the left, and 4) back flat bump, which is flat bump located on the back. For the second case, we set up more challenging scenario where we use diverse real-world objects for real dense packing as shown in Fig. 1, and the robot needs to fit a glass into crowded box. In this setup,

---

In-hand cup (150 ml)
Fixed cup (250 ml)
Dans
Electronic weight scale
1-1
Base flom
hani donned homp
burap
hurp
(a) Dense packing
(b) Pouring
Figure 4: Illustration of our task setup for the dense packing task and the pouring task.
have different objects including thin metal board, soft stuffed toy, towel, and plastic plate. During the process, the glass will interact with these objects, and the robot must respond differently to avoid the obstacles.
In this task, the robot predicts the (I,, y, 2) displacement of the end-effector, with each dimension can be positive, negative, or zero. The robot moves along axes to adjust where to insert in the horizontal plane, and vertically along the axis insert the object.
Pouring. The goal of this task is to pour tiny beads from one cylinder container to the other and reach the target level. To complete the task, the robot first needs to locate and align the two cups, then rotate the cup hand to start pouring, and stop the right moment to avoid overflow.
We use small beads of diameters of Imm to resemble water in the pouring task for ease of exper- iments. As shown in Fig. 4b, we use two cylinder containers as cups. One cup is grasped by the robot, with either 60g or 100g beads in it initially before pouring. Another cup is fixed on the table catch the dropped beads. The goal is to pour 40g beads into the fixed cup. The closer the weight of the beads is to the target amount, the better this task is accomplished. An electronic weight scale placed below the fixed cup accurate quantitative measurements.
In this task, the robot predicts 2-dimensional action (r, Φ), where the horizontal movement & is required align the cups, and is the rotation angle that corresponds the pouring action. Same as the dense packing task, the action in each dimension can be positive, negative, or zero.
See Supp. for details of the setup and how collect human demonstrations for these two tasks.
Experiments
We validate the benefit of multisensory perception for robotic manipulation on two challenging tasks: dense packing and pouring.
Baselines:
We compare our model with the following baselines:
Direct Concat: A baseline method that directly concatenates the extracted feature vectors of all time steps from all three modalities. This is commonly used in the literaure [13, 52, 10, 57, 58] for modality fusion, making it broadly representative baseline for standard practice.
Du et al. [6]: A recent mudio-visual imitation learning method that uses audio to mitigate visual occlusion. We adapt their LSSM-based model to fuse the three modalities for action prediction.
MULSA (V): A variant of our model that only uses visual sensory data (V).
MULSA (V+A/V+T): Two variants of our model that only use two of the sensory modalities. The model complements vision with either audio (V+A) or touch (V+T).
MUL.SA (A+T): variant our model that only learns over audio and touch, and adopts heuristic visual policy complete the aligning process during the task.
5

---

Attention senres
Vision
Audio
Touch
30 Time (sec)
Before insertion
Touching bump
Passed bump
(a) Sensory observations of three representative moments for dense packing; no contact happens before inser- tion (left), the peg touches the bump and generates informative acoustic and tactile signals (middle)) and the peg has passed the bump and just needs to pushed down (right).
Attention scores
Vision
Audio
Touch
Time (scc)
Aligning cups to pour out the first bead
Stop rotation
Stop pouring
(b) Sensory observations of three representative moments for pouring: the robot aligns the two cups and the very first head drops into the cup (left), the rohot stops rotating the held cup pervent beads from overflowing (middle), and cup is retrieved to end pouring (right).
Figure 5: Visualization of the aggregated attention scores for each modality the robot completes (a) the dense packing task and (b) the pouring task in two test trials.
4.1 Physical Setup and Implementation Details
Robot Setup. We run all experiments on a Franka Emika Panda Arm, which is equipped with an inverse kinematies solver that maps 6-DoF Cartesian space displacement command input to the 7- DoF joint action. For both tasks, we generate Cartesian space displacement commands at a policy frequeney of 10 Hz. Then the low-level control loop maps the displacement to the force and torque input in the joint space to the robot at 200 Hz.
Data Collection. As shown in Fig. 2a, the visual data are recorded by an Intel RealSense D435 camera with a resolution 320 240 at a frequency 60Hz. The acoustic data are recorded with HOYUJI TD-11 11 piezo-electric contact microphone at sampling rate of 14.44..k0 1kHz. The microphone connected to a TASCAM US-4 x 4HR audio interface, which transmits the signal to worksta- tion. The tactile data are recorded by GelSight [59] sensor with resolution of 400 300 and frequency 30Hz.
We implement our multisensory self-attention model in PyTorch. We downsample the visual and tactile frames to dimension 140 105, and randomly crop regions of size 128 x 96 during training. For audio, we resample the received signal from our piezo-electric contact microphone at 16kHz. The short-time Fourier transform is computed using a Hann window of length 25ms, hop length of 10ms, and mel size of 64 to generate a mel spectrogram of size 64 x 50 for a 0.5s audio segment at each time step. We use = 6 observations for three modalities. The 6 frames of the visual and tactile obseravtions span time interval of 3 seconds to align with the audio input. See Supp. for details of our model architecture.
Model Details.
4.2 Dense Packing
We collect 10 episodes of human demonstrations for dense packing for each of the four designed bases (hard slanted, soft slanted, back flat, left flat). We train a single policy for all four scenarios

---

Hard slanted
Soft slanted
Back flat
Left flat
Avg. succ. rate
Direct Coneat
0.90
0.50
0.00
1.00
0.60
Du et al. [6]
0.50
0.30
0.50
0.60
0.48
MULSA (V)
0.00
0.00
0.00
1.00
0.25
MULSA (V+A)
0.80
1.00
0.10
0.50
0.60
MULSA (V+T)
0.00
1.00
1.00
1.00
0.75
MULSA (A+T)
0.20
1.00
1.00
1.00
0.80
MULSA (ours, V+A+T)
1.00
1.00
1.00
1.00
1.00
Dense packing results using 3D printed objects. We report the success rate for testing on each type of the designed four bases and the average performance.
Table 1:
and run 10 test trials on each base with random starting locations. A trial is considered successful if the robot can pass the bump and insert the peg into the base.
Table I shows the results. our model MULSA using all three modalities outperforms all the baselines and achieves a 100% success rate in all four scenarios. In contrast, MULSA with vision (V), vision and audio (V + A), and vision and touch (V + T) only achieve an average success rate of 25%, 60%, 75,, respectively, demonstrating the usefulness of each modality and the benefit of multisensory perception. Particularly, from this ablation study, see that the visual and tactile modalities turn out to be sufficient for the model to learn the correct policy on the flat bumps. The tactile sensor can easily distinguish these different in-hand object poses, therefore extra information in the audio is unnecessary. However, we observe that the model cannot tell the difference between the hard and the soft slanted bumps if not using audio, because the two bumps produce very similar visual and tactile feedbacks, while the sound generated when contacting the hard surface can easily distinguish them. On the other hand, the tactile signal can be very important to distinguish the two flat bumps. When pushed onto these two bumps, the peg tilts in different directions, revealing critical local geometry to avoid the obstacles. When the model only learns over audio and touch and adopts a heuristic visual policy that moves the end-effector to spot very close to the base, the major failure mode observed occurs on the hard slanted bump. where the robot successfully avoids the bump but fails to continue inserting downward after the signal disappears in audio and touch. Our model also has large gains compared to Direct Concat and Du et al. [6], showing the power of our multisensory self attentono model for modality fusion.
Our model uses self-attention mechanism dynamically attend useful modalities at each time step, and fuses information from the multisensory data. In Fig. 5a, we illustrate the change of the aggregated attention scores of the three modalities in one test trial as the robot inserts the peg into the base with a hard slanted bump. At the beginning of the trial, the peg location is randomly initialized, therefore the model focuses on vision to locate the correct position to insert. After the peg is partially inserted into the base, vision suffers from occlusions by the walls of the base, shifting the model's attention towards both audio and touch. Specifically, there is an obvious contact sound, alerting that the peg is touching hard bump. Then touch tells how the peg is tilting, helping the model to infer in which direction the robot should move next avoid the bump. At the end of the trial, the peg has already passed the bump, so the attention is back to vision to move the robot arm downward. See Supp. video for more qualitative examples.
The controlled experiments with 3D printed objects above help us analyze the characteristics of each modality in these different settings. Table shows our results on dense packing using real-world objects as shown in Fig. to further test the effectiveness of our model and demonstrate the power of multisensory perception. Note that this is a challenging setting due to the diverse set of objects of different shapes and material properties. One major failure mode observed in this setting is that the robot cannot recognize whether the obstacle is hard or soft, thus pushing too hard onto a hard surface and getting stuck. When touching hard surfaces, such as the plastic plate, the robot should move to avoid breaking them. If the object soft, such as the toy or the towel, the robot should squeeze down because they can deform. Another typical failure mode of the baseline models is that the robot misses the correct range of the empty space in the box during the initial aligning phase, and the glass is moved out of the box. Still, our model consistently outperforms all the baselines.
7

---

Initial weight (g)
60
100
Average
Direct Concat
2.24 + 1.34
2.69 + 1.85
2.46 + 1.54
Methods
Suce, rate
Du et al. [6]
7.35 + 3.59
2.70 + 2.19
5.02 + 3.72
Direct Concat
0.60
MULSA (V)
20.87 = 0.0
60.87 = 0.0
40.87 = 0.0
Du al. [6]
0.30
MULSA (V+A)
8.89 - 0.96
21.51
15.20 + 7.90
MULSA (ours) 0.80
MULSA (V+T)
2.81 = 1.30
8.35 = 1.14
5.58 + 3.14
MULSA (V+A+T)
1.06 + 0.83
1.76 1.16
1.41 + 1.02
Table 2: Dense packing re- sults with real-world objects.
Table 3: Mean weight error of beads poured into the cup (mean = standard deviation) (g).
4.3 Pouring
We collect 10 human demonstrations with two initial settings: 60g or 100g beads held in the cup. The goal is to pour 40g beads into the fixed cap. We train single policy for these two settings and run 10 test trails for each setting with the fixed cup slightly shifted from the training position.
Table 3 summarizes the results. We show the mean weight error of the beads poured into the fixed cup compared to the target weight. MULSA with only vision (V) always keeps pouring all beads into the fixed cup since it has clue about the current status due to the highly similar visual feedback across the entire trial. With the help of one extra modality (V + A or V + T), the model obtains large gains. Our MULSA model using all three modalities achieve the best results, outperforming the two baseline methods that use a different mechanism for fusing the three modalities.
In Fig. 5b, we visualize three key moments when model attends to different modalities. When pouring is about to begin, the model attends to vision to align the two cups, with some attention on audio to capture the sound of the first few dropping beads. Later, more beads start to drop and make sounds in the fixed cup. The pitch of the sound increases as more beads are poured into the cup. The center of mass of the in-hand cup also begins to vary, making changes to the gel deformation of the GelSight sensor. Therefore, the model's attention scores are gradually shifted to audio and touch. To prevent the overflow of the beads, the robot needs to stop rotation and transit to a static pose at the right moment. The last critical moment is to predict when to stop pouring, which heavily relies on the acoustic feedhack shown in the last example. This also agrees with experience when testing without the audio feedhack- We are more likely to see a lag in prediction at the end of the whole pouring process, leading to more beads being poured out than desired.
5 Conclusion and Limitations
We presented a multisensory robot learning framework that leverages visual, acoustic, and tactile sensory data to solve two challenging manipulation tasks: dense packing and pouring. Our multi- sensory self-attention model strongly outperforms series of baseline methods by effectively fusing and filtering out the most useful information from the three sensory modalities. We have the fol- lowing observations for the roles of vision, audio. and touch for robotic manipulation tasks: Vision usually provides general positional information, yet it often suffers from occlusion. Audio provides immediate feedback and also indicates the characteristics of the objects such material properties, but is less informative when there is no contact. Touch captures in-hand object dynamics, which best indicates the local geometry as well as the direction and magnitude of the contact force. We hope our work can inspire new research to use multisensory perception for robot learning.
Our current framework uses behavioral cloning for learning the expert policy, which requires collecting human demonstrations. Though we have shown encouraging results using im- itation learning, the way robot perceives multisensory signals might be different from how humans leverage them during demonstrations. It would be interesting future work to use reinforce- ment learning to have the robot automatically uncover the best way of fusing multisensory data streams and tackle more challenging manipulation tasks. Another potential improvement is to es- tablish more general robot setup that can be easily adapted to more general tasks. For example, we can design a tailored robot gripper that has the contact microphone and the GelSight sensor built into it to receive the acoustic and tactile signals simultaneously in any robot manipulation tasks.
Limitations:
8

---

Acknowledgments
We thank Chen Wang, Kyle Hsu, Yunzhi Zhang, and Koven Yu for helpful discussions or feedback on paper drafts. The work is in part supported by the Stanford Institute for Human-Centered Al (HAT), the Toyota Research Institute (TRI), NSF RI #2211258, ONR MURI N00014-22-1-2740, Adobe, Amazon, Meta, and Samsung.
References
[1] O. Kroemer, S. Niekum, and G. D. Konidaris. A review of robot learning for manipulation: Challenges, representations, and algorithms. JMLR, 2021.
[2] T. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P. Abbeel, J. Peters, et al. An algorithmic perspective on imitation learning. Foundations and Trends@ in Robotics, 2018
[3] A. Nagabandi, K. Konolige, S. Levine, and V. Kumar. Deep dynamics models for learning dexterous manipulation. In CoRL, 2020.
[4] H. Shi, H. Xu, Z. Huang, Y. Li, and J. Wu. Robocraft: Learning see, simulate, and shape elasto-plastic objects with graph networks. In RSS, 2022.
[5] O. M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz. B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 2020.
[6] M. Du, O. Lee, S. Nair, and C. Finn. Play it by ear: Learning skills amidst occlusion through audio-visual imitation learning. In RSS, 2022.
[7] C. Matl, Y. Narang, D. Fox, R. Bajesy, and F. Ramos. Stressd: Sim-to-real from sound for stochastic dynamics. In CoRL, 2020.
[8] D. Gandhi, A. Gupta, and L. Pinto. Swoosh! rattle! thump!-actions that sound. In RSS, 2020.
[9] A. Sawhney, S. Lee, K. Zhang, M. Veloso, and O. Kroemer. Playing with food: Learning food item representations through interactive exploration. International Symposium on Experi- mental Robotics, 2020.
[10] R. Calandra, A. Owens, D. Jayaraman, Lin, W. Yuan, J. Malik, E. H. Adelson, and S. Levine. More than a feeling: Learning to grasp and regrasp using vision and touch. RA-I., 2018
[11] L. Pinto, D. Gandhi, Y. Han, Y-L. Park, and A. Gupta. The curious robot: Learning visual representations via physical interactions. In ECCV, 2016.
[12] R. Calandra, A. Owens, M. Upadhyaya, W. Yuan, Lin, E. H. Adelson, and S. Levine. The feeling of success: Does touch sensing help predict grasp outcomes? In CoRL, 2017.
[13] M. A. Lee, Y. Zhu, K. Srinivasan, P. Shah, S. Savarese, L.. Fei-Fei, A. Garg, and J. Bohg. Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks. In ICRA, 2019.
[14] E. J. Smith, R. Calandra, A. Romero, G. Gkioxari, D. Meger, J. Malik, and M. Drozdzal. 3d shape reconstruction from vision and touch. NeurIPS, 2020.
[15] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NeurIPS, 2012.
[16] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
[17] D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.
[18] A. Vaswani, N. Shazeer, Parmar, J. Uszkoreit, L. Jones, N. Gomez, t.,. Kaiser, and I. Polo- sukhin. Attention is all you need. In NeurIPS, 2017.

---

[19] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810004805, 2018.
[20] J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A. Y. Ng. Multimodal deep learning. In ICML, 2011.
[21] Liang, Lyu, X. Fan, Z. Wu, Y. Cheng, Wu, L. Y. Chen, P. Wu, M. Lee, Y. Zhu, et al. Multibench: Multiscale benchmarks for multimodal representation learning. In NeurlPS Datasets and Benchmarks Track, 2021.
[22] A. Owens and A. A. Efros. Audio-visual scene analysis with self-supervised multisensory features. In ECCV, 2018.
[23] H. Akbari, L. Yuan, R. Qian, W.-H. Chuang, S.-F. Chang, Y. Cui, and B. Gong. Vatt: Trans- formers for multimodal self-supervised learning from raw video, audio and text. In NeurIPS, 2021.
[24] Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. Reid, S. Gould, and A. Van Den Hengel. Vision-and-language navigation: Interpreting risually-groundee naviga- tion instructions in real environments. In CVPR, 2018.
[25] C. Chen, U. Jain, C. Schissler, S. V. A. Gari, Z. Al-Halah, V. K. Ithapu, P. Robinson, and
K. Grauman. SoundSpaces: Audio-visual navigation 3d environments. In ECCV, 2020.
[26] C. Chen, S. Majumder, A. Ziad, R. Gao, S. Kumar Ramakrishnan, and K. Grauman. Learn- ing set waypoints for audio-visual navigation. In ICLR, 2021.
[27] C. Gan, Schwartz, S. Alter, M. Schrimpf, J. Traer, De Freitas, Kubilius, A. Bhandwal- dar, N. Haber, M. Sano, et al. Threedworld: A platform for interactive multi-mode physical simulation. In NeurlPS Datasets and Benchmarks Track, 2021.
[28] R. Zellers, J. Lu, X. Lu, Y. Yu, Y. Zhao, M. Salehi, A. Kusupati, J. Hessel, A. Farhadi, and Y. Choi. Merlot reserve: Multimodal neural script knowledge through vision and language and sound. In CVPR, 2022.
[29] R. Gao, T.-H. Oh, K. Grauman, and L. Torresani. Listen to look: Action recognition by previewing audio. In CVPR, 2020.
[30] A. Nagrani, S. Yang, A. Amab, A. Jansen, C. Schmid, and C. Sun. Attention bottlenecks for multimodal fusion. In NeurlPS, 2021.
[31] Y. Zhang, H. Doughty, L. Shao, and C. G. Snock. Audio-adaptive activity recognition across video domains. In CVPR, 2022.
[32] K. Simonyan and A. Zisserman. Two-stream convolutional networks for action recognition in videos. In NeurlPS, 2014.
[33] Y. Li, J.-Y. Zhu, R. Tedrake, and A. Torralba. Connecting touch and vision via cross-modal prediction. In CVPR. 2019.
[34] S. Antol, A. Agrawal, Lu, Mitchell, D. Batra, C. Zitnick, and D. Parikh. Vqa: Visual question answering. In ICCV, 2015.
[35] R. Garg, R. Gao, and K. Grauman. Geometry-aware multi-task learning for binaural audio generation from video. In BMVC, 2021.
[36] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2004.06125. 2022.
[37] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. In ICML. PMLR, 2021.
[38] M. Mason. Toward robotic manipulation. Annual Review of Control, Robotics, and Au-
tonomous Systems, 2018
10

---

[39] F. Ruggiero, V. Lippiello, and B. Siciliano. Nonprehensile dynamic manipulation: A survey. RA-L, 2018.
[40] M. Raibert and J. Craig. Hybrid position/force control of manipulators. Journal of Dynamic Systems, Measurement, and Control, 102: 1981.
[41] B. D. Argall, S. Chernova, M. Veloso, and B. Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 2009.
[42] P. Sermanet, C. Lynch, Y. Chebotar, J. Hsu, F. Jang, S. Schaal, S. Levine, and G. Brain, Time- contrastive networks: Self-supervised learning from video. In ICRA, 2018.
[43] D. E. Whitney al. Quasi-static assembly of compliantly supported rigid parts. Journal of Dynamic Systems, Measurement, and Control, 1982.
[44] J. L. Nevins and D. E. Whitney, Research on advanced assembly automation. Computer, 1977.
[45] H. Liang, S. Li, X. Ma, N. Hendrich, T. Gerkmann, F. Sun, and J. Zhang, Making sense of audio vibration for liquid height estimation in robotic pouring. In IROS, 2019.
[46] J. Sinapov, M. Wiemer, and A. Stoytchev. Interactive learning of the acoustic properties of household objects. In ICRA, 2009.
[47] Christie and N. Kottege. Acoustics based terrain classification for legged robots. In ICRA, 2016.
[48] S. Clarke, T. Rhodes, C. G. Atkeson, and O. Kroemer. Learning audio feedback for estimating amount and flow of granular material. CoRL, 2018.
[49] S. Clarke, N. Heravi, M. Rau, R. Gao, J. Wu, D. James, and J. Bohg. Diffimpact: Differentiable rendering and identification of impact sounds. In CoRL, 2021.
[50] E. Smith, D. Meger, I.. Pineda, R. Calandra, J. Malik, A. Romero Soriano, and M. Drozdzal. Active 3d shape reconstruction from vision and touch. NeurlPS, 2021.
[51] S. Suresh, Z. Si, J. G. Mangelson, W. Yuan, and M. Kaess. Shapemap 3-d: Efficient shape mapping through dense touch and vision. In ICRA, 2022.
[52] M. A. Lee, Y. Zhu, Zachares, M. Tan, K. Srinivasan, S. Savarese, L. Fei-Fei, A. Garg, and
J. Bohg. Making sense of vision and touch: Learning multimodal representations for contact- rich tasks. IEEE Transactions on Robotics, 2020.
[53] S. Dong, D. K. Jha, D. Romeres, S. Kim, D. Nikovski, and A. Rodriguez. Tactile-rl for inser- tion: Generalization objects of unknown geometry. In ICRA, 2021.
[54] S. Dong and A. Rodriguez. Tactile-based insertion for dense box-packing. In IROS, 2019.
[55] M. Lee, B. Yi, R. Martín-Martnn, S. Savarese, and J. Bohg. Multimodal sensor fusion with differentiable filters. In IROS, 2020.
[56] M. Bauza, E. Valls, B. Lim, T. Sechopoulos, and A. Rodriguez. Tactile object pose estimation from the first touch with geometric contact rendering. In CoRL, 2020.
[57] R. Gao, Y.-Y. Chang. S. Mall, L. Fei-Fei, and J. Wu. Objectfolder: A dataset of objects with implicit visual, auditory, and tactile representations. In CoRL, 2021.
[58] R. Gao, 7. Si, Y.-Y. Chang, S. Clarke, J. Bohg, I.. Fei-Fei, W. Yuan, and J. Wu. Objectfolder 2.0: multisensory object dataset for sim2real transfer. In CVPR, 2022.
[59] W. Yuan, S. Dong, and E. H. Adelson. Gelsight: High-resolution robot tactile sensors for estimating geometry and force. Sensors, 2017.
[60] Wang, Y. She, Romero, and E. Adelson. Gelsight wedge: Measuring high-resolution 3d contact geometry with compact robot finger. In ICRA, 2021.
11


Reviews:
[{"review_id": "U2W8AJOJ_Ej", "reviewer": "Reviewer_329W", "summary": "The paper proposes a robot system that fusing three sensory modalities, namely seeing, hearing, and touching, with a self-attention model. The system is deployed to accomplish two challenging tasks, dense packing and pouring. The hardware experimental results show that the robot system with such fusion of three modalities outperforms the systems with only two modalities or without the self-attention mechanism.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strengths of the paper are:\n1.\tThe training and testing of the self-attention model are done with real hardware, which makes the conclusions more convincing.\n2.\tThe experimental setup are well explained with relevant details. \n3.\tThe supplementary video is helpful in demonstrating the qualitative performances of the robot system.\n\nThe main weaknesses of the paper are:\n1.\tSome results of the validation experiments are not very intuitive to understand. The authors need to discuss a bit more to make the results more meaningful to the readers. For example, in the dense packing tasks, the MULSA (V+A+T) method outperforms the MULSA (V+T) only in the Hard slanted case. Seems it’s always the case if the robot is able to obtain more information about the environment, it is more likely to distinct different situations, given that it uses more or less the same algorithms.\n2.\tIt would be more convincing if the authors could open source their codes in the near future for reproduction of their results in the community.\n\nThe paper shows promising results in ultilizing smart sensor fusion with a self-attention model, which is inspiring for other researchers in the community.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper presents a concise yet effective design of multi-modal sensory fusion and impress the readers with interesting demonstrations in real robotic manipulation tasks. The idea is interesting to the community and the presentation is well organized. Hence, I recommend accepting the paper.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "1.\tThe authors are suggested to add the variations in addition to the average performance in Table 1, 2, and 3 to show the consistency of the proposed robot system.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "RlQIHVLISRFp", "reviewer": "Reviewer_yYyH", "summary": "The main contribution of the paper is the system setup of using 2 gelsight finger tips, 1 contact microphone, and a RealSense D435 camera looking at the robot to enable the robot to accomplish 2 tasks: placing an item into a cluttered box / putting a peg in different 3d printed boxes to simulate different packing scenarios, and pouring beads from one graduated cylinder to another. They way they accomplish this is by feeding all 3 data streams into individual resnet18 networks which are then concatenated and fed through a self-attention model that finally goes into a MLP that is used to control the robot. The main way they train the robot's policy is through collecting human demonstrations by allowing them to move the robot in delta movements and tilting the cylinder to start pouring.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think the combination of all the sensors into a combined system is a good engineering effort. Using the self-attention mechanism also makes sense to allow the robot to figure which modality would be most helpful at a certain point in the task. However, I am not really sure if the experiments were that noteworthy. If the authors had used the Franka robot's force sensing and some simple impedance control, would they have been able to achieve similar performance in the peg in box task? In addition, I have seen a couple of other interesting papers of pouring liquids into containers by using a non-contact microphone because it would be a pain to have to attach a contact microphone to each object. The paper I am referencing is \"Making sense of audio vibration for liquid height estimation in robotic pouring\" because it is true that when water fills up in certain bottles while I am refilling it, the sound will change as it rises and I can use that to figure out when to stop. Why was the contact microphone not attached to the robot instead so it could transfer across both tasks without changing the setup between the 2 tasks? Is it because the contact microphone would not be able to pick up sounds due to dampening between objects due to the gelsight sensor? Basically, I think that the experiments shown in this paper do show the useful properties of incorporating all 3 sensor modalities. However, I would have instead liked for the robot to be fully sensorized instead, so that changing tasks would not have required any significant differences. Also I wasn't sure where the contact microphone was located under the actual cardboard box during that task or if it was attached to the side because in the video, you first showed that the contact microphone was placed onto the table surface, but when I looked closely at each task, the contact microphone was attached to the 3d printed boxes, which transmits the sound well, but different materials might transmit the sound differently and may not work as well. Also the contact microphones were attached under the target graduated cylinder for the pouring task, which is again different. Also I feel like the limitations section was a bit short with just a basic description about using human demonstrations that are not optimal. I feel like you could have written some more concrete limitations about how in certain tasks, these extra modalities may not be useful as well if there is widespread occlusion from the static camera. I feel like it would have been better to also have a robot mounted wrist camera to provide additional information for the peg in the box task. It is way more useful than putting the camera outside the box where of course there will be occlusion, so I think that by not putting the camera on the wrist, the problem got inherently harder in order to demonstrate the efficacy of using the other sensors, but I think that if you had used the robot's force sensors, and a wrist camera, the task would have been much more easily solved.\nAnother minor critique is that the video was a bit too long. It was 13 minutes. Next time, please keep it short like around 5-8 minutes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall my recommendation is weak accept. I think that even though the authors used ResNet18 coupled with a self-attention model and traditional sensors (Gelsight, Realsense D435, and contact microphones), they were able to display how their method did improve on just a direct concatenation method as well as in the ablation studies. However, I still think there are many improvements that can be made in future work regarding sensor placement for actual reuse between a wide range of tasks. If other reviewers think that there was not enough originality in the work for CoRL, then I would agree with them for a weak reject. Overall, I would not mind if this paper were to be accepted at CoRL, but I do not feel strongly either way.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "So as I said in my strengths and weaknesses, I feel like there are some issues that need to be addressed by a change in the setup where you could use a wrist-mounted Realsense D435 camera and the Franka Robot's in-built force sensing to try and also complete the first peg in the box task. In addition, I feel like an external non-contact microphone would be useful in sensing pouring even without vision.\nHowever, I don't think it would be fair to you as an author to have to redo all the experiments by switching the setups. I just wanted to warn you that the paper would be much stronger if the robot sensor setup didn't have to be customized for every different situation. Otherwise we would be approaching the point of automation where if we wanted to be extra precise, you could have put an overhead camera to estimate the mesh of the box and do the planning for the cluttered box placing just like in industrial pick and pack startups or using a overhead laser and usb-connected scale in order to precisely pour the exact amount desired.\n\nSo basically, I want you to discuss these limitations in your actual limitations section instead of just talking about needing human demonstrations instead of using pure reinforcement learning in your current limitations section. \nI think you did contribute something with your setup and self-attention model; however, I think the tasks that were chosen to demonstrate the efficacy of the system were not really ideal to show the reason why you need all the different modalities. It would have been better if you had done some like previous cargo transport like graham crackers. That would show off why using the tactile sensors would prevent the robot from closing the gripper all the way in order to not break the graham cracker and a microphone could detect when the graham cracker crumbles. The vision would be helpful for locating the graham cracker for the initial grasp location."}}, {"review_id": "FO8WTAgHtn1", "reviewer": "Reviewer_2ByC", "summary": "This work proposes a new multisensory self-attention model for fusing visual, acoustic, and tactile sensory data, and uses imitation learning for action prediction. The model uses a self-attention mechanism to attend both across modalities and across time, outperforming prior fusion methods. They use a self-attention mechanism to fuse the set of features using a standard multi-head self-attention architecture.\n\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- The use of multi-sensory inputs to perform robotics tasks\n- Some level of engineering required to build and integrate the sensors and synchronize the data acorss multi modality\n- The results are compeling and the evaluations of the effect of adding each modality is also convincing\n- There is a lot of detail about their hardware setup, their 3D printed task items etc.\n- The use of multi sensory inputs in this domain is rather new and more work is exploring this area.\n\n### Weaknesses\n- The architecture on its own is very simple, its just a simple feature extraction using ResNet and feature fusion using multi-head self attention architecture.\n- The paper mostly contains engineering detail of the task setup (This is still useful but this is more of an engineering paper).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I definitely like the idea of applying sensory fusion techniques to utilize multiple sensors. The experimental setup and the results are convincing. My only thought is that, the method on its own, is not novel, its just a simple architecture for combining multiple modality. Even though it does establish a baseline to further explore other ideas to improve upon.\nFinally, this paper for me is more of a well-executed robotics engineering paper, as opposed to strong core ideas being proposed here.\nI am also not familiar with related work in this domain, but again combining multiple sensors for this particular manipulation domain is new but I imagine there might be more related work combining image and touch sensors.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- I did not come across any discussion with regards to sensor calibration and synchronizing across multi modality - Did the authors find this to be an issue? Or was combining all the sensors a smooth process in general?\n- There is reference to a method called `mel spectrogram representation` - I'm not familiar with this, perhaps the authors can provide a reference and explanation of how this data is generated."}}, {"review_id": "DScZgqYYy5", "reviewer": "Reviewer_YUYT", "summary": "This paper presents a multisensory robotic system that can leverage vision, tactile (vision-based sensor) as well as contact microphone data. They propose using a multisensory self-attention model to simultaneously leverage each of the sensory modalities and present results on a dense packing task, and a pouring task with pouring a fixed quantity of beans into a cup.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Proposes a simple attention-based architecture capable of combining visual, tactile and audio inputs.\n- Demonstrates multimodal learning from just ten human demonstrations for each task.\n\nWeaknesses:\n- Dense packing experiment in the real world is lacking significant details on how the experiment was performed. What were the types of objects used? Were there harder and easier objects? How was success defined? \n- Pouring experiment is way too specific and could just be overfitting. Even if you fix the number of beans you pour into the cup, at least vary the quantity of beans you start with.\n- There is no real justification for using vision for these tasks and no baselines that use just audio and tactile sensing and a basic heuristic visual policy. The visual aspect of these tasks seems relatively trivial and achievable without needing to be learned.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While the results demonstrated in the paper are promising, the tasks are very limited in scope to convincingly demonstrate the effectiveness of this method for more general tasks. A number of details are missing on the dense packing real-world experiment, while the pouring task operates with fixed initial and and final conditions. \n\n-------\n\nI thank the authors for their detailed response and the extra details and experiments provided. While I still believe the tasks are limited in scope and the method involves minimal novelty in terms of technical contribution, it is a commendable engineering effort and shows a promising approach for learning from multiple modalities. ", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- More details and analysis of failure modes on the real-world setting for the dense packing experiment\n- Results on the pouring task would be interesting to see in a more general setting. Can multimodal inputs actually help the policy have a better sense of what quantity of material has been poured? It is unclear if this is a well-defined task because even humans are not very good at estimating quantity poured open loop without looking at the level to which the cup has been filled\n- It is unclear how much visual input any of these policies need to be successfully executed. It would be useful to see a tactile+audio baseline on these tasks using a basic heuristic visual policy if required."}}]

Author rebuttal:

Decision: Accept (poster)