Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a multisensory robotic system that can leverage vision, tactile (vision-based sensor) as well as contact microphone data. They propose using a multisensory self-attention model to simultaneously leverage each of the sensory modalities and present results on a dense packing task, and a pouring task with pouring a fixed quantity of beans into a cup.
strengths_and_weaknesses: Strengths:
- Proposes a simple attention-based architecture capable of combining visual, tactile and audio inputs.
- Demonstrates multimodal learning from just ten human demonstrations for each task.

Weaknesses:
- Dense packing experiment in the real world is lacking significant details on how the experiment was performed. What were the types of objects used? Were there harder and easier objects? How was success defined? 
- Pouring experiment is way too specific and could just be overfitting. Even if you fix the number of beans you pour into the cup, at least vary the quantity of beans you start with.
- There is no real justification for using vision for these tasks and no baselines that use just audio and tactile sensing and a basic heuristic visual policy. The visual aspect of these tasks seems relatively trivial and achievable without needing to be learned.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: While the results demonstrated in the paper are promising, the tasks are very limited in scope to convincingly demonstrate the effectiveness of this method for more general tasks. A number of details are missing on the dense packing real-world experiment, while the pouring task operates with fixed initial and and final conditions. 

-------

I thank the authors for their detailed response and the extra details and experiments provided. While I still believe the tasks are limited in scope and the method involves minimal novelty in terms of technical contribution, it is a commendable engineering effort and shows a promising approach for learning from multiple modalities. 
issues: - More details and analysis of failure modes on the real-world setting for the dense packing experiment
- Results on the pouring task would be interesting to see in a more general setting. Can multimodal inputs actually help the policy have a better sense of what quantity of material has been poured? It is unclear if this is a well-defined task because even humans are not very good at estimating quantity poured open loop without looking at the level to which the cup has been filled
- It is unclear how much visual input any of these policies need to be successfully executed. It would be useful to see a tactile+audio baseline on these tasks using a basic heuristic visual policy if required.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The main contribution of the paper is the system setup of using 2 gelsight finger tips, 1 contact microphone, and a RealSense D435 camera looking at the robot to enable the robot to accomplish 2 tasks: placing an item into a cluttered box / putting a peg in different 3d printed boxes to simulate different packing scenarios, and pouring beads from one graduated cylinder to another. They way they accomplish this is by feeding all 3 data streams into individual resnet18 networks which are then concatenated and fed through a self-attention model that finally goes into a MLP that is used to control the robot. The main way they train the robot's policy is through collecting human demonstrations by allowing them to move the robot in delta movements and tilting the cylinder to start pouring.
strengths_and_weaknesses: I think the combination of all the sensors into a combined system is a good engineering effort. Using the self-attention mechanism also makes sense to allow the robot to figure which modality would be most helpful at a certain point in the task. However, I am not really sure if the experiments were that noteworthy. If the authors had used the Franka robot's force sensing and some simple impedance control, would they have been able to achieve similar performance in the peg in box task? In addition, I have seen a couple of other interesting papers of pouring liquids into containers by using a non-contact microphone because it would be a pain to have to attach a contact microphone to each object. The paper I am referencing is "Making sense of audio vibration for liquid height estimation in robotic pouring" because it is true that when water fills up in certain bottles while I am refilling it, the sound will change as it rises and I can use that to figure out when to stop. Why was the contact microphone not attached to the robot instead so it could transfer across both tasks without changing the setup between the 2 tasks? Is it because the contact microphone would not be able to pick up sounds due to dampening between objects due to the gelsight sensor? Basically, I think that the experiments shown in this paper do show the useful properties of incorporating all 3 sensor modalities. However, I would have instead liked for the robot to be fully sensorized instead, so that changing tasks would not have required any significant differences. Also I wasn't sure where the contact microphone was located under the actual cardboard box during that task or if it was attached to the side because in the video, you first showed that the contact microphone was placed onto the table surface, but when I looked closely at each task, the contact microphone was attached to the 3d printed boxes, which transmits the sound well, but different materials might transmit the sound differently and may not work as well. Also the contact microphones were attached under the target graduated cylinder for the pouring task, which is again different. Also I feel like the limitations section was a bit short with just a basic description about using human demonstrations that are not optimal. I feel like you could have written some more concrete limitations about how in certain tasks, these extra modalities may not be useful as well if there is widespread occlusion from the static camera. I feel like it would have been better to also have a robot mounted wrist camera to provide additional information for the peg in the box task. It is way more useful than putting the camera outside the box where of course there will be occlusion, so I think that by not putting the camera on the wrist, the problem got inherently harder in order to demonstrate the efficacy of using the other sensors, but I think that if you had used the robot's force sensors, and a wrist camera, the task would have been much more easily solved.
Another minor critique is that the video was a bit too long. It was 13 minutes. Next time, please keep it short like around 5-8 minutes.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: Overall my recommendation is weak accept. I think that even though the authors used ResNet18 coupled with a self-attention model and traditional sensors (Gelsight, Realsense D435, and contact microphones), they were able to display how their method did improve on just a direct concatenation method as well as in the ablation studies. However, I still think there are many improvements that can be made in future work regarding sensor placement for actual reuse between a wide range of tasks. If other reviewers think that there was not enough originality in the work for CoRL, then I would agree with them for a weak reject. Overall, I would not mind if this paper were to be accepted at CoRL, but I do not feel strongly either way.
issues: So as I said in my strengths and weaknesses, I feel like there are some issues that need to be addressed by a change in the setup where you could use a wrist-mounted Realsense D435 camera and the Franka Robot's in-built force sensing to try and also complete the first peg in the box task. In addition, I feel like an external non-contact microphone would be useful in sensing pouring even without vision.
However, I don't think it would be fair to you as an author to have to redo all the experiments by switching the setups. I just wanted to warn you that the paper would be much stronger if the robot sensor setup didn't have to be customized for every different situation. Otherwise we would be approaching the point of automation where if we wanted to be extra precise, you could have put an overhead camera to estimate the mesh of the box and do the planning for the cluttered box placing just like in industrial pick and pack startups or using a overhead laser and usb-connected scale in order to precisely pour the exact amount desired.

So basically, I want you to discuss these limitations in your actual limitations section instead of just talking about needing human demonstrations instead of using pure reinforcement learning in your current limitations section. 
I think you did contribute something with your setup and self-attention model; however, I think the tasks that were chosen to demonstrate the efficacy of the system were not really ideal to show the reason why you need all the different modalities. It would have been better if you had done some like previous cargo transport like graham crackers. That would show off why using the tactile sensors would prevent the robot from closing the gripper all the way in order to not break the graham cracker and a microphone could detect when the graham cracker crumbles. The vision would be helpful for locating the graham cracker for the initial grasp location.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper proposes a robot system that fusing three sensory modalities, namely seeing, hearing, and touching, with a self-attention model. The system is deployed to accomplish two challenging tasks, dense packing and pouring. The hardware experimental results show that the robot system with such fusion of three modalities outperforms the systems with only two modalities or without the self-attention mechanism.
strengths_and_weaknesses: The main strengths of the paper are:
1.	The training and testing of the self-attention model are done with real hardware, which makes the conclusions more convincing.
2.	The experimental setup are well explained with relevant details. 
3.	The supplementary video is helpful in demonstrating the qualitative performances of the robot system.

The main weaknesses of the paper are:
1.	Some results of the validation experiments are not very intuitive to understand. The authors need to discuss a bit more to make the results more meaningful to the readers. For example, in the dense packing tasks, the MULSA (V+A+T) method outperforms the MULSA (V+T) only in the Hard slanted case. Seems it’s always the case if the robot is able to obtain more information about the environment, it is more likely to distinct different situations, given that it uses more or less the same algorithms.
2.	It would be more convincing if the authors could open source their codes in the near future for reproduction of their results in the community.

The paper shows promising results in ultilizing smart sensor fusion with a self-attention model, which is inspiring for other researchers in the community.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper presents a concise yet effective design of multi-modal sensory fusion and impress the readers with interesting demonstrations in real robotic manipulation tasks. The idea is interesting to the community and the presentation is well organized. Hence, I recommend accepting the paper.
issues: 1.	The authors are suggested to add the variations in addition to the average performance in Table 1, 2, and 3 to show the consistency of the proposed robot system.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work proposes a new multisensory self-attention model for fusing visual, acoustic, and tactile sensory data, and uses imitation learning for action prediction. The model uses a self-attention mechanism to attend both across modalities and across time, outperforming prior fusion methods. They use a self-attention mechanism to fuse the set of features using a standard multi-head self-attention architecture.


strengths_and_weaknesses: ### Strengths
- The use of multi-sensory inputs to perform robotics tasks
- Some level of engineering required to build and integrate the sensors and synchronize the data acorss multi modality
- The results are compeling and the evaluations of the effect of adding each modality is also convincing
- There is a lot of detail about their hardware setup, their 3D printed task items etc.
- The use of multi sensory inputs in this domain is rather new and more work is exploring this area.

### Weaknesses
- The architecture on its own is very simple, its just a simple feature extraction using ResNet and feature fusion using multi-head self attention architecture.
- The paper mostly contains engineering detail of the task setup (This is still useful but this is more of an engineering paper).

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I definitely like the idea of applying sensory fusion techniques to utilize multiple sensors. The experimental setup and the results are convincing. My only thought is that, the method on its own, is not novel, its just a simple architecture for combining multiple modality. Even though it does establish a baseline to further explore other ideas to improve upon.
Finally, this paper for me is more of a well-executed robotics engineering paper, as opposed to strong core ideas being proposed here.
I am also not familiar with related work in this domain, but again combining multiple sensors for this particular manipulation domain is new but I imagine there might be more related work combining image and touch sensors.
issues: - I did not come across any discussion with regards to sensor calibration and synchronizing across multi modality - Did the authors find this to be an issue? Or was combining all the sensors a smooth process in general?
- There is reference to a method called `mel spectrogram representation` - I'm not familiar with this, perhaps the authors can provide a reference and explanation of how this data is generated.
review_title: null

Meta-review: metareview: The paper presents a multisensory attention model that can extract relevant features from the different modalities (RGB-D, visuo-tactile, and microphone sensors) for downstream manipulation tasks.

The paper **strengths** are: 
- The use of multiple sensors in a single and simple pipeline for downstream tasks. 
-The shown efficiency of the methods, in terms of needed demonstrations.
- The engineering effort to set up and provide real-hardware experiments.

The paper **weaknesses** are:
- Lack of comparisons and discussion of other methods that would use more than one modality, in tasks like pouring.
-Clear description of the experimental setup, and additional variations of the experiments to explain generalization of the proposed method.
-Limitations in convincing ablations stating from visual only data, to assess the significance of the added sensors on the various tasks.
-The method's limitations are not well addressed.

The authors should provide convincing answers to the reviewers' questions, and address the weaknesses of their paper.

**Post-rebuttal assessment:** After carefully inspecting the rebuttal discussion and the updated results, I appreciate the presented experiments and the addition of a new ablation study. Methodologically, I would argue that the use of Attention in Multisensory Fusion has been demonstrated previously (e.g., https://openreview.net/pdf?id=nhnJ3oo6AB), but not for the modalities and tasks presented in this paper, therefore making it an interesting application paper for the community.
confidence: 5: The area chair is absolutely certain
Decision: Accept (Poster)