Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper develops a touch-based curiosity approach for touch-intensive robot manipulation tasks with sparse rewards. The paper describes a two-stage learning approach where in the first stage, exploration is performed based on cross-modal (visual and touch) consistency and mismatch that inherently encourages robot-object physical interactions and in the second stage, the parameters from the exploration stage are re-used to achieve a downstream task with a task-specific sparse reward.

strengths_and_weaknesses: Strengths:
- Exploration based on cross-modal consistency that encourages physical interaction based on haptic modality mismatch and augmenting it with visual surprise based on a forward dynamics model
- Consideration of a variety of metrics including object movement to avoid passive physical interactions
- A benchmark with four simulated manipulation tasks with F/T sensor data: playing, pushing, opening, and picking up
- Baseline comparison with single-modality approach

Questions / Suggestions for Improvement:

- In haptics, sensing inherently depends on action. What this means is, forces during a physical interaction are not only dependent on object states (object physical characteristics) but also on robot states (what was the impact velocity etc.). The touch-control module, which maps visual signals to haptic signals for checking haptic modality mismatch does not consider robot states explicitly for this mapping. Can the paper comment on how will the cross-modal consistency generalize across different robot control behaviors (for example: different robot velocities for pushing etc.)?

- What happens if the task success reward during the second phase is actually simultaneously used with the touch interaction reward in a single stage itself? Can a comparison be done with the two-stage approach? 

- Can the parameters from the exploratory phase be reused across multiple tasks if all it’s getting is touch experience from visual inputs? Insights would be useful. For example, exploratory tasks would just focus on playing with the object and learning more about the visual-haptic mapping. And then, you use that to perform various tasks with the object such as pushing, opening, pick-and-place etc.?

- Can the paper discuss how the methods would deal with physical interactions that may require repeated make and break of contacts, such as in some in-hand manipulation scenarios?

- In a related way, do the methods bias the solutions towards more physical interaction even if they are not necessary? For example, a pushing task can also be achieved by increasing the robot’s impact velocity and letting the object slide on a table-top once the model has a good understanding of the physics. Can the paper discuss this and provide some insights?

- Some more insights about the results would make the paper stronger such as: 1) Why is TOC-future worse than TOC? 2) Why does the visual-curiosity based approach (ICM) perform better than TOC-pure (touch-based curiosity) for the pushing task? Is it because for the pushing task, object movement could probably be more crucial?

- “Each of the fingers is equipped with a simulated force/torque sensor that measures the joint reaction force applied to it” - ? Why is there a 6D F/T sensor on each finger instead of one at the wrist or a simple tactile sensor on each finger?

- Fig.1 - caption: \hat{z_t} -> \hat{z^_{t+1}}

- It would be very helpful to the readers if the graphs are made more legible with bigger fonts, larger frame sizes etc.

summary_of_recommendation: The paper describes an interesting two-stage learning approach which encourages physical interaction by considering cross-modal consistency and forward dynamics in the exploratory stage and sparse task rewards in the adaptation stage. As long as the limitations of the approach are clearly identified and the points raised above are discussed in the revised manuscript, I would be in favor of publication of this manuscript.

issues: See “Suggestions for Improvement” section above.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The work presents an approach that considers visuo-tactile representations for addressing tasks such as pushing objects and opening door, etc. (in simulation). In the proposed method, cross-modal consistency between visual and touch signals is used to guide exploration in reinforcement learning tasks, i.e., mismatches between visual features and touch representations will lead to surprises and more exploration. Two variants of the method are studied: Touch-based Curiosity (ToC) and Pure ToC, i.e., with the intrinsic reward being derived both from the visuo-tactile representations and vision-based predictive model, and visuo-tactile representations only. The experiments show that this approach can obtain better results than the standard (vision-based) Soft Actor-Critic algorithm and the state-of-the-art vision-based curiosity method ICM (SAC +  Intrinsic Curiosity Module).
strengths_and_weaknesses: Strengths: 
1. The idea of cross-modal curiosity for reinforcement learning tasks is interesting.
2. The experiments show that this approach can obtain better results than the standard (vision-based) Soft Actor-Critic algorithm and the state-of-the-art vision-based curiosity method ICM (SAC +  Intrinsic Curiosity Module).

Weaknesses:
1. Only simulation experiments are presented and no real robot data have been used in the experiments.
2. The presentation of the paper can be improved.
summary_of_recommendation: The work presents an interesting idea of cross-modal curiosity for reinforcement learning tasks but it lacks proper justification of the motivation of cross-modal curiosity and needs further validation on the real robots.
issues: 1. Title: Touch-based isn't entirely correct as the work doesn't consider any touch sensors per se and rather a force/torque vector.
2. A few settings of the methods are not clear. Why are the touch features only used to craft the intrinsic rewards? Why does the input to RL agent consists of visual features alone? If both touch features and visual features are taken as the input to the RL agent, the performance of the RL agent may be better. As for the loss functions, why was the L2 reconstruction loss selected instead of L1 loss? 
3. Is there a better way to get optimal lamda (Eq. 4) than uniformly sampling it between 0 and 1? Also, it may not be optimal to use a fixed lamda, has a changeable been attempted to maximize the performance?
4. Introduction / Paragraph 2: It would be interesting to find an argument for exploration from a computation standpoint. What's the practical advantage of animals and infants being curious? Why is it preferable to random exploration?
5. Figure 1:  the representation of the loss function should be consistent between figure (a) and (b), to ease the paper reading.
6. In the caption of Figure 1, "actual next latent state zˆ_t" should be "actual next latent state z_{t+1}".
7. Related Work / Paragraph 4 (101-107) should be revised, other works have studied visuo-tactile representations.
Lee, J.T., Bollegala, D. and Luo, S., 2019, May. “Touching to see” and “seeing to feel”: Robotic cross-modal sensory data generation for visual-tactile perception. In 2019 International Conference on Robotics and Automation (ICRA) (pp. 4276-4282). IEEE.
8. Related Work / Paragraph 5 (114-115) "This might not work in our case as touch is a more sparse signal and using a discriminator could lead to ambiguous outcomes." This sentence seems to be ambiguous, as it is dependent on the used sensor, and not backed by any evidence.
9. Problem Formulation (132) is not correct, pi_star is the optimal policy that maximizes the discounted sum of rewards.
10. Figure 2: It is not clear what (c) and (d) mean.
11. Figure 4: Which one is better? More touch information and more object movement? Why? Why are there 4 plots (either angle and mm)?
12. Figure 5(b) shows good generalization to novel objects, but there is no details about the seen dataset and unseen dataset. 
13. Table 1: The column refers to the method as ToC and ToC-fut, the caption as ToC and ToC Pure. In other parts of the paper, the same ambiguity persists e.g. 274-281.
14. Lines: 290-298: What are random features? (e.g. reference the section) and what are IDF?
15. Figure 5: It would be better to add extra spacing between figures (a) and (b) to help the interpretation, and use one single legend in figure (b)  to be consistent with figure (a).
16. Some grammar issues:
line 118: a RL agent -> an RL agent
 line 145: a L2 -> an L2
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a curiosity-based approach that leverages both vision and "touch" as a self-supervised pretext task. The task is to predict the observed haptics signal from the visual input (Fig 2).  Experiments show the approach leads to better performance on downstream sparse-reward RL tasks. The authors also contribute a benchmark (minitouch) can that be used with their method to pre-train networks and doesn't require a proprietary simulator. 
strengths_and_weaknesses: ### Major comments
In general, this paper is clear and easy-to-follow. The key idea (cross-modal learning) is simple and easy to implement, and the experiments validate the approach to a good degree (up to the points below).  

As for weaknesses, the technical advance appears to be incremental given that previous work (e.g., those in Sec 2 and [47]) have shown the benefits of cross-modal learning for learning better representations. The paper also has many minor errors that should be corrected before it can be accepted. Please see below for specific questions/concerns:

- line 147: Eq (2) and statement that "A high prediction error on a given image indicates that the agent has had few interactions like this." This is not necessarily true as the image may not be informative of the tactile sensation, e.g., consider real textured surfaces v.s. a flat printed texture. Essentially, the underlying predictive distribution may be multi-modal (as in multiple peaks and not multiple sensors) and the prediction error may not be informative. 
- Can the authors confirm that the touch information is sparse, i.e., only occurs when there is contact? If so, then the decoder would have zero output for a large amount of the state space when there is nothing in contact with the arm? 
- In the experiments and Fig 5(b), the authors claim "robustness to diverse shapes": how different are the unseen objects? The performance variance on the unseen tasks appears larger.  
- A short discussion of how the approach can be used in the real-world would help to increase the paper's impact. From the experiments, it seems a large number of steps are needed (20k)? Also, it's unclear if each "training step" is an episode or a time-step? The paper also mentions "episode steps" and it is unclear what is an episode is in the setup.
- Fig 2 (c) and (d) are unclear. Are the x,y-axis coordinates? which coordinate frame? I don't understand what is meant by "Robot’s ﬁngers are spread everywhere at starting while learn to focus near the object towards the end of training." Also, where is the object? at the top left? 
- Fig 3: the pushing task in the "Success" "Episode steps" and "Touch interaction"  seem to be over 1e6 steps rather than 1e5?
- Fig 3. The episode steps appear to start at a different number of steps for ToC and ToC-pure? Are these episode steps collected from the adaptation phase? 

### Minor issues: 
- Tuning $\lambda$ for the forward objective weight may be an issue and some discussion on how to pick it would help.
- There is related work in learning tactile skills through exploration (Pape et al, 2012) and multimodal learning (Chen et al, 2021) that attempts to learn consistent representations from multiple modalities for RL.  
- 188: "two-layerd"->"two-layered"
- 214: "Network Distiallation" -> "Network Distillation"
- Fig 3: "darker line represents mean" -> "darker line represents the mean"
- Table 1 caption: "Toc-Pure" is "ToC-future"? 
- 257: "to succeeds" -> "to succeed"
- 259: "surpasses towards" -> "surpasses RND towards" 
- 295: "figure.5a" -> "Figure 5a"
- pg. 5 . Two footnotes with the same number (1). 

### References

- Pape et al, 2012, Learning tactile skills through curious exploration, Frontiers on Neurorobotics. 
- Chen et al, 2021, Multi-Modal Mutual Information (MuMMI) Training for Robust Self-Supervised Deep Reinforcement Learning, ICRA 2021.  

summary_of_recommendation: Overall, I like the idea of doing cross-modal learning. However, the paper is borderline and falls slightly short due to the limited technical novelty and the numerous minor errors.

### Post Response 
Thank you for responding to my comments. I have amended my recommendation accordingly. 
issues: See above for specific issues.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper computes intrinsic motivation (i.e., artificial curiosity) by a vision and tactile sensing for improving the exploration in sparse reward function. Integrating tactile sensing is expected to guide the robot exploration in touch-intensive tasks better as the haptic information is more relevant to the rewards of these tasks. The proposed method trains an image encoder to transform an image input (x_t, where t denotes the timestep) to a latent vector z_t and then a decoder to reconstruct the haptic sensor inputs \hat{h}_t by the latent vector z_t. Furthermore, the author finds that incorporating forward dynamics model prediction objectives in vision inputs can improve exploration performance. 

strengths_and_weaknesses: Strengths: 
- The idea of cross-modality association is still underexplored in curiosity-driven exploration.
- The author compares with plenty of baselines of curiosity-driven exploration algorithm.

Weakness:
- The experiment setup is unclear. For example, are the state inputs in both exploration and adaptation step the same? Do the baselines take touch and vision as state inputs? 
- Some descriptions are vague. For example, L242,  "Figure 2a indicates that ToC is slightly more dynamic in the interaction
243 in contrast to ToC-Pure. ". What is "more dynamic"?
- Weak connection between the experiments and the proposed method. For example, wow does the proposed method ToC improve generalization to novel objects? Generalization is important but it seems to me ToC is not expected to improve generalization over novel objects. Also, it would be better to see the generalization ability of other methods.
summary_of_recommendation: Overall, the idea is well-motivated, and the cross-modality curiosity is promising. Most curiosity algorithms only use single modality information. I think this paper did a good job of bringing up the attention on cross-modality association in curiosity-driven algorithms. That being said, the reasoning for the experimental results is unclear to me in the current version of the manuscript. I would like to adjust my rating after rebuttal.
issues: 1. What are the inputs of the SAC, ICM, Disagreement, and RND? The text seems that they only take images as inputs for both exploration and adaptation steps. If so, the comparison might be unfair. The author tries to argue that cross-modality (touch and vision) “association” is beneficial for exploration and, therefore, improves adaptation. To claim this, the author should compare with the baselines that take both vision and tactile sensing as inputs but don’t encourage the association between tactile and visual inputs. It is necessary to compare with these baselines because the performance gain of ToC could be from using multi-modality inputs.

2.  As the introduction said, you constrain the search space by touch because the task prior tells us that haptic novelty is more relevant to the tasks. That could be true in most touch-intensive tasks, but it might be misleading in some cases. For example, visually similar but materially different objects could constantly generate novelty to the robot and introduce noises in the reconstruction loss. I would like to see more discussion about this point.
I’m not sure of the meaning of the heatmaps in Figure 2. I can’t find the reference in the text, and the caption is confusing. Could you elaborate more on these heatmaps?

3. Did you train ICM, RND, and Disagreement with haptic features? That would be an important baseline for ToC. If ToC outperforms the baselines trained with haptic features or haptic + vision features, the importance of cross-modality association will be strengthened. The current experiments are not convincing to me that cross-modality association is the key for the performance gain in the experiments.

4. In the adaptation step, is each agent trained by images only or images+touch?
review_title: null

Meta-review: This paper presents a multi-modal algorithm to improve exploration in presence of torque/force readings

Quality:

(+) The experiments seem to support the hypothesis of the manuscript

(-) Reviewer vFAA highlighted several clarifications and additional experiments that would be nice to perform.

(-) Experiments are only performed in simulation

Clarity:

(+) The paper is overall well written

(-) All reviewers suggested several changes to the text to make it more accurate and clear.

(-) As remarked by Reviewer 2Equ the paper does not indeed make use of touch sensors, but only of torque/force sensors. The  title and a large part of the nomenclature in the paper is thus misleading and should be modified to accurately reflect the content of the paper

Originality:

(+) The mixture of curiosity-based systems and multi-modal sensing (vision and force/torque) is novel

Significance:

(+) The reviewers agree that the topic tackled in this manuscript is interesting and potentially impactful.

(-) Reviewer 6jo7 had concerns regarding the significance of the contribution compared to existing work 


In summary, while the paper deal with an interesting topic, the current manuscript seems to have several limitations, from experiments that should be improved, to clarity, and overall contribution of the paper. 


Minor comment:
Recently, several simulators for tactile sensors have been open-sourced (e.g., https://github.com/facebookresearch/tacto, https://github.com/robotology/gazebo-yarp-plugins/issues/55) why not using one of them instead of force/torque sensors?

---

After rebuttal:
The authors improved the paper and addressed most of the concerns from the reviewers.
There now seems to be agreement from the reviewers that the paper is sufficiently solid.
Although the title has not been changed in the manuscript, I do expect the authors to change it as agreed in the rebuttal, and remove all mentions of "touch sensing" from the manuscript.

Minor comment: Fig 4, 5, 6 are hardly readable, and completely unreadable when printed in B&W. You should consider making the figures larger and more readable.
Decision: Accept (Poster)