ViTaS: Visual Tactile Soft Fusion Contrastive Learning for Reinforcement Learning

Authors: Yufeng Tian, Shuiqi Cheng, Tianming Wei, Tianxing Zhou, Yuanhang Zhang, Zixian Liu, Zhecheng Yuan, Huazhe Xu
Primary area: reinforcement learning
Keywords: visuo-tactile representation learning, reinforcement learning, contrastive learning

Abstract:
Tactile information plays a crucial role in human manipulation tasks and has recently garnered increasing attention in robotic manipulation. However, existing approaches struggle to effectively integrate visual and tactile information, resulting in suboptimal performance. In this paper, we present **ViTaS**, a simple yet effective framework that incorporates both visual and tactile information to guide an agent's behavior. We introduce _Soft Fusion Contrastive Learning_, an advanced version of conventional contrastive learning method, to enhance the fusion of these two modalities,  and adopt a CVAE module to utilize complementary information within visuo-tactile representation. We conduct comprehensive experiments, including $\mathbf{9}$ tasks in simulation environment, across $\mathbf{5}$ different benchmarks, to compare ViTaS with existing baselines. The results demonstrate that ViTaS achieves state-of-the-art performance, with an average improvement of $\mathbf{51}$%. Furthermore, our method significantly enhances sample efficiency while maintaining minimal parameters, underscoring the effectiveness of our approach. The code will be released upon acceptance.

Decision: reject
Reviewer scores: 8, 3, 6, 5

Reviews and discussion:
<thread>
creation date:
1730611069349

thread_type:
official_review

<thread_object>
creator: reviewer_ttav

<content>
summary:
This paper proposed a framework ViTaS, a contrastive learning based framework to enhance reinforcement learning for vision-tactile manipulation tasks in simulation. By incorporating a soft vision-tactile contrastive learning term with CVAE supervision, ViTaS helps various PPO-based manipulation tasks achieve better performance.

strengths:
1. Applied cross-modal contrastive learning method to an interesting use case
2. Conducted various experiments with many baselines, showing a significant boost in curves

weaknesses:
1. The manuscript is not well written. There are dozens of cases that
- Lacks coherence. In Sec 2, 3, and 4 it confused me a lot to see methods or results without knowing your motivation for doing so, or how to interpret the results. The entire paragraph from Line 418 doesn't belong here. 
- Figures and Tables
  + Many referred in context 2 pages before/after itself while you don't have to.
  + Details in Fig 1 are not explained, making this teaser confusing
  + Fig 2 is not clear enough, especially the soft fusion contrastive learning part. You showed lots of details in different colors, while it looks just like time contrastive to me. Also, what is the graph in the CVAE module visualization?
  + Fig 5, where is the drop you mentioned at Line 366? 
  + Fig 6 It's unclear how to evaluate the two settings. i.e. what patterns prove ViTaS is superior? 
  + Why do you use the same abbreviation rule for w/ and w/o in Table 2? Also, you mentioned K=1 and K=50, but the K value for other settings is not mentioned until the ablation for K itself.
- Regarding your contributions, from my PoV (1) and (2) are the same thing. Also, the average improvement of 51% is not shown in your experiment section.
- Missing method details. This is probably due to you mimicking Sec 3.2 of "Self-supervised Co-training for Video Representation Learning" since your Sec 3.1 is mainly adopting this method in your scenario, but I strongly suggest you revise your method sections. Your RL setting has more details to tell than copying theirs, as you showed in your pseudocode in the Appendix. For example, 
  + Your top-K is found in the buffer, which is never mentioned in context as an important implementation detail. 
  + Also, the metric for similarity is non-trivial. You should explain your usage of the encoder in formula (2). 
- Typos
  + Line 151, takes information from two modalities, or simply two modalities'.
  + Line 154, cross-modal
  + Line 215, full stop
  + Line 261, involving
  + Line 286, zero-padded
  + Line 431, results are
  + Line 437, is not explained "below"

2. Your entire framework is based on the observation "We note that when two tactile maps are resembling, the key features derived from the corresponding visual data should likewise bear a strong resemblance and vice versa". This is a strong and dangerous claim without clarifying the scope. For more broad vision-tactile manipulation tasks, I can easily construct counter-examples by replacing every part of the grasped object except the contact surfaces, not to mention the backgrounds. In this case, (1) does this claim work only in your task setting, with restricted object types that satisfy this rule? (2) will applying ViTaS in turn harm the generalizability of the RL agent? Please add more discussion on this part, elaborating either on the nature of your dataset or the limitation of your method. Please correct me if I misinterpreted your claim.

questions:
1. In your CVAE ablation you showed only the pen rotation test. From Fig 4 it seems the pen is large enough in the visual image. What about other tasks? For example, can your CVAE work in other experiments where the object of interest is small, or its visual states are subtle? Please show some ablations on this. For the same reason, the explanation in Line 381 "This token-based feature extraction is prone to overlooking critical information compared to pixel-level extraction, thus resulting in the capture of fewer essential features" is not convincing. Can you show some qualitative results?
2. The last two questions in the Weaknesses section
</content>
</thread_object>

</thread>
<thread>
creation date:
1730652515085

thread_type:
official_review

<thread_object>
creator: reviewer_385j

<content>
summary:
This paper presents ViTaS, a framework that combines visual and tactile data to improve robotic manipulation. It employs Soft Fusion Contrastive Learning and a CVAE module to effectively fuse and reconstruct these modalities. Experiments are done on a diverse set of none tasks in five different benchmarks.

strengths:
- The paper is well-written, organized, and straightforward.
- The experiments are conducted across a diverse set of simulated environments, providing robust validation.

weaknesses:
- **Lack of Motivation:** The authors incorporate a CVAE as a key component in their method, yet the rationale for this choice remains unclear. Section 3.2 includes a brief explanation, but it lacks depth and fails to adequately justify the use of CVAE. The authors should provide a more explicit motivation, detailing why CVAE combined with Contrastive Learning is preferable to the MAE objective used in [1].

- **Contrastive Objective:**  The theoretical differences between the contrastive loss used in this work and that in [2] requires further clarification. Providing a mathematical comparison between the two approaches could offer valuable insight into the unique contributions of this method, particularly regarding distinctions in formulation and the expected effect on learning outcomes.

- **Reconstruction Objective and Noise:** It is surprising that the authors opted for a pixel-based reconstruction objective. Prior research [2,3,4] has consistently shown that pixel-level reconstruction objectives often underperform in noisy conditions, as they may capture irrelevant elements from the observation. How would this approach fare in real-world scenarios with noise-prone tactile sensory data? I recommend that the authors demonstrate the model's robustness by presenting results on noisy or irrelevant observation components, like Gaussian noise, salt-and-pepper noise (similar to [1]), or missing pixels.

- **Limitations:** The limitations specific to this method are insufficiently addressed. The authors should explicitly discuss these limitations in the paper. For instance, addressing scalability to complex manipulation tasks, representation robustness under noise, and real-world applicability could provide readers with a clearer understanding of the practical constraints and areas for future development.

- **Performance on Rotate Tasks:** ViTaS significantly outperforms all other methods on the rotate tasks. The only provided explanation is that these tasks are “contact-rich and require methods to incorporate visual and tactile information jointly.” However, this characteristic seems relevant to all tasks in general. The authors should offer more insights into why ViTaS performs exceptionally well in the rotate tasks specifically.

- **Clarification Needed:** In Section 4.3, the authors claim, “This token-based feature extraction is prone to overlooking critical information compared to pixel-level extraction, thus resulting in the capture of fewer essential features.” This assertion seems counterintuitive, given that ViTs employ an attention mechanism that typically compensates by aggregating information across tokens, effectively capturing both local and global features. The authors should provide references to support this statement.

- **Experiments:** Although the method has been validated across a range of simulated environments, it has not been empirically tested or combined with tactile sensors in real-world experiments. Real-world tactile data is inherently different and would serve as a valuable addition to the validation. However, this is not a major concern, considering the wide range of simulated experiments.

**References**

[1] Sferrazza, Carmelo, et al. "The power of the senses: Generalizable manipulation from vision and touch through masked multimodal learning." (2023).

[2] Zhang, Amy, et al. "Learning Invariant Representations for Reinforcement Learning without Reconstruction."International Conference on Learning Representations.

[3] Bai, Chenjia, et al. "Dynamic bottleneck for robust self-supervised exploration."Advances in Neural Information Processing Systems 34 (2021): 17007-17020.

[4] JDeng, Fei, Ingook Jang, and Sungjin Ahn. "Dreamerpro: Reconstruction-free model-based reinforcement learning with prototypical representations."International conference on machine learning. PMLR, 2022.

questions:
Why is only the visual observation reconstructed in the CVAE, and not the tactile data?

The other Questions are in Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730673807370

thread_type:
official_review

<thread_object>
creator: reviewer_v2we

<content>
summary:
The authors introduce ViTaS, a framework for integrating visual and tactile information to improve robotic manipulation. ViTaS employs Soft Fusion Contrastive Learning to better fuse these modalities and a CVAE module to leverage complementary visuo-tactile information. The authors conduct some experiments with the proposed method.

strengths:
1. The paper is well-structured and easy to follow.
2. The authors conduct extensive experiments in simulated environments.

weaknesses:
1. Key visual RL baselines, such as DrQ-v2, are missing from the comparisons. The authors should compare their method with DrQ-v2 using both image-only and combined image-tactile observations.
2. More analysis is needed to explain how contrastive learning and the CVAE module contribute to better feature representations.
3. In Figure 5, only the proposed method performs well in the egg, block, and pen rotation environments, while it performs similarly to some baselines in other environments. Are these environments not optimized adequately for the baselines?
4. In Figure 7, the performance variance without CVAE is notably high. Does this indicate that one/multiple of the trials converges to a local optimum? If so, the authors are encouraged to run over more seeds to verify if ViTaS exhibits similar behavior.

questions:
The authors need to address my concerns in the weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730716620114

thread_type:
official_review

<thread_object>
creator: reviewer_pigr

<content>
summary:
The author introduces a framework to enhance robot manipulation by fusing visual and tactile data. They use a soft fusion contrastive learning approach and want to align these modalities in the latent space to improve RL policy.

strengths:
1. The visualization of t-SNE is good and the overall presentation is easy to read.
2. Extensive experiments and analysis of ablation study, the overall results are convincing.
3. Diverse simulation and environment.

Overall its a good paper to accept.

weaknesses:
1. No real-world experiments were included to validate the effectiveness of the method.

2. Some minor presentation issue should be fixed.

questions:
1. [line281-288]. Please specify what is 32x32x3 tactile sensor and what is 3x3x3. Please specify each channel of them. 

2. It would be better for the author to mention what kind of tactile sensors they are using and what kind of sensor signals can be received (eg. normal force, shear force)

3. [line 241,242], there are a lot of papers already showing the dexterous hand's abilities to do in-hand rotation using both vision and tactile or using solely tactile. Thus, the question seems too trivial to answer and cannot be a good point to compare with existing work.

4. To access the generalization of the approach, it would also be good for the author to list the objects used during the testing. Also, It would be better to describe the level of randomization and data distribution for that.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733311484881

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal 1/3 - Overview

comment:
We would like to thank all reviewers for their constructive suggestions, valuable advice and warm encouragement! We have highlighted the changes in blue in the revised version of our paper and here we provide an overview of our rebuttal.

# Overview 

The reviewers are impressed by our idea to fuse visual and tactile modalities in the RL process and the overall performance. In our paper, we present ViTaS, an innovative and efficient visuo-tactile fusion framework that significantly advances the integration of visual and tactile perception in reinforcement learning, yielding outstanding results. Our method leverages soft fusion contrastive learning alongside a CVAE module to enhance feature extraction and modality complementarity, as confirmed by comprehensive experiments and ablation studies.

Reviewers also propose some concerns about our method and experiment, and we have carefully considered the reviewers' feedback and addressed each point through additional experiments and clarifications. The main supplementary explanations we provide are as follows: 
1. **Motivation clarification**. We refine the presentation of the paper and clarify our motivation, providing a clearer narrative. 
2. **The effectiveness and necessity of CVAE and Soft Contrastive Learning**. We elaborate on the CVAE and Soft Fusion Contrastive Learning's influence on the latent space using t-SNE visualizations, illustrating how our approach facilitates superior fusion of visuo-tactile representations through the synergy of the two modules. Additionally, their roles are also clarified from the perspectives of reconstruction and the theoretical formula.
3. **Feature quality and robustness**. We demonstrate the reconstruction performance under varying noise conditions, comparing it with token-based baseline methods to emphasize our approach's robustness to noise and the high quality and complementarity of the visuo-tactile representations. 
4. **Reliability of experiment results**. We expand our analysis by including more tasks, more baselines, fine-tuning these baselines, and conducting more repeat experiments with different random seeds. This is coupled with an examination of the experimental environment to substantiate the robustness and reliability of our findings.

Replies to the reviews and additional experiments are listed in the following.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733311560311

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal 2/3 - Conclusion of reviews and replies

comment:
# Conclusion of reviews and replies

We would like to thank all reviewers for their time, appreciation and constructive suggestions! Specifically, we list the reviews and replies below.

`Reviewer v2wE` has raised some valuable concerns, and we have carefully addressed the issues which are listed below.
1. Lack of baseline DrQ-V2. We have added DrQ-V2 using both image-only and combined image-tactile observations, shown in **Fig 9**.
2. Further explaining the contribution of CVAE and soft fusion contrastive. We carefully explain the contribution and motivation behind them [in the reply](https://openreview.net/forum?id=IOkYP5ZxO5&noteId=trfOOaXKnH). Generally, we have conducted more experiments (e.g. **Fig 7, 12, 13, 14**) to show how the two main designs contribute to ViTaS.
3. Questioning whether we optimize baselines well. We carefully analyze why our baselines have been optimized and [here](https://openreview.net/forum?id=IOkYP5ZxO5&noteId=oyhfyTbu2N). Specifically, we why M3L exhibits large variance in **Fig 5**, which may be the inherent defect of M3L instead of failure to tuning. Then, we add extensive experiments of VTT with tuning some important parameters and designs (e.g. learning rate, number of transformer layers, coefficient in VTT loss). After [careful ablation](https://openreview.net/forum?id=IOkYP5ZxO5&noteId=aE2knL400m), we reach the conclusion that we have tuned VTT well.
4. Large variance in Fig 7. We have added a total of $10$ different seeds to reproduce results, shown in **Fig 6(b)**. It is evident from the learning curve that the variance of $10$ seeds drops while still remaining at a high level, indicating the importance of CVAE for capturing the detailed information and reducing variance.

`Reviewer TtAv` has made valuable suggestions and questions. 
1. Following the suggestion, we have refined and supplemented the presentation issues in our methodology, experimental results, and analysis. We enhanced the ablation studies of the CVAE module across more tasks, demonstrating its necessity. Additionally, we address the reviewer's concerns about pixel-level detail extraction by comparing reconstruction performance, which gained agreement.
2. The reviewer still had questions about the limitations “within one task and the same trajectory” imposed by the statement, "We note that when two tactile maps are resembling, the key features derived from the corresponding visual data should likewise bear a strong resemblance and vice versa", believing it could restrict the method's scope. We explain this point using the real-life example of plugging into a socket, along with technical illustrations of the RL paradigm and related experiment results as shown in our [reply](https://openreview.net/forum?id=IOkYP5ZxO5&noteId=igSEFkKfec), to demonstrate that this assumption does not impose additional limitations. ViTaS shares the same scope as any other task-specific encoders.

`Reviewer PiGR` appreciates our works, raising some minor presentation issues, which we have all corrected and [given a conclusion reply](https://openreview.net/forum?id=IOkYP5ZxO5&noteId=oduUygQYLI). `Reviewer 385j` raises the score after we add more clarification and experiments. It is noteworthy that **Fig 12** is a highlight to show the effectiveness of CVAE. 

Generally, we sincerely hope that our replies have clarified the issues, and we would greatly appreciate it if reviewers could consider raising their score. Thanks again for the time and constructive suggestions of our work!
</content>
</thread_object>

</thread>
<thread>
creation date:
1733311632296

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal 3/3 - Updated Experiments

comment:
# Updated Experiments
In accordance with the suggestions of reviewers, we have conducted extensive experiments to solve their concerns. All extra experiments are listed below.

- **New baselines.** As shown in **Appendix A.4**, we have added new visual RL baselines DrQ-V2 according to `Reviewer v2wE`. Specifically, we add basic DrQ-V2 and DrQ-V2 w/ tactile input compared to ViTaS, presented in **Fig 9**.
- **Extended horizons for M3L and VTT.** As shown in **Appendix A.5** and **Fig 10**, we have extended the horizons for M3L and VTT from 3M to 20M in Egg Rotate and Block Rotate to show the alignment between the original M3L paper and our paper, further showing we have adequately tuned the baselines.
- **Additional ablation on CVAE.** We have conducted more ablation study for CVAE as shown in **Fig 11** and **Fig 6(b)**. We show that CVAE could capture detailed visual information given the results between Lift and Lift w/ tiny objects. Moreover, we have run $10$ seeds in **Fig 6(b)** to address the concern Reviewer v2wE raised before about the limited seeds, and the results show that variance drops while still remaining at a high level with more different random seeds, which we believe could solve the concern.
- **Additional clarification about eq (1).** We further explain the meaning of eq (1) in **Appendix A.7** to show the implementation of $\text{Sim}$ in our formula, which successfully solves concerns of `Reviewer 385j`.
- **Comparison for reconstruction between M3L and ViTaS.** We would like to highlight our experiments of the effectiveness of the reconstruction process between ViTaS and M3L, shown in **Fig 12** and **Fig 13**. The reconstruction results further prove the necessity of our designs about CVAE and soft fusion contrastive learning.
- **Visualization of trajectory.** To give an intuitive explanation how our method works, we depict the t-SNE plots for the trajectory, as shown in **Fig 14**. The plots of visual and tactile embeddings of ViTaS show a small mean distance and could form a cohesive whole, which leads to the conclusion that ViTaS has a stronger representation learning.
- **Ablation of $\mu$ in CVAE loss.** We have conducted extra ablation studies to solve the concerns of `Reviewer v2wE`. Specifically, different coefficients of $\mu$ in eq(1) have been carefully investigated. The learning curve of $\mu=0.01, 0.1, 1, 10$ has been presented in **Fig 15** and **Table 5** to show the effectiveness of different coefficients. It is evident that $\mu=0.1$ shows a prominent result, so we test some finer value like $\mu=0.5, 0.05$. The results show that $\mu=0.1$ is optimal, echoing with the choice in ViTaS. Moreover, we also depict the embeddings in latent space when adopting different $\mu$ in **Fig 16**. With careful analysis, we believe that the experiments could solve the concerns of `Reviewer v2wE`.
- **Tuning the baselines.** As mentioned by `Reviewer v2wE`, several experiments regarding VTT are also accomplished to indicate that VTT used in our paper are well tuned. Specifically, we ablate $3$ main designs - learning rate, number of transformer encoder layers, coefficient in representation loss to balance with policy loss. The results shown in [the reply](https://openreview.net/forum?id=IOkYP5ZxO5&noteId=oyhfyTbu2N) suggest the settings we use are better than other settings. We believe it could solve the concerns of `Reviewer v2wE`.

Generally, we have many additional experiments, and we sincerely hope our effort could help to clarify the concerns for the reviewers. We would like to thank reviewers again for the time and constructive suggestions!
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper puts forward the ViTaS framework, with the aim of integrating visual and tactile data to enhance robotic manipulation. It employs soft fusion contrastive learning and a CVAE module to proficiently fuse and reconstruct multimodal information, and verifies the proposed method via experiments in a variety of simulated environment tasks.
The authors proactively carried out a substantial number of supplementary experiments and furnished explanations in response to the reviewer comments. Consequently, they effectively resolved the some of the issues raised by the reviewers and remarkably enhanced the quality of the paper. The principal concern regarding this paper is the absence of real-world experiments, and the simplicity of the simulation experiments. The reviewers are also concerned about the limits the contribution.
The AC concurs with the reviewers that the paper requires further refinement prior to acceptance.