Visuo-Tactile Transformers for Manipulation
Authors: Yizhou Chen
Mark Van der Merwe
Andrea Sipos
Nima Fazeli
Venue: CoRL 2022 Poster (2022)
Keywords: Multimodal Learning
Reinforcement Learning
Manipulation

Abstract: Learning representations in the joint domain of vision and touch can improve manipulation dexterity, robustness, and sample-complexity by exploiting mutual information and complementary cues. Here, we present Visuo-Tactile Transformers (VTTs), a novel multimodal representation learning approach suited for model-based reinforcement learning and planning. Our approach extends the Visual Transformer to handle visuo-tactile feedback. Specifically, VTT uses tactile feedback together with self and cross-modal attention to build latent heatmap representations that focus attention on important task features in the visual domain. We demonstrate the efficacy of VTT for representation learning with a comparative evaluation against baselines on four simulated robot tasks and one real world block pushing task. We conduct an ablation study over the components of VTT to highlight the importance of cross-modality in representation learning for robotic manipulation. 

Visuo-Tactile Transformers for Manipulation
Yizhou Chen
Andrea Sipos Mark Van der Merwe Department of Robotics, University of Michigan Ann Arbor, MI 48109, United States fyizhouch, asipos, markvdm, nfz] edu https: https://ww.. mmintlab. com/vtt
Nima Fazeli
Abstract: Learning representations in the joint domain of vision and touch can improve manipulation dexterity, robustness, and sample-complextty by exploit- ing mutual information and complementary cues. Here, we present Visuo-Tactile Transformers (VTTs), novel multimodal representation learning approach suited for model-based reinforcement learning and planning. Our approach extends the Visual Transformer [1] to handle visuo-tactile feedback. Specifically, VTT uses tactile feedback together with self and cross-modal attention to build latent heatmap representations that focus attention on important task features in the vi- sual domain. We demonstrate the efficacy of VTT for representation learning with comparative evaluation against baselines real world block pushing task. We conduct an ablation study over the components of VTT to highlight the importance of cross- mods-ttyyll in representation learning.
four simulated robot tasks and
Keywords: Multimodal Learning, Reinforcement Learning, Manipulation
Introduction
Deep reinforcement learning (RL) algorithms have solved numerous challenging tasks from raw observations (c.g., images) [2. 3, 4, 5]. Despite these successes, learning directly from such high- dimensional observation spaces can be numerically difficul/uunstable, sensitive to hyyer--prameet and sample inefficient [4]. These challenges have limited the utility of these methods for robotics applications because data collection time intensive, costly, and task variance is high.
To address this, recent research in representation learning aims to build compact "latent" repre- sentations of the underlying states [4, 6, 7, 8, 9, 10]. These compact representations are typically trained together with a policy using prediction and reconstruction losses along with task-dependant rewards. The policy then learns from the oow-dieesiional latent representation rather than the high- dimensional raw sensory observation. This approach offers stability and sample-efficiene, for policy learning owing the information-dent latent representations generated from rich prediction and construction supervisory signals. As such, latent representations are a promising tool for real-world robotic planning and RL.
Most research in robot latent representation learning has focused on images and proprioception [2, 11, 12, 13, 14, 15]. The sense of touch, together with vision, plays important role in robotic manipulation; however, visuo-tactile representation learning is still in its relative nascency. Recent works [10, 16, 17, 18, 19, 20, 21, 22] have proposed visuo-tactile methods; however, the latent representations used are typically vectors or clusters in I and do not exploit attention architectures [23]. These representations can smooth over fine details and struggle with locality in images. These issues are particularly troublesome for manipulation as the difference between in-contact and out- of-contact can be small in pixel space or the robot may need to move significantly contact objects.
In this paper, we propose a novel latent representation learning method that addresses these chal- lenges. Our approach extends the popular Visual Transformer (ViT) [1] to integrate multimodal feedback from vision and touch the Visuo-Tactile Transformer (VTT). Our approach focuses spatially-aware attention on important visual task features using tactile feedback. We show how this cross-modal attention creates a rich spatially distributed latent space that facilitates more effi- cient task execution. We evaluate the efficacy of our approach four simulated and one real-world contact-rich interactions, benchmarking against state-of-the-art techniques.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Trage
(2133
Mallbaay fort
Sal-Aberton
Useer Prejecion Hottoned Pasckes
Lrear Prejection of Tadil Rathhe
Rodilny Perten
Laven Hires
Locer
Locer
FLEI-Tppoaa Tepart Embedding
wotheringg
Nathooa Hordd FOS
Tospe
Image Embeading
Fastle briboding
Fosed
Figure 1: Architecture: Schematic of our proposed Visuo-Tactile Transformer. Parts of the diagram with gray backgrounds are novel contributions and differentiate our structure from ViT.
2
Related Work
The learning community has made significant progress in latent space dynamics learning, demon- strating promise in relatively low-dimensional [24, 25, 26, 27] and high-dimensional [4, 6, 7, 8, 9] spaces. The key idea in these approaches is to learn latent dynamics model that compactly rep- resents the high-dimensional observation space that is then used for planning or model-based RL. Many of these approaches consider continuous control tasks that may be applicable to the robotics domain [28, 29, 30, 31, 32, 33). Existing approaches focus on learning latent dynamics from visual feedback. Here, our work explicitly addresses multimodal learning in the visuo-tactile domain.
The robot learning community has also made progress in learning latent space dynamics. In par- ticular, moss- learning has recently gained traction [10, 16, 17, 18, 19, 20, 21, 22] where the current trend is visuo-tactile representations. Our work most similar to [10, 16] and we their encoders representation. Rather than using a vector in R", VTT learns a multimodal latent heatmap where at- tention is distributed spatially. We hypothesize that VTT's explicitly spatially-aware cross-altention is more effective for representation learning in terms of sample efficiency and expected reward than either of these approaches. We evaluate this claim by benchmarking against existing approaches.
baselines. The key difference between these works and
is the latent
Our proposed approach an extension of the Transformer [23] and VIT [1] to the disioltactile domain. Recent research has demonstrated the efficacy of attention mechanisms for sequence-to- sequence and spatially-distribute
inputs. tighter integration of encoders developed by the com- puter vision community [34, 35, 36] with robotic applications lead to significant advances in representation learning that enable efficient planning and RL.
3 Visuo-Tactile Transformers
At high level, VTT produces compact latent representations by fusing visuo-tactile inputs with self and cross-modal attention mechanisms. This enables robotic agents to leverage the complementary nature of these sensing modalities for policy learning. Additionally, VTT supplements image and tactile input patches with learned contact, alignment, and position/modality embeddings to further improve multi-modal reasoning. The overall VTT architecture is shown Fig. We present the details of the architecture and its components in the following sections.

---

3.1 Modality Patches
Before sending inputs to the transformer encoder to compute self and cross-modal attention, both the image and tactile inputs need to be patched. RGB image inputs are patched and embedded with one 2-D convolution layer as in ViT [1]. Tactile inputs from a wrist-mounted force-torque sensor are patched into reaction forces and torques, then linear projection is applied to embed them. Image and tactile modalities are notated 7 and respectively. We denote the embedded modality patch as XM IR (P))+x) where is the number of patches in each modality and is the number of features in each modality, which we assume to be identical. This embedded modality patch Xas can be written [X;; Xy] and used input to the first attention layer, as shown in Fig. 1.
3.2 Self and Cross-Modal Attention
We implement self and cross-modal attention mechanisms by adapting the multi-head attention method from Transformer architecture [23]. This adaptation is shown on the right side of Fig. I. The transformer encoder is composed of stack layer has two sub-layers:
identical attention layers. Each attention multi-head attention sub-layer and feed-forward sub-layer. When 1 1 n + S N, the output X, of the n attention layer A, will be the input for the (n + 1)th at- tention layer in the transformer encoder. To show how self and cross cross-modal attention are computed through N attention fusion layers, we first analyze An=1 and then generalize to n > layers.
To calculate the self and cross-modal attention at layer malization (LNN of embedded modality patch Xas into the Query (Q==1), (Vi=1) shown in Eq. 1, where h is the number of attention heads and 1 - S h.
1, we first project the post layer nor-
1) Key (K)=)), and Value
X1 XT
X1 XT
X XT
2- LN
W
K LN
W
V n=1
LN
W
(1)
In this formulation, W E Raddk, W E Rdxxx, and W E R are weights for Query, Key and Value, where is defined Sec. 3.1 and dx is the dimension of features the Key. Following Eq. 1, we note that the Query, Key, and Value can be written as
Q1 Q1 10
Kj Kr
V V1
Q =
K
V n=1
n=1
n=1
n=1
After projecting the embedded modality patch XM into Qu=1. Kn=1, and Vn=1, use Eq. 3 to calculate the attention at each head using scaled dot product with regularization factor Vd.
1 Atten X softmar Q ( (Ki=)))
(3)
a For clarity and brevity, we expand Eq. 3 in terms of Q, K, and V to isolate self and cross-modal attention.
[q QK QTKi QTKT
Vi VT
QQK QTKKVT
Kr QTK
1h ~ QKTV=
(4)
Attention Heatmap
Self Allention
Cross Atlention
From Eq. 4 we can see that the attention heatmap constructed with image and tactile self attentions Kr and Qr Ky on the diagonal and cross-modal attentions Q₂ and Qr in the upper and lower triangles. The attention output of Eq. is the sum of self and cross-modal attention. Finally, the multi-head attention A,=1 is formed by concatenating each attention head as shown in Eq. 5. attention encourages the model to explore the attention subspace.
A,=1 [A 4
= E (P)
(5)
To connect the sub-layers in each attention layer, we implement the residual connection method from [37]. Finally, the output X,=1 1 of the attention layer A. can be written
X = fa= (An=1, X, Xn-1
= Xax)
where fn=1 is a learned nonlinear function induced by nonlincar activation functions and operations.
To generalize to n > attention fusion layers, X,-1 is the input for attention layer A,, and is projected into Qn. K,, and V,, using Eq. 1. This proposed attention mixture allows the agent to iteratively leverage weights of self and cross-modal attention.
3

---

3.3 Learned Embeddings
To improve feature learning for RL policies, we propose three learnable embeddings to bolster the modality patches and attention mechanisms we introduced in Sec. 3.1 and Sec. 3.2. We evaluate effects of these embeddings on model performance in the ablation study presented in Sec. 4.2.
Contact Embedding: Intuitively, contact embeddings are used to pull latent codes for alike con- lact states together while pushing latent codes for dissimilar contact states apart. After N layers, the contact embedding Xc E IR Exd becomes the contact head Chead E R following the same method outlined in Sec. 3.2. We form contact loss in Eq. 7 by using Chead to predict the contact state of the system then comparing that prediction to the ground truth Cyt-
Alignment Embedding: Although contact embeddings introduce binary contact recognition, tem- poral alignment of tactile signatures to images is non-trivial [10, 16). Alignment embeddings shape the latent space to seek similarities in temporally aligned modalities and differences in temporally misaligned modalities [10, 38]. Therefore, alignment embedding XAI E Riidd introduced and be- comes the alignment head Alkead E Rixd after N layers. As in the previous section, we form a loss in Eq. 7 by using Alhead to predict whether the tactile and image data are aligned then comparing that prediction to the ground truth Alyt.
The contact and alignment losses are defined by a binary cross entropy with logits loss BC Esogits (-)-
(vTr= BC BC Fsogits (M (MLP(Alhhea)) Algt) + BC Etogits (M (MLP(Chead)) Cgt)
(7)
The final type of learnable embedding implement is posi- tion/modality embedding. We found that [Xc; XAt; Xx] do not contain the position or modality of information, so it is necessary to include position/mmdality embedding XP E RIx(2+ Pr). Inspired by [1, 391, we add this embedding to our structure form [Xc; XAi XMi + Xp.
Position/Modality Embedding:
3.4 Discussion of Compressed Representation Head
The output XN of the transformer encoder is a series of fused heads originating from the visuo- tactile embedded patches XM, contact embeddings Xc. alignment embeddings XAt. and posi- tion/modality embeddings Xp. In order to use our learned representation from VTT for RL, we need to think about the dimensionality of the latent vector If rectly, z will be too high- dinenssiiniii.... Consequently, we propose compressing all fused heads from P to +r))x with constant >> by using MLP. This compression results in R (rr) Our proposed method will allow image and tactile attention heads from VTT to be used for latent dynamics learning while introducing minimum inductive bias.
X E ((2+Pr+Pr)xd di-
3.5 Combining with Reinforvement Learning
Due to the frequency of visual occlusions and high levels of tactile perception uncertainty in our manipulation benchmarks, we formulate our manipulation tasks as partially observable Markov de- cision processes (POMDPs) [40]. There have been number of advances in model-based RL for POMDPs [41, 4, 42, 2]. From these, we chose the Stochastic Latent Actor Critic (SLAC) [4] algo- rithm because of its stability and robustness. These characteristics accommodate high stochasticity and provide an effective baseline method to compare visuo-tactile fusion mechanisms.
SLAC is composed of a model-lcarning component and policy-learning component. The model- learning piece is built with a factorized sequential variational autoencoder (VAE) fo [43]. fo uses a Gaussian distribution to form a prior model and a posterior model The prior model p(zz 1 a1-1) propagates latent dynamics from an action and the posterior model q(z;) [[=]|=;[1:20 1 (.-1, integrates observations. In this notation, z ~ fo(zo: 10) (0 To close the distribution gap between the prior and posterior models, KL divergence is introduced. Fi- nally, zz is decoded to reconstruct the visual input O and infer reward r. VTT losses from Eq. composed with SLAC losses to form the total model-learning losses in Eq. 8.
l E(O, (z a.-1) + E(r, |z a.-1) + (Kz(g pp) + EvTT
(8)
The policy-learning component is implemented using the soft actor critic (SAC) algorithm [44]. SAC is an off-policy actor-critic deep RL algorithm that maximizes future return and entropy for exploration. For further details and derivation on the model and policy learning, see [44, 4]. It is important to note that the latent representation from VTT is utilized by both components. Addi- tionally, the critic's 's value function loss is backpropagated through VTT to the attention mechanism.

---

4 Experiments and Results
IrageTactii Contect? Aiggment? (0/1)
20Conv
(0/1)
In this section, we benchmark VTT's performance against two popular multi-modality fusion tech- niques on four simulated robot manipulation tasks. Next, we present an ablation study that evaluates the importance of our contact loss and aligmment loss on simulation tasks. We then evaluate VTT on real-world pushing task. Finally, we visualize the attention maps produced by VTT on simulation and real-world tasks to illustrate the effects and intuition of cross- modal attention in the latent space.
MLP
Ruston
z
Module
MLP
Baseline Fusion Module: Concatenation
PoE
Figure Baseline Structure: We compare VTT to concatenation and product of experts (PoE) baselines. These methods are inserted at the 'fusion module'.
4.1 Simulated Experiments
To evaluate the performance of VTT, benchmark against two popular multi-modality fusion methods: concatenation and product of experts (PoE) [10, 16]. Fig. 2 shows the architecture of these baselines, where the fusion model is the key distinction between the meth- ods. Both baselines follow the same training pipeline as VTT to allow for fair comparisons. We denote the imputs of the fusion module as E₁ E IR [xdj and E1 E R 1xdr.
Baseline Structures:
In the concatenation fusion module, the latent vector Z becomes E Rix(dr+dr) = [r Er). However, the PoE fusion module treats E, and ET independently and applies a VAE-like structure to each latent code individually. E1 and ET pass through separate MLPs to generate separate means (μE, E and variances ET 4 Er) that are mapped to normal distribution using KL divergence, then fused using PoE as shown in Eq. 9 for i modalities and j features. The latent space in this method is also shaped by contact and alignment classification. Further details about each of these baselines can be found in their published works.
2
2
2
of
1
Simulation Experiment Setup:
We conduct simulated experiments of four manipulation tasks in Pybullet. The selected tasks are Pushing, Door-Open, Picking, and Peg-Insertion. Based on the continuous control manipulation benchmark developed by FlementAl [45], we construct a dense reward setting for these tasks. The agent receives an 84 x 84 x 3 RGB image and 1 x 6 wrist reaction wrench. Task rewards and parameters are outlined below.
1. Pushing Push a white block to grey target pose. The agent receives +25 reward for finishing the task. In this task, we vary the initial pose and mass of the block as well the target pose.
2. Door- Open Open a door from Adoor
to goal- When ddoor < 0gcats we penalize the agent -Xat ]. When gout = goat, the agent receives +25 reward. We incorporate curriculum learning [46] into this task by randomly initializing 0door > 0 during training only. In this task, we vary the color of the door and frame
Picking Pick up a white block. We foree the gripper to touch the block with a location-based Locrrierr Locstook)) 1) penalty. The agent receives +1 reward for grasping when the wrist feels significant downward force. The agent receives +25 reward when the block is lifted to a certain height. In this task, we vary the initial pose, mass, and size of the block.
Pns-rniert Insert peg into a hole. We force peg-hole alignment with location-based based -oCpee Lochotel) penalty. To encourage the agent to lower the peg into the hole, the agent receives +X(12mar Zpeg) )) reward after alignment. The agent receives +25 reward when the peg in fully inserted. In this task, we vary the color of the peg and the color of the hole.
We follow the SLAC training schedule to pretrain the dynamics model fo and fusion modules VTT. concatenation, and PoE. After the pretraining, SLAC and VTT are trained together. For the model we use Ir 0.0001 and batch size 32. For the policy we use Ir 0.0003 and batch size 256. We train using the Adam optimizer [47]. Further implementation details are included in the supplementary material.
Training Details:
5

---

Pushing
Door-Open
Picking
Peg-Insertion
100
100
Success Rate (%%
75
75
60
50
50
40
25
25
20
0
2
2
0.6
0.8
1.0
Training Steps 1e5
Training Steps le4
Training Steps le5
Training Steps
Concatenation
PoE
VTT(Ours)
Figure 4: Simulation Results: We compare the performance of our method baselines across trials and find that VTT outperforms the baselines in all but one task. The variance of VTT's learning curve significantly smaller than either baseline, suggesting better numerical stability of VTT.
Baseline Comparison: We compared VTT with the concatenation and PoE baselines over 8 trials. The results for each task with each fusion module are shown in Fig. 4. Compared to the baselines, our pro- posed approach achieves higher sample efficiency on all tasks and higher success rate in all but the Peg-Insertion task. We speculate that this task may not require as much spatial reasoning as the other tasks because the robot always grasping the peg and only has to make contact with the table then in- sert. The remaining tasks all require the robot first reach for and grasp/push an object, then perform the remainder of the task. Additionally, our approach shows lower variance with respect to the baselines. This suggests that the latent representation numer- ically easier and more stable to learn.
Pushing
Doar-Open
Picking
Peg: Insertion
Figure 3: Manipulation Tasks: We evalu- ate VTT on four tasks in Pybullet. We vary visual and physical parameters in each task.
4.2 Ablation Studies
In this section, we conduct three ablation studies to verify the fidelity our method. Firstly, we ab- late the contact and alignment losses in the VTT loss function. Secondly, we ablate the compression of latent vector Finally, we ablate model sizes by adjusting parameter counts (Appendix Sec. 6.4).
4.2.1 Loss Function
To show the importance of contact and alignment losses in VTT, we conduct ablation study by sequentially removing the contact and alignment embeddings and losses and performing the Door- Open and Peg- Insertion tasks. Overall, results in Fig. 5a indicate that each of these methods critical for achieving high sampling efficiency and success rate. The lack contact and align- ment losses cause miscorrelation of images and tactile feedback. The representation induced by such miscorrelations may mislead the agent's state value estimation and decision making, leading notice that the agent performs
worse overall performance. Additionally, contact loss than without the alignment loss. This behavior is likely due to the contact-rich tasks use evaluate our method. One potentially useful application of this misalignment may be as metric for evaluating just how out of distribution novel observation is.
without the
4.2.2 Latent Vector Compression
To investigate the effect of compression and the amount of compression, we conducted an ablation study. We find that our chosen compression value of 12 yielded the best results in the Door- Open and Peg-Insertion tasks. This level of compression allowed VTT to learn faster and yield generally better performance than other compression values, as shown in Fig. 6.

---

100
Door-Open
Peg-Insertion
b)
Pushing Demonstration
Success Rate (50
30
Success
01
10
Training Steps
Training Steps
104
No Alignment
No AligmmenaContact
0.6
Training Steps
0.8
1.0
1.2
164
Contacta
Full-Model
VIT(Oun)
We sequentially remove the contact and alignment losses from VTT to show performance in two simulated tasks without them. We find that the full method outperforms each partial method. h) Real-World Demonstration: We find that VTT also outperformed the baselines in the real-world Pushing task in terms of sample efficiency and success rate.
Figure 5: a) Ablation Results:
Door-Open
Reg-Insertion
100
100
c=1 (No Compression)
80
80
Success Rate (%)
(Ours)
60
18
40
20
20
9.0
1.5
2.0
2.5
3.0
3.5
4.0
9.0
1.5
2.0
2.5
3.0
4.0
4.5
Training Steps
Training Steps
Figure 6: Latent Vector Compression: We examine how compression of the latent vector z impacts the performance of VTT and find that value of yields the best results.
4.3 Real-World Demonstration
We conduct the Pushing task in the real world using a Franka Emika Panda robot arm with wrist-mounted ATI Gamma F/T sensor and custom end-effector. The block is a 2.25-in steel cube with Apriltags on its visible sides. These tags are tracked by two Intel RealSense D435 cameras located front of the robot (Camera 1) and the back right of the robot (Camera 2). We use the RGB image from Camera as input Ej and utilize Camera 2 to track the block in case of occlusion. We use the reaction force and torque from the F/T sensor input Er. This setup is shown in Fig. 7. The agent will receive +25 reward for finishing the task and -10 Locgoat| penalty when the task not finished. Each trial has 100 step maximum.
Real-World Demonstration Setup:
Real- World Baseline Comparison: Similarly to the simulated tasks, we evaluated VTT against the concatenation and PoE baselines. As in the simulation results, approach outperforms the other fusion modules in the real world as shown in Fig.
4.4 Attention Map Visualization
We illustrate the output of VTT's attention mechanism in Fig. 8 for simulated Pushing and Door- Open tasks. These heatmaps are the average the multi-head attention from Eq. We empirically observe 2 distinct phases: contact-free and in-contact. In contact-fiee free steps, we observe the attention
heatmap highlight the robot and object (Pushing / 1, Door-Open = 2). This is likely due to correlation between the goal and the change in the tactile signature that arises from contact between end-effector and object. Once contact present, the attention focuses mostly on the contact interaction and the goal. Again, this is likely due to the correlation between the reward signal and
7

---

Camera
Time
Camera
Goal
F/T Sensor
End-Effector
Block
Figure 7: Real-World Demonstration Setup: In the left pane show two successful block push- ing tasks. In the right pane we note the location of our cameras and F/T sensor, block, and goal.
Pushing t= 4
Door Open ==
t= 1
3
t=
t=7
t= 2
t
t=
t= 10
Original Image
Attention Map
Figure 8: Attention Heatmap: We overlay the attention heatmaps from Eq. onto RGB inputs. Further visualizations available in Sec. 6.3 along with plots of visual and tactile attention the timesteps visualized. Additional examples are shown in Figs. 9 and 10 (Sim and Real-world).
the tactile signature. We note that the object and the goal can be far from the robot in the visual scene, yet our attention mechanism able to correctly identify these key features.
Discussion and Limitations
We expect that VTT will enable important downstream applications such as visuo-tactile curiosity. Curiosity [48] makes exploration more effective by maximizing a proxy reward. This reward is often formulated as a series of actions that increase representation entropy. Similarly to [45]. VTT can measure potential mismatch in vision and touch and use this score to explore. VTT advantageously incorporates spatial attention to localize regions of interest in an image, which sets it apart from other multi-modal representation methods. Despite its promise, the current iteration of VTT has few limitations. Firstly, our implementation uses 6-dimensional reaction wrenches from F/T sensors as tactile feedback. This low-dimensional signal can often alias important contact events and generally not as informative collocated tactile sensing like [49, 50]. To address this, the tactile patching and embedding need be adapted to handle high-dimensional tactile signatures. Secondly, only evaluated VTT rigid-body interactions. This choice was motivated by the availability of simulation platforms that incorporate both tactile feedback and deformables and that are fast enough for RL applications. It is not entirely clear how VTT will handle deformables; though we suspect that attention will distributed over both the deformation and the contact region.

---

Acknowledgments
The authors would like to gratefully thank reviewers for giving useful comments. This material based upon work supported by the National Science Foundation Graduate Research Fellowship Program under Grant No. 1841052. Any opinions, findings, and conclusions recommendations expressed in this material are those the authors and do not necessarily reflect the views of the National Science Foundation.
References
[1] A. Dosovitskiy, Beyer, A. Kolesnikov, D. Weissenborn, Zhai, Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. CoRR, abs/2010.11929, 2020.
[2] D. Hafner, T. P. Lillicrap, J. Ba, and M. Norouzi. Dream control: Learning behaviors by latent imagination. CoRR, abs/1912.01160,, 2019.
[3] D. Hafner, T. Lillicrap, Fischer, Villegas, Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pages 2555-2565, 2019.
[4] A. X. Lee, A. Nagabandi, P. Abbeel, and S. Levine. Stochastic latent actor-critic: Deep rein- forcement learning with latent variable model. Advances in Neural Information Processing Systems, volume 33, pages 741-752. Curran Associates, Inc., 2020.
[5] D. Yarats, A. Zhang, Kostrikov, B. Amos, J. Pineau, and R. Fergus. Improving sample efficiency in model-free reinforcement learning from images, 10 2019.
[6] D. Ha and J. Schmidhuber. World models. CoRR, abs//803. 10122, 2018.
[7] De Bruin, J. Kober, K. Tuyls, and R. Babuška. Integrating state representation learning into deep reinforcement learning. IEEE Robotics and Automation Letters,
2018.
[8] D. Hafner, T. P. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamies for planning from pixels. CORR, abx/181 1.811.000 1.0455, 2018.
[9] A. Stooke, K. Lee, P. Abbeel, and M. Laskin. Decoupling representation learning from re- inforcement learning. International Conference on Machine Learning, pages 9870-9879. PMLR, 2021.
[10] M. A. Lee, Y. Zhu, K. Srinivasan, Shah, Savarese, L. Fei-Fei, A. Garg, and J. Bohg. Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks. CoRR, abs/1810. 2018.
[11] Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, Ha. J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, and S. Levine. Soft actor-critic algorithms and applications. CoRR, abs/18 1as.1812.0595, 2018.
[12] M. Zhang, S. Vikram, L. M. Smith, P. Abbeel, M. J. Johnson, and S. Levine. SO- LAR: deep structured latent representations for model-based reinforcement learning. CoRR, ab//ab///// 1801008888.00 09105.
abs/180 801.00900. 2018. URL http:://aare wow./bs////o.eeeoooo.oooaaeeee
[13] Z. Fu, A. Kumar, A. Agarwal, H. Qi. J. Malik, and D. Pathak. Coupling vision and proprio- ception for navigation of legged robots. CoRR, abs/2112.02091 2021.
[14] Torabi, G. Warnell, and P. Stone. Imitation learning from video by leveraging proprioception. 2019.
CoRR,
[15] L. Cong. H. Liang. P. Ruppel, Y. Shi, M. Görner, N. Hendrich, and J. Zhang. Reinforce- ment learning with ciioo-proprioception model for robot planar pushing. Frontiers Neuro- robotics, 16:824437, 03 2022. doi: 20..:889/foooo2222220000000

---

[16] M. A. Lee. Y. Zhu, P. Zachares, M. Tan, K. Srinivasan, S. Savarese, L. Fei-Fei, A. Garg, and J. Bohg. Making sense of vision and touch: Learning multimodal representations for contact- rich tasks. IEFF Transactions on Robotics, 36())es26333)::35555)5) 2020.
[17] Y. Li, J.-Y. Zhu, R. Tedrake, and A. Torralba. Connecting touch and vision via cross-modal prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10609-10618, 2019.
[18] N. Fazeli, M. Oller, J. Wu, Z. Wu, J. B. Tenenbaum, and A. Rodriguez. See, feel, act: Hierar- chical learning for complex manipulation skills with multisensory fusion. Science Robotics, (26):eaav3123, 2019.
[19] Y. Narang, B. Sundaralingam, M. Macklin, A. Mousavian, and D. Fox. Sim-to-real for robotic tactile sensing via physics-based simulation and learned latent projections. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 6444-6451. IEEE, 2021.
[20] W. Zheng, H. Liu, and F. Sun. Lifelong visual-tactile cross-modal learning for robotic material perception. IEEE transactions on neural networks and learning systems, 32(3): 192-1123,, 2020.
[21] D. Driess, Z. Huang, Y. Li, R. Tedrake, and M. Toussaint. Learning multi- dynamics with compositional neural radiance fields. arXiv preprint arXiv:2202.11855 2022.
[22] Y. Wi, P. Florence, A. Zeng, and N. Fazeli. Virdo: Visio-tactile implicit representations of deformable objects. arXiv preprint arXiv:2202.00868. 2022.
[23] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is all you need, 2017.
[24] M. Deisenroth and C. E. Rasmussen. Pilco: A model-based and data-efficient approach to pol- icy search. In Proceedings of the 28th International Conference machine learning (ICML- 11), pages 465-472. Citeseer, 2011.
[25] Y. Gal, R. McAllister, and C. E. Rasmussen. Improving pilco with bayesian neural network dynamics models. In Data-Efficiee Machine Learning workshop, ICML, volume 4, page 25, 2016.
[26] M. Henaff, W. F. Whitney, and Y. LeCun. Model-based planning with discrete and continuous actions. arXiv preprint arXiv: 1705.07777 2017.
[27] B. Amos, L. Dinh, S. Cabi, T. Rothörl, S. G. Colmenarejo, A. Muldal, T. Erez, Y. Tassa, N. de Freitas, and M. Denil. Learning awareness models. arXiv preprint arXiv:1804.066 06318, 2018.
[28] R. Rafailov, T. Yu, A. Rajeswaran, and C. Finn. Offline reinforcement learning from images with latent space models. In Learning for Dynamics and Control, pages 1154-166. PMLR, 2021.
[29] O. Rybkin, C. Zhu, A. Nagabandi, K. Danilidis, I. Mordatch, and S. Levine. Model-basee reinforcement learning via latent-space collocation. In International Conference on Machine Learning, pages 9190-9201. PMLR, 2021.
[30] A. Allshire, R. Martín Martín-Martin, Lin, S. Manuel, S. Savarese, and A. Garg. Laser: Learning a latent action space for efficient reinforcement learning. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 6650-6656. IEEE, 2021.
[31] A. Zhang, R. McAllister, R. Calandra, Y. Gal, and S. Levine. Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint arXiv:2006. 10742, 2020.
[32] C. Gelada, S. Kumar, J. Buckman, O. Nachum, and M. G. Bellemare. Deepmdp: Learning continuous latent space models for representation learning. In International Conference on Machine Learning, pages 2170-2179, PMLR, 2019.
10

---

[33] M. Okada and T. Taniguchi. Dreaming: Model-basee based reinforcement learning by latent imag- ination without reconstruction. In 2021 IEEE International Conference on Robotics and Au- lomation (ICRA), pages 42154 IEEE, 2021.
[34] Y.-H. H. Tsai, S. Bai, P. P. Liang, J. Z. Kolter, L.-P. Morency, and R. Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), Florence, Italy, 2019. Association for Computational Linguistics.
[35] x. Wei, T. Zhang, Y. Li, Y. Zhang, and F. Wu. Multi-modality cross attention network for image and sentence matching. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10938-10947, 2020.
[36] P. Gao, H. You, Z. Zhang, X. Wang, and H. Li. Multi-modality latent interaction network for visual question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
[37] K. He, X. Zhang. S. Ren, and J. Sun. Deep Residual Learning for Image Recognition. CoRR, abs/1512.03335 2015.
[38] T. Baltrusaitis, C. Ahuja, and L. Morency, Multimodal machine learning: survey and taxon- omy. CoRR, abs/1 705.099699 2017.
[39] M. Ryoo, A. J. Piergiovanni, A. Arnab, M. Dehghani, and Angelova. Tokenlearner: What can learned tokens do for images and videos? CoRR, abs/2106. 1297, 2021.
[40] H. Kurniawati. Partially Observable Markov Decision Processes (POMDPs) and Robotics. CoRR, abs/2107.07599, 2021.
[41] M. Igl, L. Zintgraf, T. A. Le, F. Wood, and S. Whiteson. Deep variational reinforcement learn- ing for POMDPs. In Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 21 2117-2. PMLR, 10-15 Jul 2018.
[42] D. Hafner, T. P. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. CoRR, abs/181 0ts//181100451,, 2018.
[43] D. P. Kingma and M. Welling, Auto-Encoding Variational Bayes. CoRR, abs/1312.6114, 2013.
[44] T. Haarmoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference Machine Learning. volume 80 of Proceedings of Machine Learning Research, pages 1861-1870. PMLR, 10-15 Jul 2018.
[45] S. Rajeswar, C. Ibrahim, N. Surya, F. Golemo, D. Vázquez, A. C. Courville, and P. O. Pinheiro. bouch-based curiosity for spwrse-reward tasks. CoRR, abs/2104.00442 2021.
[46] S. Narvekar, B. Peng, M. Leonetti, J. Sinapov, M. E. Taylor, and P. Stone. Curricuuum learning for reinforcement learning domains: A framework and survey. CoRR, 2as/2000300496/ 2020.
[47] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[48] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity-driven exploration by self- supervised prediction. In ICML, 2017.
[49] I. Taylor, S. Dong, and A. Rodriguez. Gelslim3. 0: High-resolution measurement of shape, force and slip in compact tactile-sensigg finger. arXiv preprint arXhv:2103.12266 2021.
[50] A. Alspach, K. Hashimoto, N. Kuppuswamy, and R. Tedrake. Soft-bubble: A highly com- pliant dense geometry tactile sensor for robot manipulation. In 2019 2nd IEEE International Conference on Soft Robotics (RoboSoft), pages 597-604. IEEE, 2019.
11

---

6 Appendix
6.1 Implementation Details of VTT
With reference to the notation in Sec. and Fig. 1, VTT uses one 2-D convolution layer to patch an 84 x 84 image into Xy E R36x33 Each patches goes through one linear projection layer to make X E R362384, In the real-world experiment, we reduce all the channel sizes to 128. Prior patching, VTT separates the
reaction force and I x 3 torque. Features are extracted through one linear projection layer to form Xr E R2x588 Contact embedding X E R x384 alignment embedding XA ! x and position embedding Xp E I 30403384 are initialized with weights based on truncated normal distribution by following the class embedding initialization [1]. We note here that d, = dr d de is required for watch-wise concatenation.
wrist tactile feedback into 1
Wq. WX K. Wy formed with 2-layer MIPs. Each MIP is constructed with 384 384 linear layers and has GELUs as activation functions after each linear layer. The feed forward block is also constructed with the same MLP. We chose the attention head number 8 to effectively spread attention distribution and N = 6 attention fusion iterations.
The contact and alignment heads from the transformer encoder's output, denoted Chead and Alpead, are passed through a 1-layer MLP. The MLP is constructed with one 384 x 1 linear layer followed by sigmoid activation function for alignment/contact binary classification. is assigned for positive pairing/contact, is assigned for negative pairing/non-contacr. When feeding negative pairs, RL's parameters frozen prevent misleading gradients for policy learning. As in [10, 16], the align- ment embedding is used alongside a training scheme that samples both aligned data and temporally misaligned data.
The fused heads that are output from the transformer encoder Fhead IR 48X have dimensions that are too large for model-based RL. Therefore, each fused head is compressed by 2-layer MLP constructed with 384 x 40 x 32 linear layers followed with RELU activation functions amid each linear layer. The resulting An R are flattened and passed through the 2-layer MLP to form z E R x288
6.2 Further Details of VTT with SLAC
With reference to notation in Sec. 3, the main objective of SAC is to maximize entropy of policy H(TT((1ae-1..4)) 2,))) while also maximize following POMDP based actor-critic objective functions:
JQ(B)
E
1
(10 a,) - (r, + +V (2
x
(10)
Jz(() = E
E [alog(TT ( [Z1 Q= (xx(2++1;)++1))
Where 3 is the parameter of soft Q-function, and 3 is delay of B3 for state value V-function. Ç is parameter of policy. Unlike typical model-based RI. instead of using belief representation, SLAC's policy learning is only conditioned on the past and current T + steps. The hyperparameters of SLAC follow its published code.
6.3 Further Attention Visualization
In addition to visualizing the attention on the RGB inputs, we plot the proportion of attention be- tween visual and tactile inputs over time during the simulated Door-Open task and the real-world Pushing task. We find that at the beginning of the task the attention is dominated by vision in both simulation and the real world. Upon instances of contact, we find that the attention heatmap contracts to the pixel-space of the RGB input where contact is located. At the same time, the proportion of tactile attention increases. In some cases, such as in Row of Fig. 10 (green/green), the modalities meet in the middle near a 50/50 split of attention. We find that in the real-world task, the attention slightly more balanced the start of the task and again becomes evenly split upon contact. In some real-world examples, the tactile attention spikes above the visual attention immediately after contact then becomes balanced around 50/50 (Row 1, Row 2). We find this result be unsurprising because it is in line with our intuition about the nature of multimodality in contact-rich settings such
12

---

Door-Open Task
Attention Modality
Visual
Tacble
Oriinnal Image
Attention (%)
60
40
Attention Map
20
11
14
80
Oriinnal
image
Attention (%)
60
Attantion
20
0
7
14
t=1
t=7
t=8
t=11
t=14
Onginal Image
Attention (%))
60
40
Attention Map
20
9
11
80
Original
Image
Attention (9))
60
Allention
20
0
7
1-1
[m8
[=]]
[=13
Original
Image
(%)
60
40
Map
t=2
t=7
t=11
t=12
Figure 9: Attention Visualization for Simulated Tasks: On the left side, we overlay the attention heatmaps from Eq. 4 onto RGB inputs for instances of the simulated Door-Open task. On the right side, we plot the visual and tactile attention during the task and note the time steps that correspond the RGB visualizations on the left.
as robotic manipulation. In general, we think of vision as being a global sensing modality and touch as being local sensing modality. Thus, stands to reason that until contact detected, vision outweighs touch.
6.4 Parameter Count
We present the exact parameter counts for each model ablated Sec. 6.4. This ablation is motivated by VTT having roughly 5-6x parameters as our implementation of the concatenation and PoE base- lines. After adjusting each of these baselines to have comparable parameters to VTT, we found that their performance did not improve.
13

---

Real World Pushiigg Tasi
Attertion Modalty B Viual Tack
Iwl
1:33
Cogesl
Abtrbbe (h))
353
Figure 10: Attention Visualization for Real-World Tasks: On the left side, overlay the tention heatmaps from Eq. 4 onto RGB inputs for 4 instances of the real-world Pushing task. On the right side, we plot the visual and tactile attention during the task and note the time steps that correspond to the RGB visualizations on the left.
Table Parameter counts for methods.
Method
Parameters
Concatenation
2.228E5
PoE
2.889E5
Concatenation w/ Adjustment
1.110E6
PoE w/ Adjustment
1.201E6
MulT
1.118E6
VIT
1.193E6
To ensure that VTT's outperformance of the baselines is not due to the expressiveness of the models, conduct ablation study over parameter count. We increase the size of the concatenation and PoE baselines to be comparable to the size of VTT. As we show in Fig. 11, we find that increasing the sizes of concatenation and PoE worsen their performance. Parameter counts are reported in Table
14

---

Door-Open
Peg-Insertion
100
100
Success Rate(%) (%)
80
80
60
60
40
40
20
20
0
4
0
2
Training Steps
1e4
Training Steps
1e4
Concat
Concat with Parameter Adjust POE with Parameter Adjust
VTT
POE
Figure 11: Parameter Adjustment:
We examine how the size of each model impacts performance and find that larger concatenation and PoE models do not yield better results.
15


Reviews:
[{"review_id": "lYxL_wspJij", "reviewer": "Reviewer_2Auu", "summary": "The paper introduces a multimodal representation learning method which combines visual and tactile information using a transformer architecture, to act as a latent representation learning model in methods such as SLAC (stochastic latent actor critic). The method is evaluated on several simulated robotic environments in which the observations are RGB images as well as signals from a force-torque sensor, and also on a real world block pushing task. Compared to other fusion baselines like naive concatenation and a product of experts, the proposed method is able to achieve better performance and sample efficiency on the studied tasks.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is well motivated and timely – the problem of how to effectively combine visual and tactile information for robot learning is one which is beginning to be explored but still quite open. \n- The idea of using attention blocks and a transformer architecture for this fusion is quite promising due to the success of methods like vision transformers. \n- The experimental results are quite strong. Although I have a few concerns about the evaluation protocol as mentioned later in the review, the results presented in Figure 4 do show substantial improvement over concatenation and a strong previous method in PoE. \n- The qualitative examples of visualized attention maps are quite convincing that the method does indeed learn to focus on the contact interactions when present. \n\nWeaknesses:\n- The presentation of the main method in section 3 makes it slightly confusing for me to understand. From my understanding, it seems like the cross-modal attention is achieved by performing self attention using a set of inputs which contains both the embedded modality patches of images as well as tactile sensing information. The demonstration of the image and tactile self and cross-modal attention in Equation 4 is nice, but many symbols like the $A_{n=i}^{self}$ are introduced and only used a couple of times which makes it hard to keep track of the computation. Additionally, I’m not sure there is a large benefit from describing the attention fusion layers as iterations rather than simply layers? The form of Equation 7 doesn’t add anything for me in addition to the previous description. \n- Particularly, I think the learned embeddings introduced in section 3.3 are a bit confusing – my understanding that a contact embedding $X_C$ is learned, not dependent on any of the image or tactile inputs, and a ground truth contact state is used to supervise the output at that position after attention layers are applied. A nitpick is that the phrasing “After N iterations, the contact embedding becomes the contact head” is rather confusing because it suggests that the contact head is somehow set to the contact embedding vector, whereas the contact head (I believe) is rather an entirely different vector, the output of a stack of attention layers. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Generally, I feel that the method is well motivated and achieves strong results. While the overall novelty of the method is somewhat limited, I think it is a nice step in the direction of multimodal fusion for robot learning. I will definitely consider raising my score if the authors address the issues particularly with evaluation in the “issues” section below.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "- What considerations are made to make the baseline comparisons as fair as possible in terms of parameter count? It could be that the performance improvements of the transformer method comes from simply having a more expressive model. \n- Experimental results are reported on the top 5 of 8 results for each method. I don’t believe this is a standard evaluation protocol so it would be good to see the full results or understand why this choice was made. \n- I assume that, as in Lee et. al, the alignment embedding is used alongside a training scheme which samples both “aligned” data as well as temporally “nonaligned” data, but I don’t think this is mentioned in the manuscript.\n- The introduction argues that “recent research in representation learning aims to build compact “latent” representations of the underlying task”, but I would argue that the representations are of the states rather than the task itself.\n"}}, {"review_id": "fJR3z98z-y", "reviewer": "Reviewer_nBFB", "summary": "This paper proposes the Visuo-Tactile Transformer (VTT) which is a neural network architecture that can be used for model-based reinforcement learning and planning. It takes in visual and tactile information. The tactile information comes from a wrist force-torque sensor. The VTT can take in the RGB image and the force torque sensor information and produce a latent variable z, and that can be the representation used for model-based RL. The paper combines VTT with Stochastic Latent Actor-Critic (SLAC). The paper evaluates on 4 simulated tasks and one physical block pushing task.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\nI like the combination of vision and touch. I think learning from multiple sensing modalities is key to unlocking generalist robots. Also, building on Vision Transformer (ViT) and showing good results is ideal as ViT is very popular which might increase the likelihood of other researchers adopting this method.\n\nThe experiments seem to be developed under well controlled conditions. In simulation the benchmark is a strong one from prior works, and the paper also has a nice set of ablations. The results from Figures 4 and 5 do seem to show benefits of their method over alternatives or over ablated variants.\n\nThe limitations section seems fair and appropriate (though I'd caution, mentioning curiosity is interesting but it feels too speculative to mention this…). \n\nWeaknesses: the robotics tasks seem to be fairy simple though I guess the same could be said for a lot of papers. In the real world the experiments are block pushing, and it seems simplistic.  I don't quite get the feeling that VTT is enabling robots to solve more complex manipulation tasks. Is there a way to address my concern?\n\nThere are also a bunch of clarification questions I have, see \"issues\".\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I am recommending a weak accept, I think this seems like a reasonably good fit for CoRL and I'm excited to see more vision and tactile work for robotics. I also can't see any glaring drawbacks to the paper at the moment.\n\n\n**Update after reading author response**\n\nI appreciate the authors taking some of my feedback into account, such as fixing the Figure 2 and adjusting the language and adding some more heat map examples (mostly in supplement). I also like the new ablation on parameter count for the baselines.\n\nStill, I feel a bit hesitant given just 1 real world demonstration tested here (I wish there was more) as they responded to my **Comment 7**. Also from their reply to my **Comment 4** I'm still not sure if it's fair to just use a different loss function to compare against the Lee et al baseline since the choice of loss function could be highly integrated into the choice of the architecture.\n\nAlso while I didn't mention this in my initial review, I felt a bit lost reading through Section 3 and the implementation details. I'm not an expert in ViT (the most closely related prior work) so that might have made it harder.\n\nTo also clarify: when I was referring to cleaning up the BibTex I was hoping to see things like the conference or journal venue, instead of repeated use of \"CoRR\".\n\nI originally gave a weak accept rating and I would prefer to keep this rating.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "Mostly I have a few clarification questions.\n\n1. In Figure 1, where are the cross-attention layers? Furthermore, can the caption clearly indicate which aspects of this figure are new and which ones are straight from ViT?\n\n2. Regarding VTT and SLAC integration: is VTT trained once and then used as a fixed architecture for getting a latent vector z for SLAC training, or are VTT and SLAC trained together? I think it's the latter, but I hope the paper can clarify (or that the authors can tell me where I missed it in the paper)\n\n3. In Section 4.1, the paper says that the baseline are the methods from Lee et al. However those papers use an optical flow loss. I don't see this in Figure 2. Can the paper clarify this? Otherwise I am concerned that the baseline is not fairly implemented. Also I don't completely buy the claim in lines 218-219 about the peg insertion task evaluated in the baselines, because those prior papers used real world peg insertion, whereas the current paper uses simulation. It feels like this writing should be made more conservative.\n\n4. Lines 212 to 214: the paper says it selects the top 5 of 8 results from each fusion module. I don't understand this comment, is this like taking the top 5 of 8 random seeds? \n\n5. Figures 4 and 5: how many random seeds are part of each curve? Also for Figure 5(b), what happened to the standard deviations? Are these just one trial each (yikes)?\n\n6. I would encourage adding more qualitative examples of the attention heat maps in the supplementary material.\n\n7. Typo: Line 223, \"w.r.t. to\" should be \"w.r.t\" or it can be spelled out entirely (there is space in the paper).\n\n8. The BibTeX really needs to be updated, e.g., [35] is a duplicate reference, and [36] through [44] need to have more complete citations. This needs to be cleaned up for the next version."}}, {"review_id": "Uz5l0SO4TxQ", "reviewer": "Reviewer_Y7uf", "summary": "In this paper, the authors propose a multimodal representation learning method named visuo-tactile transformer (VTT) to effectively exploit multimodal feedback from vision and touch. They demonstrate the effectiveness of VTT based on a comparative evaluation against baselines on both simulated and real-world tasks and on an ablation study over the components of VTT. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n-\tThe paper is well-written and clearly structured.\n-\tThe contributions are convincing. \n-\tMultimodal representation learning is getting important in the domain of robotics. The authors propose a multimodal representation learning method suited for reinforcement learning. \n\nWeakness:\n-\t The experimental evaluation should be fair. A comparative evaluation against baselines should be conducted in fair conditions.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is well-written, easy to read, and clearly structured. Although the main pipeline is the same as the baselines and extended from the prior study, the main contributions of the paper are valid. There are several issues to be addressed though. The issues are described in the issue section. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Issue 1: The references on page 1 lines 16-17 are not appropriate for reinforcement learning (RL) from raw sensory inputs. The authors need to investigate proper references in the corresponding studies. \n\nIssue 2: The appropriate references should be mentioned for the sentence “most research in robot latent representation learning has focused on images and proprioception” on page 1 line 29. \n\nIssue 3: I suggest the authors conduct an additional ablation study for the compressed latent vector z. For example, the effectiveness of compression and different dimensions of latent vector z. \n\nIssue 4: The authors use SLAC for the RL baseline. Why not Dreamer-v2, currently the state-of-the-art model-based RL method?\n\nIssue 5: The proposed method uses SLAC as an RL baseline. However, the baselines [11, 12] use TRPO as an RL baseline. These two RL algorithms, SLAC and TRPO, are very different; for example, TRPO is a model-free approach whereas SLAC is a model-based approach. It is well-known that SLAC is better sample-efficient than TRPO. The authors need to perform a comparative evaluation against the baselines on the same condition."}}, {"review_id": "GqxQ2inVV1t", "reviewer": "Reviewer_KWDi", "summary": "The paper extends vision transformers and proposes visuo-tactile transformer to perform modality fusion, and demonstrate the effectiveness of the proposed architecture on four simulated robot tasks and one real-world block pushing task, outperforming two alternative fusion methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The proposed approach extends the popular Visual Transformer (ViT) [1] to integrate multimodal feedback from vision and touch – the Visuo-Tactile Transformer (VTT), which is well-motivated.\n\n- On both simulated data and real-world experiments, the proposed Visual-Tactile Transformer architecture outperforms two alternative fusion methods: simple concatenation and Product-of-Experts.\n\n- Qualitative analysis demonstrates the attention modules in the transformer architecture successfully attends to the important regions to achieve the tasks.\n\nWeaknesses:\n\n- The paper claims to perform visuo-tactile representation learning, which is somewhat confusing. Throughout the paper, the main focus is more on using VTT to perform modality fusion instead of doing representation learning. \n\n- The proposed VTT architecture is very similar to ViT[1] with some minimal changes. The main contribution/idea is to also divide for-ce-torque data into temporal patches and use the same vision transformer architecture to process and fuse the two modalities. It is important and necessary to highlight the key differences of the proposed architecture compared to ViT (be explicit on which part is from ViT and which part is new), and show why the proposed new components are necessary instead of just applying it to tactile data.\n\n- For qualitative results, the attention map is only shown and overlaid on visual images. However, the key idea is attention on tactile data, therefore qualitative analysis of tactile data would be more useful and important. Otherwise, it's very much expected because the architecture for the vision part is similar to or almost the same as ViT.\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper extends vision transformer (ViT) and introduces VTT that also uses tactile data. Most of the network design is the same as ViT. Though the proposed approach outperforms two alternative fusion methods, the novelty of the method is somewhat limited. And the main contribution is adding tactile data (force and torque) to ViT and showing it's better than two other fusion methods. Therefore, I am leading towards rejection of the paper but happy to discuss more in the rebuttal discussion.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Please see the discussion in Strengths And Weaknesses", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)