Learning to Jointly Understand Visual and Tactile Signals
Authors: Yichen Li
Yilun Du
Chao Liu
Chao Liu
Francis Williams
Michael Foshey
Benjamin Eckart
Jan Kautz
Joshua B. Tenenbaum
Antonio Torralba
Wojciech Matusik
Venue: ICLR 2024 poster (2024)
Keywords: multi-modal perception
Primary area: visualization or interpretation of learned representations

Abstract: Modeling and analyzing object and shape has been well studied in the past. However, manipulation of these complex tools and articulated objects remains difficult for autonomous agents. Our human hands, however, are dexterous and adaptive. We can easily adapt a manipulation skill on one object to all objects in the class and to other similar classes. Our intuition comes from that there is a close connection between manipulations and topology and articulation of objects. The possible articulation of objects indicates the types of manipulation necessary to operate the object. In this work, we aim to take a manipulation perspective to understand everyday objects and tools. We collect a multi-modal visual-tactile dataset that contains paired full-hand force pressure maps and manipulation videos. We also propose a novel method to learn a cross-modal latent manifold that allow for cross-modal prediction and discovery of latent structure in different data modalities. We conduct extensive experiments to demonstrate the effectiveness of our method.

Published as a conference paper at ICLR 2024
LEARNING TO JOINTLY UNDERSTAND VISUAL AND TACTILE SIGNALS
Yichen Li', Yilun Du', Chao Liu', Chao Liu", Francis Williams", Michael Foshey', Ben Eckart?, Jan Kautz", Joshua B. Tenenbaum Antonio Torralba', Wojciech Matusik MIT CSAIL "NVIDIA
ABSTRACT
Modeling and analyzing objects and shapes has been well-studied in the past. How- ever, manipulation of these complex tools and articulated objects remains difficult for autonomous agents. Our human hands, however, are dexterous and adaptive. We can easily adapt manipulation skill on object to all objects in the class and to other similar classes. Our intuition comes from that there is a close connection between manipulations and topology and articulation of objects. The possible articulation of objects indicates the types of manipulation necessary to operate the object. In this work, we aim take manipulation perspective understand everyday objects and tools. We collect a multi-modaa visual-tactile dataset that contains paired full-hand force pressure maps and manipulation videos. We also propose novel method to learn a cross cross-modal latent manifold that allows for cross-modal prediction and discovery of latent structure in different data modalities. We conduct extensive experiments to demonstrate the effectiveness of our method. E
INTRODUCTION
Manipulating complex tools and articulated objects is a crucial yet difficult task for autonomous agents. Existing efforts modeling and analyzing objects and shapes focus appearance (Li et al., 2015; Schulz et al., 2017; Li et al., 2020b), topology (Luo et al., 2008; Duan et al., 2008; Hui et al., 2022), and articulation (Li et al., 2020a; Mo et al., 2021; Noguchi al., 2021; Yi et al., 2018). To motivate a dexterous robotic hand, we study human hand interaction with complex everyday objects that comes in different shapes and forms. key insight into the robustness and universality of human manipulation is the underlying hand synergy subspace (Santello et al., 2013). We hypothesize that hand force synergy and the articulation of tools are closely connected. In this paper, we aim to construct computational model that may discover the cross-modal structure shared through the two perceptual modalities, tactile that capture the force pressure of the hand, and vision, which captures the appearance, topology, and articulation of objects.
We focus on tactile and vision, and study ways to discover the underlying structure of force fields on our hands and to connect them with our visual senses. To achieve this, we focus on cross-modal prediction, the ability to synthesize temporal force signals from vision inputs, and predicting plausible vision signals from tactile force sequences. The main challenge lies in the disparity in spatial scale between the signals of vision and tactile and our ability to discover the shared structure across each modality. Our visual perception mechanism digests complete scene at once, whereas tactile force senses can only perceive a minute portion of an object at the scale of our hand. This gap in scale imposes challenge in the mapping mechanism to be many-to-many mapping between tactile and vision, where similar haptic force senses are plausible for different objects and one object can be manipulated using various forces.
To this end, we collect cross-modal force-vision dataset. We collect the active full-hand force pressure sequence data using specialized tactile sensor gloves (Sundaram et al., 2019b). We also use a webcam to record visual data of both hand-object manipulation sequences. In total, we collected 2,000,000 frames of paired hand foree pressure and video frames over 89 real object instances of different object categories.
In this work, we represent different modalities of information through manifold learning. However, directly using the shared latent space to represent both vision and force will cause the richer data
For further references:

---

Published as a conference paper at ICLR 2024
modality to dominate the more local and sparser data modality. To address the difference in scale, we introduce manifoldo projections to allow different sensory modalities to be represented in their local scale while preserving the close coupling across modalities through shared manifold. We notice that interesting structures appear in the learned manifold, where various manipulation properties and their possible combination with objects are organically summarized. To demonstrate the advantage of our proposed method in connecting force and vision, we evaluate our method on the two proposed tasks using our dataset. Quantitative and qualitative results are provided show benefits of our proposed method. A Additionally, we also visualize our learned manifold to better understand the latent structure captured by the model. These experiments show that our method outperforms previous multi-modal representation works. Our contributions are summarized as:
We collect a cross-modal dataset of force and vision on real-world articulated object bodies to study hand-object manipulation
We propose manifold carming-basede method to study the connection between the two modalities
We conduct extensive experiments to show the advantages of our proposed dataset and method
2 RELATED WORK
Shared Embedding Space Humans interpret our visual environment through a variety of sensory modes. This notion has inspired many efforts to explore the learning of shared embeddings across diverse domains such as words (Devlin et al., 2018), images (Frome et al., 2013), audio and video (Ay- tar et al., 2016; Chen et al., 2023a; 2022), and text and visual data (Radford al., 2021; Miech et al., 2019; Fang et al., 2021; Li et al., 2023). Recent efforts in this field have devoted to matching several different sensory modalities (Girdhar al., 2023). These advancements have been enabled by the availability of large-scale, paired cross-domain datasets. However, such datasets are yet to be developed for vision and force fields. To this end, we collect dataset of paired force fields and videos on articulated object manipulation sequences.
Multi-Sensory Data and Learning To equip robots with the diverse sensory capabilities, a variety of force, haptic, and tactile sensors have been innovated over the years, such as tactile (Dong et al., 2017; Yuan et al., 2017b), hapties (Lederman & Klatzky, 1987; 2009). force (Cutkosky al., 2008) sensors. The GelSight tactile sensor has enabled many vision and robotic applications (Gao et al., 2021; Yang et al., 2022: Li et al., 2019; Yuan et al., 2017a). These GelSight tactile sensors directly produce very local geometry texture properties on a small region of on the object. Different from these prior works, the glove sensor (Sundaram et al., 2019a; Luo et al., 2021b) we work with is a full-hand force pressure sensor that directly measure the normal force maps. Existing works that maps between GelSight tactile images and RGB images (Gao et al., 2021; Yang et al., 2022; Li et al., 2019) is difficult to directly employ in our setting. The mapping between Gelsight tactile images and RGB images is more densely coupled with one-to-one mapping. because every object can have unique surface texture property. However, the mapping between force maps and RGB images is loosely coupled many-to-many mapping, where the same object can be manipulated differently with different forces. Previous works (Zhang et al., 2021; Luo al., 2021a) has utilized the force-pressure tactile sensors to retrieve and predict other sparse data modalities, such as motion trajectory and human pose. Different from these prior effort, we aim map between sparse full hand force pressure maps and dense pixel sequences.
Manifold Learning To discover the underlying non-linear, low-dimensional subspace in which the high- high-dimensional input signals exist, many efforts have been devoted to learning manifold from complex sensory data (Tenenbaum et al., 2011; 2000; van der Maaten & Hinton, 2008; Donoho & Grimes, 2003; Coifman & Lafon, 2006; Belkin & Niyogi, 2003; Schölkopf et al., 1998). (Tenenbaum et al., 2011) proposes to use the geodesic distance between similar points, and (Chen al., 2023b) proposes to use compressed latent space to achieve model reduction on physics similuation problems. Recently, many efforts (Jia et al., 2015; Mishne et al., 2015; Pai et al., 2018) have proposed the integration of traditional manifold learning algorithms with deep networks by training autoencoders to align latents with classical manifold learning embedding spaces. In this work, we aim to discover the underlying low-dimensional structure of several different sensory modalities. By enforcing the geodesic distance on dfferent sensory manifold, we allow the underlying low- dimeesnnsio structure of various sensory information to arise. Different from previous works manifold learning and multi modal learning, we introduce geodesic distance arithmetic on latent manifold to structure the manifold of complex sensory data in meaningful ways supported by other data modalities.
2

---

Published as a conference paper at ICLR 2024
3 DATASET
We focus on common articulated object bodies for our dataset. These are object and tools prevalent in our daily life. Human hand exerts complex forces on functional parts of the object to operate them in desired way. By studying how different types of human hand haptic forces can result in different kinds of articulations, we hope to inspire interesting applications for autonomous systems to operate and manipulate these tools.
Data Collection Setup To collect the dataset, we leverage synchronized tactile glove sensor (Sun- daram et al., 2019b) and webcam video recorder, and we follow the data recording scheme inspired by (DelPreto et al., 2022). person wearing the tactile sensor glove to conduct various manipula- tions, meanwhile, we use web cam to record the entire on hand-object interaction environment. Additionally, we also take static photos of the canonical state of all object instances. We show data collection setup in Figure. 6 in Appendix.
Our hull-hand haptic force sensors are constructed following Sundaram using an embroidery- based method. We obtain 16 x 16 grid of pressure readouts distributed over the palm and finger regions of the gloves. We further calibrate the tactile sensory readings from the glove using URSe robotic arm to map tactile readings to force pressure level in Newton. The haptic force data is recorded in the format of a 2D heatmap on the hand, as shown in Figure. 6 in Appendix. This data format allows to leverage commonly used image- processinn network backbones (He et al., 2015) to extract the relevant force information.
(Sundaram et al., 2019b),
Dataset Statistics In total, we collected 150,000 frames of synchronized haptic force and video RGB sequences. The sequence data spans 89 real-world articulated object instances, including scissors, sprayers, staplers, and clips. For each object instance, we conduct several possible manipulations. Our training set contains 81 objects spanning 4 different categories, with paired tactile and video recordings of manipulation, containing 123,561 frames of data. Our test set is constructed with 3 different subset, 1) in-distribution instances, 2) in-category novel-instances, 3) out-of-category novel instances. Specifically, the in-distribution test-set is constructed with withheld sequences of the same 81 objects seen during training, containing 10,000 frames of data. The in-category test-set contains withheld objects unseen during training, containing around 6,000 frames of data. The out-of-categor test set contains 4 object instances of novel category unseen during training, which contain 5,000 frames of data. In total, the test set contains: 21,000 frames of data. We group frames into small sequences for training and testing, and each sequence contains 10 frames.
4 TACTILL MANIFOLD LEARNING
In our work, we are interested in learning the cross-modal structure between RGB manipulation videos x, tactile signal sequence h. and static images of objects c.
Given a video sequence I; E R10x64x64x3 we would like to infer the corresponding tactile sequence observation h, R 10x 16x1 and what object is being manipulated in the image e, E 664x64x3
and likewise given a tactile reading h;, infer the corresponding possible visual inputs 24, c, The chief diffieulty is the dramatic signal disparity between these modalities. We present method to capture and generate the shared structure between signals by embedding them into a shared latent space, with individual signals being decode from the latent using neural field. We further propose leverage manifold projection to encourage the desired structure to arise from the learned manifolds.
4.1 LEARNING A SHARED CROSS-MODAL VISUAL-TAATILS MANIFOLD
Given a training set of paired video, tactile, and image tuples I = (2s, hi, es ) we would like to learn and discover the shared structure between different signals. A chief difficulty is the dramatically different nature between tactile and visual signals, with visual videos being significantly higher resolution and complex than corresponding tactile signals.
Existing methods, such as TouchandGo Yang et al. (2022), ObjectFolder Gao et al. (2021), VisGel Li et al. (2019), etc., employ encoder-decoder techniques where the latent codes are constructed and obtained from encoding specific modality of information. To effectively capture structure across such dramatically different signals, we propose to compactly represent each signal in a shared latent space, where tactile, video, and image signals with three separate neural fields. Specifically, an auto-decoder technique, using different branches of neural field decoders to construct different modalities of information (tactile/vision). In this way, we allow gradients from different modalities
3

---

Published as a conference paper at ICLR 2024
M
M
Ch
M
Figure 1: Method Overview. Our approach embeds image, tactile, and video information into a shared latent manifold. This manifold is further decomposed into submanifolds for each signal.
of information to simultaneously update and optimize the latent embedding. We randomly initialize our shared latent embedding.As. shown in Figure. 1, the latent embedding on the leftmost side is the learned shared manifold. During training, we sample latent code z; to decode to different modalities of signals, I₁, h, and c, shown on the right side.
Formally, given either visual or tactile signal, we represent it efficiently as a function & which maps each coordinates u E IR' in either image tactile signals to the value e of the feature coordinate at that dimension.
(I) Such a formulation enables us to represent any signal, such as an image, shape, video sequences, or waveform continuous function t. We use MLPs to represent the neural field ø. Only the input dimension of is varied, depending on the dimensionality of the underlying signal this allows us to employ the same architectural choices across signal types.
t R' + R, + (z))
4.2 CROSS-MODAL MANIFOLD PROJECTION
In order effectively capture shared structure across image, tactile and visual signals would like our shared cross- model visual-tactile manifold to not only smoothly interpolate and project between different instances of signals to each other, but also capture the local structure in each individual signal modality. To enforce this local structure, we propose a manifold projection layer, where we linearly project our global manifold space local submanifolds for each signal. We then adopt existing manifold learning losses to ensure that each submanifold captures a space that is locally consistent and metric.
Formally, the manifold projection layer I& : M MMMMM, E {x, h, c} takes in the signal agnostic manifold to project to compressed lower dimensional submanifold that reflects aspect of signal modality. As shown in Fig. 1, the sampled latent code (red dot in the shared manifold) is projected through Т. and Tr the signal-specific manifold M, and MA. To encourage differentiability and explainability, we choose I's to be linear projection I's E R where S is the dimension of the original signal agnostic manifold and 7' is the dimension of the sub-manifold. Each data modality k E I is decoded from the submanifold M (.M and we enforce that any two samples sampled from the signal agnostic manifold {2,, 2y } S .M respects the distance between the samples I, I,, and that the same two samples sampled from the submanifold {2,, Zk.i S M₁ should respect the distance between the samples for the k-ith modality of signal In,i, In.j-
For cases where some signal modalities can be very expressive and contain rich and entangled information, this type of submanifold projection method can help disentanglement and encour- age various structures to arise from the sub- manifold focusing only on the specific signal being described. For example, in our test case with video x and tactile sequence data, the video sequences contain rich information about the manipulation and objects being manipulated, whereas the tactile h sequence data is sparse and reflects mostly the hand manipulation and less about the object geometry. Using natural language analogy, video data can be described using sim- ple sentences, the tactile information focuses on the verb of the sentence, and the noun can be captured by static image of the canonical shape. Through submanifold projection, encourage one submanifold M₁ = (MM to structured around the tactile modality and fo- manipulation. Similarly, can leverage the submanifold projection M. Te(M) to
4

---

Published as a conference paper at ICLR 2024
34
M.
Figure 2: Test-Time Manifold Manipulation. Our learned manifold can be used to retrieve possible cross-modal completions of relevant signal, including those of novel unseen test objects. a) shows the process of cross-modal prediction, and b) shows the process of inference on novel shape instances.
focus on object geometry concepts in canonical object images. We compose video latents بz.a by combining latents of manipulation 21,5 and the latents of canonical shapes Ze,s, As shown in the middle branch in Figure. Specifically, the video projection layer 9 constructed by con- catenating the image projection Se,i = [o(=;), E R556x256 layer and tactile projection layer 2h,4 = ('')),, I'x. E |226x16 žndi = (*) = (Z.))) where I'2 = (T' 3 E R226x272 or equivalently, žz (To(z));Tss I' (z.)) (Ze,z; Zh.s) This way of enforcing the compositional property in manifold space can help disentangle rich and expressive into more elementary concepts.
To further encourage structures of elementary concepts to arise from the submanifolds, would like to distill the dominant and descriptive information from each sensory modality onto their submanifolds. Inspired by Chen al. (Chen et al., 2023b), we choose to project the tactile submanifold Md to extreme low dimensions to constrain the neural field to focus on the dominant and most descriptive information. For our tactile submanifold M, we choose dimension=16.
To ensure our learned submanifolds follow the desired properties of data completeness and perceptual the following loss functions, as inspired by (Du et al., 2021). First, to ensure reconstruction
consistency, accuracy, we train each auto-decoder branch using the reconstruction loss.
LR [(0,(
2 [1](+ (Zx.s) .xe|2
Additionally, to ensure that our learned manifold achieves perceptual consistency, where similar signals are closer in the underlying manifold. We employ the manifold isometry loss:
Lise = ||o * [1(2; 2z) IZ - IIlll-
(3)
= =dd (o,,)) dz drt (2 + (d. (Zn,s, mn.)) dz h2 + (d ke.)) ) dy (z,,
(aa)) - (2nr))l- ||= eylll) + (=..) |)l-||aa z, [||) + (IIl (=x,,) (Rx,)) |||h
(5)
In this way, we enforce the distance between latents z; and zj our manifold M be locally isometric to the MSE distance of the corresponding samples Z, and Zj. Our loss function is constructed
using both terms: L = Lio + LRee.
4.3 MANIPULATING THE TACTILE- VISSLLE-VISUA MANIFOLD
Given our shared mectilevissall manifold, we next describe ways to manipulate and leverage the learned tactile-visual manifold to map between visual and tactile signal.
Cross-Modal Signal Retrieval. During test-time, given a seen tactile h, or visual 2;, e signal, we would like to retrieve the corresponding signals of the other modalities. As shown in Figure. 2, a) we leverage test-time latent code optimization to achieve cross- moosl retrieval. For example, given the RGB visual sequence T,, and we would like to see the corresponding tactile sequence h;. We first start a random location on the learned manifold εg,i, which corresponds to an rgb video signal 6_ (2..)) We use MSE loss signal to optimize for the latent, arg min;,, ||x, (Zx, 122, by fixing weight of the networks de, dz. di. The optimized latent E can be directly decomposed into latent vector for tactile sequence 2n,; and Le,i- We can use the projected latent to decode the desired tactile sequence h, ((²n.))
Similarly, given tactile sequence h, we can retrieve the corresponding video sequence x; by optimizing latent ê,,; to project back to the canonical image space M, = [r( (M,)) through the signal agnostic manifold to obtain the corresponding canonical image vector É.. We use to denote
5

---

Published as a conference paper at ICLR 2024
the reverse direction of the original projection. With the two vectors combined 2,,5 can decode into the desired sequence of visual signal, क; = Φz(2,,))
[&c,at 2n,a] we
One other advantage of our proposed method is varitational inference. Without any modification to the it can be used to predict variations of different signals our framework learns the joint manifold of visual and tactile signals. This allows us to handle multimodal settings, as each possible pairing in the manifold. This can be extracted by running test-time optimization with different latent initializations. By initializing different latents for test-time optimization, we can allow variational inference for fixed given sequence. We show an example in Section A.3 in Appendix.
Cross-Modal Prediction for Unseen instances Another advantage of our proposed method is its ability to handle novel object instances. As shown in Figure.2 b), given an image of an unseen object instance j. our proposed method can be used to predict possible manipulation force and video sequences. We can interpolate the shape manifold space to optimize for a shape latent code & that matches the image signal of the object instance. Various method exists to infer manipulation sequences hy, Sty. If desired manipulation is given through tactile sequence htj we can optimize for the tactile latent code En.j. Alternatively, we can infer the possible manipulations by projecting the shape latent code onto the manipulation manifold En. = T (f)()))) With the desired tactile manifold vector Er. we can decode tactile manipulation sequences from h, Φn (zn.)). ). and video sequence from the latent vectors L = oz ([ēc,; fn,])) When given a video or tactile manipulation sequence of novel instance, proposed method can predict the other modality by interpolating and optimizing latent E on the latent manifold.
EXPERIMENT
In this section, we show several experiments answer the following questions:
How does our proposed method perform compare existing cross-modal retrieval methods?
Can our proposed method generalize manipulation seen during training to novel object instances?
Does our learned manifolds exhibit meaningful latent structure of each signal modality?
How is our dataset and method used in robotic and vision applications? (Appendix Sec. A.2)
Does our proposed method enable variational inference? (Appendix Sec. A.3)
Table 1: Quantitative comparison cross- cross-modal prediction. Our approach substantially outper-
forms baselines across each test on cross-model signal undersanding.
cross-modal retrieval generalize new object + h h x(hh + x)
generalize to new category h
Method
h h <h,, x))
h a(h, c x))
Visgel 0.0022
0,0038
0.0024
0.0139
0.0024
0.0176
TouchGo
0.0027
0.0262
0.0028
0.0228
0,0025
0,0224
ObjectFolder
0.0030
0.0340
0.0034
0.0236
0.0026
0.0216
GEM
0.0022
0,0194
0.0023
0.0106
0.0021
0.0160
Ours
0.0019
0.0174 (0.00208)
0.0020
0.0097 (0.00710)
0.0020
0.0118 (0.01038)
5.1 BASELINES
We compare our proposed approach with several multi-sensory retrieval methods:
VisGel (Li et al., 2019): Li al. proposes to a conditional-GAN based model synthesize GelSight tactile images from input video sequences. The model takes in video sequence and conditioned on a RGB reference frame and a GelSight reference frame to predict the tactile sequence. We compare with the proposed approach by using the canonical shape image as reference to conduct cross-modal prediction on our force-visual data.
ObjectFolder (Gao et al., 2021): Gao et al. proposes to use triplet loss associate the sensory modalities RGB images, GelSight images, and sound. Inspired by them, we adapt the method as haptic force and vision data to use triplet loss anchor the force sequences,
a baseline video sequences, and canonical shape images.
TouchAndGo (Yang al., 2022): Inspired by previous works (Li et al., 2022: Tian al., 2020), Yang et al. proposes visuo-tactile CMC model to align RGB images that GelSight tactile images. We compare with their proposed approach by adapting it to work on visuo-force data.

---

Published as a conference paper at ICLR 2024
GEM (Du et al., 2021): Du et al. proposes the signal-agnostic manifold learning based method to represent and translate different sensory modalities. GEM (Du et al., 2021) was originally proposed for modal visual-audio retrieval. We modify it to work with our haptic force and vision data.
Coegrry
tepat sqaary
QT
VaGdi
Trechardo
Oreahhddar
Our
Figure 3: Qualitative Results for Video-to-Tacte Tactile Prediction. A comparison of r + h comparing our proposed method and existing methods on cross-modal retrieval and generalization new category and object instances. We show 5 frames in each example. Yellow regions corresponds to high pressure areas. Our approach is able to outperform baselines.
5.2 EVALUATION METRIC AND QUANTITATIVE RESULTS
We use L2 distance between the predicted signal and ground truth signal to evaluate the performance of various methods, ||& - 6.5h 1|2 E {c, x, h}. We show the quantitative performance in per-pixel difference for our proposed method and existing methods in Table. 1.
To compare with existing cross-modal prediction methods, we test on with several different task settings. First, following previous works (Li et al., 2019; Yang et al., 2022), and evaluate the cross- modal retrieval capabilities of different methods. This evaluation involves a subset of training object instances and manipulation sequences, depicted in the cross-modal retrieval section of Table 1. Additionally. we would like to understand the proposed method is able to extend its learned manipulation insights to novel object instances in seen object category. We show the cross-mon modal prediction performance in the context of unseen object instances of the training object category in generalize to new object section. Finally, we also conduct stress test to the ability of various method in generalizing to novel instances in unseen object category to understand if the proposed, as shown in generalize new category section.
We can see from Table. 1, our proposed method outperforms existing methods on both the cross- modal retrieval task and the tasks of cross-modal generalization to new object instances and category. The task difficulty increases from retrieval to generalization. All methods exhibits performance decrease when tested on novel instances and stress-tested novel category.
We observe a performance gap across all methods between inferring the tactile modality from videos h and predicting video sequences from tactile signals h + x. The drop in performance comes from the inherent difference between the two signal modalities. Videos contain rich and dense information about both the manipulation and the object being manipulated, whereas tactile signals only capture the miniature portion of object through the force field actuated on small areas of our hand. In this sense, predicting videos sequences from only tactile signals - is inherently a more difficult task. As Table. shows, when non-sufficient information is given, it is easy for test-time optimization based methods, such as GEM (Du et al., 2021) and our proposed approach. to stuck in local minima, whereas conditioning-bas based approach, such as VisGel (Li et al., 2019), can leverage the close coupling of conditioning vector 24 and output signal x to achieve better retrieval accuracy when predicting video from tactile. However, due to the same reason of the close coupling of the conditioning vector and output, conditioning-based approaches are limited in generalizability to new object instances and categories. Our proposed approach, on the other hand, encourage discovery of structured information embedded in each signal modality, and, therefore, is able transfer manipulation signals learned through seen data to novel instances and object categories. Other methods that leverage contrastive losses, such as TouchAndGo (Yang et al., 2022) and ObjectFolder (Gao et al., 2021). ), encourages loose coupling of latent vectors of different modalites
7

---

Published as a conference paper at ICLR 2024
Rarioval
Reriesa
New fesaace
Nov Calgary
(xpet Qpeay
Votal
GIM
Oun
Figure 4: Qualitative Results for Tactile-to- Video Prediction We illustrate - x results comparing proposed method and existing methods moosll retrieval and generalization new category and object instances. We show 5 frames of sequence data in each example.
2h. Zz. In our context of extreme scale mismatch, these metric-learning based methods fail to produce meaningful cross-modal prediction and result in mode collapse.
In Table. 1, we also show additional results of our method leveraging manifold composition capability. Instead of only leveraging tactile signals ht we compose information of both object instance e and manipulation type h to predict the video of object manipuation see from the table that our proposed approach is able to leverage the extra bit of information c to significantly improve the accuracy of our video prediction. This demonstrates that proposed approach is able to disentangle object concept and manipulation concepts leveraging different signal modalities, and compose these elementary concepts into meaningful visual sequences.
shown in h, -> x. We can
5.3 CROSS- PREDICTION QUALITATIVE RESULTS
We show the qualitative results of various method on the task of cross-modal retrieval and generaliza- tion new object instances and category in Figure. and Figure. 4
Video to Tactile x + f Predicting tactile manipulation signal is relative easier task as compared to factile-to-video task. We can from Figure. 3 that both proposed approach and Visgel (Li et al., 2019) is able to accuractely retrieve tactile sequences given video sequences seen during training. In generalizing to novel instances and category, proposed method is able to capture the manipulation in the video to and accurately predict tactile signal sequences, whereas other existing method fail to predict meaningful tactile sequences for unseen objects.
Tactile to Video - x We can see from Figure. that when directly predicting videos from the tactile information, most existing approaches, including TouchAndGo (Yang al., 2022), ObjectFolder (Gao et al., 2021), suffers from mode collapse. As the query manipulation tactile signal is extremely sparse, contrastive losses such CMC loss used in TouchAndGo Triplet loss used in ObjectFolder are unable to retrieval meaningful video latent code when the anchor latent vector žn are of extreme resemblance in metric space. VisGel (Li et al., 2019) uses conditional-GAN and retrieves seen video sequences accurately, but has limited generalizability to novel instances and categories. GEM (Du al., 2021) leverage shared latent manifold to predict one modality from the other. However, as different modality of signals are of different scale, the latent vector shared by different modality of signal become dominated by the modality with more information and weakens the model's ability to be queried by the sparser modality. As a result, the retrieved video sequences by GEM exhibits smoothing effect. However, when only tactile information is given, our proposed approach retrieves sharper videos, but the subject shape instance is often wrong. For example, with the new black scissor instance, our proposed method predicts a manipulation sequence of red scissor. Tactile force signal is not informative of the color of the object. When add the additional information with canonical object image h, c + x, our proposed method able to correctly predict the manipulation sequence of the desired object.
8

---

Published as a conference paper at ICLR 2024
CIEE
M
M
Mr
Mx
Figure 5: TSNE Visualization of Learned Latent Manifold. We visualize the manifold structure captured by our approach each submanifold and across the full manifold. On the top left corner, we show the initialization of each manifold. MM. Mu, and MM is the shape, tactile, and video manifold respectively. We color the latent vectors by their object and manipulation category. The trajectory of optimizing a latent code on the learned manifold during test-time is shown in black.
5.4 LATENT INTERPRETABILITY
We would like to understand whether our proposed method can encourage meaningful structures to naturally arise on the learned manifold without labeled data. We visualize our learned latent manifold through T-SNE embedding shown in Figure. 5. We can see from the figure that randomly initialized latent embedding are trained to accurately capture semantic concepts such as object category and manipulation type without any explicit label supervision.
We also observe that manipulation manifold M,, trained only with tactile sequence signal shows an interesting structure, where the manipulation "press" is clustered and away from the other manipula- tions of "hold" and "squeeze". We reason that the manipulation action of pressing is different from the other manipulation actions in that the actions of pressing activates contiguous area of the hand and is one directional, whereas the manipulation actions of "squeeze" and "hold" create force closure through opposing directional force actuated by multiple areas of the hand. In this sense, the actions of "squeeze" and "hold" share more similarities.
As the video sequence manifold M₂ is obtained through compositing Me and MA. it now shares the interesting property of both manifolds. Objects of similar manipulation and visual properties clustered together, such as the staplers and scissors that are both two rigid body parts connected through a rotational axis. Tools that are enabled by similar manipulation action are now also clustered together, such as the action "press" for the objects of "clip" and "stapler".
Remark. With the above experiments, we have demonstrated that our proposed approach outperforms exhisting methods in cross-modal prediction, and is able encourage meaningful latent structure to arise from different modality of signals. We do want to note several limitations of our work, we primarily focus on concept discovery and understanding manipulation and object, our decoding neural field is simple MLP and more advanced neural field architectures can be employed and combined to reconstruct high frequency details and improve the reconstruction quality. which we leave for future work. Additionally, we also show several ablation studies(Sec. A.4), robotic applications (Sec. A.2), variational inference (Sec A.3), and object classifcation experiments (Sec. A.2) in the Appendix.
CONCLUSION AND FUTURE WORK
In summary, we take an manipulation perspecitve to gain insights into objects and tools. Our primary focus is to explore the intricate connection between manipulation and objects. To this end, we curate a multi-modal dataset encompassing both visual and tactile information that contains paired full-hand force pressure maps and manipulation videos. Furthermore, we introduce an innovative technique designed to uncover a cross-modal latent manifold. This manifold enables cross-modal predictions and the revelation of hidden structures within distinct data modalities. We present a series of extensive experiments to demonstrate that our proposed method learns manifolds of meaningful latent structure that captures the principled information in each data modality. Our proposed approach outperforms existing cross- modal methods and exhibits ability to generalize learned manipulation to novel object instances.

---

Published as a conference paper at ICLR 2024
7 ACKNOWLEDGEMENT
We thank Wistron and GIST for their research grant for supporting our project.
REFERENCES
Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. In Advances in Neural Information Processing Systems, 2016. 2
Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps for dimensionality reduction and data representation. Neural Computation, (5(6))1373-133)) 2003. doi: 10. 10.11220899766032178817,, 2
Changan Chen, Ruohan Gao, Paul Calamia, and Kristen Grauman. Visual acoustic matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18858-18868, 2022. 2
Changan Chen, Alexander Richard, Roman Shapovalov, Vamsi Krishna Ithapu, Natalia Neverova, Kristen Grauman, and Andrea Vedaldi. Novel-view acoustic synthesis. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 6409-6419, 2023a. 2
Peter Yichen Chen, Jinxu Xiang, Dong Heon Cho, Yue Chang, G A Pershing, Henrique Teles Maia, Maurizio M Chiaramonte, Kevin Thomas Carlberg, and Eitan Grinspun. CROM: Continuous ueduced-order modeling of PDEs using implicit neural representations. In The Eleventh Interna- tional Conference on Learning Representations, 2023b. URL https: opwprevieeei.eeee /
forum? d=FUOR= 2 G800g. 2,5
Ronald R. Coifman and Stéphane Lafon. Diffusion maps. Applied and Computational Har- monic Analysis, 21(1):5-30, 2006. ISSN 1063-5203. doi: https://boo.org/10.010///// rect. com/ scisciiiss wwcience/artiicle/eeeee 6. Special Issue: Diffusion Maps and Wavelets. 2
2006.04.006. URL https: :epps://wwww /www.
S1063520
Mark Cutkosky, Robert Howe, and William Provancher. Force and Tactile Sensors, pp. 455-476. 01 2008. doi: 011799978---44-3000000
2
Joseph DelPreto, Chao Liu, Yiyue Luo, Michael Foshey, Yunzhu Li, Antonio Torralba, Wojciech Matusik, and Daniela Rus. ActionSense: A multimodal dataset and recording framework for human activitics using wearable sensors in a kitchen environment. In Neural Information Processing Systems (NeurIPS) Track Datasets and Benchmarks, 2022. URL https: /att:i////iionn/ action-sense. csaal. mit t.it... edu.
Jacob Devlin, Ming-Was Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.048005 2018. 2
Siyuan Dong, Wenzhen Yuan, and Edward Adelson. Improved gelsight tactile sensor for measuring geometry and slip. pp. 137-144, 09 2017. doi: 10.1 109/
David L. Donoho and Carrie Grimes. Hessian eigenmaps: Locally linear embedding techniques for high-dimensional data. Proceedings of the National Academy of Sciences, 100(10):5591-559 2003. 2
Yilun Du, M. Katherine Collins, B. Joshua Tenenbaum, and Vincent Sitzmann. Learning signal- agnostic manifolds of neural fields. In Advances in Neural Information Processing Systems, 2021. 7,8.8 17
Xian-Bao Duan, Yi-Chen Ma, and Rui Zhang. Shape-topology optimization of stokes flow via
variational level set method. Applied Mathematics and Computation, 202(():200- 200,00002,,222000 2008. 1
Han Fang, Pengfei Xiong, Luhui Xu, and Yu Chen. Clip2video: Mastering video-text retrieval via image clip. arXiv preprint arXiv:2106. 11097, 2021. 2
Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Mare' Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. In C.J. Burges. L. Bot- tou, M. Welling, Z. Ghahramani, and K.Q. Weinberger (eds.). Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013.
10

---

Published as a conference paper at ICLR 2024
Ruohan Gao, Yen- Yu Chang, Shivani Mall, Li Fei-Fei, and Jiajun Wu. Objectfolder: A dataset of objects with implicit visual, auditory, and tactile representations. In CoRL, 2021. 2, 6, 7,
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all, 2023. 2
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arXiv preprint arXiv: 2015.
Ka-Hei Hui, Ruihui Li, Jingyu Hu, and Chi-Wing Fu. Neural template: Topology-aware reconstruc- tion and disentangled generation of 3d meshes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18572-18582, 2022.
Kui Jia, Lin Sun, Shenghua Gao, Zhan Song, and Bert Shi. Laplacian auto-encoders: An explicit learning of nonlinear data manifold. Neurocomputing, 160, 07 2015. doi: 10.1016/j.euccmm22055. 02.023. 2
S Lederman and Roberta Klatzky. Haptic perception: A tutorial, Attention, perception, psychophysics, 71:1439-55 59, 10 2009. doi: 10.3758/AAPP71...4399
Susan J. Lederman and Roberta L. Klatzky. Hand movements: A window into haptic object
recognition. Cognitive Psychology, 19:342-368, 1987. 2
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023.
Tingle Li, Yichen Liu, Andrew Owens, and Hang Zhao. Learning Visual Styles from Visual Associations. In European Conference on Computer Vision (ECCV), 2022. 6
Xiaolong Li, He Wang, Li Yi, Leonidas Guibas, A Lynn Abbott, and Shuran Song. Category-Ievel articulated object pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3706-3715, 2020a.
Yangyan Li, Hao Su, Charles Ruizhongtai Qi, Noa Fish, Daniel Cohen-Or, and Leonidas J Guibas. Joint embeddings of shapes and images via cnn image purification. ACM transactions on graphics (TOG), 34(6):1-12, 2015.
Yichen Li, Kaichun Mo, Lin Shao, Minghyuk Sung, and Leonidas Guibas. Learning 3d part assembly from single image. European conference on computer vision (ECCV 2020), 2020b. 1
Yunzhu Li, Jun-Yan Zhu, Russ Tedrake, and Antonio Torralba. Connecting touch and vision via cross-modal prediction. In The IEEE Conference on Computer Vision and Putlern Recognition (CVPR), June 2019. 3, 6, 7,8
Yiyue Luo, Yunzhu Li, Michael Foshey, Wan Shou, Pratyusha Sharma, Tomas Palacios, Antonio Torralba, and Wojeiech Matusik. Intelligent carpet: Inferring 3d human pose from tactile signals. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11255-11225,,0 June 2021a. 2
Yiyue Luo, Yunzhu Li, Pratyusha Sharma, Wan Shou, Kui Wu, Michael Foshey, Beichen Li, Tomás Palacios, Antonio Torralba, and Wojciech Matusik. Learning human environmenit interactions using conformal tactile textiles. Nature Electronics, 193-221,, 2021b. 2
Zhen Luo, Michael Yu Wang, Shengyin Wang, and Peng Wei. A level set-based parameterization method for structural shape and topology optimization. International Journal for Numerical Methods in Engineering, 76(1):1-26, 2008.
Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac Makarand Tapaswi, Ivan Laptev, and Josef Sivic. HowTo100M: Learning Text- Video Embedding by Watching Hundred Million Narrated Video Clips. In ICCV, 2019.
Gal Mishne, Uri Shaham. Alexander Cloninger, and Israel Cohen. Diffusion nets. Applied and Computational Harmonic Analysis, 47, 06 2015. doi: 10. 10.1006/j.ccha.20 0.0...077..... 2
11

---

Published as a conference paper at ICLR 2024
Kaichun Mo, Leonidas J Guibas, Mustafa Mukadam, Abhinav Gupta, and Shubham Tulsiani. Where2act: From pixels to actions for articulated 3d objects. Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 68 6813-6823, 2021. 1
Atsuhiro Noguchi, Xiao Sun, Stephen Lin, and Tatsuya Harada. Neural articulated radiance field.
In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 5762-5772,
2021. 1
Gautam Pai, Ronen Talmon, Alex Bronstein, and Ron Kimmel. Dimal: Deep isometric manifold learning using sparse geodesic sampling. 11 2018. doi: 10.1 10.110///A..22009.0000 900092. 2
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary VeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 0024-8035. Curran Associates, Inc., 2019. URL http: /ttp:///ppprr...ee neurips, cc/ cc/paper/ 9015 1:15-pyter 1155--yyorch-eee an pdf. 17
style peigh--prffrmancee deep learning Seeaaniinn-ibbrae library.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. https:: : http://ppooeeee pttpp//ecceengssseeee mlr press / vvv3vvv31vv///// w139//adfordd2ta htt.dd2aa...........m........... 2
PMLR, 18-24 Jul 2021. URL
Marco Santello, Gabriel Baud-Bou Bovy, and Henrik Jömttel. Neural bases of hand synergies. Frontiers in computational neuroscience, 7:23, 2013.
Adriana Schulz, Ariel Shamir, Ilya Baran, David IW Levin, Pitchaya Sitthi-Amorn, and Wojciech Matusik. Retrieval on parametric shape collections. ACM Transactions on Graphics (TOG), 36(1): 1-14, 2017. 1
Bernhard Schölkopf, Alexander Smola, and Klaus-Robert Müller. Nonlinear Component Analy- sis as a Kernel Eigenvalue Problem. Neural Computation, 10(5):1299-1319, 07 1998. ISSN 0899-7667.
doi: 10. 0B2116200099766880000744448
URL https: / /tt////i.oo////// ott.io.org/1...1...
Subramanian Sundaram, Petr Kellnhofer, Yunzhu Li, Jun-Yan Zhu, Antonio Torralba, and Wojeiech Matusik. Learning the signatures of the human grasp using a scalable tactile glove. Nature, 569 (7758), 2019a. doi: 10.
Subramanian Sundaram, Petr Kellnhofer, Yunzhu Li, Jun-Yan Zhu Antonio Torralba, and Wojciech Matusik. Learning the signatures of the human grasp using scalable tactile glove. Nature, 569 (7758):698-702, 2019b. 1,3
Joshua B Tenenbaum, Vin de Silva, and John C Langford. A global geometric framework for nonlinear dimensionality reduction. science, 29005500):219-22223 2000.
Joshua B. Tenenbaum, Vin de Silva, and John C. Langford. Reduction a global geometric framework for nonlinear dimensionality. 2011. 2
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI, pp. 776-794, Berlin, Heidelberg, 2020. Springer-Verlag. ISBN 078-3-030-58820- doi: 10. 1007/ 978-3-030-58621-88 URL https: /oi. www.:o.g//0... 007 978 030 58621 8_..
Laurens der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Ma- chine Learning Research, 9866)25579222 2008. URL http: / jmlr.. org/ htgg/aapess/v/ vandermaaten088. html,
12

---

Published as a conference paper at ICLR 2024
Fengyu Yang, Chenyang Ma, Jiacheng Zhang, Jing Zhu, Wenzhen Yuan, and Andrew Owens. Touch and go: Learning from human-collected vision and touch. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL https:
//openreveww. ww..eeeeeeeeeee......ee.......ee
forum? Forum221-=223FesSPP o. 2,3, 6, 8
Li Yi, Haibin Huang, Difan Liu, Evangelos Kalogerakis, Hao Su, and Leonidas Guibas. Deep part induction from articulated object pairs. arXiv preprint arXiv: 07417, 2018.
Wenzhen Yuan, Shaoxiong Wang. Siyuan Dong, and Edward Adelson. Connecting look and feel: Associating the visual and tactile properties of physical materials, pp. 4494-4502, 07 2017a. doi: N0.1109/CVPR.2017.778
Wenzhen Yuan, Chenzhuo Zhu, Andrew Owens, Mandayam Srinivasan, and Edward Adelson. Shape- independent hardness estimation using deep learning and gelsight tactile sensor, pp. 951-958, 05 2017b. doi: 10.1 10.11099/CA 16. 2
Qiang Zhang, Yunzhu Li, Yiyue Luo, Wan Shou, Michael Foshey, Junchi Yan, Joshua B. Tenenbaum, Wojeiech Matusik, and Antonio Torralba. Dynamic modeling of hand-object interactions via tactile sensing. arXiv preprint arXiv: 209. 04378, 2021. 2
13

---

Published as a conference paper at ICLR 2024
A SUPPLEMENTARY
A.1 DATA COLLECTION SETUP AND TACTILE-HAND VISUALIZATION
Figure. shows our data collection setup and tactile-hand visualization facilitate understanding of the hand-tactile pressure map signal. Additional Visualization and Data Overview We add
glow
Figure 6: Data Collection Setup and Tactile-Hand Visualization
additional visualization to illustrate our data in Fig. 7. Visualization of motion sequences can be found on our supplementary website. Background on Glove The glove is made using piczoresistive
Figure Data Visualization. show demo our data with 12 samples from our objects.
film. The working principle of this material is when the material stretches, it causes the deformation of carbon nanotubes, which then induces the relative electrical resistance changes and thus difference in voltage readouts. The material in general is sturdy and the material property does not change much with wear and tear.
Glove Calibration. The glove only needs to be calibrated once to convert the sensor readouts in voltages to force magnitudes in Newton before use. We show the process of data calibration Fig.
Selup
Signal
Figure 8: Sensor Calibration. The left shows the sensor calibration setup before the embroidery of the insulation layer. The right shows the collected calibration data from 50N of continuous force applied to each of the 16 x 16 voltage readout positions.
14

---

transfer knowledge Published as a conference paper at ICLR 2024
A.2 DOWNSTREAM TASKS
To motivate downstream robotic applications, we show two experiments inspired by robotic perception applications. The high-level intuition is that our learned embedding space trained with human- collected data sufficiently represent data collected by robotic arms, from human demonstration to robotic tasks. First, we collected some data using a robotic hand show the cross-modal prediction results leveraging the pretrained embedding with human data. Additionally, we would like to see how much knowledge is transferrable from human demonstrations data robotic hand. To achieve this goal, we train a simple classifier using our trained embedding space with human data to enable manipulation and object classification through tactile sequences. While we only show two applications here, other downstream tasks can be enabled by our proposed method and dataset.
Robotics inspired application. We recognize that different robotic hands might induce different levels of domain gap. To minimize the domain gap, we use a robotic motor to drive a tenden-driven OpenBionics hand, and record tactile sequence shown below Fig. 9 a) and b). We use test- time optimization approach to find a latent code in the shared embedding space to decode into other modalities. We notice that because the robotic hand is non-conformable, the collected sequences of data are of inherent domain gap and thus result in different possible predictions that equally fit the optimization energy.
M
A
OpenBionicHand
bj Robolic Data
Video to Tactila Prodiction
d) Tectile to Video Prediction
Figure 9: Robotic application using shared manifold learning trained with human collected data, a) and b) show the task and data collection setup. c) d) shows the cross-modal prediction results of only using the video or the tactile sequence infer the other modality of signals. The top row shows the query signal, and the bottom two rows showcase different predictions made by our method.
Object classification with shared embedding. One other potential application is to help robotic tasks in low-light settings, by equipping robotic hands with tactile sensors. We can leverage our pretrained cross-modal manifold to recognize objects and even operate objects when vision is impaired. With our trained shared latent space we train a classifier classify the corresponding object category and manipulation type from the signal sequence. We withhold some latent codes from the classifier to test the baseline performance on human data. We use test-time optimization to obtain latent code in the shared embedding space using the robotic tactile data. We show the performance in Table. below.
test data type
manipulation
object
human test data robotic tactile data
0.96
0.97
0.86
0.48
Table Object classification of human and robotic data using shared embedding space We notice that human manipulation and robotic manipulation share significant amount of trans- ferrable and common knowledge. In that a classifier trained using the shared latent code that has only seen human data and also be used classify robotic manipulation data. On the other hand, the non-conformability of the robotic hand significantly restricts the information on the details of the object geometry in the tactile signal and thus limits the accuracy of classifying objects using robotic tactile data.
While these simple robotic applications of our proposed dataset and method, we note here that other more advanced robotic applications for different modalities of signals and leave more challenging robotic tasks as future work.
are possible. We focus on learning a joint representation
15

---

Published as a conference paper at ICLR 2024
A.3 MULTI-MODAL PREDICTION
We randomly initialize three different latent codes start test-time optimization for given tactile video sequence. We show that we are able to recover different cross-modal sequences with different initializations, shown in Figure. 10. We want to note that it possible that different latent initializations converge to the same mode prediction when there exists a global optima, or when two latent codes are initialized around the same neighborhood of local minima.
Query Signal
Prodiction
Prediction
Prediction
Figure 10: MultiModal Prediction via variational initialization test-time optimization.
A.4 ABLATION EXPERIMENTS
Table 3: Ablation Study
video tactile
tactile video
first frame inference w/out projection layer
0.0028
0.0247
0.0022
0.0194
dim 2n, = 256
0.0018
0.0212
dim 2,,- = 128
0.0018
0.0201
dim 24 =
0.0019
0.0196
dim 2
0.0023
0.0188
ours
0.0019
0.0174
We show our ablation experiments in Table. 3. We observe that removing the projection layer results in decrease of cross-modal prediction performance. We can also observe that by varying the latent dimension of the tactile signal 2h, the performance for video to tactile prediction does not change too much, which means that 2₁ is over wve--parameteried. Through decreasing the latent dim 2t. performance for tactile video starts increase. when latent dimension decreases, the neural field enforce the latent code to use limited space to register the same amount of information, and thus captures the principled information in sparse signals such as tactile. Our proposed method leverages latent dimension of 16, which trades off the marginal gain of accuracy reconstructing tactile signal, but captures much more principled information the tactile space. Finally, we also show that our model can also be directly adopted to only use the first frame signal during test-time optimization, we notice a performance decrease from using the full sequence data. We believe the difference resulted from an increase of variational inference, or in other words, with less optimization signal, there are now more local minima during test-time optimization that fit the optimization objective.
A.5 IMPLEMENTATION DETAILS
All our manifolds are randomly initialized with Gaussian distribution with (n, o) = (0, 1). Tactile manifold M₁ is initialized to be 16 dimensional and all other manifolds are initialized to be 256 dimensional. We use MLPs to express our neural fields 4. All MLP neural fields are of three layers and 512 hidden dimension. Specifically, the shared manifold is initialized to be an 256 dimensional embedding with Ntrain = 123, 561 randomly sampled from Gaussian distribution of (n, a) (0, 1). Pe is parameterized by 256 256 matrix, and J is parameterized using
16

---

Published as a conference paper at ICLR 2024
256 16 matrix. Neural fields des Pay Ωh all 6-layer MLPs using ReLU activation and 512 dimensional hidden-layers. We also use positional encoding for our -agnssiiant field with frequency cocfficient set to w 30.
A.6 TRAINING DETAILS
All baseline methods and proposed method implemented using the open-source Py- torch (Paszke et al., 2019) package with CUDA 11.7 backend. We use the same hardware and training setup to train all methods mentioned in this work. We
workstation equipped with NVIDIA Tesla V100 GPUs and 64-core AMD CPUs. We use Adam optimizer for training the baselines and our methods with learning rate initialized be le 3; batch size methods are trained to full convergence. For test-time optimization of latent code for GEM (Du et al., 2021) and proposed approach,
64, All
Adam optimizer run 1000 timesteps with batchsize of 32. For 1000 timesteps, we use 56 seconds in total, 17.87 iteration/second. on average every example use 1.75 seconds.
17


Reviews:
[{"review_id": "WXH0WIXrAU", "reviewer": "Reviewer_ksmF", "summary": "The authors have curated a unique visual-tactile dataset and introduced a manifold algorithm to explore the cross-modal relationship between objects and their manipulation. By visualizing the cross-modal latent structures, they showcase that their approach outperforms current methods and effectively generalizes manipulations to unfamiliar objects.", "questions": "Please see the weakness above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is articulately written, offering clarity and ease of comprehension, making it accessible even to readers unfamiliar with the subject matter.\n2. A significant contribution of this research is the introduction of a novel visual-tactile dataset, especially noteworthy given the limited datasets available in this domain.\n3. The innovative manifold learning approach presented has the potential to pave the way for subsequent research.\n4. Through experiments, the paper effectively showcases the promise of the cross-modal retrieval, prediction, and the latent structure. Compared to existing methodologies, the proposed approach holds considerable promise.", "weaknesses": "1. The dataset would benefit from enhanced visualization and in-depth details, possibly within the appendix.\n2. There's a typographical error on page 5 after equation 2; \"Additioanlly\" should be corrected to \"Additionally.\"\n3. Based on observations from figures 3 and 4, the sequences appear to have minimal variation across different frames. Displaying greater variation would add value. Additionally, considering a baseline that utilizes only the initial frame, as opposed to the entire sequence data, could provide intriguing insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iC17Qng9Kd", "reviewer": "Reviewer_u4UQ", "summary": "This paper purposed a force maps and RGB paird visual-tactile dataset. And further purpose to first represent each signal in a shared latent space, and then project the global manifold to local submanifold for each signal for reconstruction. The results demonstrate the effectiveness of purposed method.", "questions": "1. If as the author statement, force maps and RGB id many-to-many mapping, what is the advantage of using force maps as tactile signal?  Why different object will not have similar surface texture property? And the challenge of disparity in spatial scale of different signals seems also exist even if it is one-one mapping?\n2. How is the robustness of the tactile glove, will it need to calibrate a lot to make sure the tactile data is accurate?\n3. How long will it take for test time optimization? Since this might be important for robotic application?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper tactles a more challenging visual-tactile prediction task which is harder compare to previous works.\n2. The design of preoject global manifold to local submanifold force the model to capture different signals, and to futher incorporate with the test time optimization method to improve the prediction results.\n3. The experiments are comprehensive.The TSNE results also show the model learned with some semantic meaningful infomation.", "weaknesses": "1. My major concern is since the training set : testing set is aroud 12:1, and only include 4 categoreis, can this method really generalize to unseen objects? How is the diversity of the training and testing set?\n2. Presentation with Figure1: It would be better to also draw the process of getting shared latent space in the Figure1 for better understanding, it's quiet hard to undertand how to get a shared latent space from signals that are different dimension, could the author illustrate more about this ? Also projection layer of x seems missing in Figure 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "da5r30XzoL", "reviewer": "Reviewer_h9Bd", "summary": "This paper focuses on understanding everyday objects and tools from a manipulation standpoint. The authors have constructed a multi-modal visual-tactile dataset, consisting of paired full-hand force pressure maps and manipulation videos. Additionally, they introduce a unique method to learn a cross-modal latent manifold. This manifold facilitates cross-modal prediction and uncovers latent structures in various data modalities. The extensive experiments establish the efficacy of the proposed approach approach.", "questions": "1. Do you want to claim the dataset as one of the contribution? In another word, would you open source the dataset once the paper is accepted?\n\n2. Could you clarify the symbol $\\gamma$ used in Equation 2? I was unable to locate a definition for it within the text.\n\n3. Your elaboration on $I_i$ and $I_j$ would be appreciated, specifically in relation to the following sentence. How is the distance within the space of $I$ quantified, and how is the subtraction operation defined in Eq3 for $I_i$ and $I_j$, given that $I$ is a three-modality tuple? While I recognize that the manifold isometry loss is a standard loss in the manifold learning field, I would like to confirm if the subtraction operation is a simple reduction operation in the raw data format.\n> any two samples sampled from the signal agnostic manifold $\\{z_i, z_j \\} ⊆ M$ respects the distance between the samples $I_i, I_j$ \n\nMinor issues:\n1. There appears to be a typographical error in the first line of the 'Dataset: Data Collection Setup' section – the citation parentheses are empty.\n2. It would enhance clarity if Figure 1 was referenced in Section 4.2 and if further details about the components in the figure were provided. This issue is also applicable to Figure 2.\n3. Please use last names when citing authors. For instance, it should be 'Chen et al.' instead of 'Peter et al.'.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper tackles a common issue, manifold learning, through a pragmatic lens within robotics applications. The study aims to address multi-modal learning problems in the visual-tactile sensory observation context, a highly practical setup for manipulation tasks. The proposed representation learning method can be beneficial for a multitude of downstream applications within visuotactile learning in robotics.\n\n2. The method proposed in the paper is straightforward, suggesting that it does not place a heavy computational load on the system.\n\n3. The authors have gathered a substantial paired dataset for visual and tactile signals. If made publicly accessible, this dataset could prove to be a valuable resource for further research.", "weaknesses": "1. The method operates under the assumption that the sum of shape and tactile information equates to visual information. This assumption is manifested in the authors' approach of creating video latents by combining the latents of manipulation and the latents of canonical shapes. However, the tactile sequence may also encapsulate the object's geometric information. As suggested in the referenced paper 'Learning human–environment interactions using conformal tactile textiles,' tactile information can be employed to classify object geometry. Consequently, it's worth questioning the efficacy of combining shape and tactile embeddings to produce the video embedding.\n\n2. The cross-modality query necessitates an optimization process. Therefore, it's crucial to provide information regarding the time cost of these experiments. For instance, how much time would be required to employ the neural field in this inverse manner?\n\n3. The absence of videos in the paper is a notable limitation. Including video content could significantly enhance the understanding of the tasks and experiments conducted in the study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HFWqNnJFH1", "reviewer": "Reviewer_RFKG", "summary": "The paper studies the problem of learning representations for modeling visual and tactile sensor data. A large multi-modal visual-tactile dataset is presented, and a straightforward pipeline is proposed for learning the data. Experiments are performed on cross-modal prediction tasks to validate the idea.", "questions": "Providing additional technical details would enhance the reproducibility of the work, e.g., model architecture, training details, etc. \n\nIt would be interesting to see how the learned representations can be applied to downstream manipulation tasks, adding such results would further strengthen the paper.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper introduces a substantial dataset comprising paired visual and tactile sensor data, which holds the potential for significant advancements in cross-modal research; \n\nThe paper's organization is clear and easy to follow.", "weaknesses": "The proposed approach has been exclusively assessed in the context of cross-modal prediction tasks, with no concrete verification of its applicability in downstream manipulation tasks; \n\nMoreover, it is worth noting that a single video observation could potentially correspond to a wide range of tactile signals, such as variations in the force applied when touching dough. Regrettably, the study does not appear to account for the inherent multimodality in the distribution of data in this respect; \n\nThe paper lacks technical details, e.g., the learning rate, batch size, etc. \n\nThe video prediction results for new instances and new categories seem not promising.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)