Structured Bayesian Meta-Learning for Data-Efficient Visual-Tactile Model Estimation
Authors: Shaoxiong Yao
Yifan Zhu
Kris Hauser
Venue: CoRL 2024 (2024)
Keywords: Multimodal perception
tactile sensing
few-shot learning
TL;DR: Our SBML framework enables data-efficient visual-tactile model estimation by learning a prior of visual-tactile model from diverse real world objects.

Abstract: Estimating visual-tactile models of deformable objects is challenging because vision suffers from occlusion, while touch data is sparse and noisy.  We propose a novel data-efficient method for dense heterogeneous model estimation by leveraging experience from diverse training objects.  The method is based on Bayesian Meta-Learning (BML), which can mitigate overfitting high-capacity visual-tactile models by meta-learning an informed prior and naturally achieves few-shot online estimation via posterior estimation.  However, BML requires a shared parametric model across tasks but visual-tactile models for diverse objects have different parameter spaces.  To address this issue, we introduce Structured Bayesian Meta-Learning (SBML) that incorporates heterogeneous physics models, enabling learning from training objects with varying appearances and geometries.  SBML performs zero-shot vision-only prediction of deformable model parameters and few-shot adaptation after a handful of touches.  Experiments show that in two classes of heterogeneous objects, namely plants and shoes, SBML outperforms existing approaches in force and torque prediction accuracy in zero- and few-shot settings.

Structured Bayesian Meta-Learning for Data-Efficient Desual-Tactil Model Estimation
Shaoxiong Yao', Yifan Zhu?, and Kris Hauser' Department of Computer Science, University of Illinois at Shbaaaa-hampaign, IL, USA. *Department Mechanical Engineering and Materials Science, Yale University, CT, USA. {syao16, kkhauser Bileerrr@eillllli @illinois edu, yifan. zhu@yale... edu
Abstract: Estimating visual-tactile models of deformable objects is challeng- ing because vision suffers from occlusion, while touch data is sparse and noisy. We propose novel data-efficient method for dense heterogeneous model esti- mation by leveraging experience from diverse training objects. The method is based on Bayesian Meta-Learning (BML), which can mitigate overfitting high- capacity visual-tactile models by meta-learning an informed prior and naturally achieves few-shot online estimation via posterior estimation. However, BML re- quires shared parametric model across tasks but visual-tactile models for diverse objects have different parameter spaces. To address this issue, we introduce Struc- tured Bayesian Meta-Learning (SBML) that incorporates heterogeneous physics models, enabling learning from training objects with varying appearances and ge- ometries. SBML performs zero-shot vision-only prediction of deformable model parameters and few-shot adaptation after a handful of touches. Experiments show that in two classes of heterogeneous objects, namely plants and shoes, SBML out- performs existing approaches in force and torque prediction accuracy in zero- and few-shot settings. Website: https://shaae htssssaxioigggaaaa github. 1o/SBML
Keywords: Multimodal perception, tactile sensing, few-shot learning
Introduction
Fusing tactile information with vision is necessary for many manipulation tasks, including recon- structing occluded objects [1]. reaching in clutter [2], and assistive dressing [3]. A robot is expected to quickly acquire a visual-tactile model for efficient manipulation. Visual-tactile models of de- formable objects such as finite-llement models can make accurate predictions when an object's material parameters are properly identified [4, 5, 6]. However, this estimation problem challeng- ing and usually requires lot of data. Vision data has significant occlusions, leading ambiguity in the inference of physical response. Tactile data noisy and relatively sparse across object. These characteristics make the identification problems computationally expensive, susceptible to local minima, and ill-posed unless strong assumptions (e.g. homogeneity) are made [7].
SBML Pastrier
SBMAL Prfor
Touched region
Ciass Podwerr
Tactle
Sex
Figure 1: Our SBML approach enables few-shot learning of a novel plant's force response using experience interacting with other plants. Left: touched region (top) and tactile feedback (second row, norm of joint torques over time). Right: compare SBMI. against a stiffness estimator using naïve uniform Gaussian prior with the same touches. SBML prior already predicts stiffer trunks and branches compared to leaves and its predictions improve as it touches the plant (right columns).
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

We present few-shot learning method to estimate dense, heterogeneous visual-tactile models us- ing vision data and a small number of touches. Our approach is based on Bayesian meta-learning (BML) [8, 9]. which first learns a prior distribution over models, estimated offline from the experi- ence of related objects, and then adapts the model online via posterior inference. BML is appealing because can regularize high-capacity (overparameterizee)) models, which can help estimate het- erogeneous visual-tactile models in the face of data sparsity. BML also learns informed priors that improve few-shot accuracy compared to naive regularization methods. However, applying standard requires a unified model parameterization and cannot leverage the underlying physics of differ- ent deformable objects. Visual-tactile models for different objects usually have different numbers of state variables and material parameters. For example, the mesh of a large object has more elements than small object and therefore more parameters.
To enable the application of BML to setting, introduce the novel Structured Bayesian Meta- Learning (SBML) approach illustrated in Fig, 2. Each object generates structure of variable dimension and we learn a unified probabilistic model that predicts material parameters for each el- ement in the structure. The meta-prior is factorized so that it is learned as a parametric function shared across objects, even when the object is segmented into different numbers of parts. We show that SBML can be trained on diverse object sets with to enable rapid and accurate few-shot estima- tion on novel heterogeneous objects. In summary, this paper makes the following contributions:
We propose the SBML framework that enables meta-learning diverse tasks with different numbers of model parameters.
We show that SBML enables few-shot learning of dense heterogeneous tactile models on real touch data using vision priors and spatial correlation priors.
We evaluate the proposed approach on real-world plants and shoes datasets. SBML outperforms an unstructured state-of-the-art meta-learning approach and the vanilla VSF (Fig. 1).
2 Related Works
Visual-tactile model estimation Fusing visual and tactile information can improve the reliability of robot manipulation [3, 10, 11], but unifying these modalities is challenging due to visual occlu- sion and tactile sparsity. The parameters of analytical simulations, such as spring-mass and finite element models [5, 12, 6]. can be fit to observational data, but these estimation problems are com- putationally challenging and susceptible to local minima. Learning-baat
methods have also been applied, but typically assume homogeneity of the object [13, 14, 15] or restricted contact regions during tool usage [16, 17J. These assumptions reduce the representation power (i.e., capacity) of the visual-tactile models. More recently very high-capacity learning-bass based models (e.g. transformers) have been used to learn latent visual-tactile representation [18, 19], but these models are very data- hungry and are limited to simulation-onll or simulation-based pretraining. Our work enables the use of high-capacity heterogeneous visual- -tatiiee models from real-world data by learning an informed prior over model parameters, leading to data-efficient model estimation. We learn prior conditioned on visual information, which has been shown informative in predicting an object's material proper- ties [20, 21, 22] for fabric [23, 24, 25] and rigid surfaces [26. 27, 28). Our approach goes beyond current methods by predicting heterogeneous material and updating estimation online.
Meta-learning and Bayesian Meta-learning The goal of meta-learning is to simultaneously learn model that matches a given dataset and a process to adapt the model quickly novel datasets [29]. Meta-learning has enabled data-efficient robot learning in locomotion [30], manipulation [31, 32] and underactuated control [33, 34). Past methods often use gradient descent [35, 36) or a learned adaptation module for online update [37, 38, 39]. Bayesian meta-learning uses hicrarchical Bayes to create a unified learning objective for all tasks, enabling optimization with a single optimizer and can quantify uncertainty systematically [40, 8, 41, 42, 43). Whereas existing Bayesian meta-learning methods assume a shared parametric function across all the tasks, our proposed Structured Bayesian Meta-Learning leverages task-specific structures allow meta-learning over diverse tasks, such as visual- -act-le modeling setting. Meta- Metaallernnng graph-structured neural networks [44, 45] is related to approach but only addresses node classification problems.

---

Offline Meta- Meta-learning
Online Model Adaptation
Visual-tactile
Structure generation module
Online visual-tactile data
Object structure S,
RGBO Image
Compusat enal most Ma
Vacal
Material composition S
features
Time
Online model estimation 0-shot
Bayesian Meta-learning
Structured meta-prion priorrrorrroo ((
S-ohot
Object,
Figure 2: An overview of the structured Bayesian meta-learning method. For each task, structure generation module uses perception data create of structural assumptions mapping from common meta-parameter space to visual-tactile models of different objects. Online, maximum a- posteriori estimation is performed to infer material properties over the novel task's s structure.
3 Method
visual-tactile model of a deformable object captures how the robot's visual-tactile observations change during interaction. At time step t, the robot executes action a' (e.g.. joint position com- mands) and receives tactile and visual observations z' (e.g. force/torque readings and RGBD im- ages). In our formulation, we consider visual-tactile models that map a sequence of action a' to sequence of observations and we denote input-output pair = (al (a'..... a and y (z ......"! 2T) as a touch. Visual-tactile model estimation aims to identify this mapping and our SBML approach enables data-efficient estimation with few touches. In this section, we will first present background on BML, our extension SBML, our instantiation of the framework for visual-tactile modeling. and strategies for making offline and online learning more efficient and accurate.
3.1 Structured Bayesian meta-learning
Standard meta-learning considers meta- veea-training dataset D = notes a task and D. = {(Ta,, Bo.i) i = 1,..., N. task-specific dataset. Hach output follows distribution Pyy|x, a:) conditioned on both input æ and task a, and each your is condition- ally independent given Fa,; and a. The goal of meta-learning is predict P(ylær a.) for novel task as given data from support D. ((*,,)) V+,)) i 1,..., N.), particularly from the zero-shot (N, 0) the few-shot (small N.) cases. Note that a, is unknown and need infer it from the support set D..
L} where a de-
In Bayesian meta-learning the task-conditioned output distribution is approximated as P(ylæ, a) Ro f x, 8), where E TR. M are the model parameters corresponding to the task. A prior P(0) is learned offline for online estimation of the model posterior D.)) from the support set. BML con- siders model parameters of different tasks statistically independent conditioned meta-parameters sb, i.c. 0 P(O Note that f(:: -; 0) must be sufficiently rich class of functions so that every task maps to a corresponding 0, and P(0 (b) should capture the meta-training data without over- fitting. More precisely, offline learning can be formulated maximum likelihood estimation of meta-parameters ub over the meta-training dataset 1 [8].
f II
/ 0=
arg max
Here the data likelihood is evaluated as P(D.1aa) = N fVa. Fa,i, 0a). In the online phase, we find the maximum a posteriori (MAP) estimate of model parameters 0 using the Bayes rule:

---

0. = arg max P(D. P0
(2)
6_
and then the online prediction is given by P(y|ær D.) = [(v: x. 0.).
In the visual-tactile model estimation setting, the standard assumption in BML regarding fixed- dimensional inputs 2, outputs y, and model parameters 0 does not hold. Different touches have different sequence lengths Ta, and diverse objects have varying dimensional model parameters 0 E TR. In particular, represent heterogeneous deformable object as computational mesh with various particles and elements, leading to varying numbers of material parameters to estimate.
Our proposed Structured Bayesian Meta-Learning (SBML) framework addresses this problem using task-specific Bayesian network structures mapping a fixed-dimensional meta-parameter to a prior that supports diverse tasks. The structures for each task a are generated by a structure genera- tion module outside the learning pipeline in Fig. 2. We use S. denote the task-dependent structure. Conditioned on Sa, the probability distributions in Eq. (I) and (2) are factorized into parametric functions of z shared across tasks. We consider two types of structures specifically relevant to visual-tactile models. First, a hidden Markov structure can handle the dynamics and variation in sequence length Ta, governing the dimensionality of æa, (a "ox")) and Yo,, (za. a,, ). In this model, the object state sf has Markovian transition model defined by dynamies model Dym and the observations z' depend on the current state s via an observation model P(z'|s', S.). The data likelihood f(ya, Fa Da, Sa) is therefore factorized as follows:
Tois
(yaai Ta On, S.) II P(za. On, Sa), where a Dym(s' da Sa,
Here Sa provides the initial state s- For simplicity, we consider a dynamic model Dyn that is deterministic given material parameters 0a. From this formulation we can replace f in Eq. (I) and (2) with its structured counterpart.
Second, SBML addresses the structure-dependent set of model parameters 0a and the prior in Eq. (1) and (2) changes from P(O,|v) (b) to P(O., 1,, S.). We formulate decoupled latent variable structure that maps set of latent parameters da to a task-dependen prior over model parameters 6a. via linear map 0. = Ba for simplicity. The structure S., defines a set of features Va informative to 6a. which are visual features Vor mapped to each element j for a visual-tactile model. The latent parameters da has factorized distribution conditioned Va i.e. Φalvo Ua Ih P(oa, Ma,A, v/)- Ultimately, SBML learns parametric functions P(oo,,/Va..: tb)
g(Da,i Va,k () shared between tasks, e.g. neural network that outputs probability density. The latent structure depends on the instantiation of this framework, and our implementation is described below.
3.2 Visual-tactile model estimation via SBML
In visual-tactile model, we consider structure S., that captures object's geometry in computa- tional mesh Ma. with initial state sa. its appearance in visual features Vo and material correlation in composition structure C. as in Fig. 3. We will define the visual-tactile simulation model and material parameters 0,, to evaluate data likelihood in Eq. (3). The material parameter prior maps the univer- sal latent parameters prior g( i Va,k, v) to P(0a (v,, in a manner that defines a relationship between material parameters and appearance, and correlations between elements.
Object representation and simulation model. We consider particle-based based visual-tactile models that discretize the object into particles. The stafe of the object is the positions of the particles s¹ E 3 = (pi..., ..., Ph..), ), where pl. E 1R3 is particle k's position at time step t. The interaction be- tween particles is defined by a computational mesh M,, containing m., elements, and each element specifies a subset of particles with mutual interactions. The jth element, j = 1,..., Ma. has fixed d-dimensional material parameter fa.j E Rdd that enables us to simulate particle interactions and observations. The dynamics model Dyn updates the particle deformation s' using interactions in each element in Fig. 3. The visual-tactile model parameters of the object are the tuple of material parameters of each element 0,, = (0a.1,... 0o.m.).

---

Structure
Camputationa mush K
Structured meta-orior, v (ohoo Muc)
Visual-tactile simulation
generation
module
Patto
Material
RGBD
Image
features
+
Vorl
Object,
Figure 3: Overview of structures in visual-tactile model estimation. Left: construct the structure
Sa for an object: Middle: map meta-parameters & to material parameters 6er with homogeneous heterogeneous composition structure; Right: temporal structure in visual-tactile simulation.
Visual features and composition structure. We use visual features and composition structures to define the map from latent parameters prior to primary material prior. This allows our model to capture the relationship between appearance and material, and the correlation between material parameters.
First, defines visual features mapped to each element in Ma. where element has feature Va.j- Choosing expressive visual features will make the prior more informative to predict material param- eters, we use dense features from pre-trained vision models DINOv2 [46] by default.
Second, the material composition structure C. lets us accommodate correlation assumptions. The composition structure assumes the object is composed of components with similar materials, e.g- branches tree share similar high stiffness compared to soft leaf regions. We define set of com- ponents and each component has material parameters & mapped to a subset of elements C Ma. Here component material parameters & are the decoupled latent variables. We consider three types of components denoted by component type b E {HE, HO, SEG), where HEterogeneous compo- nent has a single element, HOmogeneous component has all elements, and SËGmented component has cluster of elements. Also, the visual features for single component aggregate the visual features j of all elements j E and
the average feature [o] Ejec Vo.j for simplicity.
Formally, C. {(Ca,k: ba Φa.k Va.k) Va.k))k=I..... defines a set of components by their elements, types, component material parameters, and visual features. Then, the material parameters are the perposition of components j,.j = - Iíj E (0, 1) matrix of size M,, |Ca, and Φ. = (#a,1,..... Da,,c,,)) stacks component material param- eters. Our prior expresses the knowledge that components the same type share the same material parameters distribution conditioned on the visual feature, P( Vo de; v)) 9b.. (Po. Co.k. v)).
and we define 6, = Bu where Ba is a
We provide different types of components with independent priors and with independent subset of meta peta-parrmetees: Db.... (Pa.ki ka,k, /)) Ih.., (9a, Vark, ). The material parameters prior can be evaluated by marginalizing over component material parameters Pa.
Cal
((0lWiS.) - 110. = -Boool II goa. (Da.ii Vo.ks v)d Aem
(4)
Overall, Eqs. (3) and (4) enable SBML to define a universal meta-learning problem over yb and task-dependent posterior estimation over 6g regardless of the tasks' structure.
3.3 Efficient implementation with Gaussian likelihoods
To achieve efficient implementation, instantiate the SBML framework on the Volumetric Stiff- ness Field(VSF) [47] visual-tactile model. VSF is a dense particle-based representation with inde- pendent Hookean springs and can flexibly represent heterogeneous material.
In Eq. (3), the dynamics model Dyn a particle- baseil deformation simulator the same as [47] and the observation model P(=*|s', S..) approximated as a LinearGGausiia function. Lin- earity is from the Hookean springs VSF and the additivity of forces/torques over springs. The Gaussian assumption models the tactile sensor noise normal distribution and is reasonable approximation for the sensors we evaluated.
5

---

Training abjects
Validation objects
Testing objects
Robot Setup
Figure 4: Left: Objects in the shoes (top) and plants (bottom) datasets. The scales of the shoes and plants are different; Right: Physical experiment setup collecting shoes dataset with Punyo tactile sensor and Kinova Gen3 arm.
The prior P(O,, (,) defines Gaussian distribution over the material parameter On, In par- ticular, .b.. is a neural network that outputs the mean and variance of oa.k. The integration in Eq. (4) has a closed-form form solution with a Gaussian integrand.
We evaluate likelihood in Eq. (1) using PyTorch [48] and optimize meta- meta-parameters v via gradient descent. The online parameter estimation in Eq. (2) quadratic program (QP) with nonnegativity constraints the spring stiffnesses. Further details are provided in Appendix A.
4 Experiments and Results
We acquire two benchmark datasets using the experimental platform in Fig. 4: 1) Plants dataset has 12 artificial plants each touched 50-70 times and 7-d joint torque from the Kinova arm; 2) Shoes dataset has 23 shoes each touched 20-30 times and pressure data from Punyo tactile sensor [49]- We use joint torques because the robot arm contacts the plants along its entire arm, while contact with shoes is localized to the end-effector. We show that SBML enables data-efficient visual-tactile model estimation by comparing SBML's zero- and few-shot predictions with baselines on real-world datasets. We also evaluate the effects of different meta-training datasets and material composition structures on SBML performance. Further experiments on the choice of visual features, prior capac- ity, and dataset size are presented in Appendix C.
4.1 SBML implementation on VSF model
Structure generation module The structure generation module samples VSF particles from the object's surface and interior using ray casting. Visual features Va generated by DINOv2 [46] and projected in 3D space. Our standard setting assumes a heterogeneous material composition C.- Other material composition structures are tested in Sec. 4.5.
Meta-prior The meta-prior for heterogeneous components uses a multi-layer perceptron (MLP) that outputs the mean and standard deviation of Gaussian distribution. The meta-priors for the segment and homogeneous components in Sec, 4.5 use linear mean model with learnable variance, chosen due to the small number of segments and tasks in the meta-training dataset.
Observation models The observation 2' is joint torques 7' R in plants and pressure difference from contact 8t IR in shoes. Note the observation models stiffness. Joint torques are the summation of Jacobian transpose times contact forces. For the Punyo sensor, we use simplified 1-D observation model that sums the force magnitude of each point.
inear-Gaussian in the spring
4.2 Baseline methods
Non-structured meta-learning We compare standard meta-learning method that directly maps robot action to tactile observation using object-specific structure. The model is instantiated as MLP and uses DINOv2 classification token [46] to encode visual information. The model is meta- trained and adapted using iMAML [36], where the inner loop adaptation uses gradient descent.
Structured model with naive prior We compare to the "vanilla" VSF model with naive Gaus- sian prior. Vision is not used, and the naïve prior assigns the same mean and variance to all particles. The mean and variance are estimated from estimated VSF stiffness in the meta- meta-training datasct.

---

4.3 Zero-shot stiffness estimation
0.5 N/m
100.0 N/m
We visualize the zero-shot stiffines pre- diction SBML for test objects Fig. organized in three rows: the top row shows the object, the middle row shows the SBML prior mean and the bottom row shows the estimated stiff- ness using vanilla VSF (average prior)
Test
Prior mein
given all touches in the dataset. We observe that the zero-shot stiffness es- timates are qualitatively quite close to the all-touch estimates, with stiffer es- timates the branches and trunk of
VSE all-shot
plants and the toes of shoes. Note
that VSF all-shot is an estimation with
all available support data and we can- not get the precise material parameters. The uncertainty of SBML prior visu- alized in Fig. 10.
Figure 5: Learned zero-shot priors estimated VSF from touches (w/o prior). Points never touched in the dataset are shown in gray. Stiffness color map is in the log scale.
4.4 Breadth of meta-training set
Mismatched Hetero.
Hetern. Seg.
Hetero.
Test object
Narrow
Broad
Homoge.
Next, we show the SBML k- shot predictions and evaluate how the breadth of the meta- training set affects results. We expect the best performance when the meta-training set dis- tribution is narrow and con- tains the test object within its support. The broad datasets contain all training objects as shown in Fig. 4. The plants narrow meta-training dataset contains different views of the same plant and the test set contains touches on the novel view. For the shoes dataset, we create narrow datasets by selecting shoes of similar ma- terials, specifically choosing two boots, nylon running shoes, and two sneakers for each test object.
0-sbot
I-slof
S-shor
Figure 6: Effects of the meta-training dataset on k-shot predic- tions. The first column shows the touched region in blue and the untouched region in green. Same colormap to Fig. 5. Black lines in (Hetero.+Seg, column, O-shot row) show segment boundaries. See Sec. 4.5 for details.
Fig. shows the qualitative re- sults for the boot test object, where SBML is trained with narrow, broad, and mismatched meta- training sets. The k-shot touching sequences from the test object's dataset. In the second row of Fig. 6, all priors estimate the front of the toe to be stiff, but only the narrow prior includes a stiff tongue. The broad prior has a moderately low stiffness and the mismatched prior estimates a very low stiffness due to the large domain gap. As expected, each model adapts to the observed data, but the narrowly trained model tends to shift less.
indicates the posterior estimate after observing support set of
We show quantitative results the test objects in Tab. 1 and Tab. 2. The prediction accuracy is evaluated on a query set disjoint from the support set. We also report the standard deviation of
7

---

Table I:
Joint torque prediction error (Nm) + Table 2:
Tactile prediction error (hPa) + std over query set on Shoe benchmark.
std over query set on Plant benchmark.
iMAML
Gaussian prior
SBML prior
SBML prior
iMAML
Gaussian prior
SBML prior
SBML prior
(bmad datasel)
(averige) (narrow dataset)
(brad dataset)
(broad dutaset)
(averige)
[narrow dataset)
(broad datasse))
O-shor
3.86 0.84
5.23 + 0.00
2.82
0.04
2,80 0.05
O-shot
9.95 + 6.71
8.27 0.00
5.66
0.23
7.43 + 0.25
1-shot
4.99 1 2.04
5.04 4: 0.04
2.78
0.03
2.76 0.05
I-shot
6.30 1: 1.00
7,46 1: 0.13
5,48
0.19
6.77 1: 0.26
S-shor
4.63 + 1.26
4.42
+ 0.07
2.71
+ 0.05
2.67 0.05
5-shot
6.21 + 0.96
5.54 + 0.21
4,91
+ 0.12
5.05
+ 0.20
10-sbot
1.05
3.88
0.03
2.61
10-shot
6.15
4.62 0.08
4.54
4,18 0.04
VSF all-shot 2.57
VSF all-shor 4.10
prediction error over 10 random seeds. The Gaussian prior has zero standard deviation for O-shot since its initial mean prediction is deterministic. The last row shows the VSF prediction error using all support set touches covering the entire object.
SBML not only demonstrates lower zero-shot prediction errors in the first row Tab. and Tab. 2, underscoring the value of incorporating visual information but also quickly improves with additional support data, achieving near-all-shot accuracy by 10 shots. SBML's absolute improvement is smaller due to an almost optimal zero-shot error in Tab. 1. In contrast, iMAML exhibits worse zero-shot performance and noisy adaptation. Using object-specific structures, the Gaussian prior improves the estimation consistently but has worse few-shot accuracy due to naive uniform mean. Comparing meta-training datasets, we observe that SBML performs best when the training and testing distribu- tions closely align. While the performance gap across meta-trac meta-training distributions is minor on plants, becomes significantly larger on shoes, likely due to the greater variation in shoes' appearance and material. Shoes are made of brown leather in boots, gray nylon in sneakers, and various colored plastics in slippers, while plants typically have brown branches and green leaves, as shown in Fig. 4. To evaluate the capacity of our method, we trained a unified prior using data from both plant and shoe datasets. The results in Appendix C.3 demonstrate that our method has sufficient capacity to learn prior shared between the plant and shoe categories.
Hetero.
Hetero. + Homoge.
4.5 Effects of material composition structure
Hetero.
Seg.
O-shot
9.76 0.75
9,23 0.86 9.13 1 0.77
Finally, we evaluate the effects of changing the material com- position structure and we tested heterogencous, heterogeneous segment, and heterogeneous + homogeneous. Qualitative results are shown in the right three columns of Fig. 6. We that the stiffness across segment changes together for heterogencous segment, and the stiffness of the entire object changes for heterogeneous + homogeneous, We also found the composition structure speeds up model adaptation quantitatively in Tab. 3. When the test object is out of meta-training distribution, the learned stiffness prior will predict inaccurate stiffness values, and the shared components will help learn across regions of coherent material. We test these com- position structures on narrow and broad datasets, but the improvement is marginal.
1-shot
8.58
0.68
7.57
8.10 0.57
1.38
S-shot
5.82
0.31
5.00
0.26
5.32 1 0.17
10-shot
4.73
0.10 4.36 0.07
4.49 0.07
VSF all shot 4.10
Table 3: SBML's predictions with different compositions on shoes mismatched dataset.
5 Conclusion and Limitations
We present a novel structured Bayesian meta-learning (SBML) approach to address the few-shot learning of visual-tactile models. Our innovations allow the method to transfer offline knowledge from objects of different sizes and shapes to improve prediction accuracy on novel objects. The method is highly flexible and can accommodate different material compositions, feature represen- tations, and model capacities. Applied benchmark datasets of plants and shoes, our experiments demonstrate that it outperforms non-informed estimators well as non-structured meta-learning approaches in zero- and few-shot prediction accuracy.
Limitations: The SBML framework is evaluated on the VSF model that can only predict tactile response, and we are interested in extending it to other deformable object models such as FEM and Graph Neural Dynamics Models that can predict visual deformation. The Gaussian prior assumes single mode of material parameters conditioned on visual features and cannot handle more com- plicated multi-modal distributions. Our work has only demonstrated generalization within object categories, but this approach has not yet enabled generalization between categories.
8

---

Acknowledgments
This work has partially been funded by USDA/NIFA Grant #2020-67021-32799 and we thank Toy- Research Institute for loan of the Punyo tactile sensor. We thank Joao M. C. Marques, Jing- Chen Peng, Mengchao Zhang, Patrick Naughton, and Sicong Pan for their valuable comments and suggestions. We are grateful to Yidi Hua for segmenting object images. We also thank Simon Kato, Haonan Chen, Yuan Shen, Zhen Zhu, Shuijing Liu, Hameed Abdul-Rashid, Yiqiu Sun, Mengchao Zhang, Yangfei Dai, Yixuan Wang, Kaiwen Hong, Shuhong Zheng, and Yuxiang Liu for providing objects used in the experiments.
References
[1] L. Rustler, J. Lundell, J. K. Behrens, V. Kyrki, and M. Hoffmann. Active visuo-haptis object shape completion. IEEE Robotics and Automation Letters, 7(2):5254-522 5261, 2022.
[2] A. Jain, M. D. Killpack, A. Edsinger, and C. C. Kemp. Reaching in clutter with whole-arm tactile sensing. The International Journal of Robotics Research, 32(():458 482, 2013. doi: 10.11777027836491 12471865.
[3] Z. Erickson, M. Clever, G. Turk, K. Liu, and C. Kemp. Deep haptic model predictive control for robot-assisted dressing. In 2018 IEEE international conference robotics and automation (ICRA), pages 4437-4444. IEEE, 2018.
[4] B. Lloyd, G. Székely, and M. Harders. Identification of spring parameters for deformable object simulation. IEEE Iransactions on visualization and computer graphics, 13(5):1081- 1094, 2007.
[5] B. Frank, C. Stachniss, Schmedding, M. Teschner, and W. Burgard. Learning object de- formation models for robot motion planning. Robotics and Autonomous Systems, 62(8): 1174, 2014. ISSN 0921-8890.
[6] H. Yin, A. Varava, and D. Kragic. Modeling, learning, perception, and control methods for deformable object manipulation. Science Robotics, 6(54))eabd8803, 2021.
[7] V. E. Arriola-Rios, P. Guler, F. Ficuciello, D. Kragic, B. Siciliano, and L. Wyatt. Modeling of deformable objects for robotic manipulation: A tutorial and review. Frontiers in Robotics and AI, 7, 2020. ISSN 2296-9144. 06::10.3389//rrb..000000000
[8] E. Grant, C. Finn, S. Levine, T. Darrell, and T. Griffiths. Recasting gradient-based meta- learning as hierarchical bayes. In International Conference Learning Representations, 2018. URL /ttps::/opeerevvi//
met/forum?id-BJ_U -OOD
[9] J. Yoon, T. Kim, O. Dia, S. Kim, Y. Bengio, and S. Ahn. Bayesian model-agnostic meta- learning. In Bengio, H. Wallach, H. Larochelle, K. Grauman, N. R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31, Curran Associates, Inc., 2018. URL https: https: http:/ooooeee.nnnoooooooo..eo.p. nourins. wttps/scccppppppeeeeeeeeeeeeeeee httpppppfflllssppperrr
and
[10] J. Kerr, H. Huang, A. Wilcox, R. Hoque, J. Ichnowski, R. Calandra, and K. Goldberg. Self- supervised visuo-tactile pretraining to locate and follow garment features. In Robotics: Science and Systems, 2023.
[11] N. Sunil, Wang. She, E. Adelson, and A. R. Garcia. Visuotactile affordances for cloth manipulation with local control. In K. Liu, Kulic, and Ichnowski, editors, Proceedings of The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 1596-1600 PMLR, 14-18 Dec 2023. URL attps://proceee press/v205/ sun///unllll sunil23.... 123a. html.
mlr.

---

[12] V. E. Arriola-Rios and J. L. Wyatt. A multimodal model of object deformation under robotic pushing. IEEE Transactions on Cognitive and Developmental Systems, 9(2):133-169 2017.
[13] H. Shi, H. Xu, Z. Huang, Y. Li, and J. Wu. RoboCraft: Learning to See, Simulate, and Shape Plasto-Plastic Objects with Graph Networks. In Proceedings of Robotics: Science and Systems, New York City, NY, USA, June 2022. doi:10.15607/RSS 2022.XVIII.06
[14] Y. Wang, D. Held, and Z. Erickson. Visual haptic reasoning; Estimating contact forces by observing deformable object interactions. IEEE Robotics and Automation Letters, 7(4): 1426- 11433, 2022. 100..11000RR.A222......
[15] H. Shi, H. Xu, S. Clarke, Y. Li, and J. Wu. Robocook: Long-horizon elasto-plastic object manipulation with diverse tools. 7th Anmual Conference on Robot Learning. 2023. URL
[16] Y. Wi, A. Zeng, P. R. Florence, and N. Fazeli. Virdo++:
Real-world, visuo-tactile dynamics and perception of deformable objects. In Conference on Robot Learning, 2022. URL https:
org wrgg/orrpuSD:::5550 2522762377.
[17] M. J. V. der Merwe, Y. Wi, D. Berenson, and N. Fazeli. Integrated Object Deformation and Contact Patch Estimation from Visuo- Tactile Feedback. Proceedings of Robotics: Science and Systems, Daegu, Republic of Korea, July 2023. 1oi:10.15607/RSS. 202.
[18] Y. Chen, M. der Merwe, A. Sipos, and N. Fazeli. Visuo-tactile transformers for manipula- tion. 6th Annual Conference on Robot Learning, 2022. URL Corumrid=JJq@STydRRS..
net/
[19] C. Sferrazza, Y. Seo, H. Liu, Y. Lee, and P. Abbeel. The power of the senses: Generalizable manipulation from vision and touch through masked multimodal learning, 2023.
[20] E. H. Adelson. On seeing stuff: the perception materials by humans and machines. In B. E. Rogowitz and N. Pappas, editors, Human Vision and Electronic Imaging VI, volume 4299, pages International Society for Optics and Photonies, SPIE, 2001. doi: 11001111//2.. 429489. URL https:///oi.org// 1ttp:/...11//1..11111 4294992..
[21] R. W. Fleming. Visual perception of materials and their properties. Vision Research, 94:62-75, 2014. URL https: //api. http:::://app.smmanni...... semanticscholar httpriiiiccoaaaror/rrrririoie : 18018810.
[22] F. Yang, C. Feng, 7. Chen, H. Park, D. Wang, Y. Dou, Z. Zeng, X. Chen, R. Gangopadhyay, A. Owens, et al. Binding touch to everything: Learning unified multimodal tactile representa- tions, arXiv preprint arXiv:240I. 18044, 2024.
[23] J.-T. Lee. D. Bollegala, and S. Luo. "touching to see" and "seeing to feel": Robotic cross- modal sensory data generation for visual-tactile perception. 2019 International Confer- ence on Robotics and Automation (ICRA), pages 4276 4282, 2019. URL https://api.
semanticscholat wwwoooo////o..oooooooo Prrg/CorpusID: 62841580.
[24] S. Luo, W. Yuan, E. H. Adelson, G. Cohn, and R. Fuentes. Vitae: Feature sharing between vision and tactile sensing for cloth texture recognition. 2018 IEEE International
Conference on Roboties and Automation (ICRA), pages 2722-2727, 2018. URL https:
l'api.semaniicscoolarr wwg/CorpusIDDDD http/CorpuSIDD33391... 3639913.
[25] W. Yuan, S. Wang, Dong, and E. Adelson. Connecting look and feel: Associating the visual and tactile properties of physical materials. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), July 2017.
10

---

[26] S. Cai, K. Zhu, Y. Ban, and T. Narumi. Visual-tactile cross-modal data generation using residue-fusion gan with feature-matching and perceptual losses. IEEE Robotics and Automa- tion Letters, 6:7525-7532, 2021. URL https: https///.pp.smttsssschl.aaaa. 235795039.
semanticscholar. org/CorpusID.
[27] M. Purri and K. Dana. Teaching cameras to feel: Estimating tactile physical properties of surfaces from images. In European Conference on Computer Vision, pages 1-20. Springer, 2020.
[28] K. Takahashi and J. Tan. Deep visuo-tactile learning: Estimation of tactile properties from images. In 2019 International Conference on Robotics and Automation (ICRA), pages 8951- 8957. IEEE, 2019.
[29] T. Hospedales, A. Antoniou, P. Micaelli, and A. Storkey. Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(9):5149-5169 2021.
[30] A. Nagabandi, I. Clavera, S. Liu, R. S. Fearing, P. Abbeel, S. Levine, and C. Finn. Learning to adapt in dynamic, real-world environments through meta-reinforcemenn learning. In 7th Inter- national Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-
9, 2019. OpenReview.net, 2019. URL https:/ /ttt:p:///opnrrevieee wtwp://nen///ooreeeeeeeeee htt.///orrrm//.diiiyittttoo
[31] C. Finn, T. Yu, T. Zhang, P. Abbeel, and S. Levine. One-shot visual imitation learning via meta-learning. In Conference on robot learning, pages 357-368. PMLR, 2017.
[32] Y. Zhu, P. Thangeda, M. Ornik, and K. Hauser. Few-shot adaptation for manipulating gran- ular materials under domain shift. In Proceedings of Robotics: Science and Systems. Daegu, Republic of Korea, July 2023. 15.00 2ttp://SSS..2..XXX....
[33] M. O' C'Cnnell, G. Shi, X. Shi, K. Arizzadenesheli, A. Anandkumar, Y. Yue, and S.-J. Chung. Neural-fly enables rapid learning for agile flight in strong winds. Science Robotics, 7(66): eabm6597, 2022. doi: 12t1////iinnoooticaamm65959
URL https: http:///wwwcciene.rrr...
doi/abs/ 1128/ccirobotics s.b abm6597.
[34] S. M. Richards, N. Azizan, J.-J. E. Slotine, and M. Pavone. Adaptive-conrro--orinnted meta- learning for nonlinear systems. In Robotics: Science and Systems, 2021. In press. Available at nttps://arxia: org/ abs/ 04490.
[35] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. In International Conference Machine Learning, volume 70, pages 1 1126-1101 1135, 2017.
[36] A. Rajeswaran, C. Finn, S. M. Kakade, and Levine. Meta-learning with implicit gradi- ents. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Gar- nett, editors, Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. URL https::/prooceeddngg wwwptinnn.nnuurrrrree. nwurips.cc...c.. wwttriss.ccppaaape cc/paper faaerrffiessppe files/paper/
[37] M. Garnelo, D. Rosenbaum, C. Maddison, T. Ramalho, D. Saxton, M. Shanahan, Y. W. Teh, D. Rezende, and S. A. Fslami. Conditional neural processes. In International Conference on Machine Learning, pages 1704-1713,
[38] S. Ravi and H. Larochelle. Optimization model for few-shot learning. In International Conference Learning Representations, 2017. URL https://openneview..eer net/forum? Kd=rJYO-Kcll.
[39] A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap. Meta-learning with eemmory-augmented neural networks. In International conference machine learning, pages 1842-1850. PMLR, 2016.

---

[40] T. Heskes. Empirical bayes for learning to learn. In Proceedings of the Seventeenth Interna- sional Conference on Machine Learning, ICML '00, page 367-374, San Francisco, CA, USA, 2000. Morgan Kaufmann Publishers Inc. ISBN 1558607072.
[41] C. Finn, K. Xu, and S. Levine. Probabilistic model-agnostic meta-learning. Advances in neural information processing systems, 31, 2018.
[42] Harrison, Sharma, and M. Pavone. Meta-learning priors for efficient online bayesian regression. In M. Morales, L. Tapia, G. Sánchez-Ante, and S. Hutchinson, editors, Algorithmic Foundations of Robotics XIII, pages 318-337, Cham, 2020. Springer International Publishing.
[43] Rothfuss, Heyn, J. Chen, and A. Krause. Meta-learning reliable priors in the function space. In Conference on Neural Information Processing Systems, pages 280-293, 2021.
[44] F. Zhou, C. Cao, K. Zhang, G. Trajcevski, T. Zhong, and J. Geng. Meta-gnn: On few-shot node classification in graph meta-learning. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pages 2357-2360, 2019.
[45] K. Huang and M. Zitnik. Graph meta learning via local subgraphs. Advances in neural infor- mation processing systems, 11-5862-5874, 2020.
[46] M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, R. Howes, P.-Y. Huang, H. Xu, V. Sharma, S.-W. Li, W. Galuba, M. Rabbat, M. Assran, N. Ballas, G. Synnaeve, I. Misra, H. Jegou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
[47] S. Yao and K. Hauser. Estimating tactile models of heterogeneous deformable objects in real time. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 12583-12589, 2023. doi:10.1109/1CRA4BB60 /0t.:///AAA8889122.2200000 202...001.....33.
[48] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. Veto,,, M. Raison, A. Te- jani, S. Chilamkurthy, B. Steiner, L. Fang. J. Bai, and S. Chintala. PyTorch]: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Bucc, E. Fox, and R. Garnett, editors, Conference on Neural Information Processing 2019.
Systems, pages
[49] A. Alspach, K. Hashimoto, N. Kuppuswamy, and R. Tedrake. Soft-bubble: A highly compliant dense geometry tactile sensor for robot manipulation. In 2019 2nd IEEE International Con- ference on Soft Robotics (RoboSoft), pages 597-604, 2019. 100:100.10///OBOOFTT.00 8722713.
[50] M. H. DeGroot and M. J. Schervish. Probability and Statistics. Pearson, edition, 2012.
[51] D. P. Kingma and Ba. Adam: A method for stochastic optimization. In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http: // //arxiv.org org/ awtpaabs/.....///////////////aea 6980.
[52] S. Diamond, E. Chu, and S. Boyd. CVXPY: A Python-embedded modeling language for convex optimization, version 0.2. http://evxpy org/, May 2014.
[53] J. Rothfuss, V. Fortuin, M. Josifoski, and A. Krause, PACOH: Bayes-optimal meta-learning with PAC-guarantees. In International Conference on Machine Learning, pages 9116-9122 126, 2021.
12

---

Appendices
A Efficient visual-tactile model estimation
Here we provide more details on the efficient SBML implementation on VSF with Gaussian likeli- hoods, outlined in Sec. 3.3.
Volumetric stiffiness field A VSF defines a volume of independent particles that resist displace- ments from their rest position with Hookean springs. (Note that this differs from standard spring- mass systems where springs connect particles). The dynamics model Dyn is a geometric simulator that simulates the particle displacements independently. This simulator detects collisions between the robot surface and particles and simulates the stick-slip motion of particles. Despite its simplic- ity, since the particle's ' displacement is independent, this deformation model enables highly efficient estimation by allowing parallel estimation of stiffness values.
The structure defined by VSF consists of a computational mesh Ma with na particles and m. nh springs. Each particle responds to displacement with Hookean reactive force f - Ka.i (0) -p.), with Ka,; the stiffness of this spring. In this case, the material parameters are the stiffness of each spring O₂ = (Ka,l,.... ..,,,)).
Most consequentially, the VSF implementation leads to material-independee dynamics model Dyn in Eq. 3:
(5) where s² E 3 represents the particle positions and we consider particles at rest positions for 22. naterial-independeern is unique characteristic of the VSF model and the material parameters influence the data likelihood only through the observation model P(z*|s", Sa)
S Dyn(s's) da Sw).
Lincar-Gaussian observation model Next, we approximate the observation likelihood as lincar-Gaussian model, linearly dependent on the material parameters. Here the linearity is from Hookean spring in VSF We assume z' = W'O,, + e with e N(w Σ.) Gaussian noise term with known covariance. Without loss of generality, we will assume 4e' = to simplify the subse- quent notation. Here W is known observation matrix that nonlinear function of state s i.e. W W (s*). ). In other words,
P'z'ls',
((2;; W' Ez).
For the plant dataset, the linear model of joint torques observation 2' = r¹ can be evaluated as,
Σ -(piot Ko,s - +er, W = [-JJ (vf pf)
JT (phn - 1% )] (7)
Here W has dimension na and Jacobian matrix (p) gt) = J, is computed from forward kinematics robot configuration q' and e, is Gaussian noise joint torques. For the shoe dataset, used simplified 1-D force model to model the pressure difference 2 8':
Σ Ki lpl - -pill2 + t W = [[lr - rfll2
Ilp' -h [12]
(8)
Here 4 has dimension x and Ep is Gaussian noise in pressure difference. Now, the log- likelihood of sequence of observations defined Eq. (3) has a quadratic form in Ba:
To,
To. 1
- log f (Va.i Kars, 0.) log II
0n)
Col. const. (9)
Gaussian material parameters prior We assume the component material parameters prior Ib.., ( Va.k, v defines a Gaussian distribution over da, We use probabilistic neural net- work that takes the visual feature Vo.k input and outputs the mean Po,k and variance E,,k. where sb are the weights in the network.
Ib.. (Bo.ki Vocks 1)) N Marks Do.k).
(10)
13

---

With this definition, the mean and variance of each element's parameter are sums of Gaussians, which are themselves Gaussian. Let Ha = (Ma.,,..., Ha.j..)) and Ea o Do, &o follows Gaussian distribution with mean Pa and covariance Ea:
Φa (Φ Ma: Ea).
(11)
We denote linear map from 6a. to 6 as 0a Bada in Sec. 3.1, where B. IÍ E ca Because Oa is linear transformation of Gaussian random variable &a. the material parameters prior in Eq. (4) has closed-form evaluation in this case [50]:
[Cal
P(fa W,, Sa) = /
IlAa = Badal
II
gbn, (oa, Va ddd
Arm1
(12)
10a, Ba Ea) En)dda = N(aa: BaHa: B B.
Offine MLE SGD Evaluating structured Bayesian meta-learning MLE objective in Eq. (1) requires integration over material parameters 0g. From the deterministic dynamics model Dyn in Eq. (5), we can get state trajectory (6 ... To.)) from action sequence. The observation matrix can be evaluated as W W(s''.) using Eq. (7) or (8). The task dataset likelihood in Eq. (1) can be evaluated in closed-form:
No II f(aa.iTa,i,0a 0a) E=1
No To.t
P(Da(0a)
IIII ==
(Za W 0a, = N(za; Wa Ez)) (13)
No,Na) into a single vector and all observation Here we stack all observations in 2n = (2d, 11 a.N.
a.No into single matrix. The observation noise matrix Σn ) is block diagonal matrix with each observation noise matrix. The number of No observations Σn Ta, is on the order of 10 for each task and (2a. Wa) can be loaded GPU. The ooglliieldooo function in Eq. (1) the entire dataset can be evaluated as,
matrices W. (W1,
Σ log / P(Da\0.)PP(alll.. v))0.
9.
(14)
Σ log / N(zoi WaOn, Ez)N((a;; Bafa, B, B. Bi )dO.,
Using the fact that the marginalization over a Gaussian distribution still Gaussian, the offline MLE problem in Eq. (1) becomes
の = arg min
La - WaBatallli B. BJ B...aaallll.....B....... +T --2)-1
(15)
0
log det (W, W + Σz)
Here we minimize the negative log-likelilnn likelihood and omit constant terms. This objective function is implemented using differentiable operations in PyTorch [48] and the optimization is executed using stochastic gradient descent. We use the Adam [51] optimizer with learning rate 10 to optimize the meta-parameters.
Online MAP as QP Because Gaussian prior and tiate the online MAP problem in Eq. (2) as a QP:
observation model, we instan-
arg min 2| - W_O. Σ2" + 10 B. 12 (.E.B. 1, s.t. 0., 2 0 0.
(16)
Here 2, and W. are evaluated on the support set D,. The non-negative constraint from the non- negative spring stiffness 02,; Ko, O. Finally, the estimated material parameters 0, can be used predict tactile response for novel touch. In implementation, the quadratic program in Eq. (16) is solved online using CVXPY [52].
14

---

Avoiding meta-overfitting Visual-tactile datasets are relatively small, and meta-learning is prone to meta-overfittnng where the meta-parameters memorize all the training tasks, so it important to adopt measures to avoid overfitting. We use several strategies our work:
Early stopping when the validation error increases.
Adding dropout layers in the meta-prior network.
A good prior initialization was important to avoid over-confident priors. We initialize the meta- prior uncertainty using the estimated stiffness standard deviation from the training dataset.
Although regularization via meta-prior-prior has been proposed for Bayesian meta-learning [53], find that it is challenging define appropriate regularization targets for the prior weights that discourage overconfidence. Detailed training configurations are presented in the Appendix B.
B Experimental setup
B.1 Equipment and benchmark datasets
The objects are fixed to table within the robot's workspace and the robot touches the object multiple times. A static RGBD camera is used capture view of each object before each touch. time series of touch data is recorded during each touch.
The Plants dataset consists of 12 artificial plants with different sizes, appearances, and materials. A Kinect Azure is used to capture RGBD data. We collect the 7-d joint torque data from the Kinova arm as the touch data. For each plant, we rotate the object in 4 different views (Fig. 7) and sample 50-70 touches for each view. There are total of 3, 255 touching sequences and 807, 103 individual touch readings.
Figure 7: Four different views of the artificial dracaena in plant dataset.
The Shoes dataset consists of 23 shoes including boots, running shoes, sneakers, and slippers. An Intel Realsense L515 is used to capture RGB-D data. Pressure data is collected with Punyo tactile sensor [49]. 23 shoes have very different materials and visual appearances. We sample the touched point uniformly at random and select the touch direction along the surface normal. We touch each shoe 20-30 times. There are a total of 729 touching sequences and 35, 614 individual touch readings.
For each object in the dataset, we sample target points uniformly on the visible point cloud. We visualize sampled starting points and target points in Fig. 8. We use inverse kinematics move the robot end-effector in straight line from start to end.
B.2 SBML structure generation module
VSF partiele set generation We first segment out the object using a rough bounding box relative the fixed base. We define a volumetric grid on this bounding box and do raycasting from the camera point sample
of points in the occluded region. The grid cells have size ~lem for artificial plants and ~0.5em for shoes. We visualize this procedure in Fig. 9. We downsample the points sampled from the grid to 10, 000 30, 000 particles depending the size of the object.
15

---

Figure 8: Sampled touches on an orange tree and a white sneaker. Pink balls indicate the starting points and blue balls indicate the target points. Yellow arrows indicate the straight line from start to end.
Figure 9: Left: VSF particle sampling for an artificial plant. The red ball indicates the camera point. The green points are sampled VSF points. The bounding box is defined large enough enclose the object. Right: Project DINOv2 features on the image plane to 3D points, colors are from the first three dimensions in PCA of the features.
Visual features We use DINOv2 [46] to generate dense visual features Va our experiments. We use pre-trained weights available on their website. By default, we use the small-version model that creates 384-dimensional feature per 14x 14 image patch. For visible points in VSF, project their 3D coordinates to the image plane and use linear interpolation to compute the feature. For occluded points in VSF, we use the average features of its 10-closest visible points. Therefore particle has feature i. Also, we normalize the visual features using statistics from the training and validation dataset to speed up learning.
B.3 SBML meta-prior architecture
For heterogeneous components, we choose a 4-layer MLP with dimensions [384, 256, 128, 2]. The 2-D output is the Gaussian distribution's 's mean and variance, and the ReLU operation in the last layer ensures non-negative output. The meta-parameters wH e are the weights this neural network.
The meta-prior for segmented and homogeneous components uses a linear mean model and learn- able variance. The meta- meta- parameters dSSEG e'ho are linear weights and variance. We choose this low-capacity model because of the small number of segments and tasks in the meta-training dataset.
16

---

We initialize the model uncertainty by setting an initial bias for variance output in MLP and initial variance value in the linear model.
B.4 iMAML implementation
We use an MLP to map robot action to tactile reading in iMAML implementation. For the no-vision version, the network input is the 7-d joint angle, and we have 6 hidden layers with dimensions [32, 128, 64, 32, 16]. The with vision version takes an additional 384-dimensional DINOv2 classi- fication token to distinguish different objects. The 384-dimensional features are encoded using [64, 7] MLP and the output is concatenated with the 7-d joint angle as 14-d network input. The output layer is 7-d predict joint torques for the plant dataset and 1-d to predict pressure reading for the shoe dataset.
We used gradient descent for the inner loop iMAML adaptation. For both benchmarks, we use 2.0 for the regularization strength, 0.001 as the outer loop learning rate, and a total 400 outer epochs. For the conjugate algorithm, we use 100.0 for the conjugate gradient damping and perform 10 steps. The inner loop learning rate was set 0.001 for shoe and 0.01 for plant. A total of 30 inner loop gradient descent steps taken for both benchmarks.
C Supplementary experiment results
C.1 Additional baseline results
We provide results with alternative baseline implementations in Tab. 4 and 5. For the non-structured baseline learned with iMAML, consider no-vision baseline that does not visual informa- tion. We do not see significant performance difference compared to iMAML with vision in Tab. and Tab. 2. This indicates that the visual features cannot enhance the meta-learning of tactile re- sponses without structure in such a real-world dataset. For the Gaussian prior baseline, since the average mean tends to overestimate the stiffness, we also consider a second variant (zero mean) that simply sets the mean to zero. This zero Gaussian prior has a better zero-shot prediction on the Plant benchmark, but its 10-shot error even larger than the 0-shot error of SBML prior.
Table Joint torque prediction error (Nm)
Table 5:
Tactile prediction error (hPa) std over query set on Shoe benchmark.
std over query set on Plant benchmark.
iMAML no-vision Gaussian prior (zero)
iMAML no-vision Gaussian prior
(broad dataset)
(broad dataset)
(zero)
O-shot
3.95 + 0.85
3.55 + 0.00
O-shot
7.44 + 1.93
9.21 0.00
1-shot
5.71 + 3.52
3.43 + 0.03
1-shot
7.22 + 2.77
8.08 + 0.14
5-shot
5.10 + 2.16
3.09
0.04
5-shot
6.38 + 1.23
5.65 + 0.22
10-shot
4.40 + 0.93
2.88 + 0.03
10-shot
6.22 + 0.89
4.63 + 0.09
VSF all-shot 2.57
VSF all-shot 4.10
C.2
Uncertainty of SBML prior
100. Nm
06 Kim
Figure 10: SBML prior standard deviation in stiffness. The color map is in the log scale.
17

---

The SBML prior also learns the uncertainty of stiffness in VSF model. We visualize the standard deviation of learned Gaussian prior Fig. 10. For plants, we can see the stiff branch region is predicted with high uncertainty, likely due to high variation in branch stiffness caused by detailed geometric features like thickness and distance to the plant base, the other hand, shoes' stiffness uncertainty overall higher than plants', probably caused by the more significant material variation in the shoe dataset.
C.3 Unified prior over plants and shoes
To evaluate the capacity our method, we test whether it can learn a unified VSF prior between the plant and shoe categories. We learn single prior using both plant and shoe broad datasets with the same training configuration. We visualize the unified prior mean prediction in Fig. 12. We qualitatively find that the unified prior in the last row has prediction similar the prior trained on data from individual category datasets in the second row. Quantitative results are given in Table and Table 7. Errors are comparable to the Plant-only and Shoe-only datasets, demonstrating that SBML can learn multi-category visual-tactile models without significant change in performance.
0.5 N/m
100.0 N/m
Test object
Por cluss prior mean
Unified prior mean
Figure 11: Here compare the zero-shot prediction using single prior learned from both plant
and shoe dataset data (last row) with zero-shot prediction using data of their specific class (second row). Here per-class means testing plants using prior trained broad plant dataset and testing on shoes using prior trained on broad shoe dataset.
Table 6: Joint torque prediction error (Nm) + std over query Plant benchmark using per-class prior or unified prior.
Table 7:
Tactile prediction error (hPa) std query Shoe benchmark using per- class prior or unified prior.
Plant data only Shoe + Plant data (broad dataset)
Shoe data only (broad dataset)
Shoe + Plant data (broad dataset)
(broad dataset)
0-shot
2.80 0.05
2.95 0.07
O-shot
7.43 + 0.25
7.05 0.49
1-shot
2.76 = 0.05
2.92 + 0.07
1-shot
6.77 - 0.26
6.52 = 0,41
5-shot
2.67 - 0.05
2.83 0.06
5-shot
5.05 + 0.20
5.06 0.20
10-shot
2.61 + 0.04
2.77 + 0.05
10-shot
4.18 + 0.04
4.32 0.06
VSF all-shot 2.57
VSF all-shot 4.10
18

---

C.4 Per-point stiffness estimation
In this section, we provide better qualitative stiffness estimation using per per-point touches. Note this estimation does not use the VSP model and is only for qualitative comparison. We setup another robotic system using the URS arm's built-in force/torque sensor to measure contact forces when object touched with conical probe. Touched points are selected by voxel-downsampling the surface point cloud to cm, and the probe is depressed normal the surface and stopped when force threshold is reached. The per-point stiffness value is estimated by dividing the maximum contact force over the displacement. For better visualization, interpolated estimated stiffness on a triangle mesh of the object. The learned SBML prior qualitatively matches trends in the per-point estimation, with the toe region estimated to more stiff than the tongue region.
objoce
0.5 N/m
100.0 N/m
Prior mecan
0.5 N/m
1000.0 N/m
Per-poinnt stiffwsss
Interpolated stiffesss
Figure 12: Comparing VSF prior with per-point stiffness estimation, with point probe moving normally to the object surface. interpolated mesh also shown. Per-point stiffness sampled at different density compared to VSF, and the estimated stiffness is plotted on different scales.
C.5 Material composition experiments
The experiments of Sec. 4.5 defined in detail follows. We consider three testing condi- tions for the composition structure C. heterogeneous, heterogencous segment, and heterogencous homogeneous.
Heterogeneous: contains m. components and each component contains one spring in VSF.
Heterogeneous + homogeneous: contains m. components, where ma components contain spring with stiffness K. and the last component assigns uniform stiffness across all the springs KHo. Following Section III.C, the stiffness of each spring is thus K; KHO + K,
19

---

Heterogeneous + segment: contains Ma + components, where components contain one spring and the n, components contain the springs in one segment. We use 10 segments generated by GMM clustering on DINOv2 features.
C.6 Effects of visual features and prior structure
XYZ+RGB
DINOV2-S
DINOv2-S
DINOx2-B
Linear
XYZ+RGB
DINOv2-S
DINO-2-S
DINOv2-B
MLP
Linear
MLP
MLP
D-shot
3.84 + 0.11
4,30 + 0.15
2,80 + 0.05
2.75 0.98
I-shot
3,77 1: 0.12
4.19 + 0.15
2.76 +: 0.05
2,74 0.99
5-shot
3.51 + 0.04
3.81
+ 0.10
2.67 + 0.05
2,69 0.06
10-shot
3.27 = 0.09
3.50 +: 0.11
2.61 0.04
2.64 0.05
Table 8: Left: Effects of the different visual features and prior structures on the meta-learning dataset. We visualize the zero-shot prior mean of the orange tree in the test set. Right: Joint torques prediction error (Nm) on Plants dataset, for SBML with different visual features and prior.
We evaluate how the choice of visual feature and prior architecture affect meta-learning perfor- mance. To avoid meta-overfittnng our mitigation strategies (in Sec 3.3 the last paragraph) help, but do not eliminate the risk. Since dataset relatively small, the feature relevance and prior capacity must be chosen carefully.
We explore combinations of the following:
Visual features: we use normalized XYZ coordinates + RGB colors as naive visual features. We also compare DINOv2-small (DINOv2-S) which outputs 384-dimensiona. feature vector and DINOv2-base (DINOv2-B) which outputs a 768-dimensional feature.
Prior structures: consider using linear MLP prior structures. The linear prior assumes homoscedastic uncertainty independent of input.
Tab. shows learned prior with different visual features and the quantitative results on the broad plant dataset. The smaller capacity models, XYZ+RGB / MLP and DINOv2-S linear, underfit the data. The larger capacity models (DINOv2-S MLP and DINOv2-B MPL) are more expressive and capture meaningful patterns. We chose DINOv2-S / MLP overfitting in smaller training datasets.
default model to avoid
We also observe failure cases where DINOv2 features generalize poorly. One example is the slipper class as shown Fig. 9. Even though the learned prior seems to capture that the toe stiffer than the upper (i.e., the term for the part that fits over the foot), the upper's stiffiness is underestimated. Hence, the prediction error is not significantly better than naive Gaussian prior. We hypothesize that the diverse stiffness distribution of slippers might cause the failure. Here, the stiffness depends on the thickness of the upper part and other fine geometry details that are not captured in the image.
Training objects
Testing object
Prior mean
Gaussian prior (zero mean)
SBML prior
0-shot
10.12 + 0.00
9.30 0.35
1-shot
8.96 0.46
8.43 0.63
5-shot
6.49 + 0.26
6.42 + 0.29
10-shot
5.02 + 0.11
5.12 + 0.15
VSF all-shot 4.55
Table Slipper failure case. Left: the prior poorly generalizes from meta-training objects. Right: SBML improves tactile prediction errors (hPa) marginally compared to using an uninformed prior.
20

---

C.7 Effects of training dataset size
We also evaluate the SBML prior's performance given different sizes of the training dataset. We consider reducing the dataset size by randomly subsampling the objects or subsampling touches-per- object. Results are shown in Tab. 10 and Tab. 11. As expected, as the dataset shrinks, performance tends degrade and have larger variance. We also observe that for shoes it is important to have more objects and fewer touches per object, where using 40% touching sequences and all objects achieve good performance. Because the shoes has only about 20-30 touches per object, 10% touch subsampling is extremely sparse and results are noisy. Also, it appears that subsampling at 70% touches for shoes gives optimal performance, but the results are not statistically significant.
Table 10: SBMI. zero-shot joint torques prediction error (Nm) on Plants dataset with different train- ing data.
10% seq
40% seq
70% seq
100% seq
10% ohj
4.97 + 2.17
+ 2.09
3.94
1.13
3.41
0.49
40% obj
4.07 + 1.48
3.04 + 0.19
2.85 + 0.21
3.27 + 0.66
70% obj
3.66 + 1.24
3.24 + 0.27
2.82 + 0.17
2.83 + 0.14
100% obj
3.18 1 0.36
3.33 1 0.29
2.82 1 0.07
2.80 0.05
Table 11: SBML zero-shot tactile prediction error (hPa) Shoes dataset with different training data.
10% seq
40% seq
70% seq
100% seq
obj
8.84 + 1.88
8.49
1.90
8.10
0.99
8.38 1.10
40% obj
9.68 + 2.07
8.87
+ 2.26
8.82 + 1.41
8.76
+ 0.92
70% obj
12.17 + 2.81
9.07 2.26
8.17 + 2.42
8.44
+ 1.45
100% obj
9.70 1.57
7.50 0.56
7.03 + 0.63
7.43 + 0.26
C.8 Evaluation with different viewpoints
Finally, we evaluate whether the learned prior is robust to different viewpoints. We found that DINOv2 visual features are robust to viewpoint variation and SBML predicts stiffness with semantic correspondences. We rotate a shoe left and right by 45 degrees and use the SBML prior trained on the broad shoe dataset to make predictions. As shown in Fig. 12, the learned prior predicts the semantic sole part with high stiffness between different viewpoints.
21

---

Front view
Rotate left
Rotate right
Prior mean
Prior mean
Prior mean
Table 12: SBML prior mean prediction with different viewpoints.
22


Reviews:
[{"review_id": "6KTqcfC9Z3", "reviewer": "Reviewer_57dj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper addresses the challenge of estimating visual-tactile models for deformable objects, where vision is hindered by occlusion and touch data is sparse and noisy. The proposed approach leverages Bayesian Meta-Learning (BML) to create a data-efficient method for model estimation that can generalize from diverse training objects. The paper introduces Structured Bayesian Meta-Learning (SBML) to deal with the issue of different parameter spaces for various visual-tactile models. SBML incorporates heterogeneous physics models to learn from objects with varying appearances and geometries, enabling both zero-shot vision-only prediction and few-shot adaptation with tactile data. The experiments validate the efficacy of SBML on heterogeneous objects, namely plants and shoes, demonstrating superior performance in force and torque prediction accuracy in both zero- and few-shot settings.", "summary_of_recommendation": "Overall, \"Structured Bayesian Meta-Learning for Data-Efficient Visual-Tactile Model Estimation\" is a high-quality, original, and significant contribution to the field. The proposed SBML approach addresses key challenges in visual-tactile model estimation, offering a data-efficient solution that generalizes from diverse training objects. While the method shows strong potential, future work should aim to extend its scope to other models and improve its capability to handle multi-modal distributions and generalize across different object categories.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "1. Can the authors provide additional experiments or validation results using other deformable object models, such as Finite Element Method (FEM) or Graph Neural Dynamics Models, to demonstrate the broader applicability of SBML?\n\n2. The paper mentions that tactile data is often sparse and noisy. Can the authors elaborate on the specific techniques used to mitigate these issues in the SBML framework?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "02Ud6upY1p", "reviewer": "Reviewer_oWYD", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors present a method Structured Bayesian Meta-Learning to use visual-tactile data to estimate stiffness of deformable objects. The method leverages Bayesian meta-learning with an incorporated structure generation module, allowing it to adapt to objects with different parameter spaces. The approach does a zero-shot vision prediction first and then few-shot adaptation using tactile signals.", "summary_of_recommendation": "My main concern is the generalization of the method.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "As mentioned in above review, I'm interested in more results cross object categories, even if they are not deformable. If you use some vision-based tactile sensors, it is also possible to retrieve surface stiffness from the readings. Have you tried any large multimodal datasets such as Objectfolder benchmark [1], or Touch and Go [2]? I think Objectfolder benchmark contains visual and tactile data and material properties that might be able to infer stiffness. Touch and Go has data collected in the wild from different surfaces.\n\n[1] Gao, Ruohan, et al. \"The objectfolder benchmark: Multisensory learning with neural and real objects.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n[2] Yang, Fengyu, et al. \"Touch and go: Learning from human-collected vision and touch.\" arXiv preprint arXiv:2211.12498 (2022).", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dteHPpwyKi", "reviewer": "Reviewer_HDBy", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a method for visuo-tactile mapping of objects. It employs a hierarchical Bayesian approach, where the mapping problem is factorized into learning prior knowledge about object classes and then using this object-class specific prior knowledge in the visuo-tactile mapping of individual object instances. The paper is motivated by the problem to predict the tactile behavior (stiffness) over full object surfaces, but from the sparse data collected using only few tactile interactions with the same object. The hierarchical Bayesian approach tackles this problem as it learns useful prior information that allows the framework to predict the stiffness of object patches either using only vision (zero-shot), or by extrapolating tactile information from other interaction points on the instance.", "summary_of_recommendation": "Relevant problem tackled by an approach that fits (hierarchical Bayesian approach). However the authors should provide some experiments from other perspectives to make sure the method doesn't overfit and should also ideally compare against real ground truth (and not against outputs of their own algorithm)..", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "- L214: why are these different? Please motivate.\n\n- As far as I can see the objects are always viewed from the same perspective -- both, shoes and plants. If this is true the results we see could strongly overfit to the perspective. It is necessary to provide experimental data that shows the method does not just learn \"further down in the image means more stiff\". The objects could be presented in random rotations or at least be approached from different directions while in their \"natural orientation\" (sole/stem downwards).\n\n- L239: It is not a good idea to use the output of your own method as \"proxy for ground truth\" -- especially because the results in Figure 5 do look rather erroneous. I don't believe the middle part of the brown boot is as soft as the middle part in the other two shoes. The brown boot should be generally way more stiff, I suppose. Shouldn't it be possible to collect ground truth by somewhat exhaustively mapping the objects square centimeter by square centimeter? Please provide a sensible ground truth, or at least explicitly acknowledge that you don't quantitatively assess the quality of the method. Sensible ground truth would be the way stronger option here though.\n\n- L282: Table 1 itself doesn't state it would be zero shot performance, but the text in this line implies so. Something is off here. Table says many-shot.\n\n- L146, L149: this enumeration is unexpected. what are you enumerating here?\n\n- L156-L178: This is confusing and also comes surprisingly as we did not talk about components up until this point. Please make sure that there is enough meta description before this point so that we understand how this fits into the big picture. This is also missing from the overview schematics for the method.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers found the paper to be well written with a clear and signficiant contribution. There were some concerns regarding generalization and evaluations. The rebuttal helped to strengthen the paper and increase its potential impact. The paper seems well suited for CoRL.
Decision: Accept