Structured Bayesian Meta-Learning for Data-Efficient Visual-Tactile Model Estimation

Authors: shaoxiong  yao
yifan  zhu
kris  hauser
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-06 06:48:10

Abstract: Estimating visual-tactile models of deformable objects is challenging because vision suffers from occlusion, while touch data is sparse and noisy.  We propose a novel data-efficient method for dense heterogeneous model estimation by leveraging experience from diverse training objects.  The method is based on Bayesian Meta-Learning (BML), which can mitigate overfitting high-capacity visual-tactile models by meta-learning an informed prior and naturally achieves few-shot online estimation via posterior estimation.  However, BML requires a shared parametric model across tasks but visual-tactile models for diverse objects have different parameter spaces.  To address this issue, we introduce Structured Bayesian Meta-Learning (SBML) that incorporates heterogeneous physics models, enabling learning from training objects with varying appearances and geometries.  SBML performs zero-shot vision-only prediction of deformable model parameters and few-shot adaptation after a handful of touches.  Experiments show that in two classes of heterogeneous objects, namely plants and shoes, SBML outperforms existing approaches in force and torque prediction accuracy in zero- and few-shot settings.

Main paper: 
 introduction: 
 Fusing tactile information with vision is necessary for many manipulation tasks, including reconstructing occluded objects [1] , reaching in clutter [2] , and assistive dressing [3] . A robot is expected to quickly acquire a visual-tactile model for efficient manipulation. Visual-tactile models of deformable objects such as finite-element models can make accurate predictions when an object's material parameters are properly identified [4, 5, 6] . However, this estimation problem is challenging and usually requires a lot of data. Vision data has significant occlusions, leading to ambiguity in the inference of physical response. Tactile data is noisy and relatively sparse across an object. These characteristics make the identification problems computationally expensive, susceptible to local minima, and ill-posed unless strong assumptions (e.g. homogeneity) are made [7] . Figure 1 : Our SBML approach enables few-shot learning of a novel plant's force response using experience interacting with other plants. Left: touched region (top) and tactile feedback (second row, norm of joint torques over time). Right: compare SBML against a stiffness estimator using naïve uniform Gaussian prior with the same touches. SBML prior already predicts stiffer trunks and branches compared to leaves and its predictions improve as it touches the plant (right 4 columns).We present a few-shot learning method to estimate dense, heterogeneous visual-tactile models using vision data and a small number of touches. Our approach is based on Bayesian meta-learning (BML) [8, 9] , which first learns a prior distribution over models, estimated offline from the experience of related objects, and then adapts the model online via posterior inference. BML is appealing because it can regularize high-capacity (overparameterized) models, which can help estimate heterogeneous visual-tactile models in the face of data sparsity. BML also learns informed priors that improve few-shot accuracy compared to naïve regularization methods. However, applying standard BML requires a unified model parameterization and cannot leverage the underlying physics of different deformable objects. Visual-tactile models for different objects usually have different numbers of state variables and material parameters. For example, the mesh of a large object has more elements than a small object and therefore more parameters.To enable the application of BML to our setting, we introduce the novel Structured Bayesian Meta-Learning (SBML) approach as illustrated in Fig. 2 . Each object generates a structure of variable dimension and we learn a unified probabilistic model that predicts material parameters for each element in the structure. The meta-prior is factorized so that it is learned as a parametric function shared across objects, even when the object is segmented into different numbers of parts. We show that SBML can be trained on diverse object sets with to enable rapid and accurate few-shot estimation on novel heterogeneous objects. In summary, this paper makes the following contributions:• We propose the SBML framework that enables meta-learning on diverse tasks with different numbers of model parameters. • We show that SBML enables few-shot learning of dense heterogeneous tactile models on real touch data using vision priors and spatial correlation priors. • We evaluate the proposed approach on real-world plants and shoes datasets. SBML outperforms an unstructured state-of-the-art meta-learning approach and the vanilla VSF (Fig. 1 ). 
related works: 
 Visual-tactile model estimation Fusing visual and tactile information can improve the reliability of robot manipulation [3, 10, 11] , but unifying these modalities is challenging due to visual occlusion and tactile sparsity. The parameters of analytical simulations, such as spring-mass and finite element models [5, 12, 6] , can be fit to observational data, but these estimation problems are computationally challenging and susceptible to local minima. Learning-based methods have also been applied, but typically assume homogeneity of the object [13, 14, 15] or restricted contact regions during tool usage [16, 17] . These assumptions reduce the representation power (i.e., capacity) of the visual-tactile models. More recently very high-capacity learning-based models (e.g. transformers) have been used to learn latent visual-tactile representation [18, 19] , but these models are very datahungry and are limited to simulation-only or simulation-based pretraining. Our work enables the use of high-capacity heterogeneous visual-tactile models from real-world data by learning an informed prior over model parameters, leading to data-efficient model estimation. We learn prior conditioned on visual information, which has been shown informative in predicting an object's material properties [20, 21, 22] for fabric [23, 24, 25] and rigid surfaces [26, 27, 28] . Our approach goes beyond current methods by predicting heterogeneous material and updating estimation online.The goal of meta-learning is to simultaneously learn a model that matches a given dataset and a process to adapt the model quickly to novel datasets [29] .Meta-learning has enabled data-efficient robot learning in locomotion [30] , manipulation [31, 32] and underactuated control [33, 34] . Past methods often use gradient descent [35, 36] or a learned adaptation module for online update [37, 38, 39] . Bayesian meta-learning uses hierarchical Bayes to create a unified learning objective for all tasks, enabling optimization with a single optimizer and can quantify uncertainty systematically [40, 8, 41, 42, 43] . Whereas existing Bayesian meta-learning methods assume a shared parametric function across all the tasks, our proposed Structured Bayesian Meta-Learning leverages task-specific structures to allow meta-learning over diverse tasks, such as our visual-tactile modeling setting. Meta-learning on graph-structured neural networks [44, 45] is related to our approach but only addresses node classification problems. For each task, a structure generation module uses perception data to create a set of structural assumptions mapping from a common meta-parameter space to visual-tactile models of different objects. Online, maximum aposteriori estimation is performed to infer material properties over the novel task's structure. 
method: 
 A visual-tactile model of a deformable object captures how the robot's visual-tactile observations change during an interaction. At time step t, the robot executes action a t (e.g., joint position commands) and receives tactile and visual observations z t (e.g. force/torque readings and RGBD images). In our formulation, we consider visual-tactile models that map a sequence of action a t to a sequence of observations z t and we denote input-output pair x = (a 1 , . . . , a T ) and y = (z 1 , . . . , z T ) as a touch. Visual-tactile model estimation aims to identify this mapping and our SBML approach enables data-efficient estimation with few touches. In this section, we will first present background on BML, our extension to SBML, our instantiation of the framework for visual-tactile modeling, and strategies for making offline and online learning more efficient and accurate.Standard meta-learning considers a meta-training dataset D = {D α | α = 1, . . . , L} where α denotes a task and D α = {(x α,i , y α,i ) | i = 1, . . . , N α } is a task-specific dataset. Each output y follows a distribution P (y|x, α) conditioned on both input x and task α, and each y α,i is conditionally independent given x α,i and α. The goal of meta-learning is to predict P (y|x, α * ) for a novel task α * given data from a support set D * = {(x * ,i , y * ,i ) | i = 1, . . . , N * }, particularly from the zero-shot (N * = 0) to the few-shot (small N * ) cases. Note that α * is unknown and we need to infer it from the support set D * .In Bayesian meta-learning the task-conditioned output distribution is approximated as P (y|x, α) ≈ f (y; x, θ), where θ ∈ R M are the model parameters corresponding to the task. A prior P (θ) is learned offline for online estimation of the model posterior P (θ|D * ) from the support set. BML considers model parameters of different tasks statistically independent conditioned on meta-parameters ψ, i.e. θ ∼ P (θ|ψ). Note that f (•; •, θ) must be a sufficiently rich class of functions so that every task maps to a corresponding θ, and P (θ|ψ) should capture the meta-training data without overfitting. More precisely, offline learning can be formulated as a maximum likelihood estimation of meta-parameters ψ over the meta-training dataset D [8] .EQUATIONHere the data likelihood is evaluated as P (D α |θ α ) = Nα i=1 f (y α,i ; x α,i , θ α ). In the online phase, we find the maximum a posteriori (MAP) estimate of model parameters θ * using the Bayes rule:EQUATIONand then the online prediction is given by P (y|x, D * ) ≈ f (y; x, θ * ).In the visual-tactile model estimation setting, the standard assumption in BML regarding fixeddimensional inputs x, outputs y, and model parameters θ does not hold. Different touches have different sequence lengths T α,i and diverse objects have varying dimensional model parameters θ α ∈ R Mα . In particular, we represent a heterogeneous deformable object as a computational mesh with various particles and elements, leading to varying numbers of material parameters to estimate.Our proposed Structured Bayesian Meta-Learning (SBML) framework addresses this problem using task-specific Bayesian network structures mapping a fixed-dimensional meta-parameter to a prior that supports diverse tasks. The structures for each task α are generated by a structure generation module outside the learning pipeline as in Fig. 2 . We use S α to denote the task-dependent structure. Conditioned on S α , the probability distributions in Eq. ( 1) and ( 2) are factorized into parametric functions of ψ shared across tasks. We consider two types of structures specifically relevant to visual-tactile models. First, a hidden Markov structure can handle the dynamics and variation in sequence length T α,i governing the dimensionality of x α,i = (a 1 α,i , . . . , a Tα,i α,i ) and y α,i = (z 1 α,i , . . . , z Tα,i α,i ). In this model, the object state s t has a Markovian transition model defined by a dynamics model Dyn and the observations z t depend on the current state s t via an observation model P (z t |s t , θ α , S α ). The data likelihood f (y α,i ; x α,i , θ α , S α ) is therefore factorized as follows:EQUATIONHere S α provides the initial state s 0 α,i . For simplicity, we consider a dynamic model Dyn that is deterministic given material parameters θ α . From this formulation we can replace f in Eq. ( 1) and (2) with its structured counterpart.Second, SBML addresses the structure-dependent set of model parameters θ α and the prior in Eq. ( 1) and (2) changes from P (θ α |ψ) to P (θ α |ψ, S α ). We formulate a decoupled latent variable structure that maps a set of latent parameters ϕ α to a task-dependent prior over model parameters θ α , via a linear map θ α = B α ϕ α for simplicity. The structure S α defines a set of features v α informative to ϕ α , which are visual features v α,j mapped to each element j for a visual-tactile model. The latent parameters ϕ α has a factorized distribution conditioned on v α , i.e. ϕ α |v α ∼ |ϕα| k=1 P (ϕ α,k |v α,k , ψ). Ultimately, SBML learns parametric functions P (ϕ α,k |v α,k , ψ) = g(ϕ α,k ; v α,k , ψ) shared between tasks, e.g. a neural network that outputs a probability density. The latent structure depends on the instantiation of this framework, and our implementation is described below.In a visual-tactile model, we consider structure S α that captures an object's geometry in a computational mesh M α , with initial state s 0 α , its appearance in visual features v α and material correlation in composition structure C α as in Fig. 3 . We will define the visual-tactile simulation model and material parameters θ α to evaluate data likelihood in Eq. (3). The material parameter prior maps the universal latent parameters prior g(ϕ α,k ; v α,k , ψ) to P (θ α |ψ, S α ) in a manner that defines a relationship between material parameters and appearance, and correlations between elements.Object representation and simulation model. We consider particle-based visual-tactile models that discretize the object α into n α particles. The state of the object is the positions of the particles s t ∈ R 3nα = (p t 1 , ..., p t nα ), where p t k ∈ R 3 is particle k's position at time step t. The interaction between particles is defined by a computational mesh M α containing m α elements, and each element specifies a subset of particles with mutual interactions. The jth element, j = 1, . . . , m α , has a fixed d-dimensional material parameter θ α,j ∈ R d that enables us to simulate particle interactions and observations. The dynamics model Dyn updates the particle deformation s t using interactions in each element as in Fig. 3 . The visual-tactile model parameters of the object are the tuple of material parameters of each element θ α = (θ α,1 , ..., θ α,mα ). Visual features and composition structure. We use visual features and composition structures to define the map from latent parameters prior to primary material prior. This allows our model to capture the relationship between appearance and material, and the correlation between material parameters.First, v α defines visual features mapped to each element in M α , where element j has feature v α,j .Choosing expressive visual features will make the prior more informative to predict material parameters, we use dense features from pre-trained vision models DINOv2 [46] by default.Second, the material composition structure C α lets us accommodate correlation assumptions. The composition structure assumes the object is composed of components with similar materials, e.g. branches on a tree share similar high stiffness compared to soft leaf regions. We define a set of components and each component has material parameters ϕ mapped to a subset of elements c ⊂ M α .Here component material parameters ϕ are the decoupled latent variables. We consider three types of components denoted by component type b ∈ {HE, HO, SEG}, where HEterogeneous component has a single element, HOmogeneous component has all elements, and SEGmented component has a cluster of elements. Also, the visual features for a single component aggregate the visual features v α,j of all elements j ∈ c and we use the average feature 1 |c| j∈c v α,j for simplicity. Formally, C α = {(c α,k , b α,k , ϕ α,k , v α,k )} k=1,...,(ϕ α,k |v α,k , ψ) = g b α,k (ϕ α,k ; v α,k , ψ).We provide different types of components with independent priors and with an independent subset of meta-parameters:g b α,k (ϕ α,k ; v α,k , ψ) ≡ g b α,k (ϕ α,k ; v α,k , ψ b α,k ).The material parameters prior can be evaluated by marginalizing over component material parameters ϕ α ,EQUATIONOverall, Eqs. ( 3) and ( 4) enable SBML to define a universal meta-learning problem over ψ and task-dependent posterior estimation over θ α regardless of the tasks' structure.To achieve efficient implementation, we instantiate the SBML framework on the Volumetric Stiffness Field(VSF) [47] visual-tactile model. VSF is a dense particle-based representation with independent Hookean springs and can flexibly represent heterogeneous material.• In Eq. ( 3), the dynamics model Dyn is a particle-based deformation simulator the same as [47] and the observation model P (z t |s t , θ α , S α ) is approximated as a Linear-Gaussian function. Linearity is from the Hookean springs in VSF and the additivity of forces/torques over springs. The Gaussian assumption models the tactile sensor noise as a normal distribution and is a reasonable approximation for the sensors we evaluated. • The prior P (θ α |ψ, S α ) defines a Gaussian distribution over the material parameter θ α . In particular, g b α,k is a neural network that outputs the mean and variance of ϕ α,k . The integration in Eq. ( 4) has a closed-form solution with a Gaussian integrand. We evaluate likelihood in Eq. ( 1) using PyTorch [48] and optimize meta-parameters ψ via gradient descent. The online parameter estimation in Eq. ( 2) is a quadratic program (QP) with nonnegativity constraints on the spring stiffnesses. Further details are provided in Appendix A. 
experiments and results: 
 We acquire two benchmark datasets using the experimental platform in Fig. 4 : 1) Plants dataset has 12 artificial plants each touched 50-70 times and 7-d joint torque from the Kinova arm; 2) Shoes dataset has 23 shoes each touched 20-30 times and pressure data from a Punyo tactile sensor [49] . We use joint torques because the robot arm contacts the plants along its entire arm, while contact with shoes is localized to the end-effector. We show that SBML enables data-efficient visual-tactile model estimation by comparing SBML's zero-and few-shot predictions with baselines on real-world datasets. We also evaluate the effects of different meta-training datasets and material composition structures on SBML performance. Further experiments on the choice of visual features, prior capacity, and dataset size are presented in Appendix C.Structure generation module The structure generation module samples VSF particles from the object's surface and interior using ray casting. Visual features v α are generated by DINOv2 [46] and projected in 3D space. Our standard setting assumes a heterogeneous material composition C α . Other material composition structures are tested in Sec. 4.5. Meta-prior The meta-prior for heterogeneous components uses a multi-layer perceptron (MLP) that outputs the mean and standard deviation of a Gaussian distribution. The meta-priors for the segment and homogeneous components in Sec. 4.5 use a linear mean model with learnable variance, chosen due to the small number of segments and tasks in the meta-training dataset. Observation models The observation z t is joint torques τ t ∈ R 7 in plants and pressure difference from no contact δ t ∈ R in shoes. Note the observation models are linear-Gaussian in the spring stiffness. Joint torques are the summation of Jacobian transpose times contact forces. For the Punyo sensor, we use a simplified 1-D observation model that sums the force magnitude of each point.Non-structured meta-learning We compare a standard meta-learning method that directly maps robot action to tactile observation using no object-specific structure. The model is instantiated as MLP and uses DINOv2 classification token [46] to encode visual information. The model is metatrained and adapted using iMAML [36] , where the inner loop adaptation uses gradient descent. Structured model with naïve prior We compare to the "vanilla" VSF model with a naïve Gaussian prior. Vision is not used, and the naïve prior assigns the same mean and variance to all particles. The mean and variance are estimated from estimated VSF stiffness in the meta-training dataset. We visualize the zero-shot stiffness prediction of SBML for test objects in Fig. 5 organized in three rows: the top row shows the object, the middle row shows the SBML prior mean and the bottom row shows the estimated stiffness using vanilla VSF (average prior) given all touches in the dataset. We observe that the zero-shot stiffness estimates are qualitatively quite close to the all-touch estimates, with stiffer estimates at the branches and trunk of plants and the toes of shoes. Note that VSF all-shot is an estimation with all available support data and we cannot get the precise material parameters. The uncertainty of SBML prior is visualized in Fig. 10 .Test object Narrow Broad Mismatched Hetero.Hetero. + Seg.Hetero. + Homoge. We show quantitative results on the test objects in Tab. 1 and Tab. 2. The prediction accuracy is evaluated on a query set disjoint from the support set. We also report the standard deviation of prediction error over 10 random seeds. The Gaussian prior has zero standard deviation for 0-shot since its initial mean prediction is deterministic. The last row shows the VSF prediction error using all support set touches covering the entire object.SBML not only demonstrates lower zero-shot prediction errors in the first row of Tab. 1 and Tab. 2, underscoring the value of incorporating visual information but also quickly improves with additional support data, achieving near-all-shot accuracy by 10 shots. SBML's absolute improvement is smaller due to an almost optimal zero-shot error in Tab. 1. In contrast, iMAML exhibits worse zero-shot performance and noisy adaptation. Using object-specific structures, the Gaussian prior improves the estimation consistently but has worse few-shot accuracy due to naive uniform mean. Comparing meta-training datasets, we observe that SBML performs best when the training and testing distributions closely align. While the performance gap across meta-training distributions is minor on plants, it becomes significantly larger on shoes, likely due to the greater variation in shoes' appearance and material. Shoes are made of brown leather in boots, gray nylon in sneakers, and various colored plastics in slippers, while plants typically have brown branches and green leaves, as shown in Fig. 4 .To evaluate the capacity of our method, we trained a unified prior using data from both plant and shoe datasets. The results in Appendix C.3 demonstrate that our method has sufficient capacity to learn a prior shared between the plant and shoe categories.Hetero.Hetero. + Seg.Hetero. + Homoge. 0-shot 9.76 ± 0.75 9.23 ± 0.86 9.13 ± 0.77 1-shot 8.58 ± 0.68 7.57 ± 0.57 8.10 ± 1.38 5-shot 5.82 ± 0.31 5.00 ± 0.26 5.32 ± 0.17 Finally, we evaluate the effects of changing the material composition structure and we tested heterogeneous, heterogeneous + segment, and heterogeneous + homogeneous. Qualitative results are shown in the right three columns of Fig. 6 . We can see that the stiffness across a segment changes together for heterogeneous + segment, and the stiffness of the entire object changes for heterogeneous + homogeneous. We also found the composition structure speeds up model adaptation quantitatively in Tab. 3. When the test object is out of meta-training distribution, the learned stiffness prior will predict inaccurate stiffness values, and the shared components will help learn across regions of coherent material. We test these composition structures on narrow and broad datasets, but the improvement is marginal. 
conclusion and limitations: 
 We present a novel structured Bayesian meta-learning (SBML) approach to address the few-shot learning of visual-tactile models. Our innovations allow the method to transfer offline knowledge from objects of different sizes and shapes to improve prediction accuracy on novel objects. The method is highly flexible and can accommodate different material compositions, feature representations, and model capacities. Applied to benchmark datasets of plants and shoes, our experiments demonstrate that it outperforms non-informed estimators as well as non-structured meta-learning approaches in zero-and few-shot prediction accuracy. Limitations: The SBML framework is evaluated on the VSF model that can only predict tactile response, and we are interested in extending it to other deformable object models such as FEM and Graph Neural Dynamics Models that can predict visual deformation. The Gaussian prior assumes a single mode of material parameters conditioned on visual features and cannot handle more complicated multi-modal distributions. Our work has only demonstrated generalization within object categories, but this approach has not yet enabled generalization between categories. 
 
 Appendix: Here we provide more details on the efficient SBML implementation on VSF with Gaussian likelihoods, outlined in Sec. 3.3.Volumetric stiffness field A VSF defines a volume of independent particles that resist displacements from their rest position with Hookean springs. (Note that this differs from standard springmass systems where springs connect particles). The dynamics model Dyn is a geometric simulator that simulates the particle displacements independently. This simulator detects collisions between the robot surface and particles and simulates the stick-slip motion of particles. Despite its simplicity, since the particle's displacement is independent, this deformation model enables highly efficient estimation by allowing parallel estimation of stiffness values.The structure defined by a VSF consists of a computational mesh M α with n α particles and m α = n α springs. Each particle responds to displacement with a Hookean reactive force f t i = -K α,i • (p t i -p 0 i ), with K α,i the stiffness of this spring. In this case, the material parameters are the stiffness of each spring θ α = (K α,1 , ..., K α,n ).Most consequentially, the VSF implementation leads to a material-independent dynamics model Dyn in Eq. 3:EQUATIONwhere s t α,i ∈ R 3nα represents the particle positions and we consider particles at rest positions for s 0 α,i . Material-independence is a unique characteristic of the VSF model and the material parameters influence the data likelihood only through the observation model P (z t |s t , θ α , S α ).Linear-Gaussian observation model Next, we approximate the observation likelihood as a linear-Gaussian model, linearly dependent on the material parameters. Here the linearity is from Hookean spring in VSF. We assume z t = W t θ α + ϵ t z with ϵ t z ∼ N (µ t , Σ z ) a Gaussian noise term with known covariance. Without loss of generality, we will assume µ t = 0 to simplify the subsequent notation. Here W t is a known observation matrix that is a nonlinear function of state s t , i.e. W t = W (s t ). In other words,EQUATIONFor the plant dataset, the linear model of joint torques observation z t = τ t can be evaluated as,EQUATION)Here W t has dimension 7 × n α and Jacobian matrix J(p t i , q t ) = J i is computed from forward kinematics at robot configuration q t and ϵ τ is Gaussian noise in joint torques. For the shoe dataset, we used a simplified 1-D force model to model the pressure difference z t = δ t :EQUATION)Here W t has dimension 1 × n α and ϵ p is Gaussian noise in pressure difference. Now, the loglikelihood of a sequence of observations defined in Eq. ( 3) has a quadratic form in θ α :EQUATION)Gaussian material parameters prior We assume the component material parameters prior g b α,k (ϕ α,k ; v α,k , ψ) defines a Gaussian distribution over ϕ α,k . We use a probabilistic neural network that takes the visual feature v α,k as input and outputs the mean µ α,k and variance Σ α,k , where ψ are the weights in the network.EQUATION)With this definition, the mean and variance of each element's parameter are sums of Gaussians, which are themselves Gaussian. Let µ α = (µ α,1 , ..., µ α,|Cα| ) and Σ α = diag(Σ α,1 , ..., Σ α,|Cα| ), ϕ α follows a Gaussian distribution with mean µ α and covariance Σ α :EQUATIONWe denote linear map from ϕ α to θ α asθ α = B α ϕ α in Sec. 3.1, where B α [j, k] = I[j ∈ c α,k ].Because θ α is a linear transformation of Gaussian random variable ϕ α , the material parameters prior in Eq. ( 4) has a closed-form evaluation in this case [50] :EQUATIONOffline MLE via SGD Evaluating structured Bayesian meta-learning MLE objective in Eq. ( 1) requires integration over material parameters θ α . From the deterministic dynamics model Dyn in Eq. ( 5), we can get state trajectory (s 0 α,i , ..., s Tα,i α,i ) from action sequence. The observation matrix can be evaluated as W t α,i = W (s t α,i ) using Eq. ( 7) or (8) . The task dataset likelihood in Eq. ( 1) can be evaluated in closed-form:EQUATION)Here we stack all observations in z α = (z 1 α,1 , ..., z T α,Nα α,Nα ) into a single vector and all observation matricesW α = (W 1 α,1 , ..., W T α,Nαα,Nα ) into a single matrix. The observation noise matrix Σ Z = diag(Σ z , . . . , Σ z ) is block diagonal matrix with each observation noise matrix. The number of observations Nα i=1 T α,i is on the order of 10 4 for each task and (z α , W α ) can be loaded on GPU. The log-likelihood function in Eq. ( 1) on the entire dataset can be evaluated as,EQUATION)Using the fact that the marginalization over a Gaussian distribution is still Gaussian, the offline MLE problem in Eq. ( 1) becomesEQUATION)Here we minimize the negative log-likelihood and omit constant terms. This objective function is implemented using differentiable operations in PyTorch [48] and the optimization is executed using stochastic gradient descent. We use the Adam [51] optimizer with learning rate 10 -4 to optimize the meta-parameters.Online MAP as QP Because of Gaussian prior and linear-Gaussian observation model, we instantiate the online MAP problem in Eq. ( 2) as a QP:EQUATIONHere z * and W * are evaluated on the support set D * . The non-negative constraint is from the nonnegative spring stiffness θ α,i = K α,i ≥ 0. Finally, the estimated material parameters θ * can be used to predict tactile response for a novel touch. In our implementation, the quadratic program in Eq. ( 16) is solved online using CVXPY [52] .Avoiding meta-overfitting Visual-tactile datasets are relatively small, and meta-learning is prone to meta-overfitting where the meta-parameters memorize all the training tasks, so it is important to adopt measures to avoid overfitting. We use several strategies in our work:• Early stopping when the validation error increases.• Adding dropout layers in the meta-prior network.• A good prior initialization was important to avoid over-confident priors. We initialize the metaprior uncertainty using the estimated stiffness standard deviation from the training dataset.Although regularization via a meta-prior-prior has been proposed for Bayesian meta-learning [53] , we find that it is challenging to define appropriate regularization targets for the prior weights that discourage overconfidence. Detailed training configurations are presented in the Appendix B.The objects are fixed to a table within the robot's workspace and the robot touches the object multiple times. A static RGBD camera is used to capture a view of each object before each touch. A time series of touch data is recorded during each touch.The Plants dataset consists of 12 artificial plants with different sizes, appearances, and materials. A Kinect Azure is used to capture RGBD data. We collect the 7-d joint torque data from the Kinova arm as the touch data. For each plant, we rotate the object in 4 different views (Fig. 7 ) and sample 50-70 touches for each view. There are a total of 3, 255 touching sequences and 807, 103 individual touch readings. For each object in the dataset, we sample target points uniformly on the visible point cloud. We visualize sampled starting points and target points in Fig. 8 . We use inverse kinematics to move the robot end-effector in a straight line from start to end.VSF particle set generation We first segment out the object using a rough bounding box relative to the fixed base. We define a volumetric grid on this bounding box and do raycasting from the camera point to sample a set of points in the occluded region. The grid cells have size ∼1cm for artificial plants and ∼0.5cm for shoes. We visualize this procedure in Fig. 9 . We downsample the points sampled from the grid to 10, 000 ∼ 30, 000 particles depending on the size of the object. Visual features We use DINOv2 [46] to generate dense visual features v α in our experiments. We use pre-trained weights available on their website. By default, we use the small-version model that creates a 384-dimensional feature per 14x14 image patch. For visible points in VSF, we project their 3D coordinates to the image plane and use linear interpolation to compute the feature. For occluded points in VSF, we use the average features of its 10-closest visible points. Therefore particle i has a feature v α,i . Also, we normalize the visual features using statistics from the training and validation dataset to speed up learning.For heterogeneous components, we choose a 4-layer MLP with dimensions [384, 256, 128, 2] . The 2-D output is the Gaussian distribution's mean and variance, and the ReLU operation in the last layer ensures non-negative output. The meta-parameters ψ HE are the weights of this neural network.The meta-prior for segmented and homogeneous components uses a linear mean model and a learnable variance. The meta-parameters ψ SEG or ψ HO are linear weights and variance. We choose this low-capacity model because of the small number of segments and tasks in the meta-training dataset.We initialize the model uncertainty by setting an initial bias for variance output in MLP and initial variance value in the linear model.We use an MLP to map robot action to tactile reading in iMAML implementation. For the no-vision version, the network input is the 7-d joint angle, and we have 6 hidden layers with dimensions [32, 128, 64, 32, 16] . The with vision version takes an additional 384-dimensional DINOv2 classification token to distinguish different objects. The 384-dimensional features are encoded using a [64, 7] MLP and the output is concatenated with the 7-d joint angle as 14-d network input. The output layer is 7-d to predict joint torques for the plant dataset and 1-d to predict pressure reading for the shoe dataset.We used gradient descent for the inner loop iMAML adaptation. For both benchmarks, we use 2.0 for the regularization strength, 0.001 as the outer loop learning rate, and a total of 400 outer epochs.For the conjugate algorithm, we use 100.0 for the conjugate gradient damping and perform 10 steps. The inner loop learning rate was set 0.001 for shoe and 0.01 for plant. A total of 30 inner loop gradient descent steps are taken for both benchmarks.We provide results with alternative baseline implementations in Tab. 4 and 5. For the non-structured baseline learned with iMAML, we consider a no-vision baseline that does not use visual information. We do not see a significant performance difference compared to iMAML with vision in Tab. 1 and Tab. 2. This indicates that the visual features cannot enhance the meta-learning of tactile responses without structure in such a real-world dataset. For the Gaussian prior baseline, since the average mean tends to overestimate the stiffness, we also consider a second variant (zero mean) that simply sets the mean to zero. This zero Gaussian prior has a better zero-shot prediction on the Plant benchmark, but its 10-shot error is even larger than the 0-shot error of SBML prior. The SBML prior also learns the uncertainty of stiffness in VSF model. We visualize the standard deviation of learned Gaussian prior in Fig. 10 . For plants, we can see the stiff branch region is predicted with high uncertainty, likely due to high variation in branch stiffness caused by detailed geometric features like thickness and distance to the plant base. On the other hand, shoes' stiffness uncertainty is overall higher than plants', probably caused by the more significant material variation in the shoe dataset.To evaluate the capacity of our method, we test whether it can learn a unified VSF prior between the plant and shoe categories. We learn a single prior using both plant and shoe broad datasets with the same training configuration. We visualize the unified prior mean prediction in Fig. 12 . We qualitatively find that the unified prior in the last row has a prediction similar to the prior trained on data from individual category datasets in the second row. Quantitative results are given in Table 6 and Table 7 . Errors are comparable to the Plant-only and Shoe-only datasets, demonstrating that SBML can learn multi-category visual-tactile models without significant change in performance. In this section, we provide a better qualitative stiffness estimation using per-point touches. Note this estimation does not use the VSF model and is only for qualitative comparison. We setup another robotic system using the UR5 arm's built-in force/torque sensor to measure contact forces when the object is touched with a conical probe. Touched points are selected by voxel-downsampling the surface point cloud to 1 cm, and the probe is depressed normal to the surface and stopped when a force threshold is reached. The per-point stiffness value is estimated by dividing the maximum contact force over the displacement. For better visualization, we interpolated estimated stiffness on a triangle mesh of the object. The learned SBML prior qualitatively matches trends in the per-point estimation, with the toe region estimated to be more stiff than the tongue region. The experiments of Sec. 4.5 are defined in more detail as follows. We consider three testing conditions for the composition structure C α : heterogeneous, heterogeneous + segment, and heterogeneous + homogeneous.• Heterogeneous: contains m α components and each component contains one spring in VSF.• Heterogeneous + homogeneous: contains m α + 1 components, where m α components contain one spring with stiffness K i,HE and the last component assigns a uniform stiffness across all the springs K HO . Following Section III.C, the stiffness of each spring is thus K i = K HO + K i,HE .• Heterogeneous + segment: contains m α + n s components, where m α components contain one spring and the n s components contain the springs in one segment. We use 10 segments generated by GMM clustering on DINOv2 features. We evaluate how the choice of visual feature and prior architecture affect meta-learning performance. To avoid meta-overfitting, our mitigation strategies (in Sec 3.3 the last paragraph) help, but do not eliminate the risk. Since our dataset is relatively small, the feature relevance and prior capacity must be chosen carefully.We explore combinations of the following:• Visual features: we use normalized XYZ coordinates + RGB colors as naive visual features. We also compare DINOv2-small (DINOv2-S) which outputs a 384-dimensional feature vector and DINOv2-base (DINOv2-B) which outputs a 768-dimensional feature.• Prior structures: we consider using linear or MLP prior structures. The linear prior assumes homoscedastic uncertainty independent of input. We also observe failure cases where DINOv2 features generalize poorly. One example is the slipper class as shown in Fig. 9 . Even though the learned prior seems to capture that the toe is stiffer than the upper (i.e., the term for the part that fits over the foot), the upper's stiffness is underestimated. Hence, the prediction error is not significantly better than a naive Gaussian prior. We hypothesize that the diverse stiffness distribution of slippers might cause the failure. Here, the stiffness depends on the thickness of the upper part and other fine geometry details that are not captured in the image. We also evaluate the SBML prior's performance given different sizes of the training dataset. We consider reducing the dataset size by randomly subsampling the objects or subsampling touches-perobject. Results are shown in Tab. 10 and Tab.11. As expected, as the dataset shrinks, performance tends to degrade and have a larger variance. We also observe that for shoes it is more important to have more objects and fewer touches per object, where using 40% touching sequences and all objects can achieve good performance. Because the shoes has only about 20-30 touches per object, 10% touch subsampling is extremely sparse and results are noisy. Also, it appears that subsampling at 70% touches for shoes gives optimal performance, but the results are not statistically significant. Finally, we evaluate whether the learned prior is robust to different viewpoints. We found that DINOv2 visual features are robust to viewpoint variation and SBML predicts stiffness with semantic correspondences. We rotate a shoe left and right by 45 degrees and use the SBML prior trained on the broad shoe dataset to make predictions. As shown in Fig. 12 , the learned prior predicts the semantic sole part with a high stiffness between different viewpoints. 


Decision: The reviewers found the paper to be well written with a clear and signficiant contribution. There were some concerns regarding generalization and evaluations. The rebuttal helped to strengthen the paper and increase its potential impact. The paper seems well suited for CoRL.

Reviews: review: title: title: Quick Overview of Structured Bayesian Meta-Learning: A Data-Efficient Approach for Visual-Tactile Model Estimation
paper_summary: paper_summary: The paper addresses the challenge of estimating visual-tactile models for deformable objects, where vision is hindered by occlusion and touch data is sparse and noisy. The proposed approach leverages Bayesian Meta-Learning (BML) to create a data-efficient method for model estimation that can generalize from diverse training objects. The paper introduces Structured Bayesian Meta-Learning (SBML) to deal with the issue of different parameter spaces for various visual-tactile models. SBML incorporates heterogeneous physics models to learn from objects with varying appearances and geometries, enabling both zero-shot vision-only prediction and few-shot adaptation with tactile data. The experiments validate the efficacy of SBML on heterogeneous objects, namely plants and shoes, demonstrating superior performance in force and torque prediction accuracy in both zero- and few-shot settings.
main_review: main_review: ### Quality
The quality of this paper is high, with robust empirical validation and a well-structured methodology. The authors provide a comprehensive explanation of the proposed Structured Bayesian Meta-Learning (SBML) approach, detailing its components and the rationale behind its design. The experimental setup and results are clearly presented, showcasing the effectiveness of SBML in comparison to existing methods.

### Clarity
The paper is generally clear and well-written.

### Strengths:
1.  The introduction of SBML addresses a critical challenge in visual-tactile model estimation, offering a novel solution that leverages Bayesian meta-learning.
2.  The method's capability for zero-shot and few-shot learning is particularly valuable in practical applications with limited data.

### Weaknesses:
1.  The evaluation is currently limited to the VSF model, which only predicts tactile response.
2.  The use of a Gaussian prior assumes a single mode of material parameters conditioned on visual features, which may not capture more complex, multi-modal distributions effectively.
questions: questions: 1. Can the authors provide additional experiments or validation results using other deformable object models, such as Finite Element Method (FEM) or Graph Neural Dynamics Models, to demonstrate the broader applicability of SBML?

2. The paper mentions that tactile data is often sparse and noisy. Can the authors elaborate on the specific techniques used to mitigate these issues in the SBML framework?
limitations: limitations: 2
review_summary: review_summary: Overall, "Structured Bayesian Meta-Learning for Data-Efficient Visual-Tactile Model Estimation" is a high-quality, original, and significant contribution to the field. The proposed SBML approach addresses key challenges in visual-tactile model estimation, offering a data-efficient solution that generalizes from diverse training objects. While the method shows strong potential, future work should aim to extend its scope to other models and improve its capability to handle multi-modal distributions and generalize across different object categories.
title: title: Meta-Learning with Multisensory Data; Stiffness Estimation
paper_summary: paper_summary: The authors present a method Structured Bayesian Meta-Learning to use visual-tactile data to estimate stiffness of deformable objects. The method leverages Bayesian meta-learning with an incorporated structure generation module, allowing it to adapt to objects with different parameter spaces. The approach does a zero-shot vision prediction first and then few-shot adaptation using tactile signals.
main_review: main_review: The paper is well-written and clear to read. The graphs and illustrations are good. The idea and method are well-explained. 

Strengths:
- The methods is novel.
- Sufficient robot demonstrations on two different datasets
- Data efficiency is good. The model can do a accurate estimation with zero-shot visual prediction and few-shot adaptation with tactile.

Weakness:
- My main concern is the generalization of this method. 
- As the authors point out in limitations, this approach cannot generalize cross object categories, and even in the shoe dataset, since the shoe appearance and material are quite different from each other, thus the performance is worse across shoes. 
- The two datasets are all not large. I can imagine that even if the model is trained on tree dataset and then fine-tuned on shoe dataset, it will probably "forget" the features it learned in tree datasets and perform bad on trees. Perhaps large models can be leveraged to identify object categories beforehand to enable better generalization.
questions: questions: As mentioned in above review, I'm interested in more results cross object categories, even if they are not deformable. If you use some vision-based tactile sensors, it is also possible to retrieve surface stiffness from the readings. Have you tried any large multimodal datasets such as Objectfolder benchmark [1], or Touch and Go [2]? I think Objectfolder benchmark contains visual and tactile data and material properties that might be able to infer stiffness. Touch and Go has data collected in the wild from different surfaces.

[1] Gao, Ruohan, et al. "The objectfolder benchmark: Multisensory learning with neural and real objects." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
[2] Yang, Fengyu, et al. "Touch and go: Learning from human-collected vision and touch." arXiv preprint arXiv:2211.12498 (2022).
limitations: limitations: 1
review_summary: review_summary: My main concern is the generalization of the method.
title: title: Hierarchical Bayesian Learning for Soft Object Mapping. Good approach but requires clarification.
paper_summary: paper_summary: This paper presents a method for visuo-tactile mapping of objects. It employs a hierarchical Bayesian approach, where the mapping problem is factorized into learning prior knowledge about object classes and then using this object-class specific prior knowledge in the visuo-tactile mapping of individual object instances. The paper is motivated by the problem to predict the tactile behavior (stiffness) over full object surfaces, but from the sparse data collected using only few tactile interactions with the same object. The hierarchical Bayesian approach tackles this problem as it learns useful prior information that allows the framework to predict the stiffness of object patches either using only vision (zero-shot), or by extrapolating tactile information from other interaction points on the instance.
main_review: main_review: The paper tackles a relevant and rather under explored problem: inferring the compliance of objects over their surface area. The problem is well motivated. I like the discussion of differences between tactile and vision in the introduction. It is sensible to formulate the problem as one of hierarchical Bayesian inference as the authors do. The need for priors is well motivated with the unavoidable sparsity of touch data.
The method as explained seems sensible, however there are a few open questions regarding the design of the method and also given the results that need to be clarified and potentially require additional work beyond writing. Please see these issues below.
Beyond these issues I think it would greatly benefit the work if the authors actually went further and tested transfer between categories, as they mention in the limitation section. (I want to mention this here and don't include it in the list below because I don't think it necessarily needs to be addressed. However, it would be nice).
questions: questions: - L214: why are these different? Please motivate.

- As far as I can see the objects are always viewed from the same perspective -- both, shoes and plants. If this is true the results we see could strongly overfit to the perspective. It is necessary to provide experimental data that shows the method does not just learn "further down in the image means more stiff". The objects could be presented in random rotations or at least be approached from different directions while in their "natural orientation" (sole/stem downwards).

- L239: It is not a good idea to use the output of your own method as "proxy for ground truth" -- especially because the results in Figure 5 do look rather erroneous. I don't believe the middle part of the brown boot is as soft as the middle part in the other two shoes. The brown boot should be generally way more stiff, I suppose. Shouldn't it be possible to collect ground truth by somewhat exhaustively mapping the objects square centimeter by square centimeter? Please provide a sensible ground truth, or at least explicitly acknowledge that you don't quantitatively assess the quality of the method. Sensible ground truth would be the way stronger option here though.

- L282: Table 1 itself doesn't state it would be zero shot performance, but the text in this line implies so. Something is off here. Table says many-shot.

- L146, L149: this enumeration is unexpected. what are you enumerating here?

- L156-L178: This is confusing and also comes surprisingly as we did not talk about components up until this point. Please make sure that there is enough meta description before this point so that we understand how this fits into the big picture. This is also missing from the overview schematics for the method.
limitations: limitations: 2
review_summary: review_summary: Relevant problem tackled by an approach that fits (hierarchical Bayesian approach). However the authors should provide some experiments from other perspectives to make sure the method doesn't overfit and should also ideally compare against real ground truth (and not against outputs of their own algorithm)..