Fine-Tuning Generative Models as an Inference Method for Robotic Tasks
Authors: Orr Krupnik
Elisei Shafer
Tom Jurgenson
Aviv Tamar
Venue: CoRL 2023 Poster (2023)
Keywords: robotic learning
fine-tuning
generative models

Abstract: Adaptable models could greatly benefit robotic agents operating in the real world, allowing them to deal with novel and varying conditions. While approaches such as Bayesian inference are well-studied frameworks for adapting models to evidence, we build on recent advances in deep generative models which have greatly affected many areas of robotics. Harnessing modern GPU acceleration, we investigate how to quickly adapt the sample generation of neural network models to observations in robotic tasks. We propose a simple and general method that is applicable to various deep generative models and robotic environments. The key idea is to quickly fine-tune the model by fitting it to generated samples matching the observed evidence, using the cross-entropy method. We show that our method can be applied to both autoregressive models and variational autoencoders, and demonstrate its usability in object shape inference from grasping, inverse kinematics calculation, and point cloud completion.

Fine-Tuning Generative Models as an Inference Method for Robotic Tasks
Orr Krupnik
Elisei Shafer
Tom Jurgenson
Aviv Tamar
Department of Electrical and Computer Engineering Technion Israel Institute of Technology
Haifa, Israel
ok1 ok1@campus- techniin..... iochiion.ac.. ac.
Abstract: Adaptable models could greatly benefit robotic agents operating in the real world, allowing them to deal with novel and varying conditions. While ap- proaches such as Bayesian inference are well-studied frameworks for adapting models to evidence, we build on recent advances in deep generative models which have greatly affected many areas of robotics. Harnessing modern GPU accelera- tion, investigate how to quickly adapt the sample generation of neural network models to observations in robotic tasks. We propose simple and general method that is applicable to various deep generative models and robotic environments. The key idea is to quickly fine-tune the model by fitting it to generated samples matching the observed evidence, using the cross-entropy method. We show that our method can be applied to both autoregressive models and variational autoen- coders, and demonstrate its usability in object shape inference from grasping, in- verse kinematics calculation, and point cloud completion.
Introduction
Humans and other animals maintain powerful mental models of the world [1] for navigation, object manipulation, social interaction and other day-to-day tasks. These mental models imperfect and tend to be inaccurate. However, they are highly adaptable and are frequently updated upon arrival of new information. Robotic agents operating in diverse and unstructured environments may also be required to adjust their behavior, and would therefore benefit from adaptable models similar to the ones humans hold. Recent studies in robot manipulation and navigation have focused on scenarios where an accurate model can be learned and used as-is in downstream planning tasks [2, 3, 4]. In contrast, inspired by biological mental models, this paper focuses on how efficiently adapt model novel information.
Natural approaches to updating models given new evidence such as Bayesian inference have been used extensively in computer vision and robotics [5, 6). When the modeled probability distributions are -in-dimensional, Bayesian inference may have closed form solutions or the posterior can be numerically estimated using methods such as Markov chain Monte Carlo (MCMC, [7]). However, there is growing interest in using high-dimensional deep generative models for roboties, which can represent complex and diverse data. Advances such variational autoencoders [8] and diffusion models [9] make it possible to learn diverse and high-dimensional distributions. For such expressive models, Bayesian inference techniques are not suitable, and they cannot operate on the time-scale required for robotic tasks [10]. Another approach, which has been found effective with deep gener- ative models, to train them to adapt by conditioning on possible observations [11, 12]. However, this paradigm may fall short when faced with nut-of-distribution evidence test-time.
We propose simple approach for updating the parameters of deep generative models given empir- ical observations, to approximate complex posterior distributions. Our method requires forward
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

simulation of the environment which can produce observations given a model, and a similarity func- tion for observations. We build on GPU-based physics simulation [13] and model training to perform fast inference in a novel robotic scenario by fine-tuning the generative model weights. Considering the parametric nature of deep generative models, we use a version of the cross-entropy method (CEM. [14]) to quickly update the parameters generate observations conforming with the avail- able evidence. We dub method MACE, for Model Adaptation with the Cross- Cross-Entropy method.
To showcase MACE, we focus on robotic domains where the forward simulation step is easily per- formed using off-the-shelf physical simulators. We demonstrate the versatility of MACE on sev- cral robotic manipulation tasks that we frame as model adaptation, using two different types of deep generative models. In particular, we demonstrate results on object identification from position measurements of multi-fingered robot gripper, on recovery of object point clouds given partial measurements (as generated by depth sensors) and ence of obstacles. In all of these environments, the posterior has rich multi-modal structure. We demonstrate that MACE is indeed capable of producing diverse posterior samples for a variety of observations, and that outperforms baseline approaches in diversity and accuracy. In terms of speed, we show that by exploiting GPU-based simulation and inference, our fine-tuning can be per- formed online in a competitive time frame. For example, the IK tasks, we find that MACE can outperform the Movelt [15] library in quickly finding IK solutions for complex scenarios.
inverse kinematics (IK) task in the pres-
2 Related Work
Bayesian inference is concerned with computing the posterior distribution of data given observa- tions. Exact computation is possible for simple models which admit conjugate priors, and Markov chain Monte Carlo (MCMC) can be used for general models [7]. Approximate Bayesian compu- tation (ABC, [16]) allows sampling from the posterior without an exact likelihood, but with some similarity function between observations. MACE is inspired by these approaches, and expedites the search for better samples using CEM [14]. Recent work by Engel et al. [17] introduces a Bayesian model update scheme using CEM related to ours. However, their method requires the true likelihood and is limited to relatively small models, while ours can be applied to deep generative models.
Bayesian inference has been used extensively in robotics in the context of state estimation, localiza- tion, and mapping [5, 18]. In recent work, Marlier et al. [19] use Bayesian estimation of posterior distribution of grasp poses for multi-finger object grasping, and Pastor al. [10] use Bayesian inference with LSTM [20] to classify objects using tactile sensors. Both the above are application- specific, while MACE is general approach applicable to a variety of tasks and generative models.
Another approach is to amortize inference by learning an approximate posterior using data from the joint distribution of models and observations p(z, o) such as in conditional variational autoen- coders (CVAE, [21]). We compare MACE with CVAE baseline, and show that while inference procedure is slower, MACE produces more diverse posterior. Furthermore, MACE can work for observations that are out of distribution with respect to p( o), unlike the CVAE. Finally, MACE can tune the same prior model with different modalities of observations without retraining.
Meta-learning, and meta-RL. in particular, an alternative approach to quickly adapt behavior to new evidence [22]. However, meta-RL is typically model-free, and learns how adapt a policy [22, 23]. Model-based meta-RL approaches such as Zintgraf et al. [12] CVAE to condition on the history of observations. Whether such methods could be improved using inference method is an interesting direction for future research.
3 Model Adaptation with the Cross-Entropy Method
We now describe MACE, our method for adapting deep generative models environment observa- tions using the cross-entropy method. We begin by describing the setup and the types of tasks aim to solve (Sec. 3.1); next, we discuss our update rule and present the full algorithm (Sec. 3.2).

---

3.1 Problem Formulation
In the robotics context, an inference problem involves the recovery of lask parameters given obser- vations of the environment. We assume some distribution p(æ) over task representations æ X. task description æ could be point cloud (PC) of an object as in grasping task; an image input for more complex manipulation; or desired joint configuration in a reaching task. MACE quires access generative model representing parametric distribution over task representations p(æ; 0). We assume the generative model is initially trained to represent the prior, i.e. for some initial parameter 0, it holds that p(æ; 06) p(z).
Although the task representation a may be unknown, some information about can be observed. We denote this information in grasping task, o could be partial PC obtained from depth sensor (while is the full object model); in reaching task, could represent obstacles for the robot avoid.
O (for observation) and note that it can be of any form. For example,
A central component of MACE a simulator of the task. Exact environment simulators are often hard to design, leaving gaps to the reality they attempt to simulate [18]. However, for our purposes, we only require the simulator to emit observations of a similar modality to the ones emitted by the environment. Therefore, the simulator can be viewed probability distribution p(o (>), providing observations given task representations.
The goal of the inference task is to train the parametric model p(z; 0) to produce a distribution closely resembling the posterior over task representations given observations, p(x)o).
3.2 Updating the Model
We aim to update the model parameters so that the generative model p(æ; 0) more closely sembles the posterior p(z]o). We can do this by minimizing the Kullback-Leibler (KL) divergence between the posterior and the parametric model:
arg min DKL. (p(=)))]]p((;; 0)) arg min f p(ar) log / p(ar ()) log p(x; O)dr.
9
The parametric model is only present in the second term, therefore we can maximize minimize the entire KL divergence. Since the posterior p(ælo) is unknown, we use Bayes' rule replace it with the likelihood p(o|æ)): arg max, f p( log 0) arg maxo. p(o) p((()) ((;;(>;;0 d) = arg max (eppp()) (p(o)æ) logp(æ; 0)]
p(a)
The likelihood of the observation p(o|æz) is also an unknown quantity. However, we may approxi- mate it using the forward simulator, which can produce observations o given æ. We define a score O O + [0, 1] indicating similarity between pairs of observations,
function as any function and assume Bor= o) is an approximation of p(o]z). An intuitive case to justify this as- sumption is when observations o are discrete, and S(o', o) 1o'=o is an indicator of whether o' is equal the evidence !. When using deterministic simulator, the expectation over observations o S(o', o) can be replaced by the score of single sample S(o', ((o',o) o) 2. Plugging in this score function, the optimization problem becomes:
arg max E Eppa)))) [S(o', 0) log p(:; 0)]
Recalling our assumption that p(:; 0o) p(x), Eq. can be optimized using importance sampling: arg max N E p(æ,:10) S(o,, o) log p(z,; 0), where x; p(:; 0), the sampling distribution, and o is the observation matching x,, One question is how to choose an effective sampling dis- tribution which places enough mass on high-scoring æ values. Inspired by the iterative approach of Engel al. [17], we optimize this objective iteratively using stochastic gradient descent (SGD).
p(m::0)
1 For other examples of score functions, see the environment descriptions Sec. 4.
As the simulators in our experiments are deterministic, we simplify notation by using the single-sample score function. The derivation for stochastic simulators follows the same lines, using expected scores.
3

---

At each iteration, we use the previous parametric model as the sampling distribution and take a few gradient steps to obtain the next model parameters. The objective at iteration is given by:
N
p(a;;; 0o) pi 0,1) S(o, o) logp(æ;;;0)
arg max
with 2, ~ p(z; 0c-1). In practice, we find that the importance sampling term in this objective makes it difficult optimize due to large discrepancies between the parametric distributions". Instead, introduce another approximation and remove the importance sampling term
alæ) from Eq. 2.
p(a:0]
To further improve performance and shorten training times, we follow the cross-entropy method (CEM) formu- lation described by Botev et al. [24]. We view the objective in Eq. as the problem of finding a distribu- tion p(=; 0) which produces samples with high scores o)) To op- timize S(o', o) (which implicitly function of æ through the simula- tor), treat p(:; 0)
Algorithm 1: MACE (Model Adaptation with the Cross-Entropy method)
Input: Prior model p(æ; Qb). simulator p(0)z),
observation quantile parameter gradient steps parameter M
for ( <- 1, 1,..,,, 7 do
Sample æ1, NN p( 0.-1)
Obtain 0n, o....., ON ~ p(o x using the simulator
Calculate S(o,, o) for all 01 01...., ON
Select top gN samples, set d ((0 o) Optimize
impor- tance sampling distribution and ad- just it such that it samples values of a that are close to the ones implic- itly maximizing S(o', o). At each it- eration we sample a batch {z;
arg max Ei1 1s(o,,0266 log pog 0,) using M steps of SGD
return updated model p(æ: 0r)
p(:: 0.-1) obtain observations using the simulator {o, ~ )) and calculate their re- spective scores () The top qN samples with the best scores selected, where q is pre-selected quantile hyperparameter. We define 8 = S(o [ o), the score function value for the J-th sample. The complete CEM-inspired MACE objective is given by:
arg max
1
0).
(3)
Botev et al. [24] solve stochastic program for each iteration of 9. Instead, we optimize this objec- tive via stochastic SGD as described above. The full MACE algorithm is shown in Alg. 1.
3.2.1 Implementation
Although MACE is suitable in principle for any generative model, some considerations must be made for specific types of models.
Autoregressive models provide an explicit likelihood value for a sample æ using the chain rule: with MACE by denoting
p(æ)= = II, paa,/_/-,, ((..))) .n)). Therefore, they are straightforward the neural network weights 0 and directly optimizing the objective in Eq.
VAES do not provide an explicit likelihood value p(x). However, they are trained with a lower bound estimate of p(x), namely the Evidence Lower Bound (ELBO) log p(=|2;; vp)
DEL (q(2/æ;; ())lp(z)), where p(z) zb) is the VAE decoder parameterized by vb. (<>[;; Φ) is the encoder parameterized by Φ and p(z) is the prior Gaussian distri- bution of the latent variables parameterized by Mz. oz. We can the ELBO in Eq. as lower bound of the log-likelihood term log p(æ;0) optimizing the VAE parameters: arg max. & Σn= 0))2 [log ((,,,,/;, |2,; vb) DKL (g(z, (6 ([(=))) We found the opti- mization of the entire set of VAE weights th, Φ to be difficult. Instead, use O [Mz, σ²) ) as tuned parameters, and keep the encoder and decoder weights frozen. This choice of 0 only affects the
We compare MACE to this objective as baseline; see experiment results in Sec. 4.1 and Sec. 4.2

---

second term of the objective, and specifically p(z). The objective of this version of MACE, which dub MACE-VAE, then becomes: arg maxg g Σ =i=1 1 0)22 1 1-DKL (1(2))=4)lp(( 0))]
Experiments
To demonstrate the potential of MACE robotic applications, we consider several domains in which generative models are used to capture complex data distributions and show that MACE leads to practical solutions, improving on the alternatives. We describe two of these environments and how they fit our setting, followed by summary of results. A third environment, in which MACE to recover object shapes from partial point cloud observations, described in Appendix C.
4.1 Inferring Object Shapes by Grasping
Inferring object shapes from tactile measurements [10] vital when visual sensors unavailable or limited. We investigate whether details about object can be inferred using only the contact points between it and the robot gripper fingertips. Differing from the approach of Pastor et al. [10), which aims to classify objects into 36 classes, we consider an expressive prior distribution over possible shapes represented by a deep generative model.
Dataset. We use the "airplane" class from the ShapeNet [25] dataset as representative collection of objects with complex shapes. This class contains 4045 objects, represented as 2048-point PCs.
Model. The prior generative model p(:; 0o) is a VAE trained on full PCs of objects. We use MACE-VAE described in Sec. 3.2.1. Our VAE architecture uses PointNet [26] in the encoder and fully connected decoder, and is based on the implementation used by Daniel and Tamar [27]. Additional training details and hyperparameters can be found in Appendix A.
Fottenor (12)
Coke
Fouberor
11 1a00nittn
Dun Onderaation
(a) Average
Simulator: For ease of implementation, we use sim- ple geometric simulator calculate contact points. Details can be found in Appendix A.
Model Diversity (Painiise Chamrer Distance)
Score Function. The score function for k-fingered grasps, aggregating distances between contact points and clipped the range of [0, 1), is defined
Aoy Karbbr
Ince
S(o', o) max -1- ビラ= - 11,1) 11. where Pj
(o) is the j-th contact point of observation
8-
Inferring Object Shapes by Grasping: Results
(b) Sample diversity
We adapt the distribution of airplane models by ob- taining single observation o representing contact points of k 5 robot fingers with an unknown ob- ject. To evaluate the tuning process, objects æ; from the prior and another 49 from the posterior, and calculate the average score for the matching observations o; obtained from the simulator using S(o,, o). In addition, we compute the pairwise Chamfer distances between every two objects in each sampled set and take their mean as measure of sample diversity. Scores and sample diversity are presented in Fig. 1.
Figure I: Mean results for the multi- fingered grasping experiments; error bars represent standard deviation over 100 seeds.
sample 49
As a baseline, we conduct the same experiment using the importance sampling loss described in Eq. 2. We replace the ligeliidd term with the ELBO as in LACE-VAE (see Sec. 3.2. with the rest of the algorithm components in Alg. 1. We find that this objective performs poorly
5

---

(a) Samples from the prior
(b) Samples from the posterior
(c) Samples from the CVAE
Figure 2: Tuning for the multi-finged grasping domain. Model samples are shown in white and finger positions and contact points of the given observation o are represented by orange cylinders.
compared MACE (see Fig. 1). Moreover, optimizing is an order of magnitude (up to 20x) slower than using MACE.
As an additional baseline, we train a CVAE conditioned on contact points calculated by grasping each training-set object in simulation. We tune our model and compare results to the CVAE baseline using 100 objects from the held-out test set of the ShapeNet "airplane" class, all with the same observation o, in which the robot finger directions are the four diagonal corners of the zy plane, and a fifth along the axis. Quantitative results can be seen Fig. 1. The CVAE baseline outperforms the tuned posterior in scores, but produces distribution with lower diversity.
In addition to the quantitative results, we present samples from the prior, posterior and CVAE models in Fig. 2. These showcase the diversity of the model tuned by MACE compared to the CVAE distribution. Additional samples and tuning hyperparameters can be found in Appendix A.
Out-of-distribution experiment. Albeit its advantage of fast amortized inference, the CVAE is limited to observations seen in its training set. We demonstrate this disadvantage with out- of-distribution experiment. We run the entire set of experiments a second time using different observation, with the fifth finger pointing along the opposite direction of the axis. This is out of the joint distribution p(x; o) which the CVAE baseline was trained on. Consequently, its results greatly deteriorate. Conversely, the model tuned by MACE outperforms it both in diversity and in scores. Results can be seen Fig. Visuals of samples from the CVAE and the MACE-tune posterior model with the new OOD observation be viewed in Fig. in Appendix A.
4.2 Inverse Kinematics with Obstacles
Inverse kinematics (IK) is the calculation of the configuration of robot joints given desired pose in Cartesian space. IK calculation is an especially challenging optimization problem when obstacles are involved and has no closed-form solution in the general case. While previous work has attempted to learn IK using generative models [28, 29, 30], we focus on tuning a pre-trained IK model to con- sider novel obstacles. We view the obstacle-constrained IK problem as an inference problem, where the prior p(:; 0o) is a generative model trained represent distribution of joint configurations conditioned on the end-effector position" when obstacles are present. Note that this is a complex and multi-moda. distribution which accounts both for self-collisions and for conditioning the desired pose. The observation is obstacle configuration, and the posterior captures a distribution over non-colliding joint configurations.
Dataset. We collect 1M random valid configurations Franka Emika Panda 7-DoF robotic ma- nipulator using PyBullet physics simulation [31], and record their matching end-effector positions. We collect these configurations with obstacles present; therefore, valid configurations which conform to the joint limits of the robot, and where the robot is not in collision with itself.
Model. We train an autoregressive model with joint positions generated sequentially, conditioned on the previous joints as well as the end-effector position: p(qı Pee). p(gel q1. Pee) etc. Probability
TThe pose also include the end-effector orientation; in this work focus position-only IK.

---

(a) Window task: prior
(b) Window task: tuned
(c) Wall task: tuned
(d) Box task
Figure 3: Tuning for the robot inverse kinematics domain. The obstacle is shown in blue, while the goal end-effector position is shown by a green ball. While Fig. 3a shows the prior distribution overlayed with the window obstacle, the same prior was used for all tasks Fig. 3b displays samples from the distribution tuned with MACE for the window obstacle, while Fig. 3c shows the same for the wall obstacle. In both cases, the posterior rarely admits configurations colliding with the obstacles, while remaining diverse.
distributions for each joint are represented by Gaussian mixture models. Further architecture and training details can be found in Appendix B.
Simulator: We use open-source physical simulation environments, optionally with obstacles in the robot workspace. The simulated robot can be set to specific joint configuration q. The simulator returns whether the robot is in collision with itself or the obstacles, as well as the distance between the desired position and the actual end-effector position obtained by setting the robot to q.
Score Function. We opt for score function that penalizes collisions harshly, and therefore set ((o o) if the robot is in collision in given configuration. Otherwise, the score is proportional to the distance between the generated end-effector position and the desired position: S(o', Pee,actual ))
exp( Pee,desired
Inverse Kinematics with Obstacles: Results
We run two experiments in this domain, with different types of obstacles the workspace, both using the PyBullet simulation environment [31]. In the first experiment, the obstacle is a vertical window, with the desired end-effector positions located beyond it. A qualitative sample from the prior model p(z; 0o) (trained with obstacles) can be viewed in Fig. 3a, where it is clear that many of the sampled configurations are in collision with the obstacle. Fig. 3b shows samples from the posterior model tuned with MACE, which almost never collide with the obstacle.
To show that the result does not depend on obstacle shape, we conduct simi- lar experiment with wall obstacle, with the target end-effector position behind it. Samples from the model tuned by MACE can be seen in Fig. 3c, again di- verse and non-colliding. We verify this result quantitatively by sampling 10 goal model with the respective score functions. As baseline, we also tune the model with the impor- tance sampling (IS) objective of Eq. 2. We report the mean scores over 1000 samples from the prior, the posterior tuned with MACE and the posterior tuned with the IS baseline in Table 1. We additionally report the success rate, calculated as the percentage of sampled configurations which are not in a collision state. The results clearly show improvement when tuning with MACE. Tuning hyperparameters and additional samples can be found in Appendix B.
Table I: Inverse Kinematics Results
Model
Score
Success Rate
Prior
0.129 0.015
0.132 + 0.015
Posterior (MACE) Posterior (IS)
0.937 + 0.079
0.941 - 0.079
0.317 0.121
0.339 + 0.122
end-effector positions behind the wall, and tuning the
Comparison to Movelt Inverse Kinematies. The Movelt [15] motion planning framework in- cluded with ROS has standard IK service, used infer goal positions for motion planning algo- rithms. While it is powerful tool, we demonstrate that MACE improve its solutions where
7

---

Table 2: Inverse Kinematics Comparison to Movelt for the Box Task
MACE
Movelt
MACE Movelt
Time (s)
Ace. (cm)
Time (s)
Acc. (cm)
Time (s)
Acc. (cm)
0.106 + 0.008
1.92 + 3.01
1.553 + 1.103
10
0.641 + 0.925
< 10
it struggles to find them quickly. We construct scenario of a box in front of the robot, with the desired end-effector position inside it (see Fig. 3d). Using our prior model only (no tuning steps, for maximal speed), we sample 20 batches and test them for collisions in the IsaacGym [13] GPU- based simulator. Using our score function, we take the configurations with the maximum score as our IK solution. Table 2, we report calculation time as well as solution accuracy for our method compared to Movelt. In addition, since Movelt depends on the initial robot position for the IK cal- culation, use the position sampled from model initial position for Movelt, thus reaping the benefits of both methods. Time and accuracy for this setting are reported in the third column of Table 2. Experiment details (including MACE adaptation experiment for the box domain) and additional visual results are available in Appendix B.
To alleviate rim-lo-real concerns, we test the solutions found by MACE on a real Franka Panda robot with box matching the IsaacGym simulator, in two different box configurations (and matching poses). Visual results of this experiment can be found https: : //www//wwwwww/////////////// orrkrup com/mace.
5 Limitations
Forward simulator. A simulator that emits observations similar to the environment may not always be available, causing a sim-to-real gap which could hurt performance. Approaches such as domain randomization [32, 18] may mitigate this problem; in addition, we show that in some cases this gap does not affect performance (see real robot experiment at https:/ //www. http//wwwwrkkkuppp com/mace)
Inference speed. In our experiments, MACE inference takes between 7 65 seconds (depending on the domain) which still not fast enough for real-time inference applications. While the sequential nature of MACE optimization is an unavoidable computational limitation, code optimizations as well as faster hardware could dramatically speed up computation.
Quality of the prior. The quality of the tuned posterior depends greatly on the quality of the pre- trained deep generative model: if high-scoring samples have low probability under the prior, MACE may not find them. In our experiments, found that deep generative models provide priors accurate enough for the domains investigated.
6 Conclusion and Outlook
We presented MACE, method for adapting deep generative models using the cross-entropy method, and demonstrated its usage for multiple robotic tasks. MACE allows the model to quickly adapt to previously unseen conditions while producing diverse posterior distributions. Our promising results for inverse kinematics show that deep generative models, when tuned appropriately using MACE, may help speed up robotic problems that are typically solved using non-learning-bant approaches.
In future work, intend to explore ways to expedite the optimization process and improve the usability of MACE in robotic tasks. Additionally, in this work we only considered the inference problem. However, in realistic scenario the agent may also have control over which observations to acquire. In this case, it would be interesting to extend MACE to an active sampling method. An- other related direction is to use MACE as an inference method for meta-RL, replacing the currently dominating CVAE-based approaches [12, 34].
$ "our experiments used unoptimized PyTorch [33] code and a single Nvidia GTX 1080 Ti GPU. As a simple proof of concept, we also ran an instance of the multi-fing fingerierrrr grasping experiment (Sec. 4.1) on single Nvidia A100 GPU, which tuning time from 25 seconds to 11 seconds.

---

Acknowledgments
The authors would like thank Tal Daniel for sharing his considerable array of VAE training tips and tricks. In addition, we acknowledge the anonymous reviewers of the paper for their thoughtful comments and suggestions for improvement. This work was funded by the European Union (ERC, Bayes-RL, Project Number 101041250). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council Exccutive Agency. Neither the European Union nor the granting authority can be held responsible for them.
References
[1] P. N. Johnson-Laird. Mental models and human reasoning. Proceedings of the National Academy of Sciences, 107(43):18243-18250 2010.
[2] J. Redmon and A. Angelova. Real-time grasp detection using convolutional neural networks. In 2015 IEEE international conference on roboties and automation (ICRA), pages 1316-1322. IEEE, 2015.
[3] A. Nagabandi, G. Kahn, R. S. Fearing, and S. Levine. Neural network dynamics for model- based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE international conference on robotics and automation (ICRA), pages 7559-7566. IEEE, 2018.
[4] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, Jang, D. Quillen, Holly, M. Kalakrishnan, V. Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision- based robotic manipulation. arXiv preprint arXiv: 10293, 2018.
[5] S. Thrun, W. Burgard, and D. Fox. Probabilistic Robotics. MIT Press, 2005.
[6] R. Szeliski. Computer vision: algorithms and applications. Springer Nature, 2022.
[7] D. Barber. Bayesian reasoning and machine learning. Cambridge University Press, 2012.
[8] D. P. Kingma and M. Welling. Auto-encoding variational bayes. In International conference on LEarning representations, 2013.
[9] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[10] Pastor, J. ProrzfaaGozzález,1 J. M. Gandarias, Medina, Closas, A. Cereza, and J. M. Gómez-de Gabriel. Bayesian and neural inference on lstm-based object recognition from tactile and kinesthetic information. IEEE Robotics and Automation Letters, 6(1):231-236) 2020.
[11] W. Yu, Tan, C. K. Liu, and G. Turk. Preparing for the unknown: Learning universal policy 2017.
with online system identification. arXiv preprint
[12] L. Zintgraf, K. Shiarlis, M. Igl, S. Schulze, Y. Gal, K. Hofmann, and S. Whiteson. Varibad: A very good method for bayes-adaptive deep rl via meta-learning. arXiv preprint axX1v:1910008348, 2019.
[13] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State. Isaac gym: High performance gpu-based physics simula- tion for robot learning, 2021.
[14] R. Y. Rubinstein and D. P. Kroese. The cross-entropy method: unified approach combi- natorial optimization, Monte-Carlo simulation, and machine learning, volume 133. Springer, 2004.

---

[15] D. Coleman, I. Sucan, S. Chitta, and N. Correll. Reducing the barrier to entry of complex robotic software: a moveit! case study. arXiv preprint arXiv: 1404.3758, 2014.
[16] J.-M. Marin, P. Pudlo, C. P. Robert, and R. J. Ryder. Approximate bayesian computational methods. Statistics and computing, 22(6): (:6))1100 167--180, 2012.
[17] M. Engel, O. Kanjilal, I. Papaioannou, and D. Straub. Bayesian updating and marginal like- lihood estimation by cross entropy based importance sampling. Journal of Computational Physics, 473:1117444 2023.
[18] Ramos, R. C. Possas, and D. Fox. Bayessim: adaptive domain randomization via probabilis- tic inference for robotics simulators. arXiv preprint (rXiv:1906.01720
2019. 2019.
[19] N. Marlier, O. Brüls, and G. Louppe. Simulation-based bayesian inference for multi- multi-fingered robotic grasping. arXiv preprini arXiv:2109. 14275, 2021.
[20] Hochreiter and Schmidhuber. Long short-term memory. Neural computation, 97))
1780, 1997.
[21] K. Sohn, H. Lee, and X. Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28, 2015.
[22] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. International conference on machine learning, pages 1126-1133 1135. PMLR, 2017.
[23] Y. Duan, J. Schulman, X. Chen, P. L. Bartlett, I. Sutskever, and P. Abbeel. RI "2 Fast rein- forcement learning via slow reinforcement learning. arXiv preprint 1611.0:6111.27777 2016.
[24] Z. I. Botev, D. P. Kroese, R. Rubinstein, and P. L'Ecuyer. The cross-entropy method for optimization. In Handbook of statistics, volume 31, pages 35-59. Elsevier, 2013.
[25] A. X. Chang, Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li. S. Savarese, M. Savva, S. Song, H. Su, al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv: 15120003, 2015.
[26] C. R. Qi, H. Su, K. Mo, and Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017.
[27] Daniel and A. Tamar. Soft-introvae: Analyzing and improving the introspective variational autoencoder. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4391 4400, 2021.
[28] H. Ren and Ben-Tzvi. Learning inverse kinematics and dynamies of a robotic manipulator using generative adversarial networks. Robotics and Autonomous Systems, 124:103386, 2020.
[29] C.-M. Hung, S. Zhong, W. Goodwin, O. P. Jones, M. Engelcke, I. Havoutis, and I. Posner. Reaching through latent space: From joint statistics to path planning manipulation. JEEE Roboties and Automation Letters, 7(2):5334 -3411, 2022.
[30] R. Bensadoun, S. Gur, N. Blau, T. Shenkar, and L. Wolf. Neural inverse kinematics, 2022.
[31] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. 2016.
[32] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 23-30. IEEE, 2017.
[33] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang. Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differeeniiaionn in pytorch. In NIPS-W, 2017.
10

---

[34] K. Rakelly. A. Zhou, C. Finn, S. Levine, and D. Quillen. Efficient off- meta- reinforcement learning via probabilistic context variables. International conference on ma- chine learning, pages 5331-5340. PMLR, 2019.
[35] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXivv1412.6980, 2014.
[36] Lundell, F. Verdoja, and V. Kyrki. Robust grasp planning over uncertain shape completions. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1526-1532. IEEE, 2019.
[37] Chen, X. Ma. Lu, and D. Hsu. initio particle-based object manipulation. arXiv preprint arXiv:2107 08860886,,8 2021.
[38] Y. Sun, Y. Wang. Z. Liu, J. Siegel, and S. Sarma. Pointgrow: cloud generation with self-attention. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 61-70, 2020.
Autoregressively learned point
[39] L. Zhou, Y. Du, and J. Wu. 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5826-
5835, 2021.
[40] Y. Cai, K.-Y. Lin, C. Zhang, Q Wang, X. Wang, and H. Li. Learning structured latent space for unsupervised point cloud completion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5543-5553, 2022.

---

Appendix
A Details for the Object Shape Inference Domain
Prior VAE Training Details
The prior used in this domain is VAE trained on 2048-point PCs of ShapeNet "airplane" objects. As it is based on the architecture used Daniel and Tamar [27]. we refer the reader to their work for architectural details". We train the VAE for 2000 iterations, augmenting the dataset with random rotations around the vertical (2) axis in the range of [- : ] Both the encoder and decoder are trained with the Adam optimizer [35], with a learning rate of 0.0005 and batch size of 64. The prior standard deviation is set to d² 0.2, and the weighting parameters for the loss are set to Pree = 50, 3xL. = 1. The latent space dimension is 128.
Grasping Simulator Implementation Details
To simplify implementation, we use a hand-crafted geometric simulator to calculate contact points between a theoretical robot hand and object point clouds (PCs). We assume each finger is moved along vector pointing the origin (which is located inside the object PC), and mark the point in the PC furthest from the origin along this vector direction the contact point. To simulate the width of the finger, consider points within certain radius around the vector for contact calculation. When grasping with k fingers, the observation Rx is the subset of contact points from the PC a.
Tuning Experiment Details
The prior p(z; 0o) is tuned for 2500 gradient steps (considering the value of M below, this means = 78 in Alg. 1), which takes approximately 25 seconds on single Nvidia GTX 1080 Ti GPU. We resample a new batch of N 256 samples from the updated model every M 32 gradient steps, each taken on half of the batch due to memory constraints. We use the Adam optimizer with learning rate 0.0002. We calculate the optimization objective with a quantile value of 1
CVAE Baseline Hyperparameters
The CVAE baseline uses architecture similar to the VAE prior model described above, with an additional encoder to encode the condition contact points to a 128 dimensional latent pprior, priion- In addition to its usage in the KL divergence loss, the prior mean perior also used as input to the decoder (concatenated to the latent noise sampled). The CVAE baseline is trained with the same hyperparameters as the VAE described above, with two differences: BKL and learning rate of 0.0002.
cattof-Distribution Experiment Visuals
Fig. 4 shows visual samples from the MACE-tuned prior and from the CVAE in the OOD experiment (with the observation constituting a condition out of the distribution the CVAE was trained on).
Additional Model Samples
In this section we display additional samples for all the distributions discussed in Sec. 4.1. All visuals follow the same color scheme as in the main text, with samples shown white and the grasp positions represented by orange cylinders.
'See their public code at https: :top:://giii //gg:://gtttttbt http.////aaaaaee/oooooottoooo--o wwwettoo-vve-ytorooe we plan to release code publicly at later time.
12

---

(a) Samples from posterior
Samples from CVAE
Figure 4: Samples from the posterior distribution tuned by MACE (left) and the CVAE baseline (right) when using an observation that an OOD condition for the CVAE note the gripper finger at the tail of the airplane. Results for MACE are similar to the in-distribution task, while the CVAE unable to generate meaningful samples.
Figure 5: Samples from the prior
Fig. 5 shows samples from the pre-trained VAE prior. Figs. 6,7 display additional samples for the first (in-distribution) experiment, for the posterior tuned by MACE and the CVAE baseline respec- tively.
Figs. 8,9 display additional samples for the second (OOD) experiment, for the posterior tuned by MACE and the CVAE respectively.
B Details for the Inverse Kinematics Domain
Architecture of the Prior Model
As mentioned in See. 4.2, we train an autoregressive model produce joint configurations condi- tioned on end-effector positions. We use 10M data points collected using the PyBullet simulator, and train the model end-to-end with the Adam optimizer in supervised manner, using maximum- likelihood objective over joint configurations.. Joint probabilities are represented by Gaussian mix-
13

---

Figure 6: Samples from the posterior in the in-distribution experiment
Figure 7: Samples from the CVAE the in-distribution experiment
14

---

Figure Samples from the posterior in the OOD experiment
Figure 9: Samples from the CVAE in the OOD experiment
15

---

Figure 10: Samples from the prior overlayed with the box obstacle. Many of them collide with the walls of the box.
ture models with two components, each parameterized using fully-connected NN with layers of 200 neurons, and Leaky ReLU activation functions.
Experiment Details
PyBullet Experiments. The first two experiments described in Sec. 4.2 are conducted with the PyBullet physies simulation environment, with the wall and window obstacles. We tune the pre- trained prior for 1500 fine-tuning steps (which is equivalent to 375 in Alg. considering the single Nvidia GTX 1080 Ti GPU.
value of M below), which takes approximately 65 seconds We resample new batch of N = 64 samples from the updated model every M = 4 gradient steps. We use the Adam optimizer with learning rate 0.00002. We calculate the optmizzation objective 1 with a quantile value q 16'
Movelt and IsaacGym Experiment. For the box environment experiment and comparison to Movelt, use the same prior model, but instead use the GPU-based IsaacGym simulation vironment to expedite scoring the samples. To calculate the results described in Table 2 of Sec. 4.2, we sample 20 batches of 4096 configurations each, and test them for collisions in IsaacGym. Ob- taining the scores, we select the best configurations and report the mean and standard deviation of their distances from the goal in the accuracy column. The same configurations are used as initial positions for the "MACE + Movelt" method the third column, with the time constituting the total duration sampling. testing for collisions with IsaacGym and finding solutions with Movelt. The middle column reports times for Movelt with standard initial position. As Movelt explicitly solves an optimization problem for the IK, its accuracy is very high; however, in some cases it takes much longer to find valid solutions.
Tuning Experiment for the Box Domain. addition to the timing experiment, we conduct a tuning experiment with MACE on box domain using IsaaeGym. The experimental procedure similar the PyBullet experiments. We tune the model for 500 tuning steps (with M this single Nvidia GTX 1080 Ti GPU
means T = 125 in Alg. 1), taking approximately 10 seconds with the faster IsaacGym simulator. We resample batch of N = 4096 configurations every M gradient steps, and use quantile of = 128 Fine-tuning is conducted using the Adam optimizer, with learning rate of 0.0001. Samples from the prior can be found in Fig. 10, while samples from the tuned model can be seen in Fig. 11.
Additional Model Samples for the PyBullet Experiments
In this section, provide additional samples for the distributions described in Sec. 4.2. Fig. 12 and Fig, 14 provide samples from the prior model, trained with no obstacles present in the workspace.
16

---

Figure 11: Samples from the posterior tuned with MACE to match the box obstacle.
Figure 12: Samples from the prior overlayed with the wall obstacle.
This is the same distribution in both sets of samples, overlayed with different objects to show that many configurations collide with each of them.
Fig. 13 shows samples from the posterior tuned with MACE in the presence of the wall obstacle.
Fig. 15 shows samples from the posterior tuned with MACE and the window obstacle.
C The the Point Cloud Completion Domain
PC completion is an important component of manipulation pipelines, which allows robots to reason about their environment when partial information is available from sources such as depth sensors [36, 37]. Previous work typically focuses on scenarios in which a model can be faithfully recovered given the partial PC, i.c., when the dataset is small or the partial information is indicative of the object [38, 39, 40]. Instead, we consider case where posterior can extremely multi-modal, and must therefore model a highly diverse distribution.
Given partial PC as the observation we infer a posterior distribution over possible full PCs æ. We include this domain as a proof-of-concept, and present qualitative results on a relatively simple dataset.
Dataset. We use dataset of 10K symmetrical 3D boxes generated with random edge lengths, placed on the zy plane and centered around the axis. Each PC consists of 2048 points, uniformly sampled the box faces.
17

---

Figure 13: Samples from the posterior tuned to match observations wall obstacle.
Figure 14: Samples from the prior overlayed with the window obstacle.
Figure 15: Samples from the posterior tuned to match observations of the window obstacle.
18

---

Model. We use a the same VAE architecture described in Sec. 4.1. The VAE is trained for 2000 iterations with training samples augmented by random rotation around the vertical (z) axis in the range of [-T, x]. Both the encoder and decoder are trained with the Adam optimizer [35], with learning rate of 0.0002 and a batch size of 64. The prior standard deviation is set to o₂ = and the weighting parameters for the loss are to Bree = 3KL 0.1. The latent space dimension is 128.
Simulator. We require a simulator that can produce partial PCs given a full PC model. For this simple dataset, we obtain partial PCs by applying a random cut to each box, using a randomly sampled hyperplane. Note that this shape of the partial PC can fit a variety of different boxes, leading to a diverse posterior.
Score function. To measure similarity between PCs, S(o', o) is calculated using the Chamfer dis- tance between PCs of and o. As suggested by Chen et al. [37]. we find that calculating the distance to the top k > 1 nearest points produces better results than = 1, and therefore use = 5 when cal- culating the score function. Considering PCs æ and w' with points labeled as [ and {pf h respectively, the original Chamfer distance is given by:
N
M Σ
CD = Σ min lp!
Pa
min Ip; - -/112 P.Ea
Exe
The k-wise Chamfer distance replaces the min operation with a selection of the top-k nearest neigh- bors, denoted by the sets (A)) and (A));
M 1pp: - -Pill
CD
Σ (k)
1P - p.112 +
Σ (k)
To obtain scores in [0, 1] with being the maximum score, we set S(o'. = exp(-rCD,(o',o)) where T is a temperature parameter, set to r = 0.1 in our experiments.
Point Cloud Completion: Results
We use MACE- VAE (see Sec. 3.2.1) to tune the prior distribution parameters of the VAE latent space for 4000 fine-tuning steps (in Alg. 1, T 31), which take approximately 40 seconds on a single Nvidia GTX 1080 Ti GPU. We resample
batch of N 256 samples from the updated model every M 128 gradient steps, each taken on batch of half of the samples. We use the Adam optimizer with learning rate 0.001. We calculate the optimization objective with quantile value of 32 Fig. 16a shows samples from the prior distribution p(z; 0o) overlayed with the partial PC observation o, while fig. 16b shows samples from the posterior model P(x; 0r) tuned with MACE. We observe that MACE can produce diverse completions the partial PC, Additional samples can be found in Figures 17,18.
Additional Model Samples
Fig. 17 shows samples from the pre-trained VAE prior. Fig. 18 displays additional samples for the posterior tuned by MACE.
19

---

(a) Samples from the prior
(b) Samples from the posterior
Figure 16: Tuning for the PC completion domain. Samples from the prior and posterior models shown in white. Partial PC observation o is overlayed over all samples in orange. While the prior model extremely diverse and exhibits many different box sizes and rotations, the posterior tuned with MACE converges to samples which more closely match the evidence, while still producing plausible distribution of objects.
Figure 17: Samples from the prior
20

---

Figure 18: Samples from the posterior
21


Reviews:
[{"review_id": "o95_w5Zeka1", "reviewer": "Reviewer_EPGP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This study proposes a fast and high-performance fine-tuning method for various deep generative models to adapt to robotic agents operating in the real world. The effectiveness of the method is demonstrated in tasks such as object shape inference from grasping, inverse kinematics calculation, and point cloud completion.\n\n", "summary_of_recommendation": "The proposal in this paper is intriguing, and I am inclined toward acceptance. However, I have concerns about the insufficient discussion regarding practical applications and the lack of thorough validation of the proposed method.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "- As the authors have acknowledged, the inference of MACE is still slower compared to real-time applications. Therefore, when considering real-world robotic applications, one could argue that adopting faster inference methods like amortized inference, even with some performance sacrifice, might be more practical. In this sense, how significant is the performance difference shown in Figure 1 for practical applications? Is the performance gap between the proposed method and amortized inference substantial, even considering the speed difference? Additionally, what are the actual inference speeds for each method?\n- The proposed method involves several hyperparameters. How were they determined? Specifically, choosing values for $T$ and $M$ in Algorithm 1 seems crucial for both speed and performance. How do the results change when these values are varied?\n- Minor comment: The authors mentioned, \"we abuse notation by referring to $\\mathbb{E}_{\\boldsymbol{o}^{\\prime} \\sim \\hat{p}(\\boldsymbol{o} \\mid \\boldsymbol{x})} S\\left(\\boldsymbol{o}^{\\prime}, \\boldsymbol{o}\\right)$ as $S\\left(\\boldsymbol{o}^{\\prime}, \\boldsymbol{o}\\right)$.\" However, this may cause some confusion in the subsequent sample approximation (Eq. (2)) since it is not clear how $o_i$ is coming. It would be better to distinguish between these two notations.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- The paper is well-structured and easy to read.\n- The problem of quickly fine-tuning deep generative models in robotics is intriguing, and the proposed method is novel. Its effectiveness is evaluated in experiments, showing significant improvements in performance and sample diversity, especially in out-of-distribution scenarios compared to fast amortized inference.\n- However, the paper lacks sufficient discussion on the trade-off between speed and performance. I raise questions about this and request further clarification from the authors.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "kXMf5GOKJEg", "reviewer": "Reviewer_LQd2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a new method called MACE for efficiently adapting deep generative models to new observations or evidence from the environment. The key idea is to frame model adaptation as fitting the model distribution to the posterior given observations, and use an iterative cross-entropy-based fine-tuning approach (called MACE) to update the model parameters. MACE is demonstrated on several robotics domains including shape completion, tactile sensing for grasping, and inverse kinematics with obstacles. Experiments show it can produce accurate and diverse posteriors adapted to observations across these tasks.", "summary_of_recommendation": "The paper presents a novel method for efficient adaptation of generative models that has the potential for high impact in robotics. Experiments across diverse simulation domains demonstrate the approach can produce accurate and diverse posteriors adapted to observations. While relying on simulation and requiring optimization, the general framework offers useful tools for enabling robots to update mental models online.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Some potential issues that could be addressed for the revision:\n\n* Provide more intuition and analysis on why cross-entropy optimization is effective for this problem setting\n* Expand the discussion of simulation accuracy and potential impacts on real-world performance\n* Test on more complex observation spaces to further validate generality\n* Demonstrate on real robotic platforms beyond just simulation experiments\n* Investigate ways to improve inference speed through optimizations and hardware\n* Consider theoretical analysis of the properties of the iterative fine-tuning procedure\n* Release code and datasets to allow for easier adoption and validation", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Originality:\nThe approach of iteratively fine-tuning deep generative models using the cross-entropy method is novel. Related work on Bayes-adaptive RL is cited, but MACE differs in focusing on posterior inference rather than policy optimization. Its model-agnostic nature and reliance on simulation rather than true labels also distinguishes it from prior work.\n\nQuality:\nThe method is technically sound, with a clearly motivated objective function and update rule. The experiments thoroughly evaluate both quantitative metrics and qualitative samples from MACE across multiple domains. Limitations are discussed, including inference speed and reliance on simulation. The writing clearly separates proposed contributions from analysis of results.\n\nClarity:\nThe method is technically sound with a clearly motivated objective function. Appendix provides helpful details on model architectures, training procedures and hyperparameters. Thorough experiments evaluate quantitative metrics and sample quality across domains. Limitations around inference speed and simulation fidelity are discussed.\n\nSignificance:\nEfficient adaptation of internal models is an important capability for real-world robot learning. MACE provides a simple and general approach to achieve this goal for complex generative models. The method's flexibility across different models, tasks, and observation modalities demonstrates its usefulness for robotics researchers.\n\nRelevance:\nThe paper is relevant to the CoRL audience. It addresses a core robot learning problem using generative models, which are seeing increasing use for robotics. The experiments cover simulation and real-world tasks like grasping.\n\nLimitations:\nThe limitations around inference speed and simulation quality are adequately discussed. Testing on more complex observation spaces could further validate the generality.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "DUddCgFYgR", "reviewer": "Reviewer_KLCv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes an approach to adapt pre-trained generative models to approximate posterior distributions over task representations given new observations in robotic tasks. The approach, named model adaptation with the cross-entropy method (MACE), is based on an application of the cross-entropy method [24] to deep generative models. It is assumed that observations can be generated via deterministic simulations given samples from the generative model. A score function is then used to measure the similarity between simulated observations and the data. Implementations based on autoregressive models and VAEs are assessed in problems involving inverse kinematics and object shape inference from tactile feedback.", "summary_of_recommendation": "The paper addresses an interesting problem, but I have a few concerns regarding details of the methodology and experiments.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "### Sec. 2\n1. Regarding the claim in lines 73-75 on out-of-distribution data, it's not clear to me how the fact that MACE uses a similarity function would make a difference in these settings.\n\n### Sec. 3.2\n2. In line 115, it is assumed that the expected value of the score function w.r.t. the simulations is an estimate of the likelihood $p(\\mathbf{o}|\\boldsymbol{x})$. It is not clear what kind of \"estimate\" it refers to. Would it need to be unbiased, for example?\n3. In line 118, it is stated that simulators are deterministic, which would make the likelihood no longer correspond to a conditional probability density and become degenerate, making the original Bayesian inference formulation fall apart. I was wondering if the authors would see that as something leading to complications.\n4. Although inspired by Engel et al. [17], the formulation in Eq. 2 does not involve tempering as originally proposed [17, Sec. 3.1]. Have the authors considered using a tempering scheme?\n5. How can the hyper-parameter $q$ be selected in general for a new problem setting?\n\n### Sec. 4\n6. The problems this method addresses seem to also fall within a likelihood-free inference framework. I was wondering if the authors considered comparisons against this type of methods, such as the ones mentioned above.\n\n#### Sec. 4.1\n7. How many training points were used to train the prior model? How many different examples was the model tested on?\n8. What do the error bars in Fig. 1 correspond to (e.g., 1 std. deviation)?\n9. How are samples from the posterior obtained? Are they exact samples? Or are they obtained by some approximate inference scheme?\n\n#### Sec. 4.2\n10. Are the initial IK solutions provided by MACE to MoveIt collision-free? Are they usually close to the goal configuration, given that MACE approximates the posterior?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is mostly well written and addresses an interesting problem. However, I have a few concerns regarding details of the methodology and the experimental evaluation.\n\n**Strengths**\n* Presentation is mostly clear and well organised.\n* The method is scalable to problems involving high-dimensional domains and large datasets.\n* The method seems reasonably simple to implement within modern deep learning frameworks.\n* Experiments are demonstrated on problems in robotics where deep generative models have shown significant potential.\n\n**Weaknesses**\n* The proposed method seems to be a straightforward application of CEM [24] to deep generative models, which would make the technical contribution somewhat incremental. The only difference is that the optimisation step involved in CEM is now replaced by stochastic gradient descent, instead of involving closed-form solutions or an application of expectation-maximisation, as in the original method.\n* The method is only derived for deterministic simulations, and it's unclear if the method would be applicable and perform well with more general stochastic simulations (e.g., involving commonly present observation noise).\n* The only non-trivial baseline in experiments is CVAE, while other classic approaches in statistics and ML could be applicable to these problem settings, e.g., likelihood-free inference methods (see references below).\n\n**References**\n* Greenberg, D., Nonnenmacher, M. &amp; Macke, J.. (2019). Automatic Posterior Transformation for Likelihood-Free Inference. *Proceedings of the 36th International Conference on Machine Learning*, in *Proceedings of Machine Learning Research* 97:2404-2414. Available from https://proceedings.mlr.press/v97/greenberg19a.html.\n* Ong, V. M. H., Nott, D. J., Tran, M. N., Sisson, S. A., & Drovandi, C. C. (2018). Likelihood-free inference in high dimensions with synthetic likelihood. *Computational Statistics & Data Analysis*, 128, 271-291.\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "892obSryHwB", "reviewer": "Reviewer_KQ64", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a framework called MACE (Model Adaptation with the Cross Entropy Method). The goal of MACE is to adapt the parameters of generative models. Using so-called cross entropy method, MACE fine tunes deep generative models. Forward simulations of the environments and some approximations of similarity functions as probability distributions, MACE provides a method to use available simulators in robotics, for the purpose of fine tuning deep generative models via approximate Bayesian inference. Two robotic experiments are considered in simulation/dataset. First is inferring object shapes by grasping, and another one is inverse kinematics with obstacles. Experiments demonstrate versatility and usability of the proposed framework for certain robotic tasks in simulation.", "summary_of_recommendation": "My recommendation is a weak reject. There are rooms for significant improvements regarding the derivations made in the proposed method. Solid justification of each steps and assumptions would be needed, as typical in Bayesian inference papers. Also, there is no \"wow factors\" in the experiments, and hardware experiments could verify that the use of simulation for the likelihood term, and the assumptions made here, if they would work in the real world.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "See the section above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I noted the following strengths:\n\n- the paper is generally polished very well. Especially the structure of the method section, and algorithm 1 are clear and easy to understand.\n\n- coping with out-of-distribution scenarios is interesting, and therefore, the benefits over CVAE are made clear. \n\n- adapting existing bayesian inference methods for robotic tasks, is a relevant topic at CoRL. The ideas of using simulation to perform bayesian inference are also trendy topic in robotics as well.\n\nOn the other hand, I noted the following potential shortcomings:\n\n- Several assumptions made during derivations, may need better justifications either empirically or theoretically.\n\nFor example, the assumption that the simulators are deterministic and thus the expectation is replace with a single term, the assumption that the importance sampling term in equation 2 can be simply removed, the assumption that the score function is an estimate of the observation given tasks, etc, should be better justified. Could be interesting to see the influence of these assumptions in the experiments, and further, explain in more depth, why these can be justified.\n\n- Experimental results are sometimes confusing. I also wonder if there was an option to use more standardized benchmarks for generative models in robotics. There is no hardware experiments, and thereby, the validation that this method would work in real world is missing. \n\nFor example, in figure 1, CVAE significantly outperforms MACE for in-distribution scenarios. The relevance of sample diversity might be explained more intuitively.  In table 2, MACE alone is significantly less accurate than MoveIT. Given that no standardized benchmark tasks are considered to compare the generative models, some hardware experiments, validating the method in real world setting would make the paper stronger. The last point is especially interesting, given that the paper builds on simulation based inference methods, and there might be needed interplay between simulation and real world.\n", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a method to finetune a trained deep generative model with newly-obtained observation evidence. The method is demonstrated on a simulated grasping pose estimation domain and a motion planning domain both in simulation and the real world. 

The reviewers have raised a number of concerns, including the assumption of having access to a deterministic sample-to-observation simulator and the unfavorable trade-off between performance and computation time. Reviewer KQ64 has also pointed out that the derivation would require further clarification.

At the same time, the reviewers have all agreed that the general problem of adapting a trained generative model to new evidence is of great interest to the community and that the proposed method is interesting and well-presented.

Therefore based on the discussion and the comments above, I recommend accepting the paper as a poster presentation. I recommend the authors to strengthen their experimental evaluation, especially towards more challenging real-world domains, and incorporate the clarification comments by the reviewers.
Decision: Accept (Poster)