Bayesian Object Models for Robotic Interaction with Differentiable Probabilistic Programming
Authors: Krishna Murthy Jatavallabhula
Miles Macklin
Dieter Fox
Animesh Garg
Fabio Ramos
Venue: CoRL 2022 Poster (2022)
Keywords: Simulation
Probabilistic programming
Differentiable physics
Bayesian inference
Causal inference

Abstract: A hallmark of human intelligence is the ability to build rich mental models of previously unseen objects from very few interactions. To achieve true, continuous autonomy, robots too must possess this ability. Importantly, to integrate with the probabilistic robotics software stack, such models must encapsulate the uncertainty (resulting from noisy dynamics and observation models) in a prescriptive manner. We present Bayesian Object Models (BOMs): generative (probabilistic) models that encode both the structural and kinodynamic attributes of an object. BOMs are implemented in the form of a differentiable probabilistic program that models latent scene structure, object dynamics, and observation models. This allows for efficient and automated Bayesian inference -- samples (object trajectories) drawn from the BOM are compared with a small set of real-world observations and used to compute a likelihood function. Our model comprises a differentiable tree structure sampler and a differentiable physics engine, enabling gradient computation through this likelihood function. This enables gradient-based Bayesian inference to efficiently update the distributional parameters of our model. BOMs outperform several recent approaches, including differentiable physics-based, gradient-free, and neural inference schemes. Further information at: https://bayesianobjects.github.io/

Bayesian Object Models for Robotic Interaction with Differentiable Probabilistic Programming
Krishna Murthy Jatavallabhula* Miles Macklin', Dieter Fox', Animesh Garg', and Fabio Ramos
INVIDI
Abstract: A hallmark of human intelligence is the ability to build rich mental models of previously unseen objects from very few interactions. To achieve true, continuous autonomy, robots too must possess this ability. Importantly, to in- tegrate with the probabilistic robotics software stack, such models must encap- sulate the uncertainty (resulting from noisy dynamics and observation models) in prescriptive manner. We present Bayesian Object Models (BOMs): gen- erative (probabilistic) models that encode both the structural and kinodynamic attributes of an object. BOMs are implemented in the form a differentiable probabilistic program that models latent scene structure, object dynamics, and ob- servation models. This allows for efficient and automated Bayesian inference samples (object trajectories) drawn from the BOM are compared with small set of real-world observations and used compute likelihood function. Our model comprises differentiable tree structure sampler and differentiable physics en- gine, enabling gradient computation through this likelihood function. This enables gradient-bard based Bayesian inference to efficiently update the distributional parame- ters of our model. BOMs outperform several recent approaches, including differ- entiable physics-based, gradient-free and neural inference schemes. (Webpage)
Keywords: Simulation, Probabilistic programming, Differentiable physics
Introduction
When presented with novel object, humans are quickly able discover number of structural properties and functional attributes. As children, they play around with objects, garnering evidence from each interaction to gradually build powerful mental representations of objects, their constituent parts, and the physical constraints imposed by their configuration. Most robots, however, rely on pre-specified kinodynamic object models for interaction planning.
We propose interactive mechanism by which a autonomous agent can build "Bayesian object models" (BOMs) probabilistic representations of scene/object structure and dynamics an analysis-by-synthesi framework [1]. In our study, robots are equipped with rich world models (forward predictive models of kinodynamics) in the form of differentiable probabilistic programs. We formulate the construction of BOMs as a Bayesian inference problem over this probabilistic program (generative model), given small number of real-world interactions (evidence).
probabilistic program is program written in special purpose language- probabilistic pro- gramming language that augments general-purpose programming languages by providing explicit constructs for sampling and conditioning over random variables. The primary goal of PPL is not just to provide a flexible specification mechanism for probabilistic models; but also to automate Bayesian inference. An increasing number of modern PPLs [2-5] natively support auto- matic differentiation, due to the widespread adoption of differentiable computing in machine learn- ing applications [6-8]. We refer to programs written such PPL leverage gradient-bar based probabilistic inference-as differentiable probabilistic programs.
Work done as part of an internship NVIDIA.
6th Conference Robot Learning (CoRL. 2022), Auckland, New Zealand.

---

Differentiable probabilistic program
(c) Robot interacts with oljects the scame (b) Olterress object/purt brajectoris
(c) infer Bayesian World Model (swiee) to best explairo the observations by learning
a probubiility distribution over scene structure (trec) and physical properties
Figure 1: Bayesian Object Models (BOMs): Robots interacting with the world observe the dynamics of previously unseen objects. We encode explicit understanding of the world (a gemerative model) the form probubilistic program that models latent scene structure, dynamics, observation models. Probabilistic programming enables us to automate several aspects of Bayesian inference, resulting in the construction of BOMs from a small number of robot-cnviromment interactions (8-20 actions in our experiments).
Our generative model is one such program, relating latent scene structure (often a tree) and kino- dynamic attributes governing object states. The state-space trajectories generated by the model, in conjunction with small set of trajectories obtained by interacting with the scene, are used to formu- late an observation likelihood for Bayesian inference. Importantly, this likelihood differentiable, allowing for efficient gradient-based probabilístic inference schemes. This is in contrast the set of approaches recently beind unified under the term iinallation-based inference SBI, where the generative model includes black-box simulator.
While object 'dynamics' can be differentiated through (enabled by the emerging volume of work on differentiable physics simulation [9-18]), these methods cannot compute gradients with respect to the object's 'structure'. We overcome this by devising a differentiable procedure for sampling tree- structured random variables. Our generative model includes the following latent variables: graph structure, edge types. object physical properties. Our framework is the first learn distribution over both the scene structure and its physical parameters essential for interoperation with current robot software stacks that rely heavily on probabilistic modeling and inference.
Related work
Many approaches have tackled interaction planning for objects with unknown articulations. They typically only capture point estimates of articulation parameters [19-28], probabilistic models of articulation kinematies [29-31]. Recent approaches leverage differentiable simulation for recovering articulation parameters [32-35, 16). Our BOMs capture distributions over tree structures of objects, and their kinodynamic parameters via efficient gradient-bart probabilistic inference.
Our work related to the causal inference and structure discovery literature [36-44], specifically building on the characterization of directed acyclic graphs (DAG) presented [39-41], extending these ideas to differentiably sample tree-structures. Other approaches to causal structure discovery include neural inference [37, 38, 42-44] and symbolic regression [45-47]. We rely explicit, differentiable parameterization and demonstrate its efficacy in robotics problem domains.
Probabilistic programming languages (PPLs) have long been used in the Bayesian inference com- munity for model specification and inference [48-50] Extensive efforts into modern differentiable programming frameworks [2-5] has also resulted in the development of modern PPLs with native support for autodifferentiation. We use the Pyro PPL [2] due to its interoperability with Pytorch [6].
Modern PPLs have yet to significantly impact problem domains in robotics. 3DP3 [51] proposes generative model depth images to recover distributions over object poses; VS [52] enables unsupervised visual servoing; and Mirchev et al. [53] enable 3D SLAM.
Simulation-based inference (SBI is recently- term [54] that attempts unify multiple lines of work leveraging simulation models for probabilisitic inference. Originally referred to as likelihood-free inference, SBI approaches have thus far assumed black-box simulators. In this work, however, we propose a new approach to SBI, leveraging white-box simulators interfaced with dif- ferentiable probabilistic programs.

---

p(r
Tobs
P(T
8 enfibeeniable sre sampling
Diffcrentiable physics
Sampled trajectories
Likelihood
Observed trajectoriec
(IJ)
engine
Figure 2: Differentiable probabilistic programming for learning BOMs: A probabilistic program (shown above) samples random variables encoding a scene structure (a graph (). physical constraints J, and ob- ject physical properties e. These sampled variables used to initialize differentiable physics engine that simulates the dynamics of the environment and computes likelihood function. A key idea here involves dif- graph-structured variables, which allows us to perform gradient-bast
ferentiable sampling procedure (variational) inference of the posterior over the random variables of interest.
Bayesian Object Models
Problem definition: A robot interacting with an object (containing multiple moving parts) applies control actions u and records part trajectories Tobs (here, point cloud streams). We represent the scene as graph (specifically, tree) g = (V,E), E). as is the norm in simulation engines for robotics. Each node v E V in the graph denotes an object part (if o is leaf node), an object part collection or an entire object (if v is an internal node). Each edge E & in the graph denotes a joint-type Je E (rigid, prismatic, revolute) with parameters 0.. Let & denote the set of all physical parameters over all edges in the graph, and J denote an enumeration of joint types, i.e., {01,... 111) }. J (2, . } We aim recover distribution over scene structures, and the physical parameters of the structure @ (object relationships, joint types, joint parameters) that best explain the observations i.e., p(G, J, + Tobs).
p(9, J, Tob p(T|), J, θ) (()) p( ((O)
(n) is normalizing constant)
(1)
Bayesian Object Models (BOMs) are generative (probabilistic) models representing the distribu- tion (,(T J, θ) over object structure and kinodynamic parameters. Specificallyy, BOM is differentiable probabilistic program 96 parameterized by ø. Samples T drawn from this model rep- resent object part trajectories, and are used to compute likelihood functions Ps (T(9 J, θ) (differ- entiable w.r.t. Φ). They can therefore be employed in gradient-based probabilistic inference schemes to infer o conditioned on real-world observations Tabs-
3.1 Likelihood function
An optimal choice of model parameters & for our BOM should best explain the observed trajec- tories Tots (evidence). Hence, straightforward choice of likelihood function would one where the maximum likelihood estimate amounts minimizing mean-squared error between sampled trajectories (from the model) and observed trajectories. We therefore assume that observation errors are Gaussian distributed with inverse variances 3., which results in the following likelihood
p(TotslT) = p(Toss G.rasslG.,, G. = II exp (-Bell - -obb,, where T; 9o(T19. (2) t=0
Gradient-free Bayesian inference in the parameter space & of this model is computationally expen- sive and performs poorly in practice (see Sec. 4). The tree involves several discrete-structtuee random variables. Further, each edge in G has several properties (joint type, friction, damping, axis parameters), resulting in high sample complexities. BOMs alleviate this complexity by presenting a differentiable likelihood computation scheme, including a differentiable greedy mechanism for sampling tree-structured random variables, differentiable physics engine model time-evolution of object state, and observation models.
3.2 Choice of prior distributions
We assume the following choices of prior distributions within our probabilistic program. These families of prior distributions general and are only used scaffolding for our inference engines. One may also explore other specialized choices in return for further performance improvements.

---

P C A PEC
Figure Our differentiable tree sampling procedure builds the differentiable DAG sampler proposed in [41] and imposes ad- ditional constraints to enforce tree structure. (Left): A tree and its upper triangular adjacency mat (here, P = I. (Right): Adding the dashed edge makes the tree DAG. Our sampling procedure per- forms opportunistic rounding step prevent edges like this
Prior over tree structure: We assume that all scene structures (trees) are equally likely. This results in uniform distribution over the set of (N 1)(N-1) trees with directed edges rooted at given node (Cayley's s theorem). The prior probability of any sampled tree with N nodes (excluding the root) is thus, P( G) (N+1)(N-1))
Prior over joint types: We parameterize joint types using categorical distribution p(J) Cat(zigid, prisiippppmmmttt
revolute]. While gradient-based optimization through categorical distribution techically requires enumeration (i.e., duplicating the computation graph for all possible choices of J), we find that using softmax function and a straight through estimator instead works well in practice.
Prior over kinodynamic parameters: Each edge has continuous physical parameters indicating its friction, damping, and axis parameters (a 6D screw vector). Each node has center-of-ng -fasssss parameter. These parameters over the entire graph are lumped into vector O which is assumed to be a standard normal distribution θ ~ OO, I). We also attempted using uniform distribution and observed that this leads to slightly inferior performance.
3.3 Generative Model
Our generative model ( (,, I, Q) relates the observed trajectories Tabs to latent scene structure g and kinodynamic parameters J, O. We implement all modeling aspects in the Pyro PPL [2, 4].
Overview: Our generative model samples object part trajectories follows. First, indepen- dently sample graph structure G and kinodynamic parameters J, θ from the prior distributions P(G), P(T), and P(@) respectively. Next, we evolve the object state using physics engine y given external input forces. The physics engine produces object states, from which we sample a trajectory sequence r by adding white Gaussian observation noise.
The key challenge in constructing this generative model is ensure that the output trajectories all differentiable with respect to the parameters of gø. We do this by (a) proposing differentiable mechanism to sample tree-structured variables, and (b) leveraging advances in differentiable physics simulation to compute gradients through the physics engine V.
Recently, multiple schemes have been proposed to efficiently sample DAG-structured random variables [39-41]. All of these ap- proaches rely the fact that graph g is DAG if and only if its adjacency matrix A is nilpo- tent [ This means A can be factorized into a permutation matrix P and an upper-triangular matrix U, such that A = PU PT [39, 41].
Efficient differentiable sampling tree-structured variables:
Our differentiable tree sampler based on the differentiable DAG sampler DP-DAG [41], extending it using greedy projection step to cast the DAG tree. We first sample a directed acyclic graph (similar DP-DAG [41]) and present greedy projection step to obtain tree.
1. Differentiable DAG sampling: Each element of U is sampled using the Gumbel-Softman trick; Wi (j 2 1 & i) are reparameterized as
Tso +og (TT,)
exp
Uij
Σg exp
"/e -og (ag)
'A square matrix A is nilpotent if A 0 for positive integer In terms of graphs this means that powers of adjacency matrices (which indicate n-hop neighbours) do not result in cycles.

---

where y are drawn from the Gumbel distribution Y ~ Gumbel(O, 1). Autodiffereniiation through this is enabled by the straight-through estimator [56, 57J. While the above approach results in a (soft) adjacency matrix, a hard thresholding typically results in a DAG.
2. Opportumistic rounding: To reduce a DAG tree, one needs impose additional constraints; specifically that unique path exists to each node in the DAG from the root node. The entries Maj E U must be structured such that, all entries Ukkj (k: > i) are set to zero, if Mij is 1. We achieve this by an opportunistic rounding scheme; for each column j. we find the first entry unj in the soft upper-triangular matrix U above a threshold and round it up to 1, while simultaneously rounding all entries Umj (m < () to zeros. If no entry in column j exceeds 8, we pick the first non-zero value in the column to round up to Gradients through the rounding step are propagated, again, using the pathwise derivative (straight-through estimator). This scheme provably samples tree.
Differentiable physics engine: Given a graph structure 9 and kinodynamic parameters J, O, differentiable physics V engine simulates the time-evolution of object states. Our discrete-time physics engine simulates phenomena such as articulations, mass-spring systems, clectrodynamics, and produces a trajectory т. We employ a deterministic dynamics model all of the stochasticity in our generative model arises due to the latent scene structure and kinodynamics. To account for any unmodeled dynamics effects, we add an observation noise term to each state in the trajectory. We find that such stochastic observation model works across simulated environments, and also in real-world scenes. A distinct aspect of our approach (compared to prior work in differentiable physics simulation) is that our differentiable physics engine is embedded within a PPL. This allows the additional flexibility of sampling from the generative model, conditioning on observed values of random variables, and also computing gradients through the sampled execution trace.
Algorithm 1: Learning BOMs
Imput: Probuhilistic model W Price, Obervation
Result: BOM inferred paramesers & while comeerged do
Inference loop
① 4- SAMPLE-STRUETEES
@ <- SANPLE- KINUDYNAMICS
Tase SASTLLE-DISSUVVITINNNS
loss e- COAFTTTSUUSEL..OOSS.....L..
Updatee We : end
Za Update ROH
3.4 Gradient-based Bayesian inference of structure and scene parameters
Having specified our model ge and appropriate priors over the latent variables, we perform Bayesian inference to recover the posterior p(g. J, θ Tatx) given the observed trajectories Tahs (Eq 1).
A key benefit of using modern PPLs (such as Pyro in our case) is that nearly all of the effort is concentrated on model, prior, and likelihood specification. As such, inference is automated by a single function call, which enables us to experiment with a variety of inference schemes. We exper- iment with the following inference schemes, each of which presents trade offs between modeling assumptions, speed, and solution quality:
SVI (Stochastic variational inference 3): The posterior is approximated using a well- behaved distribution (called the variational distribution), allowing us to formulate lower bound (ELBO), and optimize by gradient descent.
HMC (Hamiltonian Monte Carlo): A gradient-based MCMC scheme that uses Hamilto- nian dyanmics to propose samples close following target distribution.
NUTS (No-U-Turn Sampler): A variant of HMC that adaptively tunes step sizes of the HMC sampler, reducing manual tuning effort.
*SVI is stochastic in that it computes an approximation to the ELBO using a few samples drawn from the generative model. Eliminating the stochasticity would entail computing the full ELBO, which is computation- ally intractable in most real-world problems of interest.
5

---

OpenCabineDDoor Graph MSE
OpenCabinesDrawez Graph MSE
Approuch
Oracie?
Success( (6)
Joint( %
IdE
Success (%)
Joint %
IdE
Disconnected
0.772
Clique
X
0.188
0.229
Owrs Privileged
(SNPE)
SBI (SNLE)
19.05
100
0.53
7.14
100
0.54
Diff, sim.
95.23
0.01
100
Diff. sim. (tail inir)
9.52
100
0.93
100
BOM-SVI (Ours)
0.019
BOM-HMC (Ours)
100
0.081
90.58
0.23
100
0.063
93.92
0.21
ROM-NUTS (Ours)
100
0.080
90.33
0.22
100
0.061
93.58
0.21
Table 1: Evaluating structure discovery and kinodynamic prediction capabilities of BOMs. Success(%) in- dicates task success. Graph MSE measures the mean-squared error between the predicted and graphs. Joint(%) evaluates the fraction joints that were correctly characterized, IdE an 'ddntification error' metric that measures the absolute relative error in kinodynamics parameters. We see that BOMs achieve higher suc- cess rates compared to privileged strategies (SBI, Diff. sim.) that assume access to ground truth object models.
Experimental Results
We conduct experiements to investigate the following questions about BOMs.
1. How effective are Bayesian object models (BOMs) at probabilistically modeling the struc- ture of objects with unknown kinodynamics?
2. How effective are BOMs at characterizing distributions over kinodynamic parameters?
3. How do BOMs help real robots interact with previously unseen objects?
4. Are the inferred BOMs useful in downstream tasks such tracking object parts?
4.1 Evaluating the structure and kinodynamic prediction capabilities of BOMs
We first evaluate the quality of tree-structured distributions estimated by BOMs for variety of inference techniques, and against various inference techniques and relational inference schemes representative of current art.
To learn BOM, we first record a 2-5 second pointcloud sequence of an action (external force) being applied on an object of interest. Each pointcloud passed to an object part segmentation network. The tracked parts are used initialize prior distributions for sampling center of mass and axis parameters. For each object, use 8-20 such interaction trajectories to learn the parameters of the BOM.
Experiment Setup:
We use 3D assets provided by PartNet-Mobility [58]; particular the Cabinet class for results reported in this section. We also use the Dishwasher and Microwave classes for additional experiments. These assets have ground-truth 3D meshes and kinodynamic parameters. Further, to evaluate robotic interaction capabilities, tasks from the ManiSkill [59] benchmark. While the tasks in Maniskill use 67 train and 20 test instances, we modify this setup and use 42 instances for testing, owing to the sample efficiency of BOM model optimization. We use the remaining (45) instances to compute statistics that inform our data-driven prior. In all, we experiment on 142 simulated object instances from categories commonly found in household environments.
Dataset:
use the OpenCabinetDoor and OpenCabinetDrawer
Approachess evaluated: We consider three broad classes of approaches. Blackbox approaches include recent work in simulation-based inference [60, 61] implemented as part of the SBI tool- box [62]. As a strong baseline that assumes access to privileged information, we also use a differen- tiable simulator [63] equipped with an oracle physically- accurrae kinematic and mesh model of the scene, albeit with unknown dynamics parameters. Our suite of inference engines for learning BOMs include SVI, HMC, and NUTS.
Discussion:
Table 1 compares all the approaches number of metrics. Graph MSE evaluates the mean-squared error between the true adjacency matrix and the predicted adjacency distribu- tion. Join((%) computes the accuracy by comparing the estimated joint type with the greatest like- lihood against ground-truth. IdE measures the relative identification error in physical parameters (for Bayesian approaches, we take the median estimate of each parameter). Success %) measures

---

30
-1.0
-0.
(a) Despite poor initil guess (blue). SVI converges (green) to the true distribution (orange)
(b) close-up view the converged distribution (blue)
Estimated posterior over friction
Estimated damping HMC
posterior
parameters by HMC
distribution
Figure 4: Quality of physical parameters estimated by BOMs.
Qualitative results: Joint types plotted against the corresponding axis. Yellow indicates pris- matic joint, while magenta indicates revolute joint.
Figure 5:
the percentage of task success rates in ManiSkill. Note that Diff. sim. (and variants). BOM-SVI, BOM-HMC, and BOM-NUTS all leverage gradients of the observation likelihood w.r.t. the param- eters of the generative model. However, they have completely different objectives. While Diff. sim. variants attempt to find local minimum of the observation (negative log) likelihood, BOM variants attempt to estimate a posterior distribution that can be used to draw samples. We use the median from a set of 10000 samples drawn from BOM-HMC and BOM-NUTS as the representative point for evaluation.
BOMs infer accurate kinodynamic parameters. We see from the IdE columns in Table that BOMs infer physical parameters more accurately compared to gradient-free probabilistic inference methods (SBI-SNPE, SBI-SNLE). The only approach that outperforms BOMs on this metric is differentiable simulation however, this approach uses privileged information the form of ground- truth meshes, parts, and centers of mass. The primary source of error in BOMs is precisely because of errors induced estimating centers of mass from pointcloud observations instead.
BOMs accurately characterize distributions over dynamics parameters. To evaluate the learned distribution of dynamics parameters in BOMs, we plot the inferred distributions over friction and damping parameters of a cabinet asset from PartNet in Fig. 4(c,d). Notice how the density of samples drawn from BOM-NUTS is concentrated near the true point estimate.
BOMs are well-suited for robotic interaction. From the Success(%) columns in Table 1, and the qualitative results in Fig. 5, observe that BOMs accurately capture kinematic structure, which enables high success rates on robotic interaction tasks. We assume known, deterministic pol- icy is employed for each of the tasks. Object kinodynamic parameters in the (nondifferentiable) SAPIEN [64] simulator are initialized by sampling from the learned BOMs. Therefore, task success rates depend crucially on accurate joint type classification. For example, misclassifying a revolute joint cabinet door as a rigid joint instead will render the OpenCabinetDoor task impossible.
BOMs robust to initialization errors. To investigate the robustness of BOMs and to assess the sensitivy of differentiable simulators (strongest-performing approach the IdE metric, we devise a baseline "differentiable simulation (tail init)". In this approach, we wilfully set the initial guesses of a gradient-descent routine to be sampled from the tail of the true distribution. We see, at once, that the differentiable simulation approach is quite brittle success rates drop by over 80%. BOMs, on the other hand, are robust to such initialization errors. Fig. 4(a.b) show the learned distribution over damping coefficients of a cabinet from PartNet. Notice how the initial guess distribution in
*Note that in our setting, task failures are primarily due to structure and/or kinodynamic identification errors and not due to the interaction policies themselves. Action selection (i.c., choosing the amount of force to be applied, and where apply it) good avenue for future work.
7

---

RCED camera
Segmentadionas and node nissusstion
agpcency matik
Interied hoft)
Sounding adjaeency mabri tree
Predicted ol Strecture
Figure 6: Real-world BOMs from robot interacting with cabinet to infer distribution over its structure.
panel (a) has an extremely high-variance (a poor initial guess for variational inference), but upon optimization, the learned distribution is near-identical to the true underlying distribution.
4.2 Structure discovery in other physical systems
We also compare the ability of BOMs to re- cover structure against modern causal structure discovery methods- neural relational inference (NRI) [37] and variant, factorised-NRI (f-
Approach
Known structure
Learned structure
NRI [37]
2.98c-8
5.2c-5
f-NRI [38]
2.33e-8
4.7c-5
NRI) [38]. We consider the spring-mass and charged particle system environments in [37]. Given set of observed trajectories of particles (coupled by unobserved springs or charges), the aim is to recover the latent structure that best ex- plains the trajectories. The quality of the recovered structure evaluated by using the probabilistic model to predict future trajectories of each particle, and evaluating mean-squared error with respect ground-truth. NRI [37] recovers a point estimate of the underlying scene structure. f-NRI [38] an improved variant that extends NRI to model multiple interaction types. The 'known structure' setting only recovers physical parameters (masses, spring constants, charges), whereas the 'learned structure' setting recovers both structure and physical parameters. Notice how BOM-SVI perfroms significantly better than both these approaches.
Ours
2.21e-8
1.78e-6
Table 2: Structure discovery evaluation
4.3 woal-world experiments
We also evaluate the structure learning abilities of BOMs in real scene. In this experiement, a robotic manipulator (Franka Emika Panda arm) interacts with a cabinet. Color and depth im- ages are captured by a pair of Intel RealSense cameras. Part segmentation on the color images currently performed manually for the first frames (and subsequently tracked using the Lucas- the node structure for BOM. In Fig.
Kanade (lgorithm)--nitialll setup and the inferred scene structure. While the observed trajectory (a single interaction) is insuf- ficient to explain the structure of all movable links in the object, we observe that the tree structure recovered quite accurately models the moving components of the object.
visualize the experiment
4.4 Deploying BOMs for object part tracking
Approach
MSE (metres)
Const. vel. Const. BOM-SVI
10.128
We evaluate the applicability of learned BOMs in part tracking. Once trained, BOMs can be used to analyze interventions, predict- ing the outcomes of novel actions. Table to the right shows the performance of BOMs on tracking the parts of cabinet by plot- ting the mean squared error of the locations of the center of masses. We compare this against two baseline approaches, using constant velocity model and constant acceleration model respectively (common choices in Bayes filters).
37.025
0.1704
Table 3: Tracking object parts with BOMs
Discussion and Limitations
This work presented BOMs an approach to build computational models of everyday objects while characterizing uncertainty over structural and kinodynamic parameters. BOMs are effective and sample-efficient; they be built from even single real-world interaction. However, BOMs cur- rently assume static kinematic tree structures, and rely generic inference schemes (SVI. HMC), and require 7-30 minutes of runtime per interaction sequence. We refer the interested readers to our supplemental material for more details.

---

Acknowledgments
Software available from KMJ acknowledges support from the NVIDIA fellowship.
References
[1] D. Kersten, P. Mamassian, and A. Yuille. Object perception as bayesian inference. Annu. Rev. Psychol., 55:271-304, 2004.
[2] E. Bingham, J. Chen, M. Jankowiak, F. Obermeyer. N. Pradhan, T. Karaletsos, R. Singh, P. Szerlip, P. Horsfall, and N. D. Goodman. Pyro: Deep Universal Prohabilistic Programming. Journal of Machine Learning Research, 2018.
[3] M. F. Cusumano- Towner, F. A. Saad, A. K. Lew, and V. K. Mansinghka. Gen: A general-purpose prob- abilistic programming system with programmable inference. In Proceedings of the 40th ACM SIGPLAN Conference o Programming Language Design and Implementation. ACM, 2019.
[4] D. Phan, N. Pradhan, and M. Jankowiak. Composable effects for flexible and accelerated prohahilistic programming numpyro. International Conference on Probabilistic Progranming, 2020.
[5] J. Salvatier, T. V. Wiecki, and C. Fonnesbeck. Probabilistic programming in python using pymc3. PeerJ Computer Science, 2:c55, 2016.
[6] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang. Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and
A. Lerer. Automatic differentiation in pytorch. 2017.
M. Abadi, A. Agarwal, P. Barham, Brevdo, Z. Chen, C. Citro, G. Corrado, Davis, Dcan,
M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser,
M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals. P. War- den, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL https: http://wwww////// www.tensorflo...oooomooo/ wow.esooooooo../...o.... tensortlow.org.
[8] J. Bradbury, R. Frostig, P. Hawkins, M. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. Van- derPlas, S. Wanderman-Milne, and Q 7hang. JAX: composable transformations of Python+NumPy pro- grams, 2018. URL http: /tgtt/bggttt///g// com/ google/ogeeeeooooo jax.
F. de Avila Beleute-Peres, K. Smith, Allen, J. Tenenbaum, and J. Z. Kolter. In Advances Neural Information Processing Systems (NewrIPS), 2018.
[10] J. Degrave, M. Hermans, J. Dambre, et al. A differentiable physics engine for deep learning in robotics. Frontiers in nearorobotics, page 6, 2019.
[11] D. Hahn, P. Banzet, J. M. Bem, and S. Coros. Real2sim: Visco-clastic parameter estimation from dynamic motion. ACM Transactions on Graphics (TOG), 38(6):1-13, 2019.
[12] Y. Hu, Liu, A. Spielberg,
Tenenbaum, T. Freeman, J. Wu, D. Rus, and W. Matusik. Chain- queen: A real-time differentiable physical simulator for soft robotics. In 2019 International conference on robotics and automation (ICRA), pages 6265 6271. IEEE, 2019.
[13] Y. Hu, I.. Anderson, T.-M. Li, Q. Sun, N. Carr, J. Ragan-Kelley, and F. Durand. Difftaichi: Differentiable programming for physical simulation. 2020.
[14] Y.L.. Qiao, Liang, V. Koltun, and M. Lin. Scalable differentiable physies for learning and control. In International Conference on Machine Learning, pages 7847-7856 PMLR, 2020.
[15] Y.L. Qiao, J. Liang, V. Koltun, and M. C. Lin. Efficient differentiable simulation of articulated bodies.
In International Conference on Machine Learning, pages 8661-8671. PMLR, 2021.
[16] E. Heiden, D. Millard, E. Coumans, Y. Sheng, and G. S. Sukhatme. NeuralSim: Augmenting dif- ferentiable simulators with neural networks. In IEEE ICRA, 2021. URL https://github. com/ booogee-reeearchhtii------- ii simmlatoor.
[17] J. Krishna Murthy, M. Macklin, E. Golemo, V. Voleti. L. Petrini, M. Weiss, B. Considine, J. Parent- Lévesque, K. Xie, K. Erleben, L. Paull, F. Shkurti, S. Fidler, and D. Nowrouzezahrai, gradsim: Differen- tiable physics and rendering engines for physical parameter estimation from video. In ICLR, 2021.

---

[18] K. Werling, D. Omens, J. Lee, I. Exarchos, and C. K. Liu. Fast and feature-complett differentiable physics engine for articulated rigid bodies with contact constraints. In Robotics: Science and Systems, 2021.
[19] S. Niekum, S. Osentoski, C. G. Atkeson, and A. G. Barto. Online bayesian changepoint detection for articulated motion models. In 2015 IEFE International Conference on Robotics and Automation (ICRA), pages 1468-148 1475. TEFE, 2015.
[20] X. Li, H. Wang. L. Yi, L. J. Guibas, A. L. Abbott, and S. Song. Category-level articulated object pose estimation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5706-3715, 2020.
[21] Y. Weng, H. Wang, Q. Zhou, Y. Qin, Y. Duan, Q. Fan, B. Chen, H. Su, and L. J. Guibas. Captra: Category- level pose tracking for rigid and articulated objects from point clouds. In CVPR, pages 13209-13 13218, 2021.
[22] A. Jain and S. Nickum. Learning hybrid object kinematics for efficient hierarchical planning under certainty. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5253-5260. IEEE, 2020.
[23] A. Jain, Lioutikov, C. Chuck, and S. Nickum. Screwnet: Category-independeet articulation model estimation from depth images using screw theory. In IEEE ICRA, 2021.
[24] N. Heppert, T. Migimatsu, B. Yi, C. Chen, and J. Bohg. Category-inddeendeet articulated object tracking with factor graphs. arXiv, 2022.
[25] S. Y. Gadre, K. Ehsani, and S. Song. Act the part: Learning interaction strategics for articulated object part discovery. In CVPR, 2021.
[26] K. Mo, I.. J. Guibas, M. Mukadam, A. Gupta, and S. Tulsiani. Where2act: From pixels to actions for articulated 3d objects. In CVPR, 2021.
[27] K. Mo. Y. Qin, F. Xiang, H. Su, and L. Guibas. 02o-afford: Annotation-fiee free large-scale object-object affordance learning. In CoRL, 2022.
[28] S. Pillai, M. R. Walter, and S. Teller. Learning articulated motions from visual demonstration. RSS, 2015.
[29] Sturm, C. Stachniss, and W. Burgard. A probabilistic framework for learning kinematic models of articulated objects. Journal of Artffciial Intelligence Research, 41:477-526, 2011.
[30] B. Abbatematteo, S. Tellex and G. Konidaris. Learning to generalize kinematic models novel objects. In CoRL, 2019.
[31] A. Jain, S. Giguere, R. Lioutikov, and Nickum, Distributional depth-based estimation of object articu- lation models. CoRL, 2022.
[32] M. Lutter, C. Ritter. and J. Peters. Deep lagrangian networks: Using physics model prior for deep learning. ICLR, 2018
[33] M. Raissi, P. Perdikaris, and G. E. Kamiaaakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
[34] G. Sutanto, A. Wang. Y. Lin, M. Mukadam, G. Sukhatme, A. Rai, and F. Meier. Encoding physical constraints differentiable newton-culer algorithm. In Learning for Dynamics and Control, pages 804- 813. PMLR, 2020.
[35] M. Ding, Z. Chen, T. Du, P. Luo, Tenenbaum, and C. Gan. Dynamic visual reasoning by learning differentiable physics models from video and language. NEURIPS, 2021.
[36] P. Battaglia, R. Pascanu, M. Lai, D. Jimenez Rezende, al. Interaction networks for learning about objects, relations and physics. NEURIPS, 29, 2016.
[37] T. Kipf, E. Fetaya, K.-C. Wang, M. Welling, and R. Zemel. Neural relational inference for interacting systems. In ICML, 2018.
[38] E. Webb, B. Day, H. Andres-Teree and P. Lió. Factorised neural relational inference for multi-interaction systems. arXiv, 2019.
[39] C. Cundy, A. Grover, and S. Ermon. Bed nets: Scalable variational approaches for bayesian causal discovery. NEURIPS, 34, 2021.
10

---

[40] L. Lorch, J. Rothfuss, B. Schölkopf, and A. Krause. DiBS: Differentiable Bayesian structure learning. NEURIPS, 2021.
[41] B. Charpentier, S. Kibler, and S. Günnemann. Differentiable dag sampling. In ICLR. 2022.
[42] Y. Li, A. Torralba, A. Anandkumar, D. Fox, and A. Garg. Causal discovery in physical systems from videos. NEURIPS, 2020.
[43] X. Zheng, B. Aragam, Ravikumar, and E. P. Xing. DAGs with NO TEARS: Continuous Optimization for Structure Learning, In Advances in Neural Information Processing Systems, 2018.
[44] Nasa: Neural articulated shape approximation. Proceedings of the European Conference on Computer Vision, 2020.
[45] M. D. Cranmer, Xu, P. Battaglia, and S. Ho. Learning symbolic physics with graph networks. arXiv preprint arXiv: 909.05862,....... 2019.
[46] M. Cranmer, A. Sanchez Gonzalez, P. Battaglia, R. Xu, K. Cranmer, D. Spergel, and S. Ho. Discovering symbolic models from deep learning with inductive biascs. Advances in Neural Information Processing Systems, 13:17429-177 17442, 2020.
[47] P. Lemos, Jeffrey, M. Cranmer, S. Ho, and Battaglia. Rediscovering orbital mechanics with machine learning. arXiv preprint arXiv:220..02366, 2022.
[48] D. Spiegelhalter, A. Thomas, N. Best, and W. Gilks, Bugs 0.5: Bayesian inference using gibbs sampling manual (version ii). MRC Biostatistics Unit, Institute of Public Health, Cambridge, UK, pages 1-59, 1996.
[49] B. Carpenter, A. Gelman, M. D. Hoffman, D. Lee, B. Goodrich, M. Betancourt, M. Brubaker, J. Guo, P. Li, and A. Riddell. Stan: A probabilistic programming language. Journal of statistical software, 76(1), 2017.
[50] N. D. Goodman and A. Stuhlmüller. The Design and Implementation of Probabilistic Programming Languages. httt:://ippl..ooo 2014, Accessed: 2022-6-10.
[51] N. Gothoskar, M. Cusumano-Towner, B. Zinberg, M. Ghavamizadeh, F. Pollok, A. Garrett, J. B. Tenen- baum, D. Gutfreund, and V. K. Mansinghka, 3dp3: 3d scene perception via probabilistic programming. NEURIPS, 2021.
[52] N. Gothoskar, M. Lázaro- Gredilla, Bekiroglu, A. Agarwal, J. B. Tenenbaum, K. Mansinghka, and D. George. Durablevs: Data-efficient unsupervised recalibrating visual servoing via online learning in a structured generative model. IEEE ICRA, 2022.
[53] A. Mirchev, B. Kayalibay, P. van der Smagt, and Bayer. Variational state-space models for localisation and dense 3d mapping in 6 dof. In ICLR, 2021.
[54] Cranmer, Brehmer, and Louppe. The frontier simulation based inference. Proceedings of the National Academy of Sciences, 117(48):3005533000 2020.
[55] R. A. Horn and C. R. Johnson. Matrix analysis. Cambridge university press, 2012.
[56] E. Jang, S. Gu. and B. Poole. Categorical reparameterization with gumbel-sofimax. arXiv preprint 1110144 2016.
[57] Y. Bengio, N. Léonard, and A. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: /308.343,, 2013.
[58] K. Mo, S. Zhu, A. X. Chang, L. S. Tripathi, L. J. Guibas, and H. Su. PartNet: A large-scale benchmark for fine-grained and hierarchical part-level 3D object understanding. In CVPR, 2019.
[59] T. Mu, Z. Ling, F. Xiang, D. C. Yang, X. Li, S. Tao, Z. Huang, Z. Jia, and H. Su. Maniskill: Generaliz- able manipulation skill benchmark with large-scale demonstrations. In Thirty-fitth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[60] D. Greenberg, M. Nonnenmacher, and J. Macke. Automatic posterior transformation for likelihond-free inference. In International Conference on Muchine Learning, pages 2404-2414 PMLR, 2019.
[61] G. Papamakarios, D. Sterratt, and I. Murray. Sequential neural likelihood: Fast likelihood free inference with autoregressive flows. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 837-848. PMLR, 2019.
11

---

[62] A. Tejero-Cantero, J. Boelts, M. Deistler. J.MM. Lueckmann, C. Durkan, P. J. Gonçalves, D. S. Greenberg, and J. I. Macke. sbi: A toolkit for simulation-based inference. Journal of Open Source Software, 5(52):
2505, 2020. doi:10.21 2tt5//oss.0200. URL https: / / /doi. org/ 10 2:105/ joss 02505.
[63] M. Macklin. Warp: A high-performance python framework for gpu simulation and graphics. https: //gitnub.. com/ nvidia/ /warrp. March 2022. NVIDIA GPU Technology Conference (GTC).
[64] F. Xiang, Y. Qin, K. Mo, Y. Xia, H. Zhu, Liu, M. Liu, H. Jiang, Y. Yuan, H. Wang, L. Yi, A. X. Chang, L. J. Guibas, and H. Su. SAPIEN: simulated part-based interactive environment, In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
12


Reviews:
[{"review_id": "jtmLeuZ4pDz", "reviewer": "Reviewer_b3K8", "summary": "In robotics applications, there are often multiple moving parts which may be connected in different ways, \nfor example, with rigid, prismatic or revolute joints.\nFurther to this, robots are often accompanied by forward kinodynamics models that can be used to simulate trajectories given \nthe scene structure (models of the moving parts) and an action sequence.\nThis paper proposes a method for inferring the scene structure as a graph (part types, joint types) and kinodynamic parameters given trajectories observed from the true system.\nThey present a method for approximating a likelihood function using a simulator.\nAs such, I believe that their method heavily leverages ideas from simulation-based inference, however, \nthis is not stated in the paper.\nThey also present a novel differentiable mechanism for sampling tree-based random variables which they exploit \nfor gradient-based approximate inference strategies.\n\n", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. Given a simulator which can simulate trajectories given different combinations of scene structure and kinodynamic parameters, their method is able to infer a posterior distribution over the scene structure and kinodynamic parameters given trajectories observed from the true system.\n    - In comparison to previous methods, which can update simulation parameters given observations from the true system, their method also infers structure. For example, given two objects that rotate relative to each other, their method would infer that they are likely attached via a revolute joint.\n    - I think this is pretty neat.\n2. This relies on defining tree-structured random variables and they propose a novel differentiable mechanism for sampling such tree-structured variables. As such, they can adopt gradient-based inference schemes.\n    - Although many dynamics models are differentiable, they are often not differentiable with respect to an object's structure.\n\n**Weaknesses**\n\n1. The paper is not positioned well against the relevant literature on simulation-based inference which makes for a confusing read. This paper presents a simulation-based inference method but does not mention simulation-based inference until the experiments section, where they compare their method to two alternative simulation-based inference approaches.\n    - I was initially very confused by their likelihood function in Equation 1 and what they refer to as their generative model \n        $g_{\\phi}(\\tau \\mid \\mathcal{G}, \\mathcal{J}, \\Theta)$.\n        In my eyes, this work is simulation-based inference, as they do not have access to a likelihood function that relates \n        observed trajectories $\\tau_{\\text{obs}}$ to their latent variables ($\\mathcal{G}$, $\\mathcal{J}$ and $\\Theta$).\n        Instead, they have a simulator $g_{\\phi}(\\tau \\mid \\mathcal{G}, \\mathcal{J}, \\Theta)$ which depends on \n        their latent variables ($\\mathcal{G}$, $\\mathcal{J}$ and $\\Theta$).\n        The goal is to obtain the posterior over the latent variables given the observed trajectories $p(\\mathcal{G}, \\mathcal{J}, \\Theta \\mid \\tau_{\\text{obs}})$. Usually this is calculated with Bayes rule as follows:\n        \\begin{equation}\n        p(\\mathcal{G}, \\mathcal{J}, \\Theta \\mid \\tau_{\\text{obs}}) = \\frac{p(\\tau \\mid \\mathcal{G}, \\mathcal{J}, \\Theta) p(\\mathcal{G}, \\mathcal{J}, \\Theta)}{\\int\\int\\int p(\\tau \\mid \\mathcal{G}, \\mathcal{J}, \\Theta)p(\\mathcal{G}, \\mathcal{J}, \\Theta)\\text{d}\\mathcal{G} \\text{d}\\mathcal{J} \\text{d}\\Theta}\n        \\end{equation}\n        However, they do not have access to a likelihood function $p(\\tau \\mid \\mathcal{G}, \\mathcal{J}, \\Theta)$ and must \n        resort to simulation-based inference.\n    - If I understand correctly, $g_{\\phi}$ simulates a trajectory $\\tau$ for a sampled set of latent variables and then their \"likelihood\" is approximated using the exponential of the MSE, given by,\n        \\begin{equation}\n        p(\\tau \\mid \\mathcal{G}, \\mathcal{J}, \\Theta)\\propto\\prod_{t=0}^{T} \\exp \\left(-\\beta_{t}\\left\\|\\tau_{i}-\\tau_{\\text {obs }, i}\\right\\|^{2}\\right) \\quad \\text{where} \\quad \\tau_0, \\ldots, \\tau_{T} \\sim g_{\\phi}(\\tau \\mid \\mathcal{G}, \\mathcal{J}, \\Theta).\n        \\end{equation}\n        I think this is a form of simulation-based inference known as approximate Bayesian computation (ABC). See Cranmer et al., 2020.\n    - However, I could be wrong as I am not well versed in simulation-based inference.\n\n**References**\n\nCranmer, Kyle, Johann Brehmer, and Gilles Louppe. \"The frontier of simulation-based inference.\" Proceedings of the National Academy of Sciences 117.48 (2020): 30055-30062.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper presents a simulation-based inference (SBI) approach without explicitly stating their approach is SBI.\nNot only does the paper lack a discussion of the relevant literature but at least one reviewer (me) has been assigned \nthis paper with little to no experience in SBI. \nUnfortunately, I think this is due to an oversight by the authors which needs to be addressed before publication.\nHowever, I do think the approach presented in this paper has the potential to be a good piece of work, \nthe authors just need to position their work against the relevant SBI literature and resubmit, stating clearly that \ntheir work is SBI.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "1. The likelihood in Equation 1 looks a little funny to me.\n    - $p(\\tau \\mid \\mathcal{G}, \\mathcal{J}, \\Theta)$ is the likelihood where $\\tau$ represents the observed variable and the latent variables are $\\mathcal{G}$, $\\mathcal{J}$ and $\\Theta$.\n        - So what is $p(\\tau \\mid \\tau_{\\text{obs}})$? I don't think $p(\\tau \\mid \\tau_{\\text{obs}})$ makes sense. It looks like some form of predictive posterior with the latent variables integrated out.\n    - Also, shouldn't there be a normalising constant if using equals sign?\n        - I think it should be either $p(\\tau \\mid \\mathcal{G}, \\mathcal{J}, \\Theta)= \\frac{1}{Z}\\prod_{t=0}^{T} \\exp \\left(-\\beta_{t}\\left\\|\\tau_{i}-\\tau_{\\text {obs }, i}\\right\\|^{2}\\right)$ or $p(\\tau \\mid \\mathcal{G}, \\mathcal{J}, \\Theta)\\propto\\prod_{t=0}^{T} \\exp \\left(-\\beta_{t}\\left\\|\\tau_{i}-\\tau_{\\text {obs }, i}\\right\\|^{2}\\right)$\n    - I am also not sure what the $i$ subscript represents? I assume it is meant to represent the time step and should be $t$?\n\n2. I think that this work needs to be reformulated through the lens of simulation based inference.\n    - Firstly, the relevant literature needs to be cited in the introduction.\n    - Secondly, I found some of the language used throughout quite confusing, so I think reformulating this work as simulation based inference would make it much easier to follow.\n\n3. I find Algorithm 1 quite misleading. Surely the algorithm changes depending on the inference scheme? For example, SVI computes the Evidence Lower BOund (ELBO), not the likelihood like Algorithm 1 suggests.\n\n4. I also find details of the SVI implementation to be lacking.\n    The success of SVI heavily relies on the variational approximation and the evidence lower bound.\n    I think it is important to detail the variational posteriors that are being used along with a list of \n    the parameters which are being optimised and which are fixed (if any).\n    It would also be nice to see the objective (lower bound) that is being used as I am slightly unsure about using SVI \n    in this setting.\n    How are the KL divergence terms calculated, what are the local variables (factorised over data points) and what are the global variables?\n\n5. Line 252: Is this a poor initial guess? It looks like a good initial guess to me! The mean is close to the true value and the high variance means that the true value has some probability mass, which usually allows variational inference to easily find a mode of the posterior close to the true value.\n\n6. It would also be helpful to know how the inverse variance $\\beta_{*}$ in the likelihood (Equation 1) is picked? Or is it learned?\n\nMinor fixes:\n\n- Algorithm 1 is not referenced?\n- Figure 2 is not referenced?\n- Line 157: $u_{l}j$ looks like a typo?\n- Line 161: \"This scheme provably samples a tree\". Where is the proof? Add a reference.\n- Line 200: Sentence doesn't make sense. Have you repeated the various inference techniques part?\n- Line 201: should \"current art\" be \"current state-of-the-art\"?\n- Line 204: Space before \".The\"?\n- Line 222: Should NLL be MSE?\n- Line 230: What are SBI-SNPE and SBI-SNLE? Introduce acronyms.\n- Figure 4: Increase legend and axis font size\n- Figure 4c/4d are referenced before 4a/4b. Consider switching the paragraphs around or switching the order of sub figures.\n- Figure 6: Increase font size of sub captions\n- Figure 6: I can't read the axis or colorbar values in the middle two plots!\n- Table 2: what do the numbers represent? Add units and more info in caption."}}, {"review_id": "aFibhEH8UaB", "reviewer": "Reviewer_rwAv", "summary": "This work presented a probabilistic object representation, called Bayesian object models (BOMs) for robotic interaction. This representation encodes the structural properties such as scene structure or part structure of articulated objects (e.g. a cabinet with multiple drawers), and the physical kinodynamics attributes necessary for interaction (e.g. joint category and joint parameters of cabinet drawers). Concretely, relying on probabilistic programming with a proposed tree structure sampler and differential physical engines, BOMs infer the posterior distribution over the aforementioned types of certain objects, i.e. scene structure, physical properties given the prior of them and the likelihood of the object trajectories during task execution. Its usefulness on scene structure discovery, physical parameter estimation and interactive robotic manipulation has been empirically demonstrated on different setups such as Maniskill data set, other physical systems and a real robotic arm.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, the reviewer is highly in favor of the idea of building a rich computational mental model of objects for robotic interaction by utilizing the recent advances of probabilistic programming and differentiable physical engines. This work on a deeper combination of these two fields through integrating probabilistic programming into a robotic software stack is inspiring and beneficial for both communities on developing better theories and methodologies that are suited for wider scenarios in practice and able to addressed urgent concerns like robustness and trust of the system. In order to achieve this idea, the authors formulate the problem in a probabilistic manner, design the core elements within the pipeline such as the prior, likelihood as well as the inference engines and embed them in to a differential physics engines. Specifically, confronted with in-differentiability of tree sampling in generating scene structure, the authors overcome this by enforcing intuitive constraints (i.e. a opportunistic rounding scheme) to a differentiable DAG sampler. The extensiveness of empirical validation in terms of effectiveness on different aspects of the proposed model is also kindly appreciated by the reviewer. Though the real-world experiment is not convincing enough for a large credit, the reviewer considers this as a last-but-not-least complementary part to the experiment.\n\nNevertheless, the reviewer would still like to express several concerns regarding the idea and technical details, and provide suggestions for improving the paper and making it ready for publication.\n\n### Presentation\nRegarding the presentation clarify, the reviewer conveys the concern pressingly and tries to list the suggestions in the following.\n- nearly all figures are in a super low resolution.\n- it would be great to explain the term `generative model`, which means normally a model describing an unconditional random variable in contrast to the discriminative one for the conditional random variable. Because the main task of this work is to infer the posterior distribution over the scene structure and physical parameters of the task-specific objects, which the reviewer would not consider it as a generative model.\n- Sec. 1: the concrete usefulness (e.g. which exact tasks to apply) is not clearly described.\n- Sec. 3: \n     - the definition of notations such as $theta$, $J$ is unclear and ambiguous, i.e. multiple names for one letter (e.g. $J$). \n     - in the problem definition, it is so unexpected to see a sudden appearance of unseen terms such as \"a joint type\", \"joint parameters\" and so on in the text, the reviewer suggests the authors to make a distinctive **separation** between the description of a problem formulation and an example of the problem. It seems that these two parts are mixed in this sub-section.\n     - Eq (2) is more clear to explain the problem, which should be moved into the problem definition part.\n     - in sub-sec 3.1., there is a sentence \"BOMs exacerbate...\", should \"exacerbate\" be replaced by some words like \"resolve\"?\n\n- Sec. 4.: The order of the problems raised at the beginning does not match with the organization of the following sub-sections, which confuses the reader a lot. Besides the mis-matching problem, the clarify of presenting the experimental results has large room to improve. \n\n- There are some more minor typos in Sec4. please correct them.\n\n### Methods\n- Since there is no clear definition of the physical parameters ($J$ and $\\theta$) and edge types ($e$) of the structure graph, the first concern is the generalibility of this idea. How general can it be applied to other tasks instead of just the cabinet drawer/dishwasher/microwave (no results for the latter two in the paper) opening?  \n\n- There is no clear definition of trajectories ($\\tao$) used in the likelihood computation. Is it only the position or 6D pose of the point cloud? Is it possible to use another way to represent the evidence, such as segmentation of bounding box of the moving parts in 2D images? What's the motivation and benefits to use the trajectories of 3D point cloud while it might be more computational intensive?\n\n- There is no clear definition of the physical parameters (i.e. the kinodynamics), though they are details. It would be great to describe them and improve the self-containness of the paper. And the reviewer has the feeling that they might be instructive on specifying a suitable prior and inference method. \n\n- What differential physics engine is used? Is it possible to visualize them (e.g. the drawer opening task) for both the prior and posterior?\n\n- What's the insight for choosing the inference methods in this contexts? The reviewer asks this question because HMC is generally considered better than SVI, which is not shown in the results. The second related question is that which approximate distribution is used in SVI, which matters a lot for the performance.\n\n### Experiments\n- The results in Table 1. makes the reviewer confused because the better Joint accuracy and $IdE$ of Diff. sim leads to a lower success rate BOMs. On the other hand, the authors explain the results with an argument that \"misclassifying a revolute joint of a cabinet door as rigid joint instead will render the `OpenCabinetDoor` task impossible\", but the BOMs can achieve 100% success rate with only ~90% Joint prediction accuracy. Please make this part more clear.\n\n- Which question does sub-section 4.3 answer?\n\n- The reviewer considers the claimed feature of `sample-efficient` as weak, because there is no comparable experimental result to show it.\n\n### Limitation  \n- this part would be great if the overconfidence behavior of SVI could be explained in more details.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Even though there are several concerns on the clarify and method parts (as explained above), the novelty of the idea and the potential impact on promoting probabilistic programming in robotics persuade the reviewer to accept this work weakly. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "As listed in the section of Strengths And Weaknesses, the reviewer would suggest the authors to put higher priority on the clarify of the paper and then the methodological details. ", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "IUBILRd3EYd", "reviewer": "Reviewer_JsgS", "summary": "This paper presents a tree-based probabilistic model for estimating the kinodynamic properties of a scene. The tree's nodes are the objects, their links, etc, and edges are properties like joint type, mass, friction, damping, and 6D transformations. Authors propose a differentiable generative model for sampling trees given some prior distribution. Next, the parameters of the prior distributions are updated to match observed trajectories. The proposed approach can estimate the kinodynamic properties of interactive scenes like cabinets, dishwashers, etc. Comparative experiments against more privileged methods like differentiable simulators show that the proposed approach is more robust to poor initializations.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. An interesting, sensible approach for estimating properties of an unknown scene\n2. Competitive baselines, and good simulation experiments that show the strengths of the work\n3. Real-world experiments that show the applicability to real robots\n\nWeaknesses:\n1. Unclear motivation of the problem: I am unclear on why can't we assume that we know the joint types in the scene? Simple categories like prismatic versus revolute are known about most object types, and can be estimated (even without interaction). Overall, I found the basic premise of the paper somewhat ill-motivated.\n2. Missing analysis of experimental results. Authors show that proposed approach can find good solutions even when initialized from a bad prior, while diff sim cannot. Why? Isn't the proposed approach also gradient based? So why is it more robust to poor initialization?\n3. Discussion of limitations: How does this approach deal with situations where there might be multiple trees (robot is not in contact with drawers)? And what happens when the number of trees in the scene change (robot makes contact, and the trees combine)? I think assuming just one tree is kind of limiting for most robotic scenes, so this should be discussed in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think this is an interesting idea, and the paper presents nice experiments. However, I find experimental analysis lacking, and I don't have a good intuition for why the proposed approach does better than the comparisons. Additionally I found the motivation and limitations of the work lacking.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "1. Experimental analysis that explains the performance of the proposed approach over baselines.\n2. Discussion of limitations like variable, unknown number of trees in the scene\n3. Better motivation for the chosen problem. When can we not assume that we know simple kinematic things like joint types, and why not just infer dynamic properties of the scene?"}}, {"review_id": "EnBqkqrig-N", "reviewer": "Reviewer_8ymB", "summary": "In this article, the authors present Bayesian Object Models (BOMs) which are generative probabilistic models that encode both the structural and kinodynamic attributes of an object. It is formulated as a Bayesian inference problem over a generative model, which is implemented in a Probabilistic programming languages, given a number of real-world interactions. At the end, the framework allows to learn a distribution over both the scene structure and its physical parameters. Several simulations and experiments who the benefit of the proposed approach.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- In general, the paper is well written but some more details would improve it\n- Strong experiments with comparison to state-of-art are techniques\n- The robustness against initialization errors of BOM are very beneficial in practice I guess\n\nWeaknesses:\n- P5, 158; \"we find the first entry ulj in the soft upper-triangular matrix U above a threshold δ and round it up to 1\" Why are you using the first element here? It seems like there might be better ways to achieve a tree.\n- The \"limitation section\" discusses the general limitations of inference approaches rather than limitations of the actual proposed method\n- The presented approach seems to be computational expensive but there is no evaluation/comparison on this.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is well written and presents an interesting idea with potential impact on the community. However, the limitations of the proposed approach are rarely addressed.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "major:\n- see above\n\nminor:\n- Fig 3: \"r triangular adjacency mat\" matrix?", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)