Distributional Depth-Based Estimation of Object Articulation Models
Authors: Ajinkya Jain
Stephen Giguere
Rudolf Lioutikov
Scott Niekum
Venue: CoRL2021 Poster (2021)
Keywords: Articulated Objects
Model Learning
Uncertainty Estimation
Kinematics
Deep Learning

Abstract: We propose a method that efficiently learns distributions over articulation models directly from depth images without the need to know articulation model categories a priori. By contrast, existing methods that learn articulation models from raw observations require objects to be textured, and most only predict point estimates of the model parameters. Our core contributions include a novel representation for distributions over rigid body transformations and articulation model parameters based on Screw theory, von Mises-Fisher distributions and Stiefel manifolds. Combining these concepts allows for an efficient, mathematically sound representation that inherently satisfies several constraints that rigid body transformations and articulations must adhere to. In addition, we introduce a novel deep-learning based approach, DUST-net, that efficiently learns such distributions and, hence, performs category-independent articulation model estimation while also providing model uncertainties. We evaluate our approach on two benchmarking datasets and three real-world objects and compare its performance with two current state-of-the-art methods. Our results demonstrate that DUST-net can successfully learn distributions over articulation models and their parameters for novel objects across articulation model categories with better accuracy than state-of-the-art methods.

Distributional Depth-Based Estimation of Object Articulation Models
Ajinkya Jain" UT Austin
Stephen Giguere UT Austin
Rudolf Lioutikov Karlsruhe Institute of Technology
Scott Niekum UT Austin
Abstract: We propose a method that efficiently learns distributions over articula- tion model parameters directly from depth images without the need to know artic- ulation model categories a priori. By contrast, existing methods that learn articu- lation models from raw observations typically only predict point estimates of the model parameters, which are insufficient to guarantee the safe manipulation of ar- ticulated objects. Our core contributions include a novel representation for distri- butions over rigid body transformations and articulation model parameters based screw theory,
Mises-Fisher distributions, and Stiefel manifolds. Combin- ing these concepts allows for an efficient, mathematically sound representation that implicitly satisfies the constraints that rigid body transformations and articu- lations must adhere to. Leveraging this representation, we introduce a novel deep learning based approach, DUST-net, that performs calegory-nndependent articula- tion model estimation while also providing model uncertainties. We evaluate our approach on several benchmarking datasets and real-world objects and compare its performance with two current state-of-the. methods. Our results demon- strate that DUST-net can successfully learn distributions over articulation models for novel objects across articulation model categories, which generate point esti- mates with better accuracy than state-of-the-art methods and effectively capture the uncertainty over predicted model parameters due to noisy inputs. [webpage]
Keywords: Articulated Objects, Model Learning, Uncertainty Estimation
Introduction
Articulated objects, such as drawers, staplers, refrigerators, and dishwashers, are ubiquitous in hu- man environments. These objects consist multiple rigid bodies connected via mechanical joints such hinge joints or slider joints. Robots in human environments will need to interact with these objects often while assisting humans in performing day-to-day tasks. To interact safely with such objects, robot must reason about their articulation properties while manipulating them. An ideal method for learning such properties might estimate these parameters directly from raw observations, such RGB-D images while requiring limited or priori information about the task. The ability to additionally provide a confidence over the estimated properties, would allow such a method to be leveraged in the development of safe motion policies for articulated objects [1].
The majority of existing methods to learn articulation models for objects from visual data either need fiducial markers to track motion between object parts [2-5] or require textured objects [6-10]. Recent deep-leaming based methods address this by predicting articulation properties for objects from raw observations, such as depth images [11-14] PointCloud data [15, 16]. However, the majority these methods [11, 12, 15, 16] require knowledge of the articulation model category for the object (c.g., whether it has revolute or prismatic joint) which may not be available in many realistic settings. Alleviating this requirement Jain et al. [14] introduced ScrewNet, which uses a unified representation based screw transformations to represent different articulation types and performs cereggor--indeenneen articulation model estimation directly from raw depth images. How- ever, ScrewNet [14] and related methods 1-13, 15, 16] only predict point estimates for an object's
"Corresponding author: aj ijikyauuteaas.... edu Equal contribution, presented alphabetically
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

Imput: of mages 201
sequence distrbutiinn transtommtions
Output:
calegory
distribuion
contigurations (9,il)
Revolute 5,6 pf Los Wessror
Gemeralizainn acroes different objects Instances and articulatonn model categories
DUST-net
Priamatic 5.d - pf Low Versooe
Sequence 2
the distibution
S. Paee tlen (x (290
Revolute 5,0 p High Yustace
Sequincee 3
Using von Mises-Fisher distribution on a Stiefal manifold
Figure 1: DUST-net sequence of images I1:n to compute the parameters, o, of the conditional distribution over the joint parameters S and configurations {0, d This distribution allows for inference and reasoning, such uncertainty and confidence, over both the parameters and the configurations. Using a von Mises-Fisher distribution on a Stiefel manifold allows for an efficient reparameterization that inherently obeys multiple constraints that define rigid body transformations.
articulation model parameters. Nonetheless, reasoning about the uncertainty the estimated param- eters can provide significant advantages for ensuring success in robot manipulation tasks, and allows for further advancements such as robust planning [1], active learning using human queries [17], and the learning of behavior policies that provide safety assurances [18]. Motivated by these advantages, we propose a method for learning articulation models, which estimates the uncertainty over model parameters using a novel distribution over the set of screw transformations based on the matrix von Hiscs-Fisher distribution over Stiefel manifolds [19]. We introduce DUST-net. Deep Uncertainty estimation Screw Transforms network, novel deep learning-basee method that, in addition to providing point estimates of the object's 's articulation model parameters, leverages raw depth images to provide uncertainty estimates that can be used to guide the robot's behavior without requiring to knowledge of the object's 's articulation model category a priori.
DUST-net garners numerous benefits over existing methods. First, DUST-net estimates articulation properties for objects with uncertainty estimates, unlike most current methods [11-16]. These un- certainty estimates, apart from helping robots to manipulate objects safely [1], could allow a robot to take information-gathering actions when it is not confident and enhance its chances of success in completing the task. Second, similar to ScrewNet [14], DUST-net can estimate model parame- ters without the need to to know the articulation model category a priori, by leveraging the unified representation for different articulation model types. Third, this unified representation helps DUST- net to be more computationally and data-efficient than other state-of-the-an methods [11, 12], as it uses single network to estimate model parameters for all common articulation models, unlike other methods that require a separate network for each articulation model category [11, 12, 15, 16). Empirically, DUST-net outperforms other methods even when trained using only half the training data in comparison. Fourth, the distributional learning setting yields more robustness to outliers and noise. Fifth, DUST-net is able to reliably estimate distributions over articulation model parameters for objects in the robot's camera frame. By contrast, ScrewNet [14), the most closely related ap- proach ours, can only predict point estimates for articulation model parameters in the object's local frame.
We evaluate DUST-net through experiments two benchmarking datasets: simulated articulated objects dataset [11] and the MartNet-Mobille dataset [20-22], as well as three real-world objects: a microwave, drawer, and a toaster oven. We compare DUST-net with two state-of oftte-of-thharr methods, namely ScrewNet [14] and an MDN-based method proposed by Abbatematteo et al. [11], as well baseline methods. The experiments demonstrate that the samples drawn from the distribu- tions learned by DUST-net result in significantly better estimates for articulation model parameters in comparison the point estimates predicted by other methods. Additionally, the experiments show that DUST-net can successfully and accurately capture the uncertainty over articulation model parameters resulting from noisy inputs.

---

2 Related Work
Articulation model estimation from visual observations: A widely used approach for estimating articulation models based on the probabilisticc framework proposed by Sturm et al. [2]. It uses the time-series observations of 6D poses of different parts of an articulated object to learn the re- lationship between them [2, 5, 6, 10]. More recently, Abbatemattco et al. [11] and Li et al. [12] proposed methods to learn articulation properties for objects from raw depth images given articu- lation model category.
a related body of work on object parts mobility estimation, Wang et al. [15] and Yan et al. [16] proposed approaches to segment different parts of the object in input point cloud and estimate their mobility relationships, given a known articulation model category. Alleviating the requirement of having known articulation model category, Jain et al. [14] recently proposed ScrewNet that performs category-ince teneerr--nddeeendent articulation model estimation from depth images. However, these methods only predict point estimates for the articulation model parameters, while DUST-net predicts a distribution over their values.
Our contributions are related to existing work on estimating distribu- tions describing the orientation of rigid bodies. Gilitschenski et al. [23], Arun Srivatsan et al. [24], Srivatsan et al. [25] and Rosen et al. [26] propose strategies that can be used to estimate the rigid body transformation of an object using a combination of Bingham and Gaussian distributions, and the von Mises-Fisher distribution, respectively. The mathematical model used by our approach is inspired by these works, but 1) extends them to also represent uncertainty over the configuration of articulated object components about screw axes, and 2) integrates them into deep learning model that is capable of learning these configurations from raw depth images. In addition, while these approaches use distributions over orientations and rigid body transformations produce estimates, DUST-net directly outputs a distribution that can be used to facilitate further applications such as uncertainty-aware behavior planning.
Rigid Body Pose Estimation:
Katz and Brock [3] introduced IP as a method to leverage a robot's interaction with objects to generate a rich perceptual signal for articulation model estimation for planar objects, and extended it to learn 3D articulation models for objects [4]. Martín-Martín et al. [8] used hierarchical recursive Bayesian filters make estimation more robust and developed online methods for articulation model estimation from RGB images [7-9]. A comprehensive survey on IP methods in robotics was presented by Bohg al. [27]. While IP presents a powerful tool for estimating articulation properties for objects, a wide majority of existing IP methods require textured objects, unlike DUST-net, which learns these properties using depth images.
Interactive perception (IP):
Further approaches: Articulation motion models can be viewed as geometric constraints imposed on multiple rigid bodies. Such constraints can be learned from human demonstrations by leveraging different sensing modalities [13, 28-31] 13. Recently, Daniele et al. [30] proposed a multimodal learn- ing framework that incorporates both vision and natural language information for articulation model estimation. However, these approaches predict point estimates for the articulation model parameters, unlike DUST-net, which predicts distribution over the articulation model parameters.
3 Problem Formulation:
Given a sequence of depth images Iten of motion between two parts of an articulated object, we estimate the parameters of a probability distribution p(o)ZI.)) representing uncertainty over the pa- rameters & of the articulation model M governing the motion between the two parts. Following Jain et al. [14], we define the model parameters Ø as the parameters of the screw axis of motion, S= (1, m). where both and m are elements of 3 This unified parameterization can be used in ar- ticulation models with at most one ofgrree-of-rreddon (DoF), namely rigid, revolute, prismatic, and helical [14]. Additionally, we estimate the parameters of distribution p(q1--- IT..) representing uncertainty over the configurations -1 identifying the rigid body transformations between the two parts in the given sequence of images Iı. under model M with parameters é. Configurations q, i E {I...- }} correspond to a set of tuples, 4i = (0., d;), defining rotation around and displacement along the screw axis S3. We assume that the relative motion between the two object parts is determined by single articulation model.
Pleasse refer to the supplementary material for further details
3

---

4 Approach
Given a sequence of depth images Iim of motion between two parts of an articulated object, DUST- net estimates parameters of the joint probability distribution p(o, 91=x-1 representing uncer- tainty over the articulation model parameters @ governing the motion between the two parts and the observed configurations 91-n-1. When deciding how to learn this distribution, two goals arise. While some parameters, such as the translation of an object part along screw axis, are defined on Euclidean space, the set of valid screw axes exhibits constraints that prevent standard distribu- tions defined on R6 from being applied without complicating the learning process. For example, a standard representation for distributions over screw axes can be the product of a Bingham distribu- tion over the line's orientation and a multivariate normal distribution over its position in space [32]. However, this representation produces non-unique estimation targets. A rotation of 0 about the screw axis with orientation results in the same transformation as a rotation of -0 about the screw axis with orientation -1. Similarly, displacement d along I results in the same transformation as a displacement -d along -1. This leads to ambiguities in the targets in the estimation problem and can hinder the performance of the trained estimator. By selecting representation that accounts for these symmetries, these non non-unique estimation targets are removed. Second, once a suitable param- eterization is chosen, we seek a parametric form for the joint distribution which can be learned by deep network.
First, we consider the problem of parameterizing the set of screw axes. As noted earlier, we define the model parameter & as the parameters of the screw axis of motion = (1, m). However, this parameterization requires that has unit norm, and that I and m are orthogonal. To eliminate these constraints, we rewrite the moment vector of a screw axis as m = ||m|] ri, where ||m]] and in rep- resent its magnitude and a unit vector along it respectively, and the Plücker coordinates for the screw axis as S = (1, m, |mll). The Plücker coordinates can then be seen as an unconstrained point in the space S: ;= V2,3 x R where (I, m) E V2,3 with V2,3 denoting the Stiefel manifold of 2-frames in RS and ||m|] E + with R++ denoting the set of positive real numbers. The Stiefel manifold Va.m is the space whose points are sets of k orthonormal vectors in IT called k-frames in RD (k S m)) [19]. Consequently, because of the one-to-one mapping from elements of V2,3 IR+ to screw axes, the non-unique estimation targets described above are eliminated. Based on this parametrization of screw axes, we define the set of valid configuration parameters as follows. We restrict the range of values for the rotation about the screw axis to be 0 E [0, 2 and restrict the displacement along the axis to be d E R+ Note that these constraints do not reduce the representational power of the screw transform (I, m, 0, d) to denote general rigid body transform, but merely ensure a unique representation.
Having described the parameterization of the set of screw axes and configurations, we now consider the task of defining a joint probability distribution over their values. We propose to represent the distribution over predicted screw axis parameters, p(S Ii=)) with S E S, as product of a matrix von Mises-Fisher distribution F(-13, F) defined on the Stiefel manifold V2,3 and a truncated normal distribution W o) with truncation interval [0, +oo) over IR+t Formally,
p(S| I = (1, m, [ |m|| II.n. F,pm 2n) = = (1, m F) A ([|m]| |/mm o
(1)
where F is x matrix representing the parameters of the matrix von Mises-Fisher distribution over V2.3, and Pm and om denote the mean and standard deviation of the truncated normal distribution.
Given the sequence of n images, we also wish to estimate the posterior over configurations Q1:n-1 = (01:n-1, 1, d1:a-1) 1) corresponding the rotations about and displacements along the screw axis S. We define the joint posterior representing the uncertainty over the screw axis S and the configurations (f1:n-1, 1, d1:n-1} } about it product of the aforementioned distribution and set of distributions defined over the configuration parameters,
p(S, 1mn di.n-1 Iim, )) - -p( Mm, oim) (1 + T(din-
(2)
where +- (F, /m, of, v. v} is the set of parameters for the distribution and T and Y represent the set of distributions having parameters 1/r and v over the configurations 0p:--1 and di respec- tively. For the sake of brevity, we present further details on modeling assumptions in the supple- mentary material (see Appendix B). In this work, we consider V and to be products of truncated normal distributions such that V = T (0, oz) and Y = I N (d, oz) with
Please refer to the supplementary material for further details

---

Mg = (pg................. ..... Pa deviations of the set of truncated normal distributions over the configurations 0in-1 and di respectively.
". }. Ma fug Hd
"i "), Ge, and aa denoting the set of means and the standard
Distribution parameter matrix F: The parameter matrix for the matrix von Mises-Fisher distribu- tion over V3,2 is a 3 x 2 matrix. F. This presents two possible parameterizations of the matrix: first, to estimate each of the elements of the 2 matrix F and second, estimate the matrices I', A, and S2 defining the SVD of F, given by F = r'n?". The second parameterization decouples the two objectives of distribution mode alignment with the ground truth labels and uncertainty repre- sentation; the mode of the distribution is given by M r??", and the concentration matrix for the distribution is given by K QA12" This decoupling allows the network to independently optimize both objectives, whereas in the first parameterization, changes in the elements of F causes changes in both components.
By definition, is a 2 x 2 diagonal matrix with two independent parameters, and Ω2 E ((2) is a rotation matrix in two dimensions with one independent parameter, the rotation angle w. The matrix TE Va.2 can be constructed from rotation matrix R E O(3) by keeping only the first two columns of R. Hence, the matrix I can be defined by three independent Euler angles, (a, 8, denoting rotation according to the ZYX convention in the rotating frame. Euler angles can suffer from the problem of gimble lock [32], which we resolve by restricting the Euler angles to be in the ranges [0, 2x), 3 E [0, n), and y E [0, 2x).
Normalization factor: One of the main challenges of using the matrix von Fisser distribution
is the calculation of its normalization factor FF ( _ _A²), which is hypergeometric function of matrix argument
[19]. In this work, we approximate this hypergeometric function using a truncated series in terms of zonal polynomials, which are multivariate symmetric homogeneous polynomials and form basis of the space of symmetric polynomials [19]. Through our preliminary experiments, found that this truncated series is good approximation of o as it converges to finite value, if the singular values of the F. i.e. λ and X2 are less than Amazz = 50.
DUST-net sequentially connects ResNet-18 CNN [33] and 2-layer MLP. ResNet- 18 extracts task-relevant features from the input images, which are used by the MLP to predict a set of parameters & for the distribution p(S, 01.n-1,,11. d1:--1 Ii:m: )) The network is trained end- to-end, with ReLU activations for the hidden fully-connecteo 40) of the last linear layer of MLP, corresponding to the parameters (a, 3,)) and w representing the matrices I and Ω respectively, are fed through ReLU-6 layer to ensure that the predictions map to their respective ranges. Remaining output is fed through a Softplus layer for non-negative output. Detailed network architecture is presented in the appendix (Fig. 7).
Architecture:
layers. The first four output (out of
Training: The training data for the model consists of sequences of depth images of ob- jects parts moving relative to each other and the corresponding screw transforms y = (I, m, ||m||
The objects and depth images rendered in Mujoco [34]. We train DUST-net by maximizing the log--robable prgpobbabbilty of the labels y under the distribution ply Ii.ms Q): L(y, +) = -ooo p(y Φ). We assume that the observed configurations in Iı.n share the same vari- ance. We use the precision parameters rather than the standard deviations, Fm. Og and aa to represent the distribution during training for better numerical stability. Following the discussion MDNs by Makansi et al. [35], we separate the training in three stages. In the first stage, we assume the dispersion of the matrix von Mises-Fisher distribution to be fixed with A diag(Ao Ao). do and learn parameters corresponding to I and Ω matrices. In the second stage, we fix the A matrix and learn the rest of the parameters in the set 4. Finally, we train to predict the complete set 4
training
5 Experiments
In this section, we evaluate DUST-net on its ability learn articulation model parameters and un- certainty estimates. We conducted three sets of experiments evaluating DUST-net's performance under different criteria: (1) how accurate point estimates of the articulation model parameters drawn from DUST-net's estimated distribution are in comparison to the existing methods, (2) how effec- tively DUST-net captures the uncertainty over parameters arising from noisy input, and (3) how effectively DUST-net transfers from simulation to real-world setting. We evaluated DUST-net's performance on two simulated benchmarking datasets: the objects dataset provided by Abbatem- atteo et al. [11], and the PartNet-Mobility dataset [20-22], as well set of three real-world
5

---

objects. From the simulated articulated object dataset [11], we considered the cabinet, microwave, and toaster oven for revolute articulations and the drawer object class for prismatic articulations. From the PartNet- Mobillty dataset [20-22], we considered five object classes: the dishwasher, oven, and microwave object classes for the revolute articulation model category, and the storage furniture object class consisting of either single column of drawers or multiple columns of drawers, for the prismatic articulation model category. Among the three sets of experiments, we conducted the first two sets of experiments the simulated datasets, while the last set of experiments were conducted on the real-world object dataset. In all the experiments, we assumed that the input depth images are semantically segmented and contain non-zero pixels corresponding only to the two objects between which we wish to estimate the articulation model.
We compared DUST-net's performance in estimating point estimates for articulation model parame- ters with two state-of-the-art methods, ScrewNet [14] and an MDNNbased approach proposed by Ab- batematteo et al. [11]. Screw Net estimates the object's 's articulation model parameters in a local frame located the center of the object, whereas DUST-net does so directly in the camera frame. We com- pare our method with ScrewNet predicting parameters both in the object local frame and the camera frame. Additionally, we propose two baseline methods that estimate distributions over articulation model parameters and compare them DUST-net. The first baseline method (vm-SoftOrtho) can be viewed as an extension of ScrewNet to a distributional setting. It represents the uncertainty over the screw axis orientation vector I and the direction of moment vector using two independent von Mises-Fistee distributions and imposes a soft orthogonality constraint over the modes the two dis- tributions. The distributions over the moment vector magnitude ||m||| and configurations 91:n-1 considered to be normal distributions. This method suffers from the same drawback as ScrewNet, i.e., the of soft orthogonality constraint during training, and therefore cannot predict valid set screw axis parameters directly, unlike DUST-net. The second baseline method (Direct F) uses the same probability distribution DUST-net to represent the uncertainty over the articulation model parameters, but estimates the individual elements of the F matrix directly, As a result, it fails capture the uncertainty over model parameters accurately.
5.1 Accuracy of Point Estimates
The first of experiments evaluated DUST-net's accuracy in predicting point estimates for artic- ulation model parameters. We use the mode of the estimated distribution as the point estimate for model parameters. We used metrics evaluate accuracy: Mean Absolute (Angular) Devia- tion (MAAD) and Screw Loss (Metric proposed in ScrewNet [14]). MAAD metric indicates how close the individual screw parameters are targets, whereas the Screw Loss indicates how close the complete predicted screw transforms is to the target transforms. The MAAD metric calculates the angular distance between the orientation of the predicted and ground-truth axis orientation vectors and the orientation vectors of the screw axis moment vectors m. For the remaining parameters (|mll d it calculates the mean absolute deviation between the predicted and ground- truth values. The screw loss reports the angular distance between the predicted and ground-truth screw axis orientation vectors as orientation error the length of the shortest perpendicular be-
d1mn-1),
MAAD: Axis oni (dog)
m
|ml]
MMAAD
dMAAD
Momest ori (dog)
Contig (deg)
Config df (cm)
Moment mag
D(SGT. Spred)
d1
Asis Oni Eir (dog)
Axis Pos Eir (cm)
Config Ear (deg)
Config Ear (cm)
ScrewNet
vm-SootDrtho
Direct-F
DUST net
(Local)
Figure 2: Mean error values on the MAAD (top) and Screw Loss (bottom) metrics for the simulated
articulated objects dataset [11] (lower values are better). Point estimates for DUST-net (violet)
correspond to the modes of the distributions predicted by DUST-net.

---

||m||
OMAAD
dMAAD
Moment mag (cm)
Config (deg)
Config (cm)
Axisori
D(SCT Spred)
di
Ori (deg))
Axis Poe Ei (em)
Conlig Eir (dag)
Config Eir (cm)
Abbatematteo al ScrewNet
vm vm-SoftOrtho
Direct-F
DUST net
ScrewNet (Locsi)
Figure 3: Mean enror values on the MAAD (top) and Screw Loss (bottom) metrics for the PartNet- Mobility dataset [20-22] (lower values are better). Point estimates for DUST-net (violet) correspond to the modes of the distributions predicted by DUST-net.
tween the predicted and ground-truth screw axes the distance between them. Configuration errors 01n-1 are reported as the difference between the predicted rotation about the predicted screw axis and the true rotation, whereas errors d1-n-1 calculated as the Euclidean distance between the points displaced by the predicted and true displacements along respective axes.
Results for the synthetic articulated objects dataset and the PartNet-Mobility dataset are shown in Figures 2 and 3, respectively. Results demonstrate that under both metrics, the estimates obtained from DUST-net are typically more accurate than those obtained from the state-of-the-ee methods. The first baseline, vm-SoftOrtho, performs comparably with DUST-net on both datasets when only MAAD estimates are considered. However, Figures 2 and 3 show that it produces very high distance ( 1 m) between the predicted and ground-truth screw axes. This error arises due to the soft- orthogonality constraint used by vm no-SootOrthh. as DUST-net and the second baseline method, both of which handle the constraint implicitly, do not report high errors on that metric. Meanwhile, the second baseline, Direct F, performs comparably with DUST-net both metrics for both datasets, but fails to capture the uncertainty over parameters with the required accuracy.
5.2 Uncertainty Estimation
The second set of experiments evaluated how effectively DUST-net's predicted distribution captures epistemic uncertainty over the pre- dicted articulation parameters. evaluate this by adding artificial noise to the training labels from the two simulated datasets while training DUST-net. As more noise is added, we expect the confidence estimates produces by DUST- net to decrease well. We add noise to the labels by sampling perturbations from a ma- trix von Mises-Fisher distribution with vary-
#
Ài SynArt SynAn
À PartNer
Āg PartNet
40-
Singular of
30-
20-
16-
ing singular values Å₁ and X2 of the distribu- tion parameter matrix F and the truncated mal distributions with varying precision param- eters B,.j E {[|m|| d). Figure show the variation of the mean of the singular values of the predicted distribution concentration matri- ces over screw axes by DUST-net with injected noise. the noiseless case, the singular values of the matrix von Mises-Fisher distribution in- creases until they reach their maximum allowed value at Amez 50. When label noise is added, our results show that DUST-net's confidence over predicted parameters decreases monotonically more noise is added to the labels, supporting hypothesis.
0.02
Noise covariance Af22
Figure 4: Variation of the mean of the singular values of predicted distribution concentration ma- trices over screw axes by DUST-net with artifi- cially injected noise. Predicted singular values de- crease monotonically with input noise, showing that the network's confidence over the predicted parameters decreases with input noise.

---

MAAD Are on tecp)
Momen on tleg)
S 5. t
(C1)
||m]|
Onlig e (oat
PMAAD
Ceeti (os)
dMNAD
(Stmn)
Am
p(SaT.Spea)
dy
Asis On Eir (deg)
Pro (ca))
.93
Config Er (dkg)
Ag
ScrewNet DUST--eet
DUST (Sima)
Rg
(a)
(b)
220
130
Procsten
Figure 5: (a) Mean error values on MAAD (top) and Screw Loss (Bottom) metrics for real-world objects when the network was trained solely using simulated data [11] (lower values better) (b) Predicted concentrations over articulation model parameters. DUST-net estimation performance on simulated data [11] (hatched green) included for comparison. DUST-net reported lower confidence in its predictions for real-workd objects than simulated data (b), analogous to its degraded estimation aceuracy(a).
5.3 Sim to Real Transfer
Lastly, evaluated how effectively DUST-net transfers from simulation real-world setting. DUST-net was trained solely on the simulated articulated object dataset [11]. Afterward, we used it to infer the articulation model parameters for three real-world objects. Results (Fig. 5(a)) report that DUST-net outperforms the current state-of-the-art method, ScrewNet, in estimating the model parameters for real-world objects. However, the estimated parameters using DUST-net accurate enough to be used directly for manipulating these objects. This sub-par performance stems from the significant differences between the training (clean and information-rich simulation data) and test datasets, which consists of noisy depth images acquired with a Kinect sensor and contain high salt-and-pepper noise, spurious features, and incomplete objects. Better performances could be achieved by either fine-tuning the network on real-world data or retraining it using a larger real- world dataset. A noteworthy insight from the results is that DUST-net also reports low confidence over the predicted parameters for real-world objects, compared to when tested on the simulated data (Fig. 5())). This clearly delineates why it is beneficial to estimate distribution over the ar- ticulation model parameters instead of point estimates. Given only point estimates of articulation model parameters, robot has no way to determine if the estimates are reliable for manipulating the object safely or not. In contrast, DUST-net's 's reported confidence over the predictions could allow the robot to develop safe motion policies for articulated objects [1, 18] or use active learning based methods [17] to reduce uncertainty over the articulation parameters.
yet
6 Conclusion
We introduced DUST-net, which utilizes novel distribution over screw transforms on a Stiefel man- ifold perform category-independent articulation model estimation with uncertainty estimates. We evaluated our approach on two benchmarking datasets and three real-world objects and compared its performance with two current state-of-the-ee methods [11, 14). Results show that DUST-net can estimate articulation models, their parameters, and model uncertainty estimates for novel objects across articulation model categories successfully with better accuracy than the state-of-the-an meth- ods. At present, DUST-net can only predict parameters for I-DOF articulation models directly. For multi-DoF objects, additional image segmentation step is required to mask out all non-relevant object parts. This procedure can be repeated iteratively for all object part pairs to estimate relative models between object parts that can be combined later to construct complete kinematic model for the object [10]. An interesting extension of DUST-net could estimate parameters for muulii-Dof ob- jects directly by learning a segmentation network along with it. Another exciting direction of future work is to use DUST-net in an active learning setting where, if the robot is not confident enough about the estimates of the articulation model parameters, can actively take information -atheiiinggthtiinggg actions to reduce uncertainty.
8

---

Acknowledgments
This work has taken place in the Personal Autonomous Robotics Lab (PeARL) at The University of Texas at Austin. PeARL research is supported part by the NSF (IIS-1724157, IIS- USS-163107, IIS- 1749204, IIS-1925082,, ONR (N00014-18-2245) AFOSR (FA9550-20-1-0077)) and ARO (78372- CS).
References
[1] A. Jain and S. Niekum. Efficient hierarchical robot motion planning under uncertainty and hybrid dynamics. In Conference Robot Learning, pages 757-766, 2018.
[2] J. Sturm, C. Stachniss, and W. Burgard. A probabilistic framework for learning kinematic models of articulated objects. Journal of Artificial Intelligence Research, 41:477-526, 2011.
[3] D. Katz and O. Brock. Manipulating articulated objects with interactive perception. In 2008 IEEE International Conference on Robotics and Automation, pages 272-277. IEEE, 2008
[4] D. Katz, M. Kazemi, A. Bagnell, and A. Stentz. Interactive segmentation, tracking, and kinematic modeling of unknown 3d articulated objects. In 2013 IEEE International Conference on Robotics and Automation, pages 5003-5010. IEEE, 2013.
[5] S. Niekum, S. Osentoski, C. G. Atkeson, and A. G. Barto. Online bayesian changepoint detec- tion for articulated motion models. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 1468-1475, IEEE, 2015.
[6] S. Pillai, M. R. Walter, and S. Teller. Learning articulated motions from visual demonstration. arXiv preprint arXiv: 502.0 2015.
[7] R. Martín-Martin and O. Brock. Online interactive perception of articulated objects with multi- level recursive estimation based task-specific priors. In 2014 IEEE/RSJ International Con- ference on Intelligent Robots and Systems, pages 2494-2501. IEBE, 2014.
[8] R. Martin-Martin, S. Höfer, and O. Brock. An integrated approach to visual perception of ar- ticulated objects. 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 5091-5097. IEEE, 2016.
[9] R. Martín-Martín and O. Brock. ception of articulated objects. The International Journal of Robotics Research, page 0278364919844 19848850, 2019.
Coupled recursive estimation for online interactive per-
[10] A. Jain and Niekum. Learning hybrid object kinematics for efficient hierarchical planning under uncertainty. IEEE International Conference on Intelligent Robots and Systems (IROS), 2020.
[11] B. Abbatematteo, S. Tellex, and G. Konidaris. Learning to generalize kinematic models to novel objects. In Proceedings of the Third Conference Robot Learning, 2019.
[12] X. Li. H. Wang, L. Yi, L. Guibas, A. Abbott, and S. Song. Category-level articulated object pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3706-3715, 2020.
[13] Q. Liu, W. Qiu, Wang, G. D. Hager, and A. L.. Yuille. Nothing but geometric constraints: A model-free method for articulated object pose estimation. arXiv preprint arXiv:2012.00088, 2020.
[14] A. Jain, R. Lioutikov, C. Chuck, and S. Niekum. Screwnet: Category-indepeedent articu- lation model estimation from depth images using screw theory. In 202/ IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021.
[15] X. Wang, B. Zhou, Y. Shi, X. Chen, Q Zhao, and K. Xu. Shape2motion: Joint analysis of motion parts and attributes from 3d shapes. In Proceedings of the IEFF/CV Conference on Computer Vision and Pattern Recognition, pages 8876-8884, 2019.

---

[16] Z. Yan, R. Hu, X. Yan, L. Chen, O. Van Kaick, H. Zhang, and H. Huang. Rpm-net: net: Recurrent prediction of motion and parts from point cloud. ACM Trans. Graph., 38(6), Nov. 2019. ISSN 0730--301. doi:10. 111334533399 3356573.
URL httphtt/p///oii //toi///ooiio org/ / 10. 1145/3355088
[17] Y. Cui and S. Niekum. Active reward learning from critiques. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 6907-6914. IEEE, 2018
[18] A. Taylor, A. Singletary, Y. Yue, and A. Ames. Learning for safety-critical control with con- trol barrier functions. Proceedings of the 2nd Conference on Learning for Dynamics and Control, 2020. URL
http::////oo httpp:://ppoceednngs..m mlr. Freess/v120/taylor00aa html.
[19] Y. Chikuse. Statistics on special manifolds, volume 174. Springer Science & Business Media, 2003. doi.https://doi.or/// 100.11.000000002100010 0017778---387--45000-
[20] Xiang, Y. Qin, K. Mo, Y. Xia, H. Zhu, F. Liu, M. Liu, H. Jiang, Y. Yuan, H. Wang, L. Yi, A. X. Chang, L. J. Guibas, and H. Su. SAPIEN: A simulated part-based interactive environ- ment. In The IEEE Conference on Computer Vision and Puttern Recognition (CVPR), June 2020.
[21] K. Mo, Zhu, A. X. Chang. L. Yi, S. Tripathi, L. J. Guibas, and H. Su. PartNet: large-scale benchmark for fine-grained and hierarchical part-level 3D object understanding. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[22] A. X. Chang. T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song. H. Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012 2015.
[23] I. Gilitschenski, G. Kurz, S. J. Julier, and U. D. Hanebeck. Unscented orientation estimation based on the bingham distribution. IEEE Transactions Automatic Control, 61():172--7 2015.
[24] R. Arun Srivatsan, M. Xu, N. Zevallos, and H. Choset. Probabilistic pose estimation using bingham distribution-base linear filter. The International Journal of Robotics Research, 37 (13-14):1610-1631, 2018.
[25] R. A. Srivatsan, G. T. Rosen, D. F. N. Mohamed, and H. Choset. Estimating se (3) elements using a dual quaternion based linear kalman filter. In Robotics: Science and systems, 2016.
[26] D. M. Rosen, L. Carlone, A. S. Bandeira, and J. J. Leonard. Se-syne: A certifiably correct algorithm for synchronization over the special euclidean group. The International Journal of Robotics Research, 38(2-3):95-125, 2019.
[27] Bohg, K. Hausman, B. Sankaran, O. Brock, D. Kragic, S. Schaal, and G. Sukhatme. Inter- active perception: Leveraging action in perception and perception in action. IEEE Transactions on Robotics, 33(6):1273- 1291, 2017.
[28] C. Pérez-D' Arpino and J. A. Shah. C-leam: Learning geometric constraints from demonstra- tions for multi-step manipulation in shared autonomy. Robotics and Automation (ICRA), 2017 IEEE International Conference on, pages 4058 4065. IEEF, 2017.
[29] Y. Liu, F. Zha, Sun, J. Li, M. Li, and X. Wang. Learning articulated constraints from one-shot demonstration for robot manipulation planning. IEEE Access, 7:172584-172596, 2019.
[30] A. F. Daniele, T. M. Howard, and M. R. Walter. A multíview approach to learning articulated motion models. In Robotics Research, pages 371-386. Springer, 2020.
[31] G. Subramani, M. Zinn, and M. Gleicher. Inferring geometric constraints in human demon- strations. Conference on Robot Learning, pages 223-236. PMLR, 2018.
[32] B. Siciliano and O. Khatib. Springer handbook of robotics. Springer, 2016.
10

---

[33] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[34] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026- 5033. IEEE, 2012.
[35] O. Makansi, E. 1lg, O. Cicek, and T. Brox. Overcoming limitations of mixture density net- works: A sampling and fitting framework for multimodal future prediction. In Proceedings the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 7 7144-7153, 2019.
[36] Y.-b. Jia. Plücker Coordinates for Lines in the Space [Lecture Notes], August 2019.
[37] K. V. Mardia and P. E. Jupp. Directional statistics, volume 494. John Wiley & Sons, 1999. doi:10 10000/998047033697799
[38] C. Khatri and K. V. Mardia. The von mises-fisher matrix distribution orientation statistics. Journal the Royal Statistical Society: Series B (Methodologicall. 39(1):55-106, 1977.
[39] A. T. James. Distributions of matrix variates and latent roots derived from normal samples. The Annals of Mathematical Statistics, 35(2):475-501, 1964.
[40] L. Jiu and C. Koutschan. Calculation and properties of zonal polynomials. Mathematics in Computer Science, pages 1-18, 2020.
[41] A. G. Howard, M. Zhu, Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 700.04481, 2017.
[42] A. Bartoli and P. Sturm. The 3d line motion matrix and alignment of line reconstructions. In Proceedings of the 2001 IEEE Computer Society Conference Computer Vision and Pattern Recognition. CVPR 200/, volume 1, pages I-I. IEEE, 2001.


Reviews:
[{"review_id": "Z_tcm4Yc6V", "reviewer": "Reviewer_tV4q", "summary": "The paper proposes a method to learn distributions over articulation models from a sequence of depth images without any known articulation model a priori. To learn the distribution, the paper first presents a novel representation for distribution over rigid body transformations and articulation model parameters using Screw theory, von Mises-Fisher distributions and Stiefel manifolds. Then the paper introduces DUST-net, a network that transforms a sequence of depth images into a distribution over the transformations between them. With experiments on two benchmarking datasets (a simulated articulated object dataset and PartNet-Mobility dataset) and three real-world objects, it is shown that DUST-net successfully learns the distribution over articulation models and their parameters. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper addresses an important problem of perceiving articulated objects from a sequence of observations without any prior knowledge of the object model or its articulation constraints.\n* The paper attempts to present a suitable representation that can capture distribution over transformations and articulation parameters. If this representation has the benefits as claimed in the paper, then it could be used for part based object estimation frameworks.\n* The presented representation lets the proposed method perform category-independent articulation model estimation along with model uncertainties.\n\nQuestions and areas for improvement:\n* In Section 3, especially the background section needs to be closely knit with the representation of an articulated model. Currently, it is hard for the reader to relate these concepts with the problem this paper is trying to address.\n* The relationship between Screw Transformations, Stiefel manifold and Matrix von Mises-Fisher distribution is unclear at Section 3. The fundamental background of these concepts can be in the supplementary material, while their relationship and relevance to the representation of articulated models can be described in the main manuscript. \n* It is unclear what Figure 2 is trying to illustrate. \n* Visuals or illustrations to support the description in Section 4 would be beneficial. \n* In Tables 1, 2, and 3, it is unclear what “Uncertainty” values indicate. It is described in the text that the uncertainty is high on real-world experiments, whereas the numbers are less in Table 3 compared to 1 and 2. \n* In Section 5.2, the text - “We evaluate this by adding artificial noise to the training labels form the two simulated datasets while training DUST-net”. It is not clear how training with noise leads to this evaluation. Is this a different training as opposed to the original for estimating the articulation models parameters? The reader would imagine that there is one training phase overall, that includes the addition of noise to the labels. \n* It is described in the Section 5.3 that the estimation is accurate in the real-world setting. How will the proposed approach handle partial observations caused due to self-occlusions and robot occlusions during the manipulation action, in addition to the depth noise from the sensor data? Is it possible to characterize the uncertainty due to sensing and the scenario differently?\n\nWeakness:\n* Main concern with the current version of the manuscript is that it is hard to follow. The paper is overloaded with concepts that require a background. Supplementary material could be used to develop basic understanding of the preliminaries presented in section 3, while the section 3 in itself can relate these concepts to the representation of an articulated model.\n* No visuals to support sections 3 and 4. \n* No qualitative examples of the articulated objects along with their model estimates and uncertainties are provided to understand the results.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper presents a novel representation for distribution over rigid body transformations and articulation model parameters. The paper attempts to describe the theoretical background in developing this background. However, the major concern with the current version of the paper is that it is dense and packed with concepts that require background. This makes it hard to judge the claims of the paper. The quantitative results presented are promising. However, the lack of qualitative results is a weakness. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "The issues are listed above. ", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "OVqQQOxR7SH", "reviewer": "Reviewer_ahNp", "summary": "This work considers the task of predicting the screw parameters of a singly-articulated object directly from a sequence of images, as well as predicting the current rotation and displacement of the joint in each image. This specific task was introduced by ScrewNet (Jain et. al., 2020). In contrast with previous work, this work proposes to learn a probability distribution over the screw parameters, rather than a direct regression of a point estimate. Previous works in other settings have proposed distributions over screw parameters (for instance, the Bingham distribution), but such distributions suffer from the double-cover problem (or more generally, “non-unique estimation targets”). To compensate for this, the work proposes a novel parameterization of Plucker coordinates that evades the double-cover problem. Their parameterization can be factorized into a point on a Stiefel manifold and a set of positive integers, over which they can further parameterize well-studied distributions (von Mises-Fisher distribution and truncated Gaussian distributions, respectively). With this parameterization in hand, the work proposes a standard neural network architecture and training procedure to learn a distribution based on sequences of data drawn from three settings: an objects dataset from Abbatematteo et al, PartNet-Mobility, and a real-world dataset. All settings contain household objects with relatively similar door/drawer structures. When compared against SOTA methods and reasonable probabilistic baselines in noise-free settings, the work demonstrates strong performance on point estimation metrics for various components of the predicted parameter space. Furthermore, the paper convincingly demonstrates the ability to capture uncertainty over predicted parameters, particularly in settings where noise is deliberately introduced to labels at training time.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n**Principled approach:**\nThe paper does a good job of considering the parameterization of distributions over Plucker coordinates, highlighting the substantial downsides of previous approaches, and proposing a principled factorization of the screw coordinates to mitigate these downsides. They choose a particularly well-understood distribution (von Mises-Fisher), which allows them to make intuitive choices for architecture and training without sacrificing theoretical soundness. Many papers fail to carefully consider the difficulties of parameterizing these structures in such a way that a neural network can learn them; the proposed method tackles the problem in a principled fashion.\n\n**Novelty:**\nA major positive aspect of this paper is its novelty and generality. While other works have variously proposed probabilistic representations of articulations, and developed category-independent articulation prediction models, this paper is the first I am aware of that unifies deep probabilistic articulation prediction by proposing 1) a parameterization that is common to all model categories, 2) a principled distribution over that parameterization, and 3) a neural network model and training algorithm that learns to predict parameters of this distribution directly from depth images. \n\n### Major Weaknesses:\n\n**Assumptions not clearly stated:**\nThe paper implicitly assumes that the probability distributions over regions of the screw space (l and m-hat; |m|) are independent; the paper also implicitly assumes that the individual displacement estimates for each input image are independent; these assumptions should be explicitly stated.\n\n**Independence assumptions are not valid:**\nThis work assumes (implicitly) that an independent factorization of the probability distributions is an appropriate representation of the label space. Particularly troubling is the assumed independence between the screw parameters and the observed displacement/rotation values; presumably, these are highly dependent. Consider a case where there is uncertainty between whether the articulation is revolute or prismatic; the displacement/rotation parameters will be quite different in these cases, and so there is naturally significant covariance of these properties. Similarly, treatment of the sequence of displacement/rotation parameters as IID is likely unrealistic, given the sequential structure of the dataset (presumably watching videos of the object actuate), and joint limits present for an object. The work should either 1) consider restructuring the probability distributions to account for possible covariance, 2) demonstrate that the neural network can adequately account for these potential covariances implicitly 3) make an explicit argument for why this independence assumption is sufficient for the task, or 4) State explicitly that this is a limitation of the proposed method.\n\n**Unclear presentation of results:**\nThe results are not clearly presented, and the significance of each metric is not clearly explained. It is difficult to interpret the duplicated columns, and it is difficult to interpret when higher/lower metrics are better (I recommend putting up / down arrows next to each metric to indicate this). Since the proposed method does not outperform all other methods across the board, it’s difficult to interpret which of the metrics are more important than the others. Furthermore, each table should have a detailed description of each column, and for readability, table captions should have a short interpretation or discussion of the results in the table.\n\n\n### Minor weaknesses:\n\nMissing experiment:\nIt would be interesting to include an ablation of DUST-Net in a local frame to see if further improvements can be obtained.\n\nMissing precise evaluation of uncertainty:\nWhen demonstrating that the network captures uncertainty (Sec 5.2), the paper only shows a monotonic correlation between input noise and certainty; no quantitative comparison is given (e.g. comparing the noise distribution with the uncertainty distribution, which may be possible since both distributions are closed form)\n\nIn the parameterization, why are d and theta considered independently at each timestep, instead of predicting the pitch h of the articulation for the whole sequence and a single h for the sequence?\n\nUsing a “ReLU-6” module in the network seems to be imprecise, since the purported limit for the parameters is 2pi, not 6.\n\nIt would be helpful to perform an ablation on the training procedure to show the benefit of their three-stage approach. \n\nHow does the work arrive at the conclusion that the vm-SoftOrtho suffers because of the soft constraint? Could the work include some measure of how much orthogonality is violated, for typical predictions?\n\nIn the uncertainty estimation section, it would be helpful to explain in more detail what the uncertainty parameters mean, particularly in the summary tables. A clearer way to present this information is by giving an estimate of the 95th-percentile confidence intervals and aggregating in some way, to ground the metrics intuitively. \n\nFor the ScrewNet performance in Table 4, is this in “local” or “camera frame” coordinates?\n\nThis paper would be stronger if it included an example of how to use the knowledge of this uncertainty in a downstream task. For instance, the paper doesn’t give a procedure for how to induce a posterior over this distribution should new observations be obtained.\n\nThe presentation of the Preliminaries (Sec 3) is quite abstract; the presentation would benefit by first grounding both the distribution and the manifolds in the practical instances (i.e. V_2,3), which is easier to grasp intuitively than V_n,k. It took a close reading to understand how the abstract form related to the problem domain.\n\nIf possible, a visualization of the uncertainty over multiple examples would lend a lot of clarity to the performance of the method.\n\nTypo:\n121: “prismatic” is repeated twice\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "As far as I can tell, this paper is the first to present a method for learning a probability distribution over screw parameters.  The proposed approach demonstrates superior performance on several metrics across tasks that are directly relevant to robotics. As such, I believe it meets the minimum standard for acceptance. However, I am hesitant to give a strong accept, because:\n- No discussion/reasoning for the independence assumptions of their proposed distribution is provided. This undermines the ability for the method to claim to be entirely principled.\n- The presentation of the results is not clear (both the text and the design of the tables), and no intuition is given for how to interpret the numerical performance on each of the provided metrics. Why are the metrics that the proposed method performs well on more important than the metrics in which it doesn’t?\n- The actual utility of modeling uncertainty is not demonstrated. Not enough motivation is given on how this information could be used in a downstream task.\n\n# Post-rebuttal Update\nI thank the authors for making the suggested changes. I believe that the clarity of the paper has improved. Most of my questions have been sufficiently resolved. \n\nHowever, I still think the limitations of the distribution approximation (discussed in the initial review and the rebuttal) should be more clearly stated.\n\n> To maintain the numerical tractability of the solution, we approximate the probability density function of the conditional distribution as a Dirac delta function centered at the expected value of the distribution over the screw axis parameters. We have included further details and discussion on the topic in the revised supplementary material for the work.\n\nThis approximation could have a significant effect on the result; I believe that it should be mentioned (at least briefly) in the main text (and then refer the reader to the supplement for additional details). For example, if the variance of the screw parameters is relatively high, then there will be much probability mass far from the expected value, and this approximation will likely be poor. This is fine, but the approximation should be explained clearly to the reader in the main text.\n\nOverall, looking at the author response and the other reviews, my score remains unchanged.\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "- Either give a rationale for the independence assumption made by the distribution design, update the distribution design to take potential covariance into account, or state this clearly as a limitation of the proposed method.\n- Improve the clarity and impact of the results section (as detailed above).\n"}}, {"review_id": "DmXczZpFiKd", "reviewer": "Reviewer_Libh", "summary": "Given a sequence of depth images depicting an object undergoing an articulation motion, this paper tackles the task of predicting: a) a parametrization of the articulation joint, and b) the per-timestep configuration. It does so by building upon a prior supervised approach \"ScrewNet\" that presents a unified representation (screw axis) which allows capturing various joint types.\n\nThis approach leverages a similar representation, supervision, architecture, with the key contribution being to predict a distribution instead of a point estimate for the screw axes and corresponding per-timestep parameters. The distribution over the screw axis is parametrized as a von mises fisher distribution, with further factorization in predicting the matrix. The other variables are all represented via truncated gaussian distributions.\n\nThe reported experiments show that this improves the performance over the baseline while also allowing capturing uncertainty in the predictions.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\na) The paper is well-motivated. The recent prior work that tackles this task via a learning-based method only predicts a point-estimate, and extending it to allow capturing the uncertainty is a promising directions.\n\n\nb) The empirical results are (mostly) convincing and demonstrate clear improvements from the proposed approach over the relevant baselines.\n\nc) Additional ablations reported e.g for directly predicting 'F' for the VMF distribution vs the proposed factorization via 4 angles and a diagonal matrix is also interesting and validates the specific choices made.\n\n\n**Weaknesses**\n\n- While capturing uncertainty is desirable, I am not sure the proposed approach is a significant contribution on its own - in particular given that the way the distribution is represented is rather straightforward (except for the screw axis) and the work is a direct extension of ScrewNet.\n\n- As a related point, while the approach does enable representing some distributions, I am concerned about the limited expressivity as:\na) that the proposed approach still only captures a unimodal distribution whereas often they might be multimodal e.g. if one is unsure of joint type (revolute vs prismatic) but sure of geometry, the screw axes may have different modes.\nb) The distribution for the per-timestep parameters is also modeled as a product of independent marginal terms and thus does not represent possible correlation among them.\nOverall, for a work explicitly aimed at modeling the uncertainty in the representations, I feel this work only takes small initial step.\n\n- A minor concern regarding the evaluations Tables 1,2 is that they claim a higher precision i.e lambda_i, and \\beta_j is desirable. However, it is not clear why they should be arbitrarily high even for non-noisy training data (perhaps it maybe only possible to recover them upto a certain degree given perceptual inputs). I think these estimates could be removed from these Tables as it is unclear if higher is better.\n\n\n----\nUpdate:\nI’d like to thank the authors for their response, and in particular for the explanation of contributions over ScrewNet. However, I am still not convinced regarding the generality of the proposed uncertainty representation. Moreover, some of the responses to the implementation “issues” further increased this feeling (e.g. an adhoc procedure for deciding a unique “ground-truth” screw axis when multiple solutions are acceptable seems suboptimal).\n\nBut more importantly, I am not convinced the independent inference of model parameters and screw transform is a benefit. The response claims that “in the cases when there are ambiguities in the object's articulation type, DUST-Net will predict distributions over both observed joint configurations to be centered at non-zero values”. However, this only works because the two types of transforms considered here only affect disjoint parameters (either \\theta, or \\d).\nConsidering an extension of this approach, say including helical joints, if the model is confused between two possible axes (say s_1 and s_2), and each of the correspondingly has different transforms the model would want to infer (say theta_1, and theta_2). The proposed approach cannot handle this for two reasons: i) the prediction over theta is multimodal, and ii) more critically, this distribution is conditional on the screw transform (e.g. theta_1 for s_1, and theta_2 for s_2).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, while the work is well-motivated in aiming to predict a distribution and also yields empirical gains, I am not excited regarding the technical contributions and also skeptical about the generality of the distributions (as they cannot capture multimodal outputs).", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {"issues": "I have some minor concerns/clarifications regarding the precise formulations and task definitions:\na) The per-timestep parameters \\theta and \\d are modeled via truncated gaussians, but I am not sure why these can't have negative values e.g. if a joint moves back and forth around an initial configuration?\n\nb) While the training objective is to maximize the likelihood of the ground-truth, I am slightly confused by how a unique screw axis is unambiguously defined as the ground-truth e.g. for a prismatic joint, wouldn't any axis parallel to the part motion suffice?\n\nc) I appreciate that both theta and d need to be predicted for a unified representation, but isn't only one of them non-zero depending on the joint type (e.g. in Figure 1)? I understand that a helical motion for example max exhibit both, but given the paper only considers revolute and prismatic joints, only one of these could be non-zero in predictions - and given the independent marginals, this is a correlation that the proposed distributions cannot model."}}, {"review_id": "4K14KQZX2IK", "reviewer": "Reviewer_41iG", "summary": "The paper proposes DUST-net, a deep-learning approach that estimates distributions over articulation models and articulated pose. \nThe model is object-category agnostic, using screw theory to represent revolute & prismatic joints with the same model. \nA novel distribution over screw axes is developed using von Mises-Fisher distributions on Stiefel manifolds which cleanly resolves issues with existing parameterizations and enables data-efficient learning.   \nThe experiments are thorough with thoughtful baselines and ablations. \n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper presents important conceptual improvement over ScrewNet in uncertainty representation, camera-frame estimation. \n- The theory is quite rich and the technical approach is sound. \n- The paper is generally well-written. \n\nWeaknesses\n- Motion of an object part isn't generally available in the wild - I'd recommend an experiment with static objects for a sense of how the approach would work without a human demonstrator for every object. This is a more appropriate comparison to the Abbatematteo paper. \n- I'd like to see a comparison to a generic feature-tracking approach to understand how significant the textured-object limitation is in practice. \n- Real results are disappointing. But, they certainly give credence to the importance of uncertainty estimation. I suspect these could improve with some care taken toward the noise model used in training, as depth images with pixel dropout have been shown to generally transfer fairly well. \n\n- Object category generally provides a useful prior over articulation type and parameters, and is generally available in the wild. This could be readily incorporated into the model.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I am recommending weak acceptance. The absence of successful real results is disappointing but I think the theory is sufficiently interesting and the simulation experiments are sufficiently thorough. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Minor issues. \nLine 92 \"Guassian\"\nLine 195 \"whit\" \nLine 318 \"form\" \nTable 4 caption lacking period", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: According to Reviewer ahNp, this is the first work to present a method for learning a probability distribution over screw parameters. The paper chooses a mathematically principled distribution (VMF) to represent a distribution over screw parameters without ambiguity. The paper empirically motivates a distributional approach over a point estimate of screws. 

Two of the reviewers believe the paper to have new ideas that will be relevant to robotics / ML, while two reviewers believe the work to be technically correct but incremental. In particular, Reviewer ahNp raises three major issues required to lift their rating, which I am inclined to agree with:
```
- No discussion/reasoning for the independence assumptions of their proposed distribution is provided. This undermines the ability for the method to claim to be entirely principled.
- The presentation of the results is not clear (both the text and the design of the tables), and no intuition is given for how to interpret the numerical performance on each of the provided metrics. Why are the metrics that the proposed method performs well on more important than the metrics in which it doesn’t?
- The actual utility of modeling uncertainty is not demonstrated. Not enough motivation is given on how this information could be used in a downstream task.
```
I would also like the authors to address the questions in the "Issues" section brought up by Reviewer Libh.

This paper does not apply distributional depth-based articulation estimates to an actual robotic control task, so it isn't obvious to the reviewers and myself what the utility of the distributional representation is until we see it on the robot. I will note that there was a concurrent work submitted (https://openreview.net/forum?id=bEito8UUUmf) that shows that uncertainty-aware refinement of depth predictions does lead to substantial performance improvements on a detection task, so perhaps something similar might go on here. The authors should suggest in the paper how exactly the distributional representation of the screw predictions can be utilized to confer benefits over point estimates of screws. For example, a distributional representation might allow for running a test-time MCMC sampling procedure to resolve the covariate issue raised by ahNp, as one way to address the modeling of co-varying screws.

---- 20210904

The authors have addressed my concerns and the majority of reviewers now vote to accept, so I will be recommending an accept. I think broadly all the reviewers agree that connecting the distributional representations to an actual robotics task is the next step in this work.
Decision: Accept (Poster)