Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: Given a sequence of depth images depicting an object undergoing an articulation motion, this paper tackles the task of predicting: a) a parametrization of the articulation joint, and b) the per-timestep configuration. It does so by building upon a prior supervised approach "ScrewNet" that presents a unified representation (screw axis) which allows capturing various joint types.

This approach leverages a similar representation, supervision, architecture, with the key contribution being to predict a distribution instead of a point estimate for the screw axes and corresponding per-timestep parameters. The distribution over the screw axis is parametrized as a von mises fisher distribution, with further factorization in predicting the matrix. The other variables are all represented via truncated gaussian distributions.

The reported experiments show that this improves the performance over the baseline while also allowing capturing uncertainty in the predictions.
strengths_and_weaknesses: **Strengths**

a) The paper is well-motivated. The recent prior work that tackles this task via a learning-based method only predicts a point-estimate, and extending it to allow capturing the uncertainty is a promising directions.


b) The empirical results are (mostly) convincing and demonstrate clear improvements from the proposed approach over the relevant baselines.

c) Additional ablations reported e.g for directly predicting 'F' for the VMF distribution vs the proposed factorization via 4 angles and a diagonal matrix is also interesting and validates the specific choices made.


**Weaknesses**

- While capturing uncertainty is desirable, I am not sure the proposed approach is a significant contribution on its own - in particular given that the way the distribution is represented is rather straightforward (except for the screw axis) and the work is a direct extension of ScrewNet.

- As a related point, while the approach does enable representing some distributions, I am concerned about the limited expressivity as:
a) that the proposed approach still only captures a unimodal distribution whereas often they might be multimodal e.g. if one is unsure of joint type (revolute vs prismatic) but sure of geometry, the screw axes may have different modes.
b) The distribution for the per-timestep parameters is also modeled as a product of independent marginal terms and thus does not represent possible correlation among them.
Overall, for a work explicitly aimed at modeling the uncertainty in the representations, I feel this work only takes small initial step.

- A minor concern regarding the evaluations Tables 1,2 is that they claim a higher precision i.e lambda_i, and \beta_j is desirable. However, it is not clear why they should be arbitrarily high even for non-noisy training data (perhaps it maybe only possible to recover them upto a certain degree given perceptual inputs). I think these estimates could be removed from these Tables as it is unclear if higher is better.


----
Update:
I’d like to thank the authors for their response, and in particular for the explanation of contributions over ScrewNet. However, I am still not convinced regarding the generality of the proposed uncertainty representation. Moreover, some of the responses to the implementation “issues” further increased this feeling (e.g. an adhoc procedure for deciding a unique “ground-truth” screw axis when multiple solutions are acceptable seems suboptimal).

But more importantly, I am not convinced the independent inference of model parameters and screw transform is a benefit. The response claims that “in the cases when there are ambiguities in the object's articulation type, DUST-Net will predict distributions over both observed joint configurations to be centered at non-zero values”. However, this only works because the two types of transforms considered here only affect disjoint parameters (either \theta, or \d).
Considering an extension of this approach, say including helical joints, if the model is confused between two possible axes (say s_1 and s_2), and each of the correspondingly has different transforms the model would want to infer (say theta_1, and theta_2). The proposed approach cannot handle this for two reasons: i) the prediction over theta is multimodal, and ii) more critically, this distribution is conditional on the screw transform (e.g. theta_1 for s_1, and theta_2 for s_2).

summary_of_recommendation: Overall, while the work is well-motivated in aiming to predict a distribution and also yields empirical gains, I am not excited regarding the technical contributions and also skeptical about the generality of the distributions (as they cannot capture multimodal outputs).
issues: I have some minor concerns/clarifications regarding the precise formulations and task definitions:
a) The per-timestep parameters \theta and \d are modeled via truncated gaussians, but I am not sure why these can't have negative values e.g. if a joint moves back and forth around an initial configuration?

b) While the training objective is to maximize the likelihood of the ground-truth, I am slightly confused by how a unique screw axis is unambiguously defined as the ground-truth e.g. for a prismatic joint, wouldn't any axis parallel to the part motion suffice?

c) I appreciate that both theta and d need to be predicted for a unified representation, but isn't only one of them non-zero depending on the joint type (e.g. in Figure 1)? I understand that a helical motion for example max exhibit both, but given the paper only considers revolute and prismatic joints, only one of these could be non-zero in predictions - and given the independent marginals, this is a correlation that the proposed distributions cannot model.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a method to learn distributions over articulation models from a sequence of depth images without any known articulation model a priori. To learn the distribution, the paper first presents a novel representation for distribution over rigid body transformations and articulation model parameters using Screw theory, von Mises-Fisher distributions and Stiefel manifolds. Then the paper introduces DUST-net, a network that transforms a sequence of depth images into a distribution over the transformations between them. With experiments on two benchmarking datasets (a simulated articulated object dataset and PartNet-Mobility dataset) and three real-world objects, it is shown that DUST-net successfully learns the distribution over articulation models and their parameters. 
strengths_and_weaknesses: Strengths:
* The paper addresses an important problem of perceiving articulated objects from a sequence of observations without any prior knowledge of the object model or its articulation constraints.
* The paper attempts to present a suitable representation that can capture distribution over transformations and articulation parameters. If this representation has the benefits as claimed in the paper, then it could be used for part based object estimation frameworks.
* The presented representation lets the proposed method perform category-independent articulation model estimation along with model uncertainties.

Questions and areas for improvement:
* In Section 3, especially the background section needs to be closely knit with the representation of an articulated model. Currently, it is hard for the reader to relate these concepts with the problem this paper is trying to address.
* The relationship between Screw Transformations, Stiefel manifold and Matrix von Mises-Fisher distribution is unclear at Section 3. The fundamental background of these concepts can be in the supplementary material, while their relationship and relevance to the representation of articulated models can be described in the main manuscript. 
* It is unclear what Figure 2 is trying to illustrate. 
* Visuals or illustrations to support the description in Section 4 would be beneficial. 
* In Tables 1, 2, and 3, it is unclear what “Uncertainty” values indicate. It is described in the text that the uncertainty is high on real-world experiments, whereas the numbers are less in Table 3 compared to 1 and 2. 
* In Section 5.2, the text - “We evaluate this by adding artificial noise to the training labels form the two simulated datasets while training DUST-net”. It is not clear how training with noise leads to this evaluation. Is this a different training as opposed to the original for estimating the articulation models parameters? The reader would imagine that there is one training phase overall, that includes the addition of noise to the labels. 
* It is described in the Section 5.3 that the estimation is accurate in the real-world setting. How will the proposed approach handle partial observations caused due to self-occlusions and robot occlusions during the manipulation action, in addition to the depth noise from the sensor data? Is it possible to characterize the uncertainty due to sensing and the scenario differently?

Weakness:
* Main concern with the current version of the manuscript is that it is hard to follow. The paper is overloaded with concepts that require a background. Supplementary material could be used to develop basic understanding of the preliminaries presented in section 3, while the section 3 in itself can relate these concepts to the representation of an articulated model.
* No visuals to support sections 3 and 4. 
* No qualitative examples of the articulated objects along with their model estimates and uncertainties are provided to understand the results.

summary_of_recommendation: The paper presents a novel representation for distribution over rigid body transformations and articulation model parameters. The paper attempts to describe the theoretical background in developing this background. However, the major concern with the current version of the paper is that it is dense and packed with concepts that require background. This makes it hard to judge the claims of the paper. The quantitative results presented are promising. However, the lack of qualitative results is a weakness. 
issues: The issues are listed above.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes DUST-net, a deep-learning approach that estimates distributions over articulation models and articulated pose. 
The model is object-category agnostic, using screw theory to represent revolute & prismatic joints with the same model. 
A novel distribution over screw axes is developed using von Mises-Fisher distributions on Stiefel manifolds which cleanly resolves issues with existing parameterizations and enables data-efficient learning.   
The experiments are thorough with thoughtful baselines and ablations. 

strengths_and_weaknesses: Strengths
- The paper presents important conceptual improvement over ScrewNet in uncertainty representation, camera-frame estimation. 
- The theory is quite rich and the technical approach is sound. 
- The paper is generally well-written. 

Weaknesses
- Motion of an object part isn't generally available in the wild - I'd recommend an experiment with static objects for a sense of how the approach would work without a human demonstrator for every object. This is a more appropriate comparison to the Abbatematteo paper. 
- I'd like to see a comparison to a generic feature-tracking approach to understand how significant the textured-object limitation is in practice. 
- Real results are disappointing. But, they certainly give credence to the importance of uncertainty estimation. I suspect these could improve with some care taken toward the noise model used in training, as depth images with pixel dropout have been shown to generally transfer fairly well. 

- Object category generally provides a useful prior over articulation type and parameters, and is generally available in the wild. This could be readily incorporated into the model.  
summary_of_recommendation: I am recommending weak acceptance. The absence of successful real results is disappointing but I think the theory is sufficiently interesting and the simulation experiments are sufficiently thorough. 
issues: Minor issues. 
Line 92 "Guassian"
Line 195 "whit" 
Line 318 "form" 
Table 4 caption lacking period
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work considers the task of predicting the screw parameters of a singly-articulated object directly from a sequence of images, as well as predicting the current rotation and displacement of the joint in each image. This specific task was introduced by ScrewNet (Jain et. al., 2020). In contrast with previous work, this work proposes to learn a probability distribution over the screw parameters, rather than a direct regression of a point estimate. Previous works in other settings have proposed distributions over screw parameters (for instance, the Bingham distribution), but such distributions suffer from the double-cover problem (or more generally, “non-unique estimation targets”). To compensate for this, the work proposes a novel parameterization of Plucker coordinates that evades the double-cover problem. Their parameterization can be factorized into a point on a Stiefel manifold and a set of positive integers, over which they can further parameterize well-studied distributions (von Mises-Fisher distribution and truncated Gaussian distributions, respectively). With this parameterization in hand, the work proposes a standard neural network architecture and training procedure to learn a distribution based on sequences of data drawn from three settings: an objects dataset from Abbatematteo et al, PartNet-Mobility, and a real-world dataset. All settings contain household objects with relatively similar door/drawer structures. When compared against SOTA methods and reasonable probabilistic baselines in noise-free settings, the work demonstrates strong performance on point estimation metrics for various components of the predicted parameter space. Furthermore, the paper convincingly demonstrates the ability to capture uncertainty over predicted parameters, particularly in settings where noise is deliberately introduced to labels at training time.
strengths_and_weaknesses: ### Strengths

**Principled approach:**
The paper does a good job of considering the parameterization of distributions over Plucker coordinates, highlighting the substantial downsides of previous approaches, and proposing a principled factorization of the screw coordinates to mitigate these downsides. They choose a particularly well-understood distribution (von Mises-Fisher), which allows them to make intuitive choices for architecture and training without sacrificing theoretical soundness. Many papers fail to carefully consider the difficulties of parameterizing these structures in such a way that a neural network can learn them; the proposed method tackles the problem in a principled fashion.

**Novelty:**
A major positive aspect of this paper is its novelty and generality. While other works have variously proposed probabilistic representations of articulations, and developed category-independent articulation prediction models, this paper is the first I am aware of that unifies deep probabilistic articulation prediction by proposing 1) a parameterization that is common to all model categories, 2) a principled distribution over that parameterization, and 3) a neural network model and training algorithm that learns to predict parameters of this distribution directly from depth images. 

### Major Weaknesses:

**Assumptions not clearly stated:**
The paper implicitly assumes that the probability distributions over regions of the screw space (l and m-hat; |m|) are independent; the paper also implicitly assumes that the individual displacement estimates for each input image are independent; these assumptions should be explicitly stated.

**Independence assumptions are not valid:**
This work assumes (implicitly) that an independent factorization of the probability distributions is an appropriate representation of the label space. Particularly troubling is the assumed independence between the screw parameters and the observed displacement/rotation values; presumably, these are highly dependent. Consider a case where there is uncertainty between whether the articulation is revolute or prismatic; the displacement/rotation parameters will be quite different in these cases, and so there is naturally significant covariance of these properties. Similarly, treatment of the sequence of displacement/rotation parameters as IID is likely unrealistic, given the sequential structure of the dataset (presumably watching videos of the object actuate), and joint limits present for an object. The work should either 1) consider restructuring the probability distributions to account for possible covariance, 2) demonstrate that the neural network can adequately account for these potential covariances implicitly 3) make an explicit argument for why this independence assumption is sufficient for the task, or 4) State explicitly that this is a limitation of the proposed method.

**Unclear presentation of results:**
The results are not clearly presented, and the significance of each metric is not clearly explained. It is difficult to interpret the duplicated columns, and it is difficult to interpret when higher/lower metrics are better (I recommend putting up / down arrows next to each metric to indicate this). Since the proposed method does not outperform all other methods across the board, it’s difficult to interpret which of the metrics are more important than the others. Furthermore, each table should have a detailed description of each column, and for readability, table captions should have a short interpretation or discussion of the results in the table.


### Minor weaknesses:

Missing experiment:
It would be interesting to include an ablation of DUST-Net in a local frame to see if further improvements can be obtained.

Missing precise evaluation of uncertainty:
When demonstrating that the network captures uncertainty (Sec 5.2), the paper only shows a monotonic correlation between input noise and certainty; no quantitative comparison is given (e.g. comparing the noise distribution with the uncertainty distribution, which may be possible since both distributions are closed form)

In the parameterization, why are d and theta considered independently at each timestep, instead of predicting the pitch h of the articulation for the whole sequence and a single h for the sequence?

Using a “ReLU-6” module in the network seems to be imprecise, since the purported limit for the parameters is 2pi, not 6.

It would be helpful to perform an ablation on the training procedure to show the benefit of their three-stage approach. 

How does the work arrive at the conclusion that the vm-SoftOrtho suffers because of the soft constraint? Could the work include some measure of how much orthogonality is violated, for typical predictions?

In the uncertainty estimation section, it would be helpful to explain in more detail what the uncertainty parameters mean, particularly in the summary tables. A clearer way to present this information is by giving an estimate of the 95th-percentile confidence intervals and aggregating in some way, to ground the metrics intuitively. 

For the ScrewNet performance in Table 4, is this in “local” or “camera frame” coordinates?

This paper would be stronger if it included an example of how to use the knowledge of this uncertainty in a downstream task. For instance, the paper doesn’t give a procedure for how to induce a posterior over this distribution should new observations be obtained.

The presentation of the Preliminaries (Sec 3) is quite abstract; the presentation would benefit by first grounding both the distribution and the manifolds in the practical instances (i.e. V_2,3), which is easier to grasp intuitively than V_n,k. It took a close reading to understand how the abstract form related to the problem domain.

If possible, a visualization of the uncertainty over multiple examples would lend a lot of clarity to the performance of the method.

Typo:
121: “prismatic” is repeated twice

summary_of_recommendation: As far as I can tell, this paper is the first to present a method for learning a probability distribution over screw parameters.  The proposed approach demonstrates superior performance on several metrics across tasks that are directly relevant to robotics. As such, I believe it meets the minimum standard for acceptance. However, I am hesitant to give a strong accept, because:
- No discussion/reasoning for the independence assumptions of their proposed distribution is provided. This undermines the ability for the method to claim to be entirely principled.
- The presentation of the results is not clear (both the text and the design of the tables), and no intuition is given for how to interpret the numerical performance on each of the provided metrics. Why are the metrics that the proposed method performs well on more important than the metrics in which it doesn’t?
- The actual utility of modeling uncertainty is not demonstrated. Not enough motivation is given on how this information could be used in a downstream task.

# Post-rebuttal Update
I thank the authors for making the suggested changes. I believe that the clarity of the paper has improved. Most of my questions have been sufficiently resolved. 

However, I still think the limitations of the distribution approximation (discussed in the initial review and the rebuttal) should be more clearly stated.

> To maintain the numerical tractability of the solution, we approximate the probability density function of the conditional distribution as a Dirac delta function centered at the expected value of the distribution over the screw axis parameters. We have included further details and discussion on the topic in the revised supplementary material for the work.

This approximation could have a significant effect on the result; I believe that it should be mentioned (at least briefly) in the main text (and then refer the reader to the supplement for additional details). For example, if the variance of the screw parameters is relatively high, then there will be much probability mass far from the expected value, and this approximation will likely be poor. This is fine, but the approximation should be explained clearly to the reader in the main text.

Overall, looking at the author response and the other reviews, my score remains unchanged.

issues: - Either give a rationale for the independence assumption made by the distribution design, update the distribution design to take potential covariance into account, or state this clearly as a limitation of the proposed method.
- Improve the clarity and impact of the results section (as detailed above).
review_title: null

Meta-review: According to Reviewer ahNp, this is the first work to present a method for learning a probability distribution over screw parameters. The paper chooses a mathematically principled distribution (VMF) to represent a distribution over screw parameters without ambiguity. The paper empirically motivates a distributional approach over a point estimate of screws. 

Two of the reviewers believe the paper to have new ideas that will be relevant to robotics / ML, while two reviewers believe the work to be technically correct but incremental. In particular, Reviewer ahNp raises three major issues required to lift their rating, which I am inclined to agree with:
```
- No discussion/reasoning for the independence assumptions of their proposed distribution is provided. This undermines the ability for the method to claim to be entirely principled.
- The presentation of the results is not clear (both the text and the design of the tables), and no intuition is given for how to interpret the numerical performance on each of the provided metrics. Why are the metrics that the proposed method performs well on more important than the metrics in which it doesn’t?
- The actual utility of modeling uncertainty is not demonstrated. Not enough motivation is given on how this information could be used in a downstream task.
```
I would also like the authors to address the questions in the "Issues" section brought up by Reviewer Libh.

This paper does not apply distributional depth-based articulation estimates to an actual robotic control task, so it isn't obvious to the reviewers and myself what the utility of the distributional representation is until we see it on the robot. I will note that there was a concurrent work submitted (https://openreview.net/forum?id=bEito8UUUmf) that shows that uncertainty-aware refinement of depth predictions does lead to substantial performance improvements on a detection task, so perhaps something similar might go on here. The authors should suggest in the paper how exactly the distributional representation of the screw predictions can be utilized to confer benefits over point estimates of screws. For example, a distributional representation might allow for running a test-time MCMC sampling procedure to resolve the covariate issue raised by ahNp, as one way to address the modeling of co-varying screws.

---- 20210904

The authors have addressed my concerns and the majority of reviewers now vote to accept, so I will be recommending an accept. I think broadly all the reviewers agree that connecting the distributional representations to an actual robotics task is the next step in this work.
Decision: Accept (Poster)