Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposed a theory for generalized manifold constrained problems in robotics and presents a framework for generative modeling of it. The generative models are constructed with Wasserstein GAN. The authors also present a graphical model for modeling the distribution of sequential problems in robotics. Experiments are shown on three robotics manipulation tasks: Pick-and-Place, Assembly, and Handover. 
strengths_and_weaknesses: Strength:

- The theory presented in this paper is novel in terms of problem formulation and solution. The theory of generative modeling on the constrained manifold is a generalization to many problems in robotics. The idea of incorporating graphical models for sequential tasks is interesting. 
- The presentation is overall clear, especially with clear math formulas.
- The experiment is extensive with 3 different scenarios to show the effectiveness of the proposed method.

Weakness:

- What is the deep model used in this paper? It is not very clear. Do both RGB images and depth images used in the experiments? What deep model is used in the experiments?
- The experiment results are hard to interpret. How to connect "Average number of trials to find a solution", precision and coverage to more commonly used metrics for manipulation tasks such as success rate? 
- The author can further show some failure cases and explain the reason for the failure.
- The visualizations of the scenarios are hard to interpret.
  - For example, how are the two middle images in Fig 1 connected by constraint manifold projection?
- The baseline compared against in the experiment is weak. The author should at least compare against a deterministic deep learned approach.



summary_of_recommendation: Though there are some technical details missing and stronger baseline(s) needed, the paper overall would interesting to the community. It re-formulates many problems in robotics as a generalized problem and solves it with generative modeling, which could be valuable and inspiring to the community.
issues: The authors can address the items under "Weakness" in my main review section.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper present a generative model to sample near a constraint manifold such that they can be used within a nonlinear optimizer to handle constraints. This is applied on sequential manipulation problems with some factorizations (to capture multi-modal distributions).

strengths_and_weaknesses: Strengths
+ Leveraging learning and working with images to solve tasks with a hybrid (discreet + continuous) action space has a lot of promising applications where pure optimization (w/o learning) approaches fall short due to strong input assumptions and high compute costs.
+ The idea to factorize the state into modular chucks is a nice way to add structure and improve data efficiency (although hand crafting this for every new task seems like a burden).

Weaknesses
- More exposition of the big picture: In Sec 1 or somewhere before Sec 3 (that jumps straight into sampling) could benefit from grounding the problem setup a bit i.e. where does the sampling fit in the larger picture of nonlinear optimization that solves some example  manipulation task?
- Can Eq. (2) alternatively be setup as a piece-wise nonlinear optimization (wherein the constraint pieces are differentiable) where the discreet decisions are either learned or sampled with heuristics or combinatorially searched? What are the trade-offs and implications here and should these be potential baselines?
- What changes between train and test settings? The variations in each of the problem scenarios seems limited so the generalization capacity is unclear.
- The overall computational time stats are helpful. It would also be helpful to get a sense for how much computation is saved by choosing to learn the sampling vs alternatives while keeping everything else the same.
summary_of_recommendation: Overall the paper has interesting ideas about using learning as 'one' tool in a box of multiple other tools and them them together towards solving a difficult problem. The current experiments brings in concerns about scaling and generalization.

issues: See weaknesses
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper presents Deep Generative Constraint Sampling (DGCS), a novel constraint manifold sampling for sequential manipulation. The proposed framework combines a deep generative sampling model -conditioned on an image-based representation of the task, and trained with a regularization using a Wasserstein distance to a reference probability distribution- and a nonlinear optimizer for sample projection onto the manifold. Moreover, the learning framework are infused with a structure based on a given Directed Graphical Models factorization of the problem, by training a sequence of conditional generative models rather a full joint generative model. The experimental results confirm that the proposed framework achieved efficiency, diversity, and precision of the sampling approach, outperforming baselines and heuristic warmstart initialization.
strengths_and_weaknesses: Strengths:
The paper presents an interesting view on a probabilistic technique that combines structure and learning to achieve superior results in robotic sequential manipulation tasks.

Moreover, the paper is also well-written, and relatively easy to be read.

Weaknesses:
Only a few minor points in the "Issues" section.
summary_of_recommendation: I think the paper present an interesting technique of mixing structure and learning in a probabilistically-sound manner which deserves a publication.
issues: (1) I suggest to list down the all the known limitations of the framework/method which open venues for future works/extension of this work.
(2) In Table 1, what is the unit of the Error (I assume it's not unitless and not normalized)? Does this "Error" corresponds to "Constraint Violation" metric as mentioned in section 6.3? Maybe it's worth to make the terminology consistent between these two, to make it more readable/understandable.
review_title: null

Meta-review: *I am the emergency meta-reviewer assigned to this paper, which is why there wasn't a meta-review provided before the rebuttal.* 

The reviewers have unanimously voted to accept the paper, so that is my recommendation as well. 

Pros:
- The authors propose using a generative model to improve the sample efficiency of sampling poses for non-linear optimization with respect to robot poses, such that the trajectory over poses solves the task ("the constraint manifold").

Cons:
- a formal definition of constraint manifolds is provided in the first sentence of the paper, but it is not clear what specific kind of robotic constraint is formulated as a constraint manifold until L99 of the paper. Overall I found the paper difficult to understand. 
Decision: Accept (Poster)