Structured deep generative models for sampling on constraint manifolds in sequential manipulation
Authors: Joaquim Ortiz-Haro
Jung-Su Ha
Danny Driess
Marc Toussaint
Venue: CoRL2021 Poster (2021)
Keywords: Generative Models
Nonlinear Optimization
Constraint Graph
Robotic Sequential Manipulation

Abstract: Sampling efficiently on constraint manifolds is a core problem in robotics. We propose Deep Generative Constraint Sampling (DGCS), which combines a deep generative model for sampling close to a constraint manifold with nonlinear constrained optimization to project to the constraint manifold. The generative model is conditioned on the problem instance, taking a scene image as input, and it is trained with a dataset of solutions and a novel analytic constraint term. To further improve the precision and diversity of samples, we extend the approach to exploit a factorization of the constrained problem. We evaluate our approach in two problems of robotic sequential manipulation in cluttered environments. Experimental results demonstrate that our deep generative model produces diverse and precise samples and outperforms heuristic warmstart initialization. 

Structured deep generative models for sampling on constraint manifolds in sequential manipulation
Joaquim Ortiz-Haro Jung-Su Ha
Danny Driess
Marc Toussaint
TU Berlin
Abstract: Sampling efficiently constraint manifolds is core problem in robotics. We propose Deep Generative Constraint Sampling (DGCS), which com- bines deep generative model for sampling close constraint manifold with nonlinear constrained optimization to project to the constraint manifold. The gen- erative model is conditioned input, and it is trained with a dataset of solutions and novel analytic constraint term. To further improve the precision and diversity of samples, we extend the approach to exploit a factorization of the constrained problem. We evaluate our approach in two problems of robotic sequential manipulation in cluttered environ- ments. Experimental results demonstrate that our deep generative model produces diverse and precise samples and outperforms heuristic warmstart initialization.
the problem instance, taking scene image
Keywords:
Generative Models, Nonlinear Optimization, Constraint Graph,
Robotic Sequential Manipulation
Introduction
We consider the problem of sampling points constraint manifold, i.e. finding diverse solutions of a nonlinear mathematical program without costs. Such problems arise throughout robotics, in particular in solving sequential manipulation problems, will be the focus in our application. We assume that the constraints are given in terms of piecewise differentiable equalities and inequalities. This allows to leverage constrained optimization methods project a randomly sampled config- urations onto the manifold. However, for highly non-linear constraints and disconnected solution manifolds, local optimization methods can get trapped in local optima and fail to find feasible solution. Further, our aim is to generate diverse and covering set of points on the constraint man- ifold, and to reduce the computation time needed by the local optimization. The crucial challenge therefore is to first sample diversely and close to the constraint manifold, so that the local optimizer can efficiently project solutions covering the constraint manifold.
We follow a learning-based approach, where we assume that a dataset of feasible points for various problem instances is available. Reusing precomputed data on similar problems is promising ap- proach [1, 2, 3]. A fundamental challenge is that the mapping between the problem instance and the feasible manifold is extremely nonlinear and discontinuous [4. 5].
Our approach, therefore, is to train a generative neural model to map a problem instance given in terms of scene image to randomized sample close to the constraint manifold for that instance. An image-based based prediction of solutions to manipulation problems has recently received attention [1, 6, 7, 9, 10] as it provides fixed size parametrization (image resolution) that encodes the obstacles and the objects to be interacted, without the need to engineer problem instance features. Importantly, it is natural way to encode non-fixed number of objects.
Generative Adversarial Networks (GANs) [11, 12) and Variational Autoencoders (VAEs) [13] have proposed a powerful methodology for training such generative model and shown great potential to represent complex distributions in high dimensional spaces. This work adopts the training objec- tive and method of GANs, which is to minimize the divergence between the generative and target distributions. That is, given the training data being the diverse samples on the constraint manifold, the deep generative model is trained produce seeds that lie close the manifold conditioned on the current problem, represented with an image of the scene. These seeds are then used as initializa-
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

tion (warmstart) of a subsequent optimization procedure that projects them to the manifold to find a feasible solution.
Despite the expressive power of function approximators and recent improvements in deep generative models, they still have limitations in generating samples from highly nonlinear and multimodal dis- tribution. Inspired by previous works exploiting an underlying sparse structure of robotic sequential manipulation problems [14, 15, 16, 17, 18] we propose an extension of our generative framework that leverages a facttoizatioon of the constraint problem to model this multimodal distribution more efficiently. Sampling decomposed into a sequence, where for each factor we can train a separate conditional generative model. Sequencing these conditional generative models increases sample efficiency of the training and the multi-modalii of the distributions it can represent.
We demonstrate the system in the context of robotic sequential manipulation, where the problem is to find a sequence of feasible mode-switch configurations, e-g., robot's poses when it picks or places an object. These problems are defined by a set of continuous variables representing sequences of robot configurations, object poses, and relative transformations, with nonlinear, piece-wise differ- entiable constraints, e.g., for grasping, placement, collision avoidance and kinematics, which give a natural factorization of the overall problem. Sampling diverse set of mode-switch configurations efficiently is essential as an inner module of task and motion planning, e.g. to provide waypoints for subsequent trajectory optimization.
In summary, our core contributions are the following:
We introduce Deep Generative Constraint Sampling (DGCS), which combines deep gen- erative models with nonlinear optimization to efficiently generate diverse solutions of con- straint problems. The generative model is trained conditional to scene image to be appli- cable to varying problem instances. As an application, we produce mode-switch configu- rations in sequential manipulation.
We extend the standard data-based training of generative models with analytical features of the feasible manifold to achieve both diversity and sample accuracy.
We extend our system to exploit a factorization of the constraint problem, if given, to fur- ther improve modelling of multi-mode distributions with disconnected support and reduce sample complexity.
2 Related Work
Generative Models in Robotics Recently, deep generative models have successfully been applied in robotics, especially in settings where the problem is represented directly with images or point clouds. For example, for generating grasps of complex objects directly from images [19, 20]. In the context of motion planning [21, 22, 23] and planning [24, 25, 26), generative models are used for sampling informative collision free configurations which greatly improve running time of sampling based algorithms. In this line of research, the general aim is to train a network to directly predict partial or full solutions to a problem. However, none of these works addresses sampling on a high- dimensional constraint manifold, or predicting full multi-robot manipulation sequences. To achieve accurate sampling, we combine learned generative models with local optimization for constraint projection.
Warmstart in Nonlinear optimization In robotics, nonlinear optimization is used to sample on constraint manifolds and optimize trajectories [27, 28, 29]. The goal of recent data based approaches learn warmstart to reduce the online computation time [2. 30]. In settings where the nonlinear programs can be represented with mixed integer constraints, [31, 3, 33] learn an assignment to the integer constraints and run subsequent optimization. In comparison to our method, integer formu- lations use a discrimative model that is easier to train but their formulation is difficult to generalize to problems without a clear integer structure. Recently, [34] applies GANs for inverse kinematics, adding forward kinematics into the network training. In contrast, we use general analytical features in the cost term. Moreover, our framework conditioned on an image representation, and scales to sequential manipulation by exploiting the factorization.

---

Figure I: Sequence of mode-switches (pick handover place) of the handover problem. Our sam- pling framework (DGCS) combines a deep generative model ä ~ Po(r) that produces approximate samples (top row) conditioned on the scene (first column) with nonlinear optimizer that projects them onto the constraint manifold (bottom row).
Sampling on a Constraint Manifold
The problem we address is to generate samples from a manifold M, parametrized by fixed di- mensional (but potentially large, c.g. images) problem parameter T,
M, = fx E IR"
s.t. (x)(=; r) 0, Dinog (2; r) S 0}
(1)
where 0eg R" 1 + R and Onneq (x) T) 18n R R are nonlinear vector- valued functions that are piecewise differentiable. E R represents the current problem instance and parameterizes all the constraints. In the context of robotic manipulation, т represents all the objects in the scene (position, size, shape...,, denotes robot's s and objects' degrees of freedom that are subject to constraints, and oeq. Pineq are (in)equality constraints that describe the problem's objectives such as collision avoidance, grasping. kinematic and pose constraints.
A generative model components (see Alg. 1): randomized seed ă generation and constrained optimization algorithm that projects the seed the solution manifold with the optimization problem (2),
IP(r) that produces feasible solutions (samples &M,) is built from two
min | 2| s.t. deg(2: r) = 0; Diney (ar; T) S 0
In our current implementation, we solve (2) approximately, by running a nonlinear optimizer from the starting point è on the feasibility problem: find x s.t. (z 7) O: (2;e < 0. The initialization and internal regularization of the optimizer provides an implicit regularization with respect ã.
The projection step is a non-convex optimiza- tion problem that has guarantees to produce a feasible sample; especially for a complex se- quential manipulation problem, the optimization landscape often has substantial local optima in- duced by constraints, making non-linear projec- tion prone to fail unless seeds are close to the solution manifold.
Algorithm 1 Sampling on a Constraint Manifold
I: Input: Problem parametrization т, number of trials N
= 0 empty list of samples
3: for i = 1, 2, ....N do
Sample v Po(T)
5: < [T)),, Project to M, with nonlin- ear optimizer (2)
To address such difficulty, we train a seeding dis- tribution (T(T) to approximate reference distri- bution of feasible solutions (7(r), so that can generate diverse samples close to the parametric manifold M, on which the optimizer can easily project. An example is shown in Fig. 1.
6: if :r feasible then
append x to L
8: Output: List of valid samples L
3

---

Training Deep Generative Models to Sample on Constraint Manifolds
Our deep generative model ā ~ Pa(T) with Gz 7), ) ~ Ps where P. is multidimennionall Gaussian distribution and Ga is neural network with parameters 0.
In contrast the standard application of adversarial generative models (image generation), in our setting we also have analytical description of the support of the target distribution, namely the features (22; 7) [dog (2; 7), max (0, (Pine r)]] that describe M,. We could consider data-free gradient based optimization of the analytical constraint violation
min E E 2m~70
[lo(2;
(3)
However, this is extremely ill-posed and can converge to the "deterministic mapping" Go(2, = Gg.r V2 where the model ignores the noise 2 and loses the potential generate diverse distribution.
To enforce sample diversity, we regularize with respect to a reference distribution (r(T) that is diverse and has its support on the manifold, i.e., Ex-P, [); O. Namely, formulate the problem as
min (T(P +
[|((;=)]|1
2~P0
where W the Wasserstein distance, and 3 E R Although the analytical term does not provide additional information than the support of (T),), as will be shown in the experiment section, its contribution is important in the context of function approximation and stochastic gradient descent in non-convex optimization.
4.1 Wasserstein Distance and Adversarial Formulation
The Wasserstein-1 (Earth Mover distance) between two probability distributions P., and Pe (5) is defined intuitively as the cost of geometrically moving the mass from one distribution to the other under an optimal transport plan,
W (P. =
inf IE || vll
(5)
>EII(P, (m.w)ry
where [T,,,, Po) denotes the set of all joint distributions 7(x,
y) with marginals and Po.
Compared to other notions distance such as Jensen-Shannon Divergence or Total Variation, adver- sarial generative models with Wasserstein distances known to have good practical stability and convergence, and are less prone to mode collapse [12, 35J. Furthermore, the Wasserstein distance provides nice interpretation in our application, because takes into account the geometric distance between two distributions: the minimization of the geometric distance (norm in configuration space) distribution supported on the manifold improves the success rate of the subsequent projection procedure.
We minimize the objective function (4) with the Wasserstein GAN [12, 35] formulation. Using Kantorovich duality, the original formulation (5) is transformed into a minimax game between the critic network D and the generator G, which are trained simultaneously with stochastic gradient descent. Specifically, the minimax problem (with our analytical feature) is:
min max
E D(a,7)- E D(ETT) - E (IVD + D z-P.
+
|| (ā; +| (6) ENPa
aNP,
where B, E Ezo and samples between V and 5. Our reference distribution IP, (P) consists of a discrete set of solution-problem pairs {(z,, These data are computed offline by solving large of problems, with focus on getting diverse samples.
Structured Generative Model by Exploiting Factorization
The previous sections treated æ as a single vector variable. While the approach for training a single generative model ã = G)(z;T), ~ P- is powerful, we can further improve the scalability to large problems by exploiting a given factorization of x and sequentially decomposing the sampling procedure into sequence of conditional sampling operations, in Bayesian networks [36].

---

Grasp
Graup
Position
Kin
Kin
Kin
Kin.
Kin
Kin
les
h
Ubs
(a) Handover
(b) Pick and Place
(c) Pick and Place
Figure 2: Constraint graphs (2b, 2a) and sampling network (2c). Additional uni-variable collision constraints are applied to p's and g's.
Figure 3: Sequence of learned deep generative of models for Pick and Place problems.
We assume the full variable x {I1, .... NND can be factored into N vector variables. Simi- larly, the constraints (equalities and inequalities) 6(x;7) are factored into L sets of constraints, {Φr. $2...., Φr.} with 0j R - Rd where aj. by E N are the dimensions of the domain and co-domain of the constraint dj,, and each constraint dyy depends only subset of variables.
x [Ez....,,NN
<(z;T) (
The set of variables and constraints now defines factored mathematical program without costs, also called constraint graph. Such factorization naturally arise in many applications, where each variable has some semantic and geometric meaning. Fig. 2 shows some examples of such constraint graphs in the context of robotic sequential manipulation. Variables correspond to robot joint configurations E I2T or IR, mobile robots base poses r E (2 or IR, relative transformations between objects and grippers t E SE(3) and object positions p E SE(3). These variables are coupled by kinematic, collision avoidance, grasp and pose constraints. The pick-and-plan mental block of sequential manipulation. Longer and complex tasks such handover (Fig. 2a) or an assembly can be represented by chaining "pick and place" with additional constraints.
graph in Fig. 2b is the funda-
5.1 Directed Graphical Model and Sequential Sampling
The factored structure directly implies factorization of the joint probability distribution ă ~ P(r) from which we want sample. As in Bayesian networks we can sequentialize sampling if decide on an ordering of variables that corresponds to a directed acyclic graph. Instead of learning single Gg to produce a full assignment with each factor by using the corresponding marginal distributions of the original data, and the subset of corresponding analytical features in the constraint graph.
Ga(2; T), ~ P... we learn a conditional model for
We illustrate the benefits of this factorization with the Pick-and- problem. The joint distribu- tion for this problem is factorized into (we omit conditioning on for elarity)
P(p, (1, (2) P(p) P(t|p) P(gilt) p(g2lt,))
where p is the placement position the object, is the relative transformation between object and gripper and Ø1. 92 the robot joint configurations at pick and place. The factorization exploits conditional independence between (q1, P) given and (41, 92) given t. We leverage this structure by training sequence of conditional samplers
PP t P.(p). 4 P. (t) , P (t, (t, Fig. 2c and Fig. 3. This factorization can be easily extended to longer manipulation sequences, with the ordering p +
q. See Appendix A for examples of sampling orders and details.
side note, using marginal distributions jointly consistent dataset is necessary, as only the marginals contain useful information of whether partial assignment will admit a full solution. For
5

---

Figure 4: Problem instances in Assembly and Handover
example, when sampling ~ Pp. we want to fulfil the two following constraints (conditioned on the scene): a) position(p) = 0; and b) 3(1, (1, 92) s.t (t, q1. -1-92.200 cannot be evaluated efficiently, but is modelled by the marginal distribution of the data.
(2. p) is feasible. The second constraint
5.2 Note the advantage of factorization for modelling mulimodality
Generating samples from a distribution with disconnected supports with a deep generative model that receives a continuous input noise P. requires infinite gradients in Ge and can only be done approximately. In these cases, training is unstable and sensitive to hyperparameters and architecture.
We can model disconnected distributions more effectively by factoring the full joint probability as a sequence of smaller conditional modules, as confirmed by our experimental results in Section 6.3. The sequencing still requires that each module is able to produce some degree of multimodality. However, once a module the sequence receives a disconnected input in the form of conditioning. it can successfully produce disconnected output. As we chain modules with the ability to generate a small amount of disconnected components given a continuous input, the number of possible dis- connected components of the output grows exponentially with the number of modules in sequence.
Furthermore, from a practical perspective, training smaller modules turned out be more effective. For instance, we observed that the analytical feature [| |) of the joint problem can provide badly-conditioned gradients when evaluated far from the manifold. This issue is alleviated when considering only subsets of constraints and variables. In our preliminary experiments, we also eval- uated GAN frameworks that have a mechanism to model disconnected distribution [37, 38] (these methods require estimate of the number of disconnected components and mechanism to cluster samples into components) but did not find significant improvements.
Experiments
6.1 Image based Problem representation
We use an image-based representation of the problem instance that consists of the depth image and masks. Specifically, - = {d, m1, m2, m,), where d the depth image, and mi, m2, my are three masks that contain, respectively, information from the initial object pose, goal pose placement region, and obstacles, see Fig. 4. In the factored approach, each generative module receives as input only the "relevant" masks, e.g. the sampler for the robot pick configuration receives mask of the obstacles and the initial configuration, but not the goal pose.
The main strength of the image representation is that it can generalize to different object shapes and changing number of obstacles and shapes. Moreover, a depth camera easily available, approximate masks can be computed with image segmentation techniques and it provides good representation of sequential manipulation problems on tabletop scenarios.
6.2 Scenarios
We consider three different manipulation tasks that involve object manipulation with stable grasps.
Pick-and-PPluce. robot has to pick an object and place on rectangular region of the table. Fig. 3.
Assembly: two mobile robots have to pick an object each and assemble them. The assembly is not unique, but modelled as manifold with constraints rotation and position: objects

---

Figure 5: Samples from the deep generative model in the same instance of the assembly scenario.
Each sample shown with two keyframes (pick and assembly). Both seeds lead feasible solution.
Seeds
Solutions
Coverage
Precision
Error
Coverage
Precision
Success Rate
Big NN
0.81
0.7=0.11
8.38
0.58+t.11
0.39:10.06
0.46
Big + analytical
0.79::011
0.53
1.21
0.75+t0.13
0.41
0.43
Structure NN
0.6
0.62:10.1
8.09+:1.35
0.41++0.05
0.44:10.06
0.56
Structure NN analytical
0.57++0.08
0.47+10.13
1,46++.79
0.44
0.28++0.08
0.78
Table 1: Ablation study in the Pick-and-Place scenario.
have to be perpendicular and make stable contact with predefined faces of the cubes (similar a T shape). Fig. 5.
Handover: Two mobile robots have to collaborate to move the object from the initial to the goal position doing handover. Fig. 1.
All these tasks are performed on cluttered table with a varying number (between and 5) of obstacles. The grasp between the gripper and the object is modelled with two finger gripper (e.g. Franka Panda Gripper) that constrains the position and orientation. The training dataset consists of 4000 pairs of problem-solution computed offline with user-defined sampling sequence to ensure diversity. The problems consist of 64x 128x4 images (input of network) and their corresponding environment (for the analytical error term in training and nonlinear optimization).
The variability between instances is defined by the number, position and size of obstacles, the size and position of the objects and the goal configuration. The instances of the evaluation and train dataset come from the same distribution. See the Appendix for details about the structure of the networks and implementation, and a collection of samples generated by our deep generative models.
6.3 Ablation Study
The Pick-and-Piaa Place scenario is used for an ablation study of the proposed generative model. We analyze the contribution of the factored structure (Big NN vs Structure NN) and analytical error term (+analytical). We evaluate precision and coverage of seed samples (output from the deep generative models) and solutions (after projection with nonlinear optimization) by generating 4000 samples for each new instance (30 in total). Results are shown in Table 1. We report the metrics:
Error: the constraint violation [| r (unitless, lower is better).
Precision: average nearest neighbour distance to a reference dataset. models how close samples are to the real data (lower is better).
Coverage: average nearest neighbour distance from the reference dataset to the computed samples. It describes how well the learned distribution covers the reference dataset (lower is better).
When analyzing the seeds (output of the generative model) two contributions essential to achieve small constraint violation (anlytical term) and good coverage (structure). Seeds from the model with structure and analytical term have higher probability of leading solution (success rate) and, after the projection, only samples from networks with structure provide good coverage.
6.4 Benchmark: Generative Model in Nonlinear optimization
The Assembly and Handover scenarios are used to compare our generative model against two base- line methods for warmstarting (seeding) nonlinear optimizers. We analyze the number of solved problems and the number of necessary optimization runs. Measuring the number of solved nonlin-

---

80
8D
Number of Instances
Number of Instances
60
60
40
Rand
Rand
20-
20-
Deep
Rand Data
Rand Data
0
Nurbber of tridds to find solution
Numbee of trials to f'ndd solution
(a) Handover
(b) Assembly
Figure 6: Histogram of the estimated number of trials necessary to solve an instance.
ear programs an indirect way to evaluate coverage and sample quality, as both are fundamental to solve a diverse set of problems with nonlinear optimizer and prevent convergence to infeasible points. We compare our complete model (deep generative model with structure and analytical error term), in short deep, with baselines:
Rand: Randomized initial guess around a reference value.
Choosing samples from the training dataset at random. The initial point is thus a feasible sample for another problem of the same family. This actually strong baseline, because it provides diverse informative initial seeds.
Rand Data:
We evaluate the generative model (deep generative model optimization) on 200 problems from the evaluation dataset. The experiments are repeated 10 times and we report mean and variance. We first report how many optimization trials (each trial has an independent starting point) are necessary to solve each of the test instances, and plot the histogram of the mean value in Fig. 6. Unsolved problems are assumed to be solved with 10 trials (maximum number trials).
In both scenarios, the proposed deep generative model outperforms the baseline warmstarts, signif- icantly reducing the number of trials required solve the instances: average (across problems), from (Rand Data, 3.86:1 to (Deep, 1.77:1 in Handover and from (Rand Data, 3.99 1.44) (Deep, 1.07 in Assembly. To complete the analysis, we also show the cumulative number of problems solved as we increase the number of optimization trials in Fig. 7 of the Appendix A. The computational overhead of evaluating the neural network small (we produce 10 samples in 8 ms with GPU), while most of the time is spent in optimization runs that converge to infeasible points. Time spent (in seconds) to complete the benchmarks are: (Deep, 949+:32), (Rand Data, 1746++76) in Assembly; and (Deep, 1246++26), (Rand Data, 1715++58) in Handover.
7 Conclusion
In this work, we propose Deep Generative Constraint Sampling (DGCS), a new approach to sample on constraint manifold to tackle problems in robotic sequential manipulation. Our framework combines deep generative sampling model, conditioned on an image based representation of the problem, and a nonlinear optimizer to project samples onto the manifold. We further extended the approach to exploit a given factorization of the problem, by training a sequence of conditional generative models rather than full joint generator. Our empirical results confirm that the trained generative models outperform heuristic warmstart strategies. Moreover, the inclusion of analytic constraints in the training of the generative model, well exploiting the factorization of given problem significantly improves the efficiency, diversity and precision the sampling approach.
A limitation of approach is that training requires dataset of solutions to different instances of the same problem class. As future work, the proposed graphical structure could be exploited to provide generalization across different problem classes (i.e. different type and number of con- straints and variables) by sharing and combining the sampling modules of the sequence. Our current framework combines generative sampling using neural network with subsequent projection using constrained optimization. A promising future direction is to explor.e whether it is possible to embed the optimization algorithm last layer of the generative model, while keeping good coverage and multimodality.

---

Acknowledgments
Joaquim Ortiz-Haro and Danny Driess thank the International Max-Planck Research School for Intelligent Systems (IMPRS-IS) for the support. This research has been supported by the German- Israeli Foundation for Scientific Research (GIF) grant 1-1491-407,660 6/2019 and the German Research -407.6/2009
Foundation (DFG) under Germany's Excellence Strategy EXC 2002/1-390523 "Science of Intelligence".
References
[1] D. Driess, J.-S. Ha, and M. Toussaint. Deep visual reasoning: Learning to predict action sequences for task and motion planning from an initial scene image. Robotics: Science and Systems 2020 (RSS 2020). RSS Foundation, 2020.
[2] N. Mansard, A. DelPrete, M. Geisert, S. Tonneau, and O. Stasse. Using memory of motion to efficiently warm-start nonlinear predictive controller. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 2986-2993. IEEE, 2018.
[3] A. Cauligi, P. Culbertson, B. Stellato, D. Bertsimas, M. Schwager, and M. Pavone. Learning mixed-integer convex optimization strategies for robot planning and control. In 2020 59th IEEE Conference on Decision and Control (CDC), pages 1698-1705. IEEE, 2020.
[4] K. Hauser. Learning the problem-optimum map: Analysis and application to global optimiza- tion in robotics. IEEE Transactions Robotics, 33(1):141-152, 2016.
[5] G. Tang and K. Hauser. Discontinuity-sensiiivee optimal control learning by mixture of experts. CoRR, abs/1 Abs//803.00m 803.02493, 2018. URL httt:://arxxv..rr//bbbs. abrrg//bb//883..... 1803. 02493.
[6] D. Driess, J.-S. Ha, R. Tedrake, and M. Toussaint. Learning geometric reasoning and control for long-horizon tasks from visual input. In 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021.
[7] A. Xie, Ebert, S. Levine, and C. Finn. Improvisation through physical understanding: Using novel objects as tools with visual foresight. In Proc. of Robotics: Science and Systems (R:SS), 2019.
[8] Ebert, C. Finn, Lee, and Levine. Self-supervised visual planning with temporal skip connections. In Conference on Robot Learning, 2017.
[9] C. Paxton, Y. Barnoy, K. D. Katyal, Arora, and G. D. Hager. Visual robot task planning. In International Conference on Robotics and Automation (ICRA), pages 8832-8838. IEEE, 2019.
[10] D. Driess, J.-S. Ha, M. Toussaint, and R. Tedrake. Learning models as functionals of signed- distance fields for manipulation planning. In Proc. of the Annual Conf. on Robot Learning (CORL), 2021.
[11] I. Goodfellow, J. Pouget-Abades M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Bengio. Generative adversarial nets. Advances in neural information processing sys- tems, 27:2672-266 22800 2014.
[12] M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning, 2017.
[13] D. P. Kingma and M. Welling. Auto-encoding variational bayes.
arXiv preprint
1rXiv:1312.6114, 2013.
[14] C. R. Garrett, T. Lozano-Pérez, and I.. P. Kaelbling. Sampling-based methods and motion planning- CoRR, abs/180 1808 300600680, 2018. URL htttt:::://xxiv.ro///.o. 1801. 00680.
for factored task
[15] F. Lagriffoul, D. Dimitrov, J. Bidot, A. Saffiotti, and L. Karlsson. Efficiently combining task and motion planning using geometric constraints. The International Journal of Robotics Re- search, 33(14): 2014.

---

[16] K. Fang, Y. Zhu, A. Garg, S. Savarese, and L. Fei-Fei. Dynamics learning with cascaded variational inference for multi-step manipulation. arXiv preprint arXiv:1910.13395, 2019.
[17] J. O. de Haro, V. N. Hartmann, O. S. Oguz, and M. Toussaint. Learning efficient constraint graph sampling for robotic sequential manipulation. CoRR, 0is/2011.04820 2020. URL https://aat http:///axvvvvvo org/ 201s 04828.
[18] R. Chitmis, T. Silver, J. B. Tenenbaum, T. Lozano- Pérez, and L. P. Kaelbling. Learning neuro- symbolic relational transition models for bilevel planning. CoRR, abs/2 14074, 2021. URL https://arxis: org/ abs/2105, 14074.
[19] A. Mousavian, C. Eppner, and D. Fox. 6-dof graspnet: Variational grasp generation for object 10520.
manipulation. CoRR, abs/ 1991. 2019, URL
[20] A. Murali, A. Mousavian, C. Eppner, C. Paxton, and D. Fox. 6-dof grasping for target-driven object manipulation in clutter. In 2020 IEEE International Conference on Robotics and Au- tomation (ICRA). pages 0232-6238 IEEE, 2020.
[21] B. Ichter, J. Harrison, and M. Pavone. Learning sampling distributions for robot motion plan- ning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 7087-7094. IEEE, 2018.
[22] J.-S. Ha, Y.-J. Park, H.-. Chae, S.-S. Park, and H.-L. Choi. Adaptive path-integral autoen- coders: Representation learning and planning for dynamical systems. Advances in Neural Information Processing Systems, 31:8927-899 -938.8 2018.
[23] G. Sutanto, M. R. Fernández, P. Englert, R. K. Ramachandran, and G. S. Sukhatme. Learning equality constraints for motion planning manifolds, 2020.
[24] Kurutach, A. Tamar, G. Yang, S. Russell, and Abbeel. Learning plannable representations with causal infogan, 2018.
[25] B. Kim, L. P. Kaelbling, and T. Lozano-Pérez. Guiding the search in continuous state- action spaces by learning an action sampling distribution from off-target samples. CoRR, abs/1711.01391, 2017. URL http://arxiv, http:///arr....r/abb/////// wrrg/bs/////////ao 01391.
[26] A. Simeonov, Du, B. Kim, F. R. Hogan, Tenenbaum, Agrawal, and Rodriguez. A long horizon planning framework for manipulating rigid pointcloud objects, 2020.
[27] M. Toussaint, Allen, K. Smith, and J. Tenenbaum. Differentiable physics and stable modes for tool-use and manipulation planning. 2018.
[28] I. Mordatch, F. Todorov, and 7. Popovié. Discovery of complex behaviors through contact- invariant optimization. ACM Transactions on Graphics (TOG), (14):11-8, 2012.
[29] A. W. Winkler, C. D. Bellicoso, M. Hutter, and J. Buchli. Gait and trajectory optimization for legged systems through maase--bsed end-effector parameterization. IEEE Robotics and Automation Letters, 3(3):1560-1567, 2018.
[30] W. Merkt, V. Ivan, and S. Vijayakumar. Leveraging precomputation with problem encoding for warm-starting trajectory optimization in complex environments. In 2018 IEFE/RS/ In- ternational Conference on Intelligent Robots and Systems (IROS), pages 5877-5884. IEEE, 2018.
[31] R. Deits, T. Koolen, and R. Tedrake. Lvis: learning from value function intervals for contact- aware robot controllers. In 2019 International Conference on Robotics and Automation (ICRA), pages 7762-7768. IEEE, 2019.
[32] D. Bertsimas and B. Stellato. Online mixed-integer optimization in milliseconds, 2020.
[33] A. M. Wells, N. T. Dantam, A. Shrivastava, and L.. E. Kavraki. Learning feasibility for task and motion planning in tabletop environments. IEEE robotics and automation letters, 4(2): 1255-1262, 2019.
10

---

[34] T. S. Lembono, E. Pignat, J. Jankowski, and S. Calinon. Generative adversarial network to learn valid distributions of robot configurations for inverse kinematics and constrained motion planning. CoRR, abs/201 1.05777, 2020. URL https: //tp:v////iivieeeeeeeeeeeeeeaeee
05717..
[35] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville. Improved training of wasserstein gans. In Advances in neural information processing systems, pages 5767-5777, 2017.
[36] D. Koller and N. Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009.
[37] X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel. Infogan: Inter- pretable representation learning by information maximizing generative adversarial nets. CoRR, abs/1606.03657, 2016. URL http: http://aaavvvv.. org/ abs////11.....3
[38] M. Khayatkhoei, M. Singh, and A. Elgammal. Disconnected manifold learning for generative adversarial networks. CoRR, abs/ 00880, 2018. URL http://arxiv..ee/ http///rxivv.rr//a/////666 00880.


Reviews:
[{"review_id": "JshJ4cgdyxK", "reviewer": "Reviewer_yooP", "summary": "The paper proposed a theory for generalized manifold constrained problems in robotics and presents a framework for generative modeling of it. The generative models are constructed with Wasserstein GAN. The authors also present a graphical model for modeling the distribution of sequential problems in robotics. Experiments are shown on three robotics manipulation tasks: Pick-and-Place, Assembly, and Handover. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n- The theory presented in this paper is novel in terms of problem formulation and solution. The theory of generative modeling on the constrained manifold is a generalization to many problems in robotics. The idea of incorporating graphical models for sequential tasks is interesting. \n- The presentation is overall clear, especially with clear math formulas.\n- The experiment is extensive with 3 different scenarios to show the effectiveness of the proposed method.\n\nWeakness:\n\n- What is the deep model used in this paper? It is not very clear. Do both RGB images and depth images used in the experiments? What deep model is used in the experiments?\n- The experiment results are hard to interpret. How to connect \"Average number of trials to find a solution\", precision and coverage to more commonly used metrics for manipulation tasks such as success rate? \n- The author can further show some failure cases and explain the reason for the failure.\n- The visualizations of the scenarios are hard to interpret.\n  - For example, how are the two middle images in Fig 1 connected by constraint manifold projection?\n- The baseline compared against in the experiment is weak. The author should at least compare against a deterministic deep learned approach.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Though there are some technical details missing and stronger baseline(s) needed, the paper overall would interesting to the community. It re-formulates many problems in robotics as a generalized problem and solves it with generative modeling, which could be valuable and inspiring to the community.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "The authors can address the items under \"Weakness\" in my main review section.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "FLHUIYJFDN-", "reviewer": "Reviewer_8mJB", "summary": "This paper present a generative model to sample near a constraint manifold such that they can be used within a nonlinear optimizer to handle constraints. This is applied on sequential manipulation problems with some factorizations (to capture multi-modal distributions).\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n+ Leveraging learning and working with images to solve tasks with a hybrid (discreet + continuous) action space has a lot of promising applications where pure optimization (w/o learning) approaches fall short due to strong input assumptions and high compute costs.\n+ The idea to factorize the state into modular chucks is a nice way to add structure and improve data efficiency (although hand crafting this for every new task seems like a burden).\n\nWeaknesses\n- More exposition of the big picture: In Sec 1 or somewhere before Sec 3 (that jumps straight into sampling) could benefit from grounding the problem setup a bit i.e. where does the sampling fit in the larger picture of nonlinear optimization that solves some example  manipulation task?\n- Can Eq. (2) alternatively be setup as a piece-wise nonlinear optimization (wherein the constraint pieces are differentiable) where the discreet decisions are either learned or sampled with heuristics or combinatorially searched? What are the trade-offs and implications here and should these be potential baselines?\n- What changes between train and test settings? The variations in each of the problem scenarios seems limited so the generalization capacity is unclear.\n- The overall computational time stats are helpful. It would also be helpful to get a sense for how much computation is saved by choosing to learn the sampling vs alternatives while keeping everything else the same.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall the paper has interesting ideas about using learning as 'one' tool in a box of multiple other tools and them them together towards solving a difficult problem. The current experiments brings in concerns about scaling and generalization.\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See weaknesses\n", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "AYhlhB3bHue", "reviewer": "Reviewer_2JcL", "summary": "The paper presents Deep Generative Constraint Sampling (DGCS), a novel constraint manifold sampling for sequential manipulation. The proposed framework combines a deep generative sampling model -conditioned on an image-based representation of the task, and trained with a regularization using a Wasserstein distance to a reference probability distribution- and a nonlinear optimizer for sample projection onto the manifold. Moreover, the learning framework are infused with a structure based on a given Directed Graphical Models factorization of the problem, by training a sequence of conditional generative models rather a full joint generative model. The experimental results confirm that the proposed framework achieved efficiency, diversity, and precision of the sampling approach, outperforming baselines and heuristic warmstart initialization.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe paper presents an interesting view on a probabilistic technique that combines structure and learning to achieve superior results in robotic sequential manipulation tasks.\n\nMoreover, the paper is also well-written, and relatively easy to be read.\n\nWeaknesses:\nOnly a few minor points in the \"Issues\" section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think the paper present an interesting technique of mixing structure and learning in a probabilistically-sound manner which deserves a publication.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "(1) I suggest to list down the all the known limitations of the framework/method which open venues for future works/extension of this work.\n(2) In Table 1, what is the unit of the Error (I assume it's not unitless and not normalized)? Does this \"Error\" corresponds to \"Constraint Violation\" metric as mentioned in section 6.3? Maybe it's worth to make the terminology consistent between these two, to make it more readable/understandable."}}]

Author rebuttal:

Meta-review: *I am the emergency meta-reviewer assigned to this paper, which is why there wasn't a meta-review provided before the rebuttal.* 

The reviewers have unanimously voted to accept the paper, so that is my recommendation as well. 

Pros:
- The authors propose using a generative model to improve the sample efficiency of sampling poses for non-linear optimization with respect to robot poses, such that the trajectory over poses solves the task ("the constraint manifold").

Cons:
- a formal definition of constraint manifolds is provided in the first sentence of the paper, but it is not clear what specific kind of robotic constraint is formulated as a constraint manifold until L99 of the paper. Overall I found the paper difficult to understand. 
Decision: Accept (Poster)