Learning Diverse and Physically Feasible Dexterous Grasps with Generative Model and Bilevel Optimization
Authors: Albert Wu
Michelle Guo
Karen Liu
Venue: CoRL 2022 Poster (2022)
Keywords: dexterous grasping
grasp planning
bilevel optimization
generative model

Abstract: To fully utilize the versatility of a multi-fingered dexterous robotic hand for executing diverse object grasps, one must consider the rich physical constraints introduced by hand-object interaction and object geometry. We propose an integrative approach of combining a generative model and a bilevel optimization (BO) to plan diverse grasp configurations on novel objects. First, a conditional variational autoencoder trained on merely six YCB objects predicts the finger placement directly from the object point cloud. The prediction is then used to seed a nonconvex BO that solves for a grasp configuration under collision, reachability, wrench closure, and friction constraints. Our method achieved an 86.7% success over 120 real world grasping trials on 20 household objects, including unseen and challenging geometries. Through quantitative empirical evaluations, we confirm that grasp configurations produced by our pipeline are indeed guaranteed to satisfy kinematic and dynamic constraints. A video summary of our results is available at youtu.be/9DTrImbN99I.

Learning Diverse and Physically Feasible Dexterous Grasps with Generative Model and Bilevel Optimization
Albert Wu Computer Science Department Stanford University, United States wanforrtttoorr..oooomoo
Michelle Guo
Computer Science Department Stanford University, United States mguo95ãcs stanf stanford... edu
C. Karen Liu Computer Science Department Stanford University, United States karenl iuêcs. stanf erd.edu
Abstract: To fully utilize the versatility of multi-fingered dexterous robotic hand for executing diverse object grasps, one must consider the rich physical constraints introduced by hand-object interaction and object geometry. We propose an inte- grative approach of combining generative model and bilevel optimization (BO) to plan diverse grasp configurations on novel objects. First, a conditional varia- tional autoencoder trained on merely six YCB objects predicts the finger place- ment directly from the object point cloud. The prediction then used seed a nonconvex BO that solves for grasp configuration under collision, reachability, wrench closure, and friction constraints. Our method achieved an 86.7% success over 120 real world grasping trials 20 household objects, including unseen and challenging geometries. Through quantitative empirical evaluations, we confirm that grasp configurations produced by our pipeline are indeed guaranteed satisfy kinematic and dynamic constraints. A video summary of our results is available at youtu. Ee//DDrIIbbNSS
Keywords: Dexterous grasping, Grasp planning, Bilevel optimization, Generative model
Introduction
Performing diverse grasps a variety of objects is a fundamental skill in robotic manipulation. Diverse grasp configurations allow flexible interaction with the objects while satisfying requirements demanded by the higher level task of interest. Dexterous grasping, which refers to object grasping with fully actuated, multi-finger dexterous robotic hand, has the potential to achieve diverse grasp configurations with applicability to wide range of objects. This is in contrast with simple grasps achieved by parallel jaw grippers or underactuated multi-finger grippers, both which have fingertip workspaces restricted to subspace of the 3D task space.
We identify two major challenges planning diverse dexterous grasps. Firstly, the solution space is multimodal due to the many finger placement possibilities and the lack of metric defining an optimal grasp among valid answers. Secondly, each valid grasp is governed by nonconvex physical constraints such as collision, contact, and force closure.
The multimodality of dexterous grasp planning motivates the use of deep learning-basse approaches. is challenging to produce diverse and multimodal grasp plans with regression or naive supervised methods. Analytical approaches such as precomputing grasp library or sampling-based grasp plan- ning, while popular for planning simple grasps, are computationally intractable on high-dimensional dexterous grasps. Nevertheless, leaming to plan grasps that strictly obey physical constraints is diffi- cult. Model-based optimization has historically been applied to enforce exact constraints. However,
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

in practice the dexterous grasp planning problem is arguably too nonconvex to solve directly. Conse- quently, previous works usually relax the constraints as scalar losses and solve the relaxed problem, sacrificing exact constraint satisfaction guarantees for practicality.
We propose integrative approach that combines learning and optimization to produce diverse physically-casiblle dexterous grasp configurations for unseen objects. Our method first predicts an initial set of contact points using conditional variational autoencoder (CVAE). The contact points are then projected onto the manifold of kinematically and dynamically feasible grasps by solving a bilevel optimization (BO) problem. Our key contributions are summarized below:
bearning-based dexterous grasp planning pipeline that integrates CVAE and BO to produce diverse, fully-specified, and sesstaiint-sttisfiing dexterous grasps from point clouds.
Bilevel grasp optimization formulation that takes learning-base grasp prediction and outputs a dexterous grasp that satisfies reachability, collision, wrench closure, and friction cone constraints. Successful and extensive hardware validation on 20 household objects. Our method achieved an overall success rate 86.7% over 1 120 grasp trials.
Figure 1: Our method can pick up different objects shapes with diverse set of grasp configurations.
2
Related Work
We limit our discussion to literature on grasping in uncluttered scenes, with an emphasis on dex- terous grasping. We exclude work that focuses on object segmentation in cluttered environments or other types of manipulation, such as in-hand manipulation.
2.1 Dexterous Grasping
Dexterous grasping is a long standing problem in the robotics community. In general, the literature can be split into learning- fit the space of possible grasps for rapid grasp generation (e.g.. [1]). More recent papers have shifted to producing grasps directly with complex model architectures, such as generative models [2. 3, 4] and convolutional neural networks [5, 6]. However, most of these works do not account for physical laws and have restricted, or even missing, hardware evaluation. This issue exacerbated by the difficulty of simulating contact-rich interactions [7, 8]. The quality of grasps produced by learning exclusively with simulation questionable. Moreover, many papers do not emphasize the ability to learn diverse grasps, thus forgoing key benefit of dexterous hands over simple grippers. No- tably, some publications on learning dexterous grasping originate from computer vision and graphics communities (e.g., [9, 10, 11, 12, 13, 14]). While these papers cover diverse grasp generation on fully dexterous hands, their ultimate objective tends to be achieving photorealistic human grasps in simulation rather than satisfying strict real-world physical constraints.
and analytical methods. Early learning- grasp planners seek to
On the other hand, analytical dexterous grasp planners are often inspired by physical constraints such force closure, friction cone, robustness external disturbance wrenches, and object contact (e.g., [15, 16, 17, 18]). Historically, analytical methods are standalone and includes both grasp modality exploration and physical constraint compliance. We point the readers to [19, 20] for detailed review on these methods. Some limitations of these approaches include dependency on a high-fidelity object model and lack of diversity in the generated grasps [20].
In recent years, some learning-based approaches have adopted a "grasp refinement" step motivated by analytical metrics [4, 18, 21]. Nevertheless, these metrics tend to be formulated as "relaxed" op-

---

timizations to keep the problem tractable. Instead of enforcing the constraints directly, the violation of each constraint cast as scalar loss and summed together. This results in an unconstrained opti- mization which is significantly easier to solve. However, there is no guarantee that the optimization output will indeed satisfy the constraints that motivated the loss design.
2.2 Simple Grasping
Due to the geometry of parallel jaw grippers, simple grasp planning can be reduced to computing 6-DoF gripper pose in space. As such, analytical simple grasp planners may directly reason about object geometry (e.g., [22]) or rank grasps using grasp quality metrics [19]. Recently, learning- based approaches that predict wrist poses have gained significant traction (e.g.. [23, 24, 25)). We refer the readers to [26, 19, 20, 27, 28] for thorough review. These approaches seldom scale directly to dexterous grasp planning, which is significantly more complex problem.
2.3 Bilevel Optimization (BO) for Planning
While BO theory is established in literature [29], application of BO motion planning is relatively
new. BO has been applied to continuous systems [30], robotic locomotion [31, 32), simple pushing and pivoting [33], collaborative object manipulation [34], and task and motion planning [35]. To our best knowledge, this work is the first application of BO on dexterous robotic manipulation.
Method
Our method consists of learned model that predicts a plausible finger placement P R3x3, and an analytical process that computes a physically feasible dexterous grasp q" E IR (Definition 1) guided by P. At inference time, the pipeline takes in an object point cloud and outputs fully specified grasp configuration q". Figure 2 gives overview of our method. We assume the object is grasped with exactly fingers. This section first formally specify the modeled physical constraints, then discuss each of the components in the pipeline. Our implementation is available at github. com com/Stanford "StanforddTML//ees _grasp.
Inference
Biievel Optimization mis 0 sdjjct to Aig
Object Point Cloud
0
CVAE Encoder
m.a
Sample s)
Decoder
CVAE
Projection
p'
Kinematics
Inverse
Gmmpp
Finger Placemen
Kíi) a 20
Overview of method, We train CVAE that predicts finger placements P E Rax given an object point cloud O. At inference time, we first obtain finger placement prediction P, which not guaranteed to be physically feasible. Next, we compute a grasp configuration initial guess q' E R from P. Finally, we apply BO to compute physically feasible grasp q E 22
Figure 2:
Definition (Physically Feasible Dexterous Grasp). Given object O in environment E, grasp configuration and a desired finger placement P = (P1-P2, Ps) E R3x3 we consider (O. E, q, P)) to be physically feasible if they satisfy following dynamic and kinematic constraints. In our setup, E 22 and E is the fixed tabletop (see Section 4.
3.1 Dynamic Constraints: Wrench Closure and Friction Cone
For static dexterous grasp, the dynamic constraints include the wrench closure constraint and the friction cone constraint. The wrench closure constraint requires the of the external wrench from
all contact points be zero: 33 f: 0 and Σ P: f: 0. f. E R is the unknown contact force applied at position p. 3 from the dexterous hand the object.
Given the static friction coefficient μ and the outward- outward-pointing pointing surface normal ni the friction cone constraint requires the normal force to be nonnegative and the contact force to be within the friction cone. Using polyhedral cone approximation ([36)) with an orthogonal basis f,; Ln, Vj E (1,2),

---

we arrive at the following approximated friction cone constraints:
0 ff- n. and [fe tt, - -Hf: n., Vi (1,2.3), Vj E (1,2}.
(1)
A grasp is dynamically feasible if it satisfies both wrench closure and friction cone constraints. Leveraging the polyhedral approximation, dynamic feasibility can be cast as quadratic program (QP):
min fa.fa.fs
E f:12+ 1 P. fill subject 0 fmin I -f: n. and (I).
(2)
The optimization in Equation (2) has objective value 0 if and only if the grasp is dynamically feasi- ble. Note that we set an arbitrary lower bound fmin RR+ on the normal force to avoid the trivial solution of f: O. The exact value of fmin is irrelevant because changing fmin represents scaling the optimal contact forces, which does not affect membership in the friction cone.
3.2 Kinematic Constraints: Reachability and Collision
The kinematic constraints include reachability constraints and collision constraints. Reachability constraints enforce the kinematic ability of the robot hand reach the contact point p; the object surface 80, i.e. 3q K2(g) = p. E 80, Vi E {1, 2, 3}. Here K; R + IR is the forward kinematics function that maps the hand state q to the position of the i-th fingertip. Assuming only fingertip contacts, collision constraints ensure that no robot link L(q) is in collision with the object
O except the fingertip or with the environment E, i.c. L()) n (0 U E} 0. Both reachability and collision constraints are nonconvex constraints.
3.3 Learning Predict Finger Placement from a Physically Feasible Grasp Dataset
We leverage generative model sample potential finger placements P given an arbitrary object observed point cloud Ó. An immediate challenge faced by the learning approach is the lack of large-scale datasets for dexterous robotic hands. While datasets for real human hands do exist, retargeting human grasping configurations to a robot hand with different kinematic structures and joint limits presents numerous challenges. We opt to synthesize a large-scale dataset of physically feasible grasping configurations for the Allegro robot hand used in this paper.
3.3.1 Creating a Physically Feasible Dexterous Grasp Dataset
We generate physically feasible grasps of six YCB objects [37] in simulation. The chosen objects "cracker box." "sugar box," "tomato soup can," "mustard bottle," "gelatin box," and "potted meat can." " We consider a realistic scenario where the object is placed on flat surface instead of free floating. As such, need to consider different object rest poses of on the surface in addition to finger placements. In summary, we first generate random object rest poses and enumerate all possible finger placements P using the corresponding simulated object point cloud Ô If kinemat- ically and dynamically feasible grasp q can be found for P, (Ö, q. P) is added to the dataset P. The detailed procedure is described in the supplementary material.
3.3.2 Training Conditional Variational Autoencoder (CVAE)
To compute diverse grasps for arbitrary objects directly from point cloud observations, we pre- dict fingertip contact points the object surface with conditional variational autoencoder (CVAE) [38]. Our model is adapted from [24] and consists of an encoder E and a decoder D that are based on the PointNet++ architecture [39]. The model seeks maximize the likelihood of producing a set of contact points P deemed feasible in Section 3.3.1 given the point cloud Ó.
The encoder E(x P. Ô) maps a grasp P and a point cloud Ó to the latent space. We assume the latent variable has a normal distribution (O, Mcanwhile, given latent sample z E, the decoder attempts to reconstruct the finger placement P. We seek to minimize the element-wise L--norm reconstruction loss Lree(?? (P, P) & || P|| for a feasible grasp from the dataset P e IP. Additionally, a KL divergence loss DEL L is applied on the latent distribution E(- I.) to ensure normally distributed latent variable. The complete loss function of the network is defined as
L
Σ EEE
(rrae(P, DDK
(F(z) PP.O) N(0.7))
33)

---

At inference time, E' is removed and a latent sample = is drawn from (0, This is passed into the decoder O z) along with the point cloud Ô to produce the grasp point prediction P.
3.4 Computing Grasps with Physical Feasibility Guarantees using Bilevel Optimization (BO)
While the grasps in the training dataset are physically feasible by construction, there is no guarantee that the CVAE output, P. is physically feasible. Additionally, P only specifies the finger placement instead of the full hand configuration. We propose BO obtain a physically feasible grasping pose given P. To seed the BO with P. we first obtain an Euclidean projection of P onto d0, denoted as P' _ {p' Pt arg mi 0.0 Po P:llz 12, i E 21, } Next, we solve an inverse kinematics (IK) problem for grasp configuration finger placement problem specified by P up to a numerical tolerance c, i.e. find q' (|);(q') pill2 S K;(q') E 80, Vi E 21. K; R222 + IR is the forward kinematics function of the ith finger. (4) E do is implemented as a constraint on the finger-object signed distance D(p, O) E [dmin dmaz]. The IK solution q' serves as the initial guess for the BO.
3.4.1 Formulating the Bilevel Optimization Grasp Refinement
Accounting for the kinematic and dynamic constraints, the naive formulation of the grasp optimiza- tion problem is given in (4).
min
0
fa.if
3 subject to Ki(g)= 80, L E) =0, 1 f = 0, E K.(g) x fill 0, (4)
3
1=1
fmin S ff n.: ff E. S -uft n. Wi (1,2,3) E {1,2}.
Equation (4) does not solve well in practice due to its complexity and nonconvexity. Additionally, while q' may serve as an initial guess for q, the initial guess for f is not obvious. Naively relaxing Equation (4), c.g., replacing the objective with ming |>> f +1 +| K,(g) x f.l.. may produce suboptimal f and, consequently, incorrect conclusion on grasp's 's dynamic feasibility.
We propose leveraging bilevel optimization [29, 32] to offload the dynamic feasibility computation from the main optimization. Define J [(q) the minimum objective value of the dynamic constraint QP in Equation (2) with p: = (q):)
J ((g)
min fa
S1B+ + Σ
f1l subject to 0 < fmån S -fs f2 and (1). (5)
i=1
We observe that the force closure constraint J(q) 0 can be abstracted away form a "lower- level" QP. (qq) = 0 can be solved to optimality with exiting QP solvers without reliance on good initial guess for fi.
Applying this abstraction to Equation (4) yields the upper-level problem:
min 0 subject to J(q) = 0, K.(q) E ao, L(q) n (OUE) =
(6)
Equation (6) is a bilevel optimization as (jq) = 0 is a constraint on the minimizer of another optimization problem (Equation (6)). While Equation (6) still defines nonconvex optimization, the choice for f; has been abstracted away entirely to the lower level QP solver. By construction, a valid solution to Equation (6), denoted as q", defines physically feasible grasp.
3.4.2 Solving the Bilevel Optimization Grasp Refinement
In practice, to solve Equation (6) with a nonconvex optimizer (e.g.. SNOPT [40]), one needs to obtain the gradient of each of the constraints with respect to q. The gradients of the kine- matic constraints can be obtained from simulators with automatic differentiation capabilities (e.g., Drake [41]). The gradient of J(q). (q)) can obtained using differentiable optimization [42]. At a high level, differentiable optimization computes V ((q) by taking the matrix differentials of the KKT conditions of the optimization problem at its solution.
We emphasize that the bilevel optimization in Equation (6) is still a highly nonconvex optimiza- tion, and the optimizer will likely return locally optimal q²' in the vicinity of q'. Intuitively, this
5

---

could be interpreted as optimizing within the grasp family of the CVAE prediction (e.g.. "left side grasp" or "top-down grasp"), which implies the necessity of CVAE in the process. This is supported empirically by our ablation studies in Section 4. In practice, the bilevel optimization provides a cer- tification of physical feasibility for the CVAE prediction. A poor q' choice will likely results in the nonconvex optimizer returning infeasibility. This serves as the condition to reject P and generate different CVAE grasp prediction with another latent variable sample.
4
Experiments
Our trained CVAE achieved a test reconstruction eπrоr of 0.5em. More training details are available in the supplementary material. The effects of applying BO is shown in Figures 3a and 3b. The finger placement prior to BO cannot form force closure Ω. n O, Vi, (1, 2, 3). BO shifted the thumb and middle fingers to an antipodal configuration, which allows for force closure. This validates our approach's 's ability make initially infeasible grasp prediction physically feasible. The rest of this section focuses on hardware experiments and evaluations.
plaamed
execured
(a) Before BO (q'). (b) After BO (q").
Hardware sctup.
(d) Planned and executed grasps.
Figure 3: 3a and 3b: Before and after BO, on bottom-up view of mustard bottle grasp. Using the color encoding of red: thumb, green: index, and blue: middle, we show P (solid spheres), P (transparent spheres) and n. (colored lines). The direction of the computed contact forces fi shown at the respective fingertip with yellow lines. The mismatch between K(q') and p' due to the numerical tolerance 3c: Hardware setup. See Section 4.1 for more details. 3d: Grasp sim-to-real. Correspondence between the planned grasp in simulation and execution on hardware.
4.1 Hardware Setup
The hardware setup and evaluated objects shown in Figure 3c.
Hardware. A 16-DoF Allegro v4.0 right hand was used for hardware grasping experiments. The 7-DoF Franka Emika Panda to realize the planned wrist pose. The object point cloud was captured by 4 stationary Intel RealSense D435 depth cameras. The implementation details are available in the supplementary material.
Allegro hand was mounted
Evaluated Objects. We evaluated our method on 20 rigid household objects resting on the table.
We categorize our objects into three sets:
3 Seen objects. (Leftmost column) mustard bottle, soup can, and sugar box from the training set. Familiar objects. féccond-from/e
column) Unseen objects with geometries similar to training objects. Includes boxes (webcam box, mask box) and cylinders (chip can, soda can).
13 Novel objects. Objects whose geometries are distinct from that of any object in the training set. From left to right, front to back: tetrahedron, massage ball, castle, pill bottle, glasses case, condiment bottle, hairspray bottle, ramen, lego, sandwich box (side), pear, sandwich box (upright), alcohol bottle.
4.2 Experiment Procedure
At the start of each trial, the object is placed in specified pose in front of the robot. After observing the point cloud and computing a grasp, the Franka arm first brings the hand the desired wrist pose using an ad hoc trajectory planner. The finger joint angles 0" R from q" is then executed on the Allegro hand. Contact forces are provided by squeezing the fingertips according to planned contact forces: 0 +- 0" + <(Vop:) 2 f.. Here Vop; is the Jacobian of the fingertip locations with respect to O, and a fixed "stiffness" constant as motivated by impedance control. The Franka then attempts to lift the object to fixed height approximately 43cm above the table. A trial is

---

Figure 4: Diverse grasps generated by method. Each image is unique grasp generated from different sampled latent variables. All grasps were computed for the same initial object rest pose.
considered successful if the object lifted and three fingers remain contact with the object.
3 repeats are performed for hardcoded policy trials as they are deterministic up to the object point cloud observation. All other trials are repeated 6 times. Our hardware pipeline was implemented with the intent to execute our planned grasp as accurately possible. Figure 3d illustrates the comulation-hardwaie grasp correspondence.
4.3 Baseline and Ablation Studies
We compare our method against the following approaches. We excluded "BO only" in our ablation studies as does not return a result without CVAE in practice. BO seldom returns a solution seeded with kinematically infeasible solution (e.g.. open hand).
Hardcoded grasp baseline. Using the set seen objects, we designed a top-down tripod grasp policy that chooses the wrist pose based the point cloud and executes fixed grasp.
CVAE-only as an ablation study. We solved collision-free inverse kinematics problem that at- tempts to match the CVAE predicted fingertip positions P. This mimics "learning only" approach.
CVAE-kinematic as an ablation study. We ablated the "lower level" part of the optimization (Equation (5)) away and use q' directly without bilevel optimization. This ablation considers kine- matic constraints but not dynamic constraints.
4.4 Grasp Trial Results
Our method achieved overall success rate of 86.77% % over 120 grasp trials on 20 objects. This superior to the hardcoded baseline, which achieved an overall success rate of 53.3%. On seen objects, CVAE-only ablation achieved a 38.9% success rate and our achieved an 88.9% success rate. The results are summarized in Table I, and the details are available in the supplementary material. A video summary is available at youtu. e
ablation
Our method can grasp challenging novel objects. This includes objects that are difficult for parallel grippers and suction cups to grasp. We discuss the challenges of some of our test objects in the supplementary material. The hardcoded baseline nearly always succeeds on objects that allow top-down grasps and are similar in size to the seen object used for tuning. However, the baseline fails on objects that either require a different grasp type or have significantly different size.
Our method can generate diverse grasps through sampling different latent x z's. Figure 4 shows 10 distinct successful grasps performed on the upright sugar box.
The median time to produce a grasp with our method is 14.4 seconds. The median repeats for each component of our method when generating grasp is two CVAE samples, IK solves for q': and two bilevel optimizations for q². Detailed timing and repetition results are available in the supplementary material.
Table 1: Object grasping statistics. Our method achieved superior grasp success rate compared to the ablations and the hardcoded baseline.
Category (object count)
Seen (3)
Familluu
Novel (13)
Overall (20)
Ours
1W18
94.4%
23/24
95.8%
64/78
82.1%
104/120
86.75
CVAB only
718
38.9%
CVAEekinematics
88.9%
Hardeoded
8/9
88.95.
9/12
75.0%
15/39
38.5%
32/60
53.3%
7

---

Table 2: Evaluation of physical constraint on various grasps. A physically feasible grasp should
satisfy D(p, O) E Imiin dmaz] =
-0.68.
-0.32) and zero force and torque ratios.
Median D
Min
Max D
Max violation
Force
Torque ralio
Ours (all obyects)
-0.32
-0.68
40.44
0.08
(0.00, 0.02)
0.01 (0.00, 2.68)
CVAE- kinemaiiess (seen objects)
-0.32
-0.64
40.32
0.04
(0.00, 0.12)
0.01 (0.00, 19,54)
CVAE-only (seen objocts)
0.33
-1.10
1.71
2.03
Table Correlation between dynamic feasibility and hardware success, Only the grasps plans with
wrench closure resulted in successful hardware execution.
Successful
Failed
Force ratio
Torque ratio
Forez ratio
Torque ratio
BO rejected (12 12 failed)
61.81 (25.51, 91.65)
48.53 (39.89, 78.68)
Kine-kinematiis (7 Iuiled) (0,00, 0.00)
0.00 (0.00, 0.01)
50.71 (19.13, 128.85)
36.75 (30,48, 131.82)
4.5 Quantitative evaluation of physical constraint enforcement
Table 2 summarizes the quantitative evaluation of physical constraints.
Kinematic constraints. The maximum finger-object distance constraint violation in grasp planned by our method is 0.08em, which is negligible in practice as is smaller than other error sources such as camera observation error. The CVAE-kinematics ablation, which only considers kinematic constraints, achieved comparable results. The CVAE-only ablation has a maximum violation of 2.032cm, confirming that while CVAE alone can produce qualitatively correct grasp, it cannot en- force kinematic constraints precisely.
Dynamic constraints. To evaluate dynamic constraint satisfaction, we chose "force (torque) ra- x 100%, and 3 3E.-12a pp.xll x 100%. If wrench closure achieved, both ratios should be zero. All ratios are reported as median, (25th percentile, 75th percentile). We computed these ratios for grasps planned with method and ablations. These ratios are not computed for CVAE-only because there are often fewer than 3 finger- object contacts. Our method achieved median of S 0.01% on both ratios, showing that BO is effective in enforcing wrench closure. We note that while the CVAE+kinematics ablation does not explicitly consider external wrench, it still produced many grasps that achieve wrench closure by coincidence. This reflects CVAE's ability to produce qualitatively correct grasps.
tio" as our metric, i.e. E f: 3E Iff:ll2
To show that physical feasibility is necessary condition for successful grasp, we examined the CVAE+kinematic trials, which may not satisfy dynamic constraints, on "ramen" and "mustard bot- tle." We also executed grasp plans that are reported to be infeasible by BO. Table 3 summarizes the results. There is a strong correlation between hardware success and wrench closure. Moreover, all grasps reported to be infeasible by BO failed hardware. This confirms that grasp refinement derived from rigorous physics-inspired metrics can significantly improve the final grasp plan quality.
5 Limitations and Conclusion
Limitations. Our method requires an observation of the full object point cloud. As our choice of physical constraint formulation does not include explicit robustness margins, estimation errors from the vision pipeline major source of grasp failures. This may addressed by introducing point cloud completion (e.g., [43, 44, 45]) or grasp robustness metrics (e.g.. [17]). Additionally, it
currently not possible to specify which type grasps to generate (e.g., "top grasp" or "side grasp"" with our method. Finally, our method currently assumes that the object to grasp is placed on flat surface and that there are no other objects in the scene.
Conclusion. This work presents novel pipeline that combines learning-based grasp generation with bilevel optimization to produce diverse and physically feasible dexterous grasps. Our method achieved a grasp success rate of 86.7% on 20 challenging real-world objects. Ablation studies demonstrated that an integrative approach combining learned models and rigorous physics-inspired metrics can achieve superior grasp output quality. Grasps initially generated by CVAE may not sat- isfy all physical constraints. However, by incorporating bilevel optimization for grasp refinement and rejection sampling, poor grasp predictions can be removed. This paradigm of combining learn- ing and physics-inspires bilevel optimization may be applied to other robotic manipulation tasks.
8

---

Acknowledgments
We would like thank Oussama Khatib, Jeannette Bohg, Dorsa Sadigh, Samuel Clarke, Elena Galbally Herrero, Wesley Guo, and Yanchao Yang for their assistance on setting up the hardware experiments. We would also like to thank Chen Wang and Jiaman Li for advice training the CVAE model. Our research is supported by NSF-NRI-2024247, NSF-FRR-2153854, Stanford- HAI-2031 12, and the Facebook Fellowship.
References
[1] B. Huang, S. El-Khoury, M. Li, J. Bryson, and A. Billard. Learning real time grasping strategy. In 2013 IEEE International Conference on Robotics and Automation, pages 593- 600. IEEE, 2013.
[2] J. Lundell, F. Verdoja, and V. Kyrki. Ddge: Generative deep dexterous grasping in clutter. IEEE Robotics and Automation Letters, 6(4):6899 6901689996696,, 2021.
[3] L. Shao, F. Ferreira, M. Jorda, V. Nambiar, J. Luo, E. Solowjow, J. A. Ojea, O. Khatib, and
J. Bohg. Unigrasp: Learning unified model grasp with multifingered robotic hands. IEEE Robotics and Automation Letters, 5(2):2286 2293, 2020.
[4] W. Wei, D. Li, P. Wang. Y. Li, W. Li, Y. Luo, and J. Zhong. Dvgg: Deep variational grasp generation for dextrous manipulation. IEEE Robotics and Automation Letters, 7(2): 1666, 2022.
[5] Q. Lu, K. Chenna, B. Sundaralingam, and T. Hermans. Planning multi-fingered grasps as prob- abilistic inference in learned deep network. In Robotics Research, pages 455-472 Springer, 2020.
[6] P. Mandikal and K. Grauman. Learning dexterous grasping with object-centric visual affor- dances. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 6169-6176. IEEE, 2021.
[7] S. Pfrommer, M. Halm, and M. Posa. Contactnets: Learning discontinuous contact dynamics with smooth, implicit representations. arXiv preprint arXiv:2009. 2020.
[8] Y. Jiang, Sun, and C. K. Liu. Data-augmented contact model for rigid body simulation. In Learning for Dynamics and Control Conference, pages 378-390 7783990. PMLR, 2022.
[9] Zhu, R. Wu, Lin, and Sun. Toward human-like grasp: Dexterous grasping via semantic representation of object-hand. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 15741-15751, 2021.
[10] S. Brahmbhatr, C. Tang, C. D. Twigg, C. C. Kemp, and J. Hays. Contactpose: A dataset of grasps with object contact and hand pose. In European Conference on Computer Vision, pages 361-378. Springer, 2020.
[11] O. Taheri, N. Ghorbani, M. J. Black, and D. Tzionas. Grab: A dataset of whole-body human grasping of objects. In European conference computer vision, pages 581-600. Springer, 2020.
[12] H. Jiang, S. Liu, J. Wang, and X. Wang. Hand-object contact consistency reasoning for human grasps generation. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11 117011116,, 2021.
[13] L. Yang, X. Zhan, Li, W. Xu, J. Li, and C. Lu. Cpf: Learning a contact potential field to model the hand-object interaction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11097-11106, 2021.
[14] E. Corona, A. Pumarola, G. Alenya, F. Moreno-Noguee and G. Rogez. Ganhand: Predicting human grasp affordances in multi-object scenes. Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 5031-5041, 2020.

---

[15] M. T. Ciocarlie and P. K. Allen. Hand posture subspaces for dexterous robotic grasping. The International Journal of Robotics Research, 28(7):851 2009.
[16) F. T. Pokorny and D. Kragic. Classical grasp quality evaluation: New algorithms and theory. In 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 3493- 3500. IEFE, 2013.
[17] C. Ferrari and J. F. Canny. Planning optimal grasps. ICRA, volume 3, page 6, 1992.
[18] T. Liu, Z. Liu, Z. Jiao, Y. Zhu, and S.-C. Zhu. Synthesizing diverse and physically stable grasps with arbitrary hand structures using differentiable force closure estimator. IEEE Robotics and Automation Letters, 7(1):470-477, 2021.
[19] M. A. Roa and R. Suárez. Grasp quality measures: review and performance. Autonomous robots, 38( (88())65-88) -8, 2015.
[20] J. Bohg, A. Morales, T. Asfour, and D. Kragic. Data-driven grasp synthesis survey, IEEE Transactions robotics, 30(2):289-309 2013.
[21] M. Liu, Z. Pan, K. Xu, K. Ganguly, and D. Manocha. Generating grasp poses for a high- dof gripper using neural networks. In 2019 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 1518-1525, IEEE, 2019.
[22] D. Berenson, R. Diankov, K. Nishiwaki, Kagami, and Kuffner. Grasp planning in complex scenes. In 2007 7th IEEE-RAS International Conference on Humanoid Robots, pages 42-48. IEEE, 2007.
[23] A. Saxena, J. Driemeyer, J. Kearns, and A. Ng. Robotic grasping of novel objects. Advances neural information processing systems, 19, 2006.
[24] A. Mousavian, C. Eppner, and D. Fox. 6-dof graspnet: Variational grasp generation for object manipulation. Proceedings the IEEE/CVF International Conference on Computer Vision, pages 2901-2910, 2019.
[25] J. Mahler, J. Liang. S. Niyaz, M. Laskey, R. Doan, X. Liu, J. A. Ojea, and K. Goldberg. Dex- net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metries. arXiv preprint arXiv: 077000331122 2017.
[26] G. Du, K. Wang, S. Lian, and K. Zhao. Vision-based robotic grasping from object localization, object pose estimation to grasp estimation for parallel grippers: a review. Artificial Intelligence Review, 54c1:,,,4(3::17777-111
2021.
[27] S. Caldera, A. Rassau, and D. Chai. Review of deep learning methods robotic grasp detec- tion. Multimodal Technologies and Interaction, 2(3):57, 2018.
[28] K. Kleeberger, R. Bormann, W. Kraus, and M. F. Huber. A survey on learning-based robotic grasping. Current Robotics Reports, 1(4):239-249, 2020.
[29] A. Sinha, P. Malo, and K. Deb. A review on bilevel optimization: from classical to evolutionary approaches and applications. IEEE Transactions Evolutionary Compulation, 22(2):276 295, 2017.
[30] Zimmermann, G. Hakimifard, M. Zamora, R. Poranne, and Coros. multi-level op- timization framework for simultancous grasping and motion planning, IEEE Robotics and Automation Letters, 5(2):2966-2972 2020.
[31] Y. Zhu, Z. Pan, and K. Hauser. Contact-implicit trajectory optimization with learned de- formable contacts using bilevel optimization. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 9921-9922 IEFF, 2021.
[32] B. Landry, Lorenzetti, 7. Manchester, and M. Pavone. Bilevel optimization for planning 2019.
through contact: A semidirect method. arXiv preprint
10

---

[33] Y. Shirai, D. K. Jha, A. U. Raghunathan, and D. Romeres. Robust pivoting: tional stability using bilevel optimization. In 2022 International Conference on Roboties and Automation (ICRA), pages 992-998. IEFE, 2022.
Exploiting fric-
[34] T. Stouraitis, I. Chatzinikolaidis, M. Gienger, and S. Vijayakumar. Online hybrid motion plan- ning for dyadic collaborative manipulation via bilevel optimization. IEEE Transactions on Robotics,
2020.
[35] Z. Zhao, Z. Zhou, M. Park, and Y. Zhao. Sydebo: Symbolic-decision-embeeld bilevel op- timization for long-horizon manipulation in dynamic environments. IEEE Access, 9; 9:1288177 7- 128826, 2021.
[36] D. E. Stewart and J. Trinkle. An implicit time-stepping scheme for rigid body dynamics with inelastic collisions and coulomb friction. International Journal for Numerical Methods Engineering, 39(15))2673-66)) 1996.
[37] B. Calli, Singh, Bruce, Walsman, Konolige, Srinivasa, Abbeel, and A. M. Dol- lar. Yale-cmu-berkelle dataset for robotic manipulation research. The International Journal of Robotics Research, 36(3):261-268, 2017.
[38] D. P. Kingma and M. Welling. Auto-encoding variational bayes. 13Xiv:1312.614 14,61111411011 2013.
arXiv preprint
[39] C. R. Qi. L. Yi, H. Su, and L. J. Guibas. Pointnet++:: Deep hierarchical feature learning on point sets in metric space. Advances in neural information processing systems, 30, 2017.
[40] P. E. Gill, W. Murray, and M. A. Saunders. Snopt: An sqp algorithm for large-scale constrained optimization. SIAM review, 47(1):99- 131, 2005.
[41] R. Tedrake and the Drake Development Team. Drake: Model-based design and verification for robotics, 2019. URL https://drake: wit.tee.m...ee... mit.edu.
[42] B. Amos and J. Z. Kolter. Optnet: Differentiable optimization layer in neural networks.
In International Conference on Machine Learning, pages 136-145. PMLR, 2017.
[43] W. Yuan, T. Khot, D. Held, C. Mertz, and M. Hebert. Pen: Point completion network. In 2018 International Conference 3D Vision (3DV), pages 728-737. IEEE, 2018.
[44] M. Sarmad, H. Lee, and Y. M. Kim. Rl-gan-net: A reinforcement learning agent controlled gan network for real-time time point cloud shape completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5898-5907, 2019.
[45] X. Han, Z. Li, H. Huang, E. Kalogerakis, and Y. Yu. High-resolution shape completion using deep neural networks for global structure and local geometry inference. In Proceedings of the IEEE international conference on computer vision, pages 85-93, 2017.


Reviews:
[{"review_id": "u9BLd3VtzX9", "reviewer": "Reviewer_GKLm", "summary": "The paper presents a grasp planning pipeline comprised of an initial guess from the object geometry and a subsequent optimization to refine its physical feasibility.\nThe initial guess of promising grasps are generated from the prior of a learned conditional VAE over 6 YCB object point clouds. \nThe refinement optimization is based on force-closure criteria and specifically decomposed into two levels to exploit pyramidal friction cones for a QP structure at the lower-level.\nThe experiments include real robot grasps for seen, novel objects and ones with adversarial features. The pipeline demonstrates success rates between 55%-70% and is compared to hardcoded and CVAE only baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\nThe paper execution is clear. The technical idea is neat for handling non-affine equality constraints via a bilevel treatment and avoiding initial guess of finger-tip forces. Real robot experiment is carried out.\n\nWeakness:\n\nSignificance and relevance to robot learning: the only learning part of the paper is training a CVAE as a generative model to sample roughly correct grasps. \nThe main technical contribution is closer to grasp planning and the paper content does not highlight the significance of CVAE and its necessity to other alternatives in the pipeline. The data and generalization setups look not very extensive.\nAlso, the idea of learning a generative model for synthesizing diverse grasps or potential actions is not novel, see some previous works using non-NN generative model [1][2].\n\nExperimental results: it is hard to tell how successful the proposed pipeline is from the main chart Table 1 and 2. Interestingly, the proposed pipeline is even not performing as well as the hardcoded baseline for the category of seen objects.\nI am always wondering to what extent qualitative results like in Figure 4 can be attributed to the exact planning results. As far as I know, the physical realization of a grasp tends to be disturbed from its nominal result due to the gap of finger contacts and inaccurate execution on the Allegro hand (lacking compensation for inertia, joint frictions, etc.).\nA static grasp does not specify how the robot hand should approach the object and whether finger placement could be well synchronized as such the grasped object will not be tipped away from the intended pose. \nThe paper can be greatly improved if it could decisively show that it is the planner instead of some controllers doing much of the heavy lifting here. This could at least help to address why we are observing a mixed message from Table 1 and 2.\n\n[1] Huang et al, Learning a real time grasping strategy, ICRA 2013\n\n[2] Sun et al, Characterizing Continuous Manipulation Families for Dexterous Soft Robot Hands, Frontiers in Robotics and AI 2021 \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper contains some good technical ideas and the execution is generally smooth. However, the paper, at least in its current form, only shows a tenuous relevance to learning methods. The results are also not very conclusive and open to interpretation. \n\n[Update on the version of Aug 28th] After seeing the authors' rebuttal, I am more leaning towards borderline so the scores are updated.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "- relevance to robot learning\n- technical novelty\n- lacking adequate evidence and discussion about the role of the contributed planning method ", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "Okq_e_W-kIg", "reviewer": "Reviewer_Fnic", "summary": "This paper presents a framework for dexterous grasp of a single object in uncluttered scenes. Specifically, the authors first trained a conditional variational auto-encoder (CVAE) to generate an initial set of contacts points given the observed object point cloud and a feasible grasp from the dataset. This will reduce the combinatorially complex contact choices when solving by an analytical approach. However, it cannot guarantee the contact points are physically feasible. Therefore, the authors proposed to use bilevel optimization to map the contact points predicted by CVAE onto the manifold of kinematically and dynamically feasible grasps. The proposed framework is evaluated in real world with a Franka Emika Panda arm and an Allegro hand. Compared with the other two baseline methods, i.e., hardcoded and CVAE-only framework, the proposed method achieved better or comparable results on seen and adversarial objects.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- This paper presented a well designed framework for dexterous grasping using raw point cloud as input, and conducted real-world robots experiments which demonstrated the proposed method can be applied to both seen and unseen objects.\n- The paper is well organized and the writing is clear and easy to understand. \n- I also appreciate the details and the videos in the supplementary materials.\n\nWeaknesses\n- Though I like the practicality of the proposed system, the novelty of the paper might be limited. Predicting contacts or contact points is not a novel idea and has been explored in many papers, such as [14, 17, 18], and [a, b]. Using CVAE to predict grasp is also used in [7], but I think the idea of generating grasps conditioned on a grasp from the dataset is interesting and can constrain the predicted grasps. The authors could add more discussion and analysis on this.\n- Bilevel optimization for dexterous grasp might be novel but using analytical approach to refine/solve the prediction/initialization from a learning-based approach is not novel.\n- Though the proposed framework seems well designed and engineered, it did not outperform the hardcoded baseline: it is hard for me to tell which one is better. Therefore it is difficult to verify the effectiveness of the proposed method.\n- I am also curious why the results of hardcoded and CVAE-only baselines are missing in Table 2.\n\nReferences\n- [a] Brahmbhatt, Samarth, et al. \"Contactdb: Analyzing and predicting grasp contact via thermal imaging.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.\n- [b] Brahmbhatt, Samarth, et al. \"Contactgrasp: Functional multi-finger grasp synthesis from contact.\" 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2019.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "**Final comment: thanks to the authors for the additional experiments and discussions to address my concerns and other reviewers' questions. I will keep my rating unchanged.**\n\n**Minor: Table 1-3. It'll be more \"fair\" if the hard-coded one also has 6 trials.**\n\n----\n(initial comment)\nThe paper is well written and the real-world experiments look good. But the technical novelty is a bit limited. My rating is more on the boarder line.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See above.\n\n\nAdditional:\n- What is the reason to train the model on only 5 objects?\n- What is the computation time of the overall system and each component?\n", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "NfdLXCb_0_n", "reviewer": "Reviewer_SirZ", "summary": "This paper adopts a CVAE network to predict grasp pose for dexterous hand and optimizes it with a bilevel optimization to force grasp pose to be more physically feasible.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1.\tThis paper considers both kinematic and dynamic constraints for a more physically feasible grasp pose which seems novel to me, as far as I know, this is the first to consider both two levels.\n2.\tThis paper conducts rich real-world experiments.\nWeaknesses:\n1.\tBaseline seems missing: Since the main contribution of this paper is bilevel optimization, the author should compare with the baseline that only uses one level to optimize. Such as [1][2], consider the collision, [3] consider force closure. However, the author seems not to compare the method with these baselines.\n[1] Wei, Wei, et al. \"DVGG: Deep variational grasp generation for dextrous manipulation.\" IEEE Robotics and Automation Letters 7.2 (2022): 1659-1666.\n[2] Liu, Min, et al. \"Generating grasp poses for a high-dof gripper using neural networks.\" 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2019.\n[3] Liu, Tengyu, et al. \"Synthesizing Diverse and Physically Stable Grasps With Arbitrary Hand Structures Using Differentiable Force Closure Estimator.\" IEEE Robotics and Automation Letters 7.1 (2021): 470-477.\n2.\tEvaluation metric: although the author mainly focuses on physically feasible grasp, the metric in this paper is success rate without metric about how to measure the grasp is more physically feasible, if only compare success rate, other methods which may not consider physically feasible may also achieve success rate. At least in simulation, I think the author should use other metrics, such as penetration for depth or volume used in [1].\n3.\tSince the main difference is bi-level optimization, ablation should show how these two levels contribute to the final result.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The author mainly proposes a bilevel optimization to generate a more physically feasible grasp pose which is novel. However, generating a grasp pose with CVAE is not novel, since other work[1] has already tried it.  Although this paper focuses on physically feasible, the metric for evaluation is only success rate, which can hardly show the “physically feasible”. There are also other methods that either consider kinematic or dynamic constraints which should be considered as the baseline, however, the author only conducts an ablation study that only compare w and w/o optimization, not showing which level is more important. Thus, I recommend this paper as a weak reject.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "1.\tIllustrate why not consider the work that considers either kinematic[1][2] or dynamic constraints[3] as the baseline, or provide the improvement of the method compared with these methods\n2.\tProvide more metrics to show “physically feasible” either in simulation or real. Such as measuring the penetration before and after optimization in simulation.\n3.\tAblation study should include the experiment to show how these two-level optimizations contribute to the final result, e.g., which one might be more important?\n"}}, {"review_id": "4BwOApGClee", "reviewer": "Reviewer_XLAr", "summary": "Proposed using bilevel optimization to hand dexterous grasping problems with three fully actuated fingers. Provided derivations to the grasp planning pipeline with further verifications in experimental results. Considered both kinematic and dynamic constraints in performing the grasping task. Unclear how learning is integrated into the proposed method, at least the integration with CVAE is relatively marginal, contributing to the proposed grasping method. But there is a lack of sufficient details to explain the bilevel optimization, which is more prone to optimization.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\nThe use of bilevel optimization in dexterous grasping is interesting and is backed by analytical reasoning from both dynamics and kinematics constraints. The application of the resultant method in robotic grasping is also a plus. \n\nWeakness:\n- It is unclear how learning is integrated based on the current content, more like optimization based on the current content. \n- The proposed method, bilevel optimization, lacks sufficient details to explain what it is. Only one short paragraph is not enough, especially when this is the core method behind this paper. The derivation part helps but not sure what it means by bilevel or how it contributes. \n- The analytical part is excellent, but more experiments are needed. The method is already complex enough based on the current content, making it suspicious to see how much time would it cost to generate a grasp (any advantage over existing analytical or learning-based methods?) At least, the grasp planning time is not reported in the current paper.\n- The authors mentioned many advantages during deriving the method but later explained that getting a working solution for grasping is not an easy task. This is also confusing but not surprising as the method is simplified and more complex. Why this method is not yet used in literature? Any superior advantage?\n- The experiment results are suspicious, many 66.7% (2/3), 83.33% (5/6), 50% (1/2), 33.3% (1/3), then another one with 94.44%. In short, the sample size is too small, and the authors should have conducted more grasping trials to further justify if this result is consistent or just luck.\n- It seems the CVAE is the only learning-related part. Not sufficient based on the current content to be learning focused on robotic manipulation or the scope of the conference.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The main problem is probably the complexity of the proposed method itself is not thoroughly learning-related, lacks comparison with other dexterous grasping methods, without sufficient experimental trails to support the claimed results (even the authors admitted that the grasp planning success rate is challenging), and not sure how would such method be used to strength contribution of learning in robotics. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- Explain clearly how learning contributes to the proposed method for robotic manipulation.\n- Explain the bilevel optimization method with further details. A brief paragraph is not enough.\n- How much time would it cost to generate a grasp planning solution? What is the success rate for generating a grasp planning?\n- The method is rather complex to reproduce without code. Please provide and justify how would others reproduce the results.\n- Please add (significantly) more grasp trails to enlarge the sample size of the reported results.\n- While the authors used dynamics in the derivation while performing the grasping, the authors explained, “A fixed, hand-designed squeezing motion is performed after q* has been achieved to provide contact forces.” If the squeezing motion is achieved through a fixed, hand-designed command, then what’s the point of incorporating dynamics in grasp planning? In other words, the proposed method does not actually provide actual use during grasping.\n- It is strange that the “massage ball” shows poor results, and no tactile sensors are used on the fingertip. So, how does the force used in performing the calculation or during the execution? How does the proposed compensate for material softness based on the presented method? How to justify that if not using the proposed method, massage balls like these are usually quite simple to be picked up by a three-finger gripper. \n"}}]

Author rebuttal:

Decision: Accept (poster)