Dynamics-Guided Diffusion Model for Sensor-less Robot Manipulator Design
Authors: Xiaomeng Xu
Huy Ha
Shuran Song
Venue: CoRL 2024 (2024)
Keywords: manipulator design
hardware optimization
diffusion model
TL;DR: Generating task-specific manipulator designs without task-specific training

Abstract: We present  Dynamics-Guided Diffusion Model (DGDM), a data-driven framework for generating task-specific manipulator designs without task-specific training. Given object shapes and task specifications, DGDM generates sensor-less manipulator designs that can blindly manipulate objects towards desired motions and poses using an open-loop parallel motion. This framework 1) flexibly represents manipulation tasks as interaction profiles, 2) represents the design space using a geometric diffusion model, and 3) efficiently searches this design space using the gradients provided by a dynamics network trained without any task information. We evaluate DGDM on various manipulation tasks ranging from shifting/rotating objects to converging objects to a specific pose. Our generated designs outperform optimization-based and unguided diffusion baselines relatively by 31.5\% and 45.3\% on average success rate. With the ability to generate a new design within 0.8s, DGDM facilitates rapid design iteration and enhances the adoption of data-driven approaches for robot mechanism design. Qualitative results are best viewed on our project website https://dgdmcorl.github.io.

Dynamics-Guided Diffusion Model for Sensor-less Robot Manipulator Design
Xiaomeng Xu Huy Ha²
Shuran Song'
Stanford University CColubiia University
https//ddgdmcorllgihbbbb
Abstract: We present Dynamics-Guided Diffusion Model (DGDM), a data-driven framework for generating task-specific manipulator designs without task-specific training. Given object shapes and task specifications, DGDM generates sensor- less manipulator designs that can blindly manipulate objects towards desired mo- tions and poses using open-loop parallel motion. This framework 1) flexi- bly represents manipulation tasks as interaction profiles, 2) represents the design space using a geometric diffusion model, and 3) efficiently searches this design space using the gradients provided by a dynamics network trained without any task information. We evaluate DGDM on various manipulation tasks ranging from shifting/rotating objects to converging objects to a specific pose. Our gener- ated designs outperform optimization-basee and unguided diffusion baselines rel- atively by 31.5% and 45.3% on average success rate. With the ability to generate a new design within 0.8s, DGDM facilitates rapid design iteration and enhances the adoption of data-driven approaches for robot mechanism design. Qualitative results are best viewed on our project website https://dgdmoorl.gihhe github.io. Keywords: manipulator design, hardware optimization, diffusion model
Sensor-less Manipulation Tasks
before interaction
after interaction
Target Object
Shift Up
Shift Left
Rotate Clockwise
Convergence (from different initial poses to the same pose)
Figure 1: Task-specific Designs without Task-specific Training. Given different input objects (Ist column), DGDM generates diverse manipulator geometrics tailored to different manipulation tasks without task-specific training, which can be deployed under the sensor-less setting with an open-loop parallel closing motion. Introduction
Mechanical intelligence refers to the utilization of mechanical design to solve tasks [1]. A substantial body of evidence in both natural [2] and artificial systems [3] has demonstrated that well-customized embodiments can significantly simplify an agent's perception and control, thereby enhancing overall robustness [4]. Despite its advantages, mechanical intelligence in robotics has recently been over- shadowed by the rapid development of its counterpart, "action intelligence", where the agent focuses on inferring different actions for different tasks, assuming fixed mechanical embodiment design.
In contrast, learning for mechanical design has largely focused on single task optimization [5, 6] or heavily engineered objective functions that could not be reused for new design task [7, 8, 9, 10, 11,
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Task & Object Conditioned Design
Sensorless Deployment
Target Object
Finger Designs
before
Dynamics-
Guided
1201
90
Diffusion Model
Task Specification
Convergence
"Align object specific pose"
O pagget
alter
Figure 2: The Convergence Task is design fingers that always reorient target object specified orienta- tion Starget (in the manipulator frame) when closing the gripper parallel. This task enables funneling objects from arbitrary poses to a specific Btarget in a sensor-less setting, and moving objects to any particular configura- tion combined with global transformation of the gripper. Despite its utiliiyy designing for convergence can be counter-intuitive it often takes expert many design cycles come up with just one design for one ohject. In contrast, DGDM can generate functional design for object in seconds.
12]. In practice, this means automating task-specific design typically involves recollecting training data for every scenario, which is too expensive be practical. Therefore, we investigate the follow- ing question: Can we automate task-specific mechanical design without task-specific training?
We introduce Dynamics-Guide Diffusion Model, a framework that generates manipulator geom- etry designs that can manipulate objects towards desired motions and poses with no task-specific training and no perception or closed-loop control only parallel jaw closing motion. From tasks simple shifting/rotating objects complex tasks requiring sequential interactions such pose convergence (Fig. 2), DGDM generates designs in seconds with geometry changes that are highly adapted to the task and object. Our framework answers two key research questions:
How to represent the task space? The task representation has to be expressive enough to capture the wide range of manipulation tasks while being compact enough be readily learned from data. Our key insight is that many manipulation tasks can be decomposed into a collection of individual motion targets that specify how each object should move under each initial pose, which we call interaction profile. While the final composed objective is specific to the task, each of the individual motions can be modeled by a generic dynamics network that is reusable across tasks.
How to facilitate efficient search? As the design space grows, the design objective landscape of- ten becomes multi-modal w.r.t. the design parameters, and generating promising yet diverse design candidates becomes challenging. To address this issue, we first represent the design space using an unconditional geometric diffusion model. Then, the interaction profile for an object and the fingers is inferred with the dynamics network. The design objective constructed by comparing the current with the target interaction profiles gives gradient how to update the finger. This dynamics guidance is incorporated into diffusion denoising steps similarly to classifier guidance [13].
We demonstrate results on both 2D and 3D objects with a variety of manipulation objectives ranging from simple to complex and single- to multi-object objectives, all under a sensor-less setting, where the initial pose of the object is unknown. Experiments in simulation and the real world demonstrate that designs generated by DGDM achieve high task performance, with 31.5% and 45.3% relative success rate improvements compared to optimization and unguided diffusion baselines.
Related work
Manual End-effector Design. The diverse array of manipulator designs we see today, including serial versus parallel, from dexterous to underactuated, typically start from many trail-and-end -eroo iterations by experts. Heavy manual efforts are needed to discover optimal and occasionally counter- intuitive designs, which hinders the development of designs for new applications. For instance, for complex manipulation tasks such convergence (Fig. 2), previous works only deal with 2D planar polygons, utilizing manual/analyticn designs of grasping policy [14, 15] or gripper geometry [16].

---

Analytical Optimization for Automatic End-effector Design. To alleviate the manual efforts, previous works have explored optimization approaches to manipulator design. Non-linear optimiza- tion approach [17] typically requires careful task-specific formulation of objectives and constraints. First-order optimization of morphology [18] or both morphology and control [17, 7, 19] more common, but requires careful initialization (task-specific parameterization [18], cage-based defor- mation [7, 8), or heuristics [19)). Further, tasks involving complex contact modes are known to yield biased and high variance gradients in differentiable simulators [20. 8). Importantly, all manual efforts involved in setting up optimization problem are typically not transferrable to new tasks.
Data-driven Robot Hardware Design. Data-driven approaches improve over optimization-ban approaches by transferring knowledge from training to reduce the cost inference. A common approach is to train value network that takes the design parameterization as input and outputs the design's 's task performance. This value network can be used to guide searchloptimization pro- cedure, which has been explored in gripper design [5] and locomotion [21, 22], to guide optimiza- tion [5, 23. 6] or graph search [21, 22, 10, 24]. Another approach is to learn a generative model of the design space, which compresses the design space into a low-dimensional continuous latent space. This makes offline optimization via gradient-descent [5, 23] or online optimization via trial-and- error rollouts of random latent-space samples [5, 10, 6] significantly more efficient. Finally, when co-optimizing morphology and control, leveraging control experience from prior embodiment evalu- ations can significantly improve the efficiency and accuracy of new embodiment evaluations [9, 12]. However, all these approaches require large amount of task-specific data, while we eliminate this requirement by leveraging dynamics as the shared structure between manipulation tasks.
Approach
3.1 Interaction Profiles as Task Specification
Requirements for a manipulation system are incredibly diverse, ranging in what initial poses are allowed, what objects are considered, and what the desired effects are. Parameterizing the space of manipulation tasks call for a representation expressive enough to capture all the diverse tasks. More- over, this representation should be compact containing only the necessary information to capture how the object interacts with the finger, such that is efficient evaluate/learn. For instance, mod- eling the detailed physics states in differentiable simulators [7] is expressive, but forward integrating the dynamics over the time horizon for every finger evaluation is expensive.
Interaction Profiles. Many manipulation tasks can be decomposed into a collection of individual motion targets that specify how each object should move under each initial pose after interacting with the manipulator. By combining motions from all objects and initial poses, we get a complete profile of how the manipulator will interact with the target objects the "interaction profile".
Denote by and m the geometry parameters of object shape and manipulator shape. When the object is at the initial planar pose = (0.xy)) closing the manipulator once will change the object's interaction dynamics 'D. We refer
pose by Ap- = (A0, Ax, Ay), dictated by the manipulator-obiee to scalar-valued functions I defined on top Ap as motion objectives and aggregate these motion objectives among all initial poses p and objects o to get the design objective
Example: Multi-object Shift Up
To design manipulator that shifts set of objects upwards, each motion objective is defined as flo,m, m. p) Ay(o,m,
(1)
where Ay is the y-translation component of Ap. The design objective is aggregated from (1) as
F(m) ΣΣ fl.
g
Interaction profiles can scale to varying ranges of initial poses and objects. Thus, using a larger set of initial poses and objects will yield an objective that is more robust to different initial poses and
3

---

tailored to more objects. Since each motion objective is conditioned on p. this approach also allows for objectives dependent on initial poses, as illustrated in the following example.
Example: Pose Convergence
The goal of pose convergence to rotate an object to a target orientation Ourget relative the manipulator (Fig. 2). The manipulator should funnel wide range of initial configurations into single target orientation with no perception, no closed-loop control, only a parallel gripper closing motion on repeat. How the object should rotate depends on the initial orientation 0 relative to the target orientation ourget:
flo m.(). -A((o,
if 0 E [Qurget n, Ouarget]
0 E [Gaggets Oarget + +
The objective for this task can then be aggregated from (3) analogously to (2).
If F can be efficiently computed, we can use this gradient optimize a pair of fingers m that achieves the task. To achieve this, we propose to represent interaction dynamics 'D as neural network and train it using data generated from interactions between random finger-object pairs.
3.2 Dynamics Network
The dynamics network D (o, m. p) + Ap aims to learn a general model of how random distribu- tion of fingers interacts with distribution of objects. Importantly, it provides gradients of the design objective with respect to the finger representation (Fig. 3).
Shape Representation. We choose cubic Bézier curves and surfaces as the manipulator shape representation [25]. Control points are grid sampled along the length (and height in 3D) of the finger while the remaining y-coordinate determines its protrusion outwards/inwards, which define as the geometry parameter of manipulator. We represent object shape 100 points from each 2D object contour and 512 3D points from 3D object surfaces.
point cloud by sampling
Motion Representation. We represent object motion under interaction as a dinee-dimensional vec- tor consisting of delta rotation along the 7-axis, delta translation along the x-axis, and delta transla- tion along the y-axis, denoted as Ap (A0, Av))
Network Architecture. First, we transform object initial poses with high-frequency positional encoding trick used to combat over smoothing of neural networks [26]. Then, o and m are passed through separate 2-layer MLPs with 256 hidden dimensions, before being concatenated with the pose embedding. Finally, the resulting embedding is passed through an 8-layer MLP with 256 hidden dimensions get the predicted object motion Ap. In 3D, we use a PointNet++ [27] encode object geometry, whereas all other parts of the network are shared between 2D and 3D tasks.
Training Data Generation. Our training data generation happens once for all tasks. We sample object and manipulator pairs, load them into MuloCo [28] simulation environment, and measure Ap after a single parallel closing interaction. We generate 321 planar object shapes from the Icons- 50 dataset [29] in 2D, and select 164 objects from Google's Scanned Objects Dataset [30, 31] in 3D. We randomly sample 1024 manipulator geometry parameters m from uniform distribution. For each object-fingers pair, grid sample 360 initial orientations and 25 initial positions, getting 321 x 1024 x30 25 training data points for 2D dynamics network and 164x (024x360x22) 25 for 3D.
Design Objective Gradient Evaluation. To design manipulators that are generalizable to more initial poses p and objects o, the design objective F (2) can be evaluated for a wider range of poses and objects. We grid-sample initial poses of set of objects and evaluate motion objectives in parallel. The design objective gradient V,, is attained by aggregating the gradients along the pose/object batch dimension. For each new design, evaluating v,,F takes 0.16 seconds on average, making it efficient to run in the inner loop of iterative design procedures. Specifically, we grid- sample 360 orientations and 5x5 positions, getting 360 5 5 dimensional motion profile.
3.3 Dynamics-Guided Diffusion Model

---

Target Object
Task Objective
Given design objective gradients from D, the obvious approach to perform gradient de- scent on the finger geometry [5. 19]. How-
Dynamics Model
Interaction Profile
Q1
Gaussian Noise me
Sampled Poses
ever, the distribution of good designs are often multi-modal, which means gradient descent ap- proaches quickly get stuck in local minima. To efficiently navigate through the large and multi- modal design space, we extend classifier guid- ance [13], an iterative diffusion model sampling approach that enables a balance between diver- sity and task-specific guidance (Fig. 3).
Final Fingers
Noisy Fingers
target
m
objective
gradients
Diffusion Model
NEL contral peints
repeat K. iterations
Figure 3: DGDM generates finger shapes given a target object and task. specified as target interaction profile 3.1). This is compared with the dynamics network's prediction of the current interaction profile, which used to construct an objective ($ 3.2). Gradients of the objective itcratively guide the reverse denoising process of a manipulator shape diffusion model ($ 3.3).
Diffusion Models [32, 33] class of proba- bilistic generative models that generate samples from an underlying distribution through itera- tive denoising. A diffusion model Eg (m)g) pre- dicts the noise added to a sample mo. We start with Gaussian noise mg and gradually predict less-noisy samples mk-1, 1 mK--,... until mo through a reverse noising process of modeling the distribution Pa ( Specifically, we sample geom- etry parameters of manipulator from a uniform distribution and train geometric diffusion model (with ID UNet architecture [34]) on this distribution once and for all tasks. We employ Denoising Diffusion Implicit Models (DDIMs) [35] for diffusion sampling process with 15 training denoising iterations and 5 inference iterations, and the Square Cosine noise scheduler [36].
Classifier Guidance [13] guides the reverse noising process with priors of an unconditional diffu- sion model. It requires classifier Po (4]mg), where my is the sample, / is the class label, and ø is the classification network. Leveraging the connection between diffusion models and score match- ing [37, 38). new noise prediction can be defined as:
E (m²) == Eg (m²) v C V Po (1
(4)
where dg = 1 can be performed with the modified noise prediction for conditioned sampling.
B. B, is the variance of Gaussian noise added to samples at step Then DDIM
Dynamics Guidance. To guide the design gen- eration towards specified manipulation tasks, we extend classifier guidance to use design objec- tives constructed from interaction profiles, which we term dynamics guidance. We replace classi- fier gradients with M F ((m)) to guide the DDIM sampling process (Algo. 1). We not only en- able guiding unconditional diffusion models with task-specific gradients, but also allow tuning the guidance scale trade off diversity and perfor- mance. Dhariwal and Nichol [13] showed that when y is larger the distribution becomes sharper and generated samples have higher fidelity, while smaller s leads to more diverse samples, which we also observed in our generated results (Fig. 4).
Algorithm 1 Dynamics-guided DDIM sam- pling, given diffusion model Ea (m))) design objective F F(m), and gradient scale
Input: design objective F(-). gradient scale s mg + sample from (0,1) for all & from K to I do
? ê Eo (m)) -VVI VF (m²)
mi-1 V
ag
0/
end for return mo
Evaluation
Manipulation Tasks & Metries. We evaluated each approach on held-out objects (8 in 2D, 6 in 3D) and manipulation tasks. Each pair of fingers is mounted to WSG50 gripper performing fixed open-close action. We categorize our suite into two difficulty levels: 1) Simple objectives in- volve single-axis object movements in SE2 space, including shifting up/downnleft/rightt and rotating Mlockwise/counterceekwwi
For each object-manipulator pair, we grid-sampled 360 planar initial object orientations and performed fixed open-close actions. The task is considered as successful if the movement of the object along the specified axis is larger than predefined threshold (0.03 rad for rotation, 3 mm for x-axis translation, 2 mm for y-axis translation). For example, manipulator
5

---

designed for the rotating objective succeeds if it rotates the object larger than 0.03 rad after the first closing action. Then, we report the average success rate over all sampled initial object orientations. 2) Complex objectives combine multiple simple objectives to parameterize a broad range of ma- nipulation tasks. For the convergence objective, we report the maximum convergence range ("), indicating the broadest range of initial object orientations that can be driven towards a consistent final orientation within small tolerance (5%). Observing continued object movement, we report the metric after the 40th open-close manipulator action. Additionally, we explored rotate clockwise and shift up/left, and rotate either way objectives to showcase DGDM's flexibility in composing conflicting objectives. These tasks were evaluated on average success rates.
Comparisons. Removing the dynamies guidance yields the Unguided baseline, which generates task-agnostic manipulators using our geometric diffusion model. Removing our diffusion model yields the GD baseline, which optimizes the manipulator control points using gradients of the de- sign objective VF via gradient descent optimization, common for many prior works [5, 7, 8. 19]. We also evaluated a gradient-free free optimization baseline CMA-ES [39] (covariance matrix adap- tation evolution strategy) that optimizes finger control points from objectives constructed from the dynamics network. Notably, we ran it with more than x 10 the compute (and x 10 time) of DGDM. To mitigate performance variance due to initialization, we ran each approach 16 times with different initializations per object-task pair and selected the best performance, then averaged among objects.
4.1 Experiment Results
Table I: Single Object Evaluation (Avg success rates and convergence range ")
Generating task-specific manipulators designs. DGDM generates tailored fin- gers for a wide variety of scenarios, sur- passing the unguided baseline across all tasks. The advantages of generating cus-
Simple Objective
Complex Objoctive
counter
clockup
drockeeh
left
Fight
clock
101812
tom fingers become more pronounced the design requirements escalate. For in- stance, DGDM exhibited a +16.6% im- provement over the unguided baseline in 2D simple objectives, a figure that ex- panded to 20. 10.0% in 2D complex objectives (see Tab. 1). A similar trend was observed when transitioning from 2D 3D objects (+18.0% over
Unguided
56.8
82.1
82.9
80.4
46,9
58.5
74.0
36.4
36.9
61.7"
GD
79.5
53.3
81.3
94.0
48.8
73.2
78.9
29.3
49.4
73.7"
2D
CMA-ES
80.7
82.2
88.1
97.0
60.5
73.9
80.3
52.2
55.8
73.4"
DGDM
88.2
92.0
96.7
97.7
60.8
72.0
79.3
62,8
63.7
83"
Unguided
43.0
43.8
80.4
87.9
41.2
33.5
64.2
30,3
33.1
63.6"
3D
GD
47.4
66.3
86.3
88.1
59.1
52.0
66.9
29.2
37.7
60"
CMA.ES
50.2
70.8
85,2
80.9
53.9
50.3
72.7
32.7
42.1
70"
DGDM
81.5 75.1
95.1
97.2
69.9
65.0
83.0
57.1
58.2
72.5
Unguided in 2D, +23.4% over Unguided in 3D, Tab. 1) and from single-object multi-object de- signs (+ 18.0% over Unguided in single-obj 2D, +18 +8.6% over Unguided in multi-obj 2D, Tab. 2).
When task complexity, design space, and the target object set grow, a human expert designer would face significantly increased time and ef- fort. DGDM handles progressively complex design requirements by aggregating gradients from individual motion objectives. A new task can be specified as long as users can articulate
Table 2: Multi-object Evaluation
Simple
Complex
clockup
clockleft
down
lelL
right
clock
1otate
Unguided
55.8
79.8
77.1
80.2
44.7
56.4
68.3
35.2
35.2
GD
78.6
50.3
79.2
93.8
46.1
71.4
74,3
25.0
49.0
2D
CMA-ES
62.2
79.3
97
73.1
22.1
37.3
how each object should move from each initial pose, and can be seamlessly incorporated into the diffusion denoising process.
DGDM
83.8
88.1
99.3
94.3
61.3
68.4
78.4
62.4
63.8
Unguided
40.1 40.8
75.8
87.9
34.7
29.2
61.7
29.2
25.2
GD
42.4
66.4
77.9
86.6
40.3
39.2
67.0
22.6
34.3
3D
CMA-ES
65.6
76.9
89.2
45.2
68.4
28.7
33.5
DGDM
89.7 66.8
96.1
95.4
69.3
58.2
77.6
44.2
37.9
Robust & efficient search with guided diffu-
sion. The GD baseline and DGDM share the same design objective gradients from our dynamics network, differing only in how this gradient information is incorporated. The baseline uses gradi- ent descent, requiring upwards of 18 and 24 minutes to converge in 2D and 3D (for 16 samples), respectively, and prone to local minima. In contrast, DGDM utilizes slassifier-uiidanee with a diffusion denoising process, which strikes a balance between exploring different modes (by intro- ducing Gaussian noise) and exploiting the current mode (using the gradient of design objective) through a guidance scaling factor (Fig. 4). This results in + +1.8% and - 10.4% higher success rates

---

Unguided
Ours
Unguided
Ours
Lvig. rarge
evig lerget
Ovig. range
evig, larges
evig. range
evrg, larget
evig. range
evig, larget
2D
30
(4)
RD 3D
(
129.3'
171.8
Figure 5: Convergence Results, each pair finger designs, we show the range of initial orientations ("evrg. range") which converges to the same convergence mode ("evrg. target").
than the baseline 2D and 3D simple objectives, respectively. Additionally, our diffusion models prove stable even with diffusion processes as short timesteps, translating to an average design time of 13 and 54 seconds in 2D and 3D, respectively.
Emergent design for convergence. What strategies do our designs employ to achieve convergence from a broader range of initial ori- entations compared to the unguided baseline (Tab. I)? We identify two emergent design pat- terns: 1) Push-and-Catch: One finger features bulge that pushes the object into the hollow cavity of the other finger (Fig. 5 a.b,d-f.h). This cavity roughly complements the object's shape at the convergence point. 2) Parallel-Align: When objects exhibit symmetric flat edges, our designs utilize two parallel surfaces to align these edges (Fig. c.g). The generated designs simul- tancously exploit object geometry and physics to achieve the effective convergence. Rotate Clockwise
xcale=0
stale-2
scale=20
Figure 4: Effect of Scaled Guidance. From left right we increase the scaled guidance in the diffusion pro- cess. The increased scale enforces more task guidance and achieves higher task performance (shifting down) while reducing the diversity of generated designs.
Shift Down
Rotate Either Direction (3D)
Multi-obj Single-obj
Figure 6: Specialized or Generalized Design Multi-objeet Scenarios. Our approach can flexibly con- ditioned on individual objects and generate specialized design for each object ['Top] or simultancously condi- tioned on multiple objects and generate one design for all objects [Bottom].
Specialized or generalized designs for multi-object scenarios. DGDM is able to generate more generic designs for multiple objects (Fig 6). In contrast, the unguided baseline lacks task-specific guidance, hindering its ability to guide its generations toward common design objective for all ob- jects. On the other extreme, the GD baseline often gets stuck a local minimum. These limitations are reflected in Tab. 2, with our approach achieving 18.6% and +23.4% higher success rates than the unguided baseline, and + 14.7% and +17.6% higher success rates than the GD baseline. Natu- rally, acknowledge that multi-object finger designs often sacrifice some performance compared to the single-object scenario (- 1-55% in 2D, -5.2% in 3D). This balance between generality and

---

Target Obj.
Shift Down
Shift Right
Rotate Counter
Convergence
T
2
Figure 7: Real-world Results. We manufacture manipulators generated by DGDM and execute the open-loop parallel closing motion. Behaviors in simulation successfully transfer the real world. Red and green masks denote object configurations before and after interaction respectively.
task-specific performance is fundamental trade-off in mechanism design in automation, with the optimal compromise dependent on the specific application.
Real-world evaluation with Sim2Real transfer. We show real-world results of all tasks for both 2D and 3D cases by mounting the 3D printed designs on WSG50 gripper (Fig. 7). The material for 3D printing objects and fingers is PLA and the molding solution is FDM. In Tab. 3 show the real-world and simulation quantitative results side by side. For the shifting down/right and rotating counterclockwise tasks, we tested with 10 random initial poses -0"-360" orientations and +5 cm from the center) of the object in the real world and reported the average success rate (%). For the convergence task, the maximum range initial orientations (?) that can be driven to the target convergence pose is reported. We observe that the performance in the real world very close and oftentimes better than the simulation result, suggesting small sim2real gap.
This is due to where we do not need perception or closed-loop control. With the same fixed parallel motion in sim and real, the transferability is only de- termined by the geometry and contact physics. Our dynamics guidance conditioned on many initial object poses, enabling the generated de- signs to be robust to different initial object poses, relying that are consistent between sim and real. Moreover, the sensor-less manipulation tasks only require the directions of individual object motions to be accurate but not the magnitudes. We see this effect when we use PLAA with a lower friction coefficient in real than sim, allowing the objects to slide more smoothly but in the same direction, leading to oftentimes better performance real.
sensor-less formulation,
Table Real-world Quantitative Results
Shift Down
Shift Right real
Rotate Counter
Convergence sim
sim
93.1
90
100
100
75.3
111"
124"
Heart 78.3
66.1
117"
119*
91.4
100
68.6
86
3D Basket 96.9
100
100
100
65.3
82"
88
prominent physical phenomena
Conclusions
We present Dynamics-Guin Guided Diffusson Model, a versatile framework for the rapid generation of diverse and tailored manipulator geometry designs for unseen tasks. With the ability to generate a new design within 0.8s, this task-agnostic framework lays the groundwork enable more rapid experimentation and future research. We hope that our framework contributes to the wider adoption of data-driven approaches in robotic mechanism design.

---

Acknowledgements
We thank Zhenjia Xu, Cheng Chi, Mandi Zhao, Zeyi Liu, Yifan Hou, Austin Patel, Chuer Pan, Yihuai Gao, Dominik Bauer, Samir Gadre, Mengda Xu and John So for their thoughtful discussions and helpful feedback on initial drafts of the manuscript. This work was supported in part by the NSF Award #2143601, #2037101, and #2132519. We would like to thank Google for the UR5 robot hardware. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the sponsors,
References
[1] Q. Lu, N. Baron, G. Bai, and N. Rojas. Mechanical Intelligence for Adaptive Precision Grasp. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 4530- 4536, 2021.
[2] D. N. Beal, F. Hover, M. S. Triantafyllou, J. C. Liao, and G. V. Lauder. Passive propulsion in vortex wakes. Journal of Fluid Mechanics, 549:385-402, 2006.
[3] T. McGeer. Passive Dynamic Walking, The International Journal of Robotics Research, 9(2): 62-82. 1990.
[4] R. Pfeifer and G. Gómez. Morphological computation computation-ccomeciing connecting brain, body, and environ- ment. Creating brain-like intelligence: From basic principles to complex intelligent systems, pages 66-83, 2009.
[5] H. Ha, S. Agrawal, and S. Song. Fit2Form: 3D generative model for robot gripper form design. In Conference on Robot Learning, pages 176-187. PMLR, 2021.
[6] T.H.. J. Wang, J. Zheng, P. Ma, Y. Du, B. Kim, A. Spielberg, J. Tenenbaum, C. Gan, and D. Rus. DiffuseBot: Breeding Soft Robots With Physics-Augmentee Generative Diffusion Models. Advances in Neural Information Processing Systems, 36, 2024.
[7] J. Xu, T. Chen, L. Zlokapa, M. Foshey, W. Matusik, S. Sueda, and P. Agrawal. An End-to-End Differentiable Framework for Cowtect-Aware Robot Design. In Robotics: Science & Systems, 2021.
[8] M. Li, R. Antonova, D. Sadigh, and J. Bohg. Learning tool morphology for contact-rich manipulation tasks with differentiable simulation. In 2023 IEEE International Conference Robotics and Automation (ICRA), pages 1865. IEEE, 2023.
[9] C. Schaff, D. Yunis, A. Chakrabarti, and M. R. Walter. Jointly learning construct and control agents using deep reinforcement learning. In 2019 International Conference on Robotics and Automation (ICRA), pages 9798-9809 IEEE, 2019.
[10] J. Hu, J. Whitman, M. Travers, and H. Choset. Modular robot design optimization with gen- erative adversarial networks. In 2022 International Conference Robotics and Automation (ICRA), pages 4282-422 4288. IEEE, 2022.
[11] K. S. Luck, H. B. Amor, and R. Calandra. Data-efficient co-adaptation of morphology and behaviour with deep reinforcement learning. In Conference Robot Learning, pages 854- 869. PMLR, 2020.
[12] T. Chen, Z. He, and M. Ciocarlie. Hardware Policy: Mechanical and Computational Co- Optimization using Deep Reinforcement Learning. In Proceedings of the 2020 Conference Robot Learning, volume 155, pages 1158-1173 1173. PMLR, 2021.
[13] P. Dhariwal and A. Nichol. Diffusion Models Beat GANs Image Synthesis. In Advances in Neural Information Processing Systems, volume 34, pages 8790-8794, 2021.

---

[14] K. Y. Goldberg. Orienting polygonal parts without sensors. Algorithmica, 10(2-4)201-225,
1993.
[15] B. Accituno-Cabezia J. Ballester, and A. Rodriguez. Certified grasping. The International Journal of Robotics Research, 42(4-5))249-2) 262, 2023.
[16] M. T. Zhang and K. Goldberg. Gripper point contacts for part alignment. IEEE Transactions on Robotics and Automation, 18(6):902-91) 2002.
[17] O. Taylor and A. Rodriguez. Optimal shape and motion planning for dynamic planar manipu- lation, Autonomous Robots, 43:327-344, 344, 2019.
[18] A. Wolniakowski, J. A. Jorgensen, K. Miatliuk, H. G. Petersen, and N. Kruger. Task and context sensitive optimization of gripper design using dynamic grasp simulation. In 2015 20th International Conference on Methods and Models in Automation and Robotics (MMAR), pages 29-34. IEEE, 2015.
[19] M. Kodnongbua, I. Good, Y. Lou, J. Lipton, and A. Schulz. Computational design of passive grippers. ACM Transactions on Graphics (TOG), 41(4):2-12, 2022.
[20] H. T. Suh, M. Simchowitz, K. Zhang, T. Pang, and R. Tedrake. Pathologies and Challenges of Using Differentiable Simulators in Policy Optimization for Contact-Rich Manipulation. In ICRA 2022 Workshop: Reinforcement Learning for Contact-Rich Manipulation, 2022.
[21] A. Zhao, J. Xu, M. Konaković-Luković, J. Hughes, A. Spielberg, D. Rus, and W. Matusik. Robogrammar: graph grammar for terrain-optimized robot design. ACM Trunsuctions Graphics (TOG), 39, 2020.
[22] J. Xu, A. Spielberg, A. Zhao, D. Rus, and W. Matusik. Multi-objective graph heuristic search for terrestrial robot design. In 2021 IEEE international conference robotics and automation IEEE, 2021.
(ICRA), pages
[23] Hu, Whitman, and H. Choset. GLSO: Grammar-guided Latent Space Optimization for Sample-efficient Robot Design Automation. In Conference on Robot Learning, pages 1321- 1331. PMLR, 2023.
[24] Whitman, R. Bhirangi, M. Travers, and Choset, Modular robot design synthesis with deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10418-10425, 2020.
[25] H. N. Fitter, A. B. Pandey, Bezier curves in CAD for manufacturing. Procedia Engineering, 97:1155-1166, 2014.
Patel, and J. M. Mistry. A review on approaches for handling
[26] B. Mildenhall, P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
[27] C. R. Qi, L. Yi, H. Su, and J. J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. Advances neural information processing systems, 30, 2017.
[28] E. Todorov, T. Erez, and Y. Tassa. MuloCo: A physics engine for model-based control. In
2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026- 5033. IEEE, 2012.
[29] D. Hendrycks and T. Dietterich. Benchmarking Neural Network Robustness to Common Cor- ruptions and Surface Variations. arXiv preprint arXiv:1807.01697, 2018.
[30] L. Downs, A. Francis, N. Koenig, Kinman, Hickman, K. Reymann, B. McHugh, and
V. Vanhoucke. Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items, 2022.
10

---

[31] K. Zakka. Scanned Objects MuJoCo Models, 7 2022.
[32] Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and Ganguli. Deep unsupervised leam- ing using nonequilibrium thermodynamics. In International conference machine learning, pages 2256-2265. PMLR, 2015.
[33] Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[34] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention, pages 234- 241, 2015.
[35] J. Song. C. Meng, and S. Ermon. Denoising Diffusion Implicit Models. In International Conference on Learning Representations, 2021.
[36] A. Q. Nichol and Dhariwal. Improved denoising diffusion probabilistic models. In Interna- tional Conference Machine Learning, pages 8162-8171. PMLR, 2021.
[37] Y. Song and S. Ermon. Improved techniques for training score-based generative models. Ad- vances in neural information processing systems, 33:12438- 2020.
[38] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-Based Generative Modeling through Stochastic Differential Equations. International Conference on Learning Representations, 2021.
[39] N. Hansen and A. Ostermeier. Adapting arbitrary normal mutation distributions in evolution strategies: The covariance matrix adaptation. In Proceedings of IEEE international conference on evolutionary computation, pages 312-317. IEEE, 1996.
[40] A. Spielherg, A. Zhao, Y. Hu, T. Du, W. Matusik, and D. Rus. Learning-in-ther optimiza- tion: End-to-end control and co-design of soft robots through learned deep latent representa- tions. Advances in Neural Information Processing Systems, 32, 2019.
[41] Y. Hu, J. Liu, Spielberg. B. Tenenbaum, W. T. Freeman, Wu, D. Rus, and W. Ma- tusik. Chainqueen: A real-time differentiable physical simulator for soft robotics. In 2019 International conference robotics and automation (ICRA), pages 6265-6271. IEEE, 2019.

---

Appendix
Manipulation tasks and metrics
We provide more details about the evaluation manipulation tasks, and introduce more metrics for each task in this section.
Simple objectives:
Shift up: Shift the object upward (along the negative x-axis in the manipulator frame) under all initial poses. For evaluation, we grid sample 360 initial object orientations with positions in the center of the manipulator, and close the manipulator around the object. The average success rate after one closure S, (%) is reported, where success is counted if the object is shifted more than 3 mm along the negative x-axis. We additionally report continuous metrics including average delta object translation along the x-axis after one closure Ax J (cm), and average final coordinate of the object after the 40th gripper closure x 4 (cm). The metrics are averaged among the 360 interaction trails with different initial object orientations. The motion objective is f(o, moo,m,) p) = -x(x,,,, p),
then the design objective is aggregated from f(o, m. p) as F(m) L. Ep f(0, m. m.
Shift down: Shift the object downward (along the positive x-axis) under all initial poses. Metries are average success rate of the object being shifted more than mm along the positive x-axis after one closure S (%). average delta translation along the x-axis Ax 1 (cm), and average final coordinate x t (cm). The motion objective is f(o.m. p) = Ax(o, ()..
Shift left: Shift the object leftward (along the negative y-axis) under all initial poses. Metrics are average success rate of the object being shifted more than mm along the negative y-axis after one closure Sy (%)), average delta translation along the y-axis Ay 4 (em), and average final coordinate y (cm). The motion objective f(o,m. p) Ay(o,m,
Shift right: Shift the object rightward (along the positive y-axis) under all initial poses. Metrics are average success rate of the object being shifted more than mm along the positive y-axis after one closure St (%). average delta translation along the y-axis Ay 1 (cm), and average final coordinate f (cm). The motion objective is foo m, p) = Ay,o,m, p).
Rotate clockwise: Rotate the object clockwise (negative delta rotation around the z-axis) under all initial poses. Metrics are average success rate of the object being rotated more than 0.03 rad around the negative z-axis after one closure Sp 1 (%). average delta rotation around the z-axis Ao + (c), and average final orientation 0 J (*). The motion objective is f(o, m. m..p) (0(0, m,,)) p)- Rotate counterclockwiee Rotate the object counterelockwise (positive delta rotation around the z-axis) under all initial poses. Metrics are average success rate of the object being rotated more than 0.03 rad around the positive z-axis after one closure S 1 (%). average delta rotation around the z-axis AO t (?), and average final orientation f (9). The motion objective is f(o.m.)) p) = (0 m, p).
Complex objectives:
Rotate the object either clockwise or counterclockwise under all initial poses. The metrics are average success rate S t (%), the average absolute value of delta rotation around the z-axis |A0| + (*), and the average absolute value of final orientation |0] + (*). The motion objective f(o.m. p) (A,
Rotate:
Rotate clockwise and shift up: Rotate the object clockwise and shift it up under all initial poses. The metrics average success rate Sp &S, (%). average delta rotation around the 7-axis AO + (?), average final orientation 0 + ("), average delta translation along the x-axis af- ter one closure At 1 (cm), and average final x coordinate x (cm). The motion objective is fl..m = (o (o, o.m.p. p) Ax(o,m.p).
Rotate clockwise and shift left: Rotate the object clockwise and shift it left under all initial poses. The metrics are average success rate S &.S, t (%), average delta rotation around the z-axis se + ("), average final orientation 0 ("), average delta translation along the y-axis Ay (cm). and average final y coordinate y (em). The motion objective is fo.mm A0 Ay(o,m,p).
12

---

Shift Up
Shift Up (3D)
Shift Down
Shift Down (3D)
Shift Left
Shift Left (3D)
Rotate Counter
Counter (3D)
Ungsided
Opt
Curs
Figure 8: Results on Simple Objectives. We generate manipulators simple objectives that involve motion along one dimension SE2 space. Red and green object masks denote object configurations before and after interaction respectively, overlaid with the image after closing in the manipulator. Compared with baseline methods, finger shapes produced by our method achieve the task much more effectively.
Reorient the object towards fixed final pose under a range of initial poses. The design objective is encouraging the object rotate in the positive direction when its initial ori- entation is smaller than the target orientation, or else rotate in the negative direction. The motion objective is:
Convergence:
(o. m. fo.m, = (A00(o if E [Qurget Orarget + π
f E Buarget π, Burgetl
(5)
To determine the best target orientation Burget, we first forward the dynamics network with the manipulator shape initialization, object shape, and sampled initial object poses to get pseudo interaction profile. We detect the initial object orientation ranges that lead consecutive positive delta rotations followed by consecutive negative delta rotations pseudo convergence ranges. Then we select the largest pseudo convergence range's corresponding convergence orientation Guaget- The metric is the maximum convergence range Rmmx 1 (9), the largest range of initial orientations leading to Oarger within a small tolerance. We report the maximum convergence range within the tolerance of 30, 50, and 10?, respectively.
2 Additional results
2.1 More metrics
We report results all the manipulation tasks and metrics described above in Tab. Tab. 5, Tab. 6, and Tab. 7. The evaluation procedure is the same as described in the main paper, where we each approach 16 times per object-task pair and select the best performance, then average among test objects. DGDM outperforms baselines consistently on both discrete metrics (e.g. average success rate) and continuous metrics (e.g. delta object transformation and final transformation).
Table 4: Single Object Simple Objectives Evaluation
down Ar
left Ay
right Ay
clock
S, f
Ar
x4
X
x
y
5!
yf
Sg
A9
01
S
AB
01
Unguided
56.8 0.2
-1.3
-13
82.1
03
829
0.5
1.2
80.4
0.5
1.4
4699
-1.5
9.5
58.5
2.3
11.1
2D
Opt.
79.5
-0.4
-22
53.3
23
81.3
-0.6
-24
94.0
0.7
1.7
48.8
-2.9
-88
73.2
3.6
9.6
DGDM
-0.4
96.7
Unguided
43.0 -0.1
-0.6143.8
-0.6
43.8
1.0
80.4
-0.2
-.2
87.9
0.2
0.9
41.2
-1.1
-49
33.5
0,5
2.7
3D
Opt.
47.4
-0.1
-1.3
66.3
1.7
86.3
40.3
1.3
88.1
1.6
59.1
-1.9
52.0
1.2
4.6
DGDM
81.5
-0.2
-1.6
75.1
1.7
95.1
40.4
4L8
97.2
0.4
1.9
69.9
-2.3
-7.7
65.0
2.2
6.3
13

---

Table 5: Single Object Complex Objectives Evaluation
clock-up
clock-left
convergence
[A8]
Sg&S,
AB
8
Ax
x
Sy&S,
A6
0
Ay
(3")
(5""
(5') (10")
Unguided
74.0
3.5
36.4
-1.5
36,9
-0.5
56.5
61.7
2D
Opt.
78.9
4.5
18.9
29.3
-1.8
-42
00.3
-1.
49.4
-29
9.0
-0.6
-211
69.6
73.7
82.1
62.7
Unguided
64.2
2.2
16.5
30.3
-11
4.9
0.1
0.6
33.1
00.5
2,7
4.22
-1.2
52.2
63.6
67.8
Opt.
66.9
2.4
15.7
29.2
1.1
-2.7
0.5
37.7
-13
-33
0.3
-1.0
50.3
60
72.3
DGDM
83.0
3,1
21.0
57.1
-24
-68
4.2
-0.9
58.2
2.9
-1.11
-0.5
-1.7
68.8
72.5
81.5
Table Multi-object Simple Objectives Evaluation
down Ax
right AvT
clock
T Ax
x4
x
St
Ay
S
y1
S
A8
01
A9
07
Unguided
-0.2
-13
77.1
80.3
13
44.7
20
Opt.
78.6
-0.4
-10
50.3
0.3
1.0
79.2
-0.6
-.0
93.8
0.7
1.7
46.1
-2.9
-788
71.4
3,5
7.4
DGDM
-0.4
-30
88.1
34
99.3
-25
94.3
0.7
21
61.3
-2.4
-10.5
68.4
Unguided
40.1
-0.1
-0.5
40.8
0.1
0.9
75.8
-..
-08
87.
0.2
0.6
34.7
-0.5
-08
29.2
0.1
2.5
Opt.
-044
66.4
86.6
0.8
40.3
DGDM
89.7
-0.2
-1.5
66.8
0.2
1.2
96.1
-0.5
-18
95.4
04
1.3
69.3
-2.0
-5.2
58.2
1.6
3.5
Table 7: Multi-object Complex Objectives Evaluation
clock-up
clock-lef L
S +
[66]
[0]
&,8.1
58
e
Ax
x
S &S;
69
e
By y
Unguided
68.3
3.2
13.4
35.2
-1.5
-3.5
-0,2
408
35.2
-1.0
-30
-0.5
-1.0
2D
Opi.
74.3
7.2
25.0
0.1
0.1
4.2
49.0
229
4.6
404
-0.
DGDM
78.4
15.8
62.4
-0.4
-36
-24
Unguided
15.8
3D
Opt.
67.0
2.3
9.5
22.6
0.1
-0.1
-0.1
-0.3
34.3
4.9
-1.2
-0.2
-0.
DGDM
77.6
21.6
44.2
-166
-0.2
-1.2
37.9
223
8.9
-0.5
-2.3
2.2 Results averaged over initializations
For each task and object shape, we generate manipulators with 16 different initializations. Aside from reporting the best of 16 trails in the main paper, we additionally report the average performance over all trails in Tab. 8. DGDM still significantly outperforms the baselines on metrics averaged over both different initialization and objects.
Table 8: Single object evaluation, metrics averaged different initializations and objects.
Simple Objective
Complex Objective
clockup
counter
down
right
clock
rotate
left
Unguided
13.3
24.5
27.3
26.4
22.4
26.8
49,2
5.4
9.7
38.49
Opt. (GD)
25.7
30.7
35.4
37.9
27.9
35.5
54.9
8.2
11.7
41.6"
CMA-ES
34.0
42.6
41.5
51.8
32.8
38.0
56.2
16.6
16.2
40.5º
DGDM
66.4
72.4
68.9
74.2
38.0
39.4
56.3
27.5
42.1
45.0"
2.3 Task progress over action horizon
We plot the progress convergence over 40 open-close actions with manipulators generated by baselines and DGDM Fig. The Unguided baseline converges the slowest, taking more than 30 steps. The Opt. baseline exhibits an unstable dip at the and only achieves consistent alignment with the target orientation after 30 steps. Manipulators generated by DGDM not only have larger convergence ranges but also converge faster (with 10 steps) than the baselines.
14

---

Unguided
Opt.
0.5
DGDM
-uungel
0.20
0.15
0.0
600
30
35
action step
Figure 9: Task progress of convergence over 40 open-close actions. The vertical axis is 10 Ogargerl denoting the absolute difference between the current object orientation with the target orientation.
3 More details on interaction profile
Interaction profile. The interaction profile between manipulator and object is defined as the distribution of the delta pose change of the object caused by the manipulator-object interaction over interaction is a simple parallel
all possible initial poses of the object. Here, the manipulators- closing action, and the object pose is represented as 2D translation and rotation. For example, in Fig. 10, when the object's 's initial pose is (0,x,y), its delta pose after the interaction is (40, Ax, Ay), which provide one data point on the interaction profile. Then, by uniformly sampling all initial poses, we obtain the complete interact profile.
In our implementation, we use simulation to obtain the ground truth interaction profile and train dynamic network to infer the predicted interaction profile given any manipulator-object pair without simulation.
before
initial pose 0, xx
after
delta pose A0, Ax, Ay
Interaction
Interaction Profile
Figure 10: Interaction profile.
Target interaction profile. target interaction profile is defined for specific task objective. For example, the target interaction profile for the convergence task is shown in Fig. 11. This interaction profile indicates the object should rotate in positive direction when its initial orientation is smaller than the convergence rotation; otherwise, it rotates in negative direction. Tab. shows examples of other target interaction profiles for different design objectives.
15

---

A0
Ptarget
+
Otarget
+
θ
F Convergence
Target Interaction Profile
Figure 11: Target interaction profile.
Target v.s. predicted interaction profile. The difference between the target interaction profile and the predicted interaction profile (inferred by the dynamics network) provides the gradients for the manipulator design (Fig. 12). Since the dynamics network is fully differentiable, can get its gradient with respect the input manipulator geometry.
gradients w.r.f control points
0 target
Fingers
Design Objective
Figure 12: Target v.s. predicted interaction profile.
List of target interaction profile plots. We provide comprehensive list of task objectives and their corresponding interaction plots in Tab. 9. For visualization purposes, we simplify the horizontal axes of interaction profile plots to include only initial orientations 0.
16

---

Table 9: Target interaction profile plots
Task
Objective Function f(o.m.
Interaction Profilc
7A
+
uurget
+
AB(O,m,p)
if 0 E [0urget R, Otarget if E [Onugett Otarget + π
converge
A0(o,m.p)
Ax
0
up
m.
Ax
0
down
Ax(o,m.p)
Ay
0
left
- m.))
Ay
0
right
Ay(o.m.p)
A0
0
clock
mo,
A0
0
counter
A0(o.m.p)
4 Discussions
Modeling Interactions instead of Modeling Contacts. Diffcrentiable simulator [7, 8, 6, 40, 41] is another popular choice for providing the gradient of design objective VnFF, but suffers from two major limitations. First, soft contact models, such as penalty-based methods used by Xu et al., are known to yield biased and high-variance gradients [20, 8]. Further, such gradients need to be computed for each simulation timestep, which computationally expensive for long-horizon interactions. Instead of modeling individual contacts, dynamics network learns to capture the temporally extended finger-object interaction. It is trained on physically accurate simulated data, avoiding the limitations associated with soft contacts. Moreover, our dynamics network generalizes to novel objects and tasks at test time, allowing for constructing new objectives without extra data generation or training.
17


Reviews:
[{"review_id": "4XbQYOxgJP", "reviewer": "Reviewer_uUTj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces a novel approach to generate finger designs by leveraging diffusion models and a tailored guidance function. The authors propose to learn a dynamics network/model that uses as input an object representation and its pose (the state), along with the manipulator representation/parameters (the action), an outputs a delta in the objects pose (the next state).  Given a new task, e.g. ”shift the object down”, this model is first used to optimise a design objective/cost function F, by running gradient descent with respect to the manipulator parameters. Noting the problems of using gradient descent for optimizing F, namely convergence to local minima, which leads to very similar designs and task failure, the authors propose to learn an (unconditional?) diffusion model on the manipulator parameters, given data obtained in simulation.  Using this diffusion model and the previously learned dynamics network, the authors employ classifier-guidance to leverage multimodality, design precision and faster inference in comparison to the gradient descent baseline.", "summary_of_recommendation": "The paper idea is very interesting and the method and contributions are clearly explained.  The results show convincing results, in particular the sim2real transfer.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Could you clarify how gradient descent is implemented? Are you doing some form of trajectory optimization using the dynamics network (which is a kind of dynamics model)? Are the gradients of the objective function “well behaved”, or would it be worth to try a gradient free optimiser (e.g. CMA-ES)?\n\nIt’s unclear to me if you have access to the initial object orientation. You state: “How the object should rotate depends on the initial orientation θ relative to the target orientation θ_target.” Are you running some pose estimation method? In that case, why not use it as a feedback signal and run an open loop? Wouldn’t it be more sensible for this kind of interaction task?\n\nCould you clarify if, for each task description (e.g., shift down, shift right, …), you train a diffusion model (and dynamics network), or is there a single model conditioned on the task description?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xjdvwMF4uW", "reviewer": "Reviewer_fWCT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents a data-driven framework for two-finger gripper design for sensor-free planar object reorientation via parallel closing action. The paper i) learns general (parallel-closing) interaction dynamics between arbitrary finger and object shapes, and ii) uses the learned dynamics model to guide a diffusion process that generates suitable gripper shape.", "summary_of_recommendation": "The paper proposes an efficient and effective method for two-finger gripper design that is agnostic to the object shape. Experiments are convincing to me and the paper is easy to follow. My only reservation centers around clarity on a couple of things that I have explained in my comments.", "technical_quality": 4, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Please address the questions and concerns raised above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nONjoUYibD", "reviewer": "Reviewer_NVZK", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Diffusion model based designing of gripper fingers for a specified set of tasks and objects", "summary_of_recommendation": "With some more clarifications and justifications, the proposed work might be a good contribution to the community.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. Clarifying what exactly is an interaction profile. For example, a list of task objectives and their corresponding interaction plots will be very helpful.\n2. Clarify figure 3 more. Particularly, is \"current interaction profile\" showing $\\Delta\\theta$ vs $\\theta$ plot? How do you know $\\theta$ as initial pose is unknown? What are \"sampled poses\"? How do you get \"sampled poses\" while reverse sampling?\n3. Is it possible to compare with any possible related work? Also, if it is possible, initializing the start of optimization baseline with the output of unguided diffusion model might help in understanding the actual contribution of classifier guidance as compared to optimization.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Post-Rebuttal Guidance**

Recommendation: ACCEPT.
Justification: Reviewers unanimously decided to accept. Rebuttal further mitigated any concerns or weaknesses the reviews had


**Pre-Rebuttal Summary**

Summary:
The authors present an approach to generate 2-fingered gripper designs for sensorless manipulation of a unknown part. The method involves a dynamics guided diffusion model, and shows promising results on both sim and real.


Strengths:
- Well written
- Well motivated
- proposed approach is novel, fast and generalizes to new shapes

Weaknesses:
- limited generalization to wider distribution of pose orientations
- lack of comparison to related work, only ablations shown
- interaction profile not well described, questionable justification/explanation for task guidance

Dear authors, looks like the reviewers have diverging opinions on the submission as well as several clarification questions. Hope you could address the reviewer concerns.
Decision: Accept