Dynamics-Guided Diffusion Model for Sensor-less Robot Manipulator Design

Authors: xiaomeng  xu
huy  ha
shuran  song
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-05-28 23:24:02

Abstract: We present  Dynamics-Guided Diffusion Model (DGDM), a data-driven framework for generating task-specific manipulator designs without task-specific training. Given object shapes and task specifications, DGDM generates sensor-less manipulator designs that can blindly manipulate objects towards desired motions and poses using an open-loop parallel motion. This framework 1) flexibly represents manipulation tasks as interaction profiles, 2) represents the design space using a geometric diffusion model, and 3) efficiently searches this design space using the gradients provided by a dynamics network trained without any task information. We evaluate DGDM on various manipulation tasks ranging from shifting/rotating objects to converging objects to a specific pose. Our generated designs outperform optimization-based and unguided diffusion baselines relatively by 31.5\% and 45.3\% on average success rate. With the ability to generate a new design within 0.8s, DGDM facilitates rapid design iteration and enhances the adoption of data-driven approaches for robot mechanism design. Qualitative results are best viewed on our project website https://dgdmcorl.github.io.

Main paper: 
 introduction: 
 Mechanical intelligence refers to the utilization of mechanical design to solve tasks [1] . A substantial body of evidence in both natural [2] and artificial systems [3] has demonstrated that well-customized embodiments can significantly simplify an agent's perception and control, thereby enhancing overall robustness [4] . Despite its advantages, mechanical intelligence in robotics has recently been overshadowed by the rapid development of its counterpart, "action intelligence", where the agent focuses on inferring different actions for different tasks, assuming a fixed mechanical embodiment design.In contrast, learning for mechanical design has largely focused on single task optimization [5, 6] or heavily engineered objective functions that could not be reused for new design task [7, 8, 9, 10, 11, 8th Conference on Robot Learning (CoRL 2024), Munich, Germany. The Convergence Task is to design fingers that always reorient a target object to a specified orientation θ target (in the manipulator frame) when closing the gripper in parallel. This task enables funneling objects from arbitrary poses to a specific θ target in a sensor-less setting, and moving objects to any particular configuration combined with a global transformation of the gripper. Despite its utility, designing for convergence can be counter-intuitive -it often takes an expert many design cycles to come up with just one design for one object.In contrast, DGDM can generate a functional design for a new object in seconds.12]. In practice, this means automating task-specific design typically involves recollecting training data for every scenario, which is too expensive to be practical. Therefore, we investigate the following question: Can we automate task-specific mechanical design without task-specific training?We introduce Dynamics-Guided Diffusion Model, a framework that generates manipulator geometry designs that can manipulate objects towards desired motions and poses with no task-specific training and no perception or closed-loop control -only a parallel jaw closing motion. From tasks as simple as shifting/rotating objects to complex tasks requiring sequential interactions such as pose convergence (Fig. 2 ), DGDM generates designs in seconds with geometry changes that are highly adapted to the task and object. Our framework answers two key research questions:• How to represent the task space? The task representation has to be expressive enough to capture the wide range of manipulation tasks while being compact enough to be readily learned from data.Our key insight is that many manipulation tasks can be decomposed into a collection of individual motion targets that specify how each object should move under each initial pose, which we call interaction profile. While the final composed objective is specific to the task, each of the individual motions can be modeled by a generic dynamics network that is reusable across tasks. • How to facilitate efficient search? As the design space grows, the design objective landscape often becomes multi-modal w.r.t. the design parameters, and generating promising yet diverse design candidates becomes challenging. To address this issue, we first represent the design space using an unconditional geometric diffusion model. Then, the interaction profile for an object and the fingers is inferred with the dynamics network. The design objective constructed by comparing the current with the target interaction profiles gives us a gradient on how to update the finger. This dynamics guidance is incorporated into diffusion denoising steps similarly to classifier guidance [13] .We demonstrate results on both 2D and 3D objects with a variety of manipulation objectives ranging from simple to complex and single-to multi-object objectives, all under a sensor-less setting, where the initial pose of the object is unknown. Experiments in simulation and the real world demonstrate that designs generated by DGDM achieve high task performance, with 31.5% and 45.3% relative success rate improvements compared to optimization and unguided diffusion baselines. 
related work: 
 Manual End-effector Design. The diverse array of manipulator designs we see today, including serial versus parallel, from dexterous to underactuated, typically start from many trail-and-error iterations by experts. Heavy manual efforts are needed to discover optimal and occasionally counterintuitive designs, which hinders the development of designs for new applications. For instance, for complex manipulation tasks such as convergence (Fig. 2 ), previous works only deal with 2D planar polygons, utilizing manual/analytical designs of grasping policy [14, 15] or gripper geometry [16] .Analytical Optimization for Automatic End-effector Design. To alleviate the manual efforts, previous works have explored optimization approaches to manipulator design. Non-linear optimization approach [17] typically requires careful task-specific formulation of objectives and constraints. First-order optimization of morphology [18] or both morphology and control [17, 7, 19] is more common, but requires careful initialization (task-specific parameterization [18] , cage-based deformation [7, 8] , or heuristics [19] ). Further, tasks involving complex contact modes are known to yield biased and high variance gradients in differentiable simulators [20, 8] . Importantly, all manual efforts involved in setting up an optimization problem are typically not transferrable to new tasks.Data-driven Robot Hardware Design. Data-driven approaches improve over optimization-based approaches by transferring knowledge from training to reduce the cost at inference. A common approach is to train a value network that takes the design parameterization as input and outputs the design's task performance. This value network can be used to guide a search/optimization procedure, which has been explored in gripper design [5] and locomotion [21, 22] , to guide optimization [5, 23, 6] or graph search [21, 22, 10, 24] . Another approach is to learn a generative model of the design space, which compresses the design space into a low-dimensional continuous latent space. This makes offline optimization via gradient-descent [5, 23] or online optimization via trial-anderror rollouts of random latent-space samples [5, 10, 6] significantly more efficient. Finally, when co-optimizing morphology and control, leveraging control experience from prior embodiment evaluations can significantly improve the efficiency and accuracy of new embodiment evaluations [9, 12] . However, all these approaches require a large amount of task-specific data, while we eliminate this requirement by leveraging dynamics as the shared structure between manipulation tasks.3 ApproachWe report results on all the manipulation tasks and metrics described above in Tab. 4, Tab. 5, Tab. 6, and Tab. 7. The evaluation procedure is the same as described in the main paper, where we run each approach 16 times per object-task pair and select the best performance, then average among test objects. DGDM outperforms baselines consistently on both discrete metrics (e.g. average success rate) and continuous metrics (e.g. delta object transformation and final transformation). - x ↑ ∆x ↓ x ↓ S + x ↑ ∆x ↑ x ↑ S - y ↑ ∆y ↓ y ↓ S + y ↑ ∆y ↑ y ↑ S - θ ↑ ∆θ ↓ θ ↓ S + θ ↑ ∆θ ↑ θ ↑ 2D Unguided 56.EQUATIONEQUATIONFor each task and object shape, we generate manipulators with 16 different initializations. Aside from reporting the best of 16 trails in the main paper, we additionally report the average performance over all trails in Tab. 8. DGDM still significantly outperforms the baselines on metrics averaged over both different initialization and objects. . 
interaction profiles as task specification: 
 Requirements for a manipulation system are incredibly diverse, ranging in what initial poses are allowed, what objects are considered, and what the desired effects are. Parameterizing the space of manipulation tasks call for a representation expressive enough to capture all the diverse tasks. Moreover, this representation should be compact -containing only the necessary information to capture how the object interacts with the finger, such that it is efficient to evaluate/learn. For instance, modeling the detailed physics states in differentiable simulators [7] is expressive, but forward integrating the dynamics over the time horizon for every finger evaluation is expensive.Interaction Profiles. Many manipulation tasks can be decomposed into a collection of individual motion targets that specify how each object should move under each initial pose after interacting with the manipulator. By combining motions from all objects and initial poses, we get a complete profile of how the manipulator will interact with the target objects -the "interaction profile".Denote by o and m the geometry parameters of object shape and manipulator shape. When the object is at the initial planar pose p = (θ , x, y), closing the manipulator once will change the object's pose by ∆p = (∆θ , ∆x, ∆y), dictated by the manipulator-object interaction dynamics D. We refer to scalar-valued functions f defined on top of ∆p as motion objectives and aggregate these motion objectives among all initial poses p and objects o to get the design objective F.To design a manipulator that shifts a set of objects upwards, each motion objective is defined asEQUATIONwhere ∆y is the y-translation component of ∆p. The design objective is aggregated from (1) asEQUATIONInteraction profiles can scale to varying ranges of initial poses and objects. Thus, using a larger set of initial poses and objects will yield an objective that is more robust to different initial poses and tailored to more objects. Since each motion objective is conditioned on p, this approach also allows for objectives dependent on initial poses, as illustrated in the following example.The goal of pose convergence is to rotate an object to a target orientation θ target relative to the manipulator (Fig. 2 ). The manipulator should funnel a wide range of initial configurations into a single target orientation with no perception, no closed-loop control, only a parallel gripper closing motion on repeat. How the object should rotate depends on the initial orientation θ relative to the target orientation θ target :EQUATIONThe objective for this task can then be aggregated from (3) analogously to (2) .If ∇ m F can be efficiently computed, we can use this gradient to optimize a pair of fingers m that achieves the task. To achieve this, we propose to represent interaction dynamics D as a neural network and train it using data generated from interactions between random finger-object pairs.The dynamics network D : (o, m, p) → ∆p aims to learn a general model of how a random distribution of fingers interacts with a distribution of objects. Importantly, it provides gradients of the design objective with respect to the finger representation (Fig. 3 ).Shape Representation. We choose cubic Bézier curves and surfaces as the manipulator shape representation [25] . Control points are grid sampled along the length (and height in 3D) of the finger while the remaining y-coordinate determines its protrusion outwards/inwards, which we define as m -the geometry parameter of manipulator. We represent object shape o as a point cloud by sampling 100 points from each 2D object contour and 512 3D points from 3D object surfaces.Motion Representation. We represent object motion under interaction as a three-dimensional vector consisting of delta rotation along the z-axis, delta translation along the x-axis, and delta translation along the y-axis, denoted as ∆p = (∆θ , ∆x, ∆y).Network Architecture. First, we transform object initial poses p with a high-frequency positional encoding -a trick used to combat over smoothing of neural networks [26] . Then, o and m are passed through separate 2-layer MLPs with 256 hidden dimensions, before being concatenated with the pose embedding. Finally, the resulting embedding is passed through an 8-layer MLP with 256 hidden dimensions to get the predicted object motion ∆p. In 3D, we use a PointNet++ [27] to encode object geometry, whereas all other parts of the network are shared between 2D and 3D tasks.Training Data Generation. Our training data generation happens once for all tasks. We sample object and manipulator pairs, load them into MuJoCo [28] simulation environment, and measure ∆p after a single parallel closing interaction. We generate 321 planar object shapes from the Icons-50 dataset [29] in 2D, and select 164 objects from Google's Scanned Objects Dataset [30, 31] in 3D. We randomly sample 1024 manipulator geometry parameters m from a uniform distribution.For each object-fingers pair, we grid sample 360 initial orientations and 25 initial positions, getting 321×1024×360×25 training data points for 2D dynamics network and 164×1024×360×25 for 3D.Design Objective Gradient Evaluation. To design manipulators that are generalizable to more initial poses p and objects o, the design objective F (2) can be evaluated for a wider range of poses and objects. We grid-sample initial poses of a set of objects and evaluate motion objectives in parallel. The design objective gradient ∇ m F is attained by aggregating the gradients along the pose/object batch dimension. For each new design, evaluating ∇ m F takes 0.16 seconds on average, making it efficient to run in the inner loop of iterative design procedures. Specifically, we gridsample 360 orientations and 5×5 positions, getting a 360×5×5 dimensional motion profile. Given design objective gradients from D, the obvious approach is to perform gradient descent on the finger geometry [5, 19] . However, the distribution of good designs are often multi-modal, which means gradient descent approaches quickly get stuck in local minima. To efficiently navigate through the large and multimodal design space, we extend classifier guidance [13] , an iterative diffusion model sampling approach that enables a balance between diversity and task-specific guidance (Fig. 3 ).Diffusion Models [32, 33] are a class of probabilistic generative models that generate samples from an underlying distribution through iterative denoising. A diffusion model ε θ (m k ) predicts the noise added to a sample m 0 . We start with a Gaussian noise m K and gradually predict less-noisy samples m K-1 , m K-2 , ... until m 0 through a reverse noising process of modeling the distribution p θ (m k-1 |m k ). Specifically, we sample geometry parameters of manipulator m from a uniform distribution and train a geometric diffusion model (with 1D UNet architecture [34] ) on this distribution once and for all tasks. We employ Denoising Diffusion Implicit Models (DDIMs) [35] for diffusion sampling process with 15 training denoising iterations and 5 inference iterations, and the Square Cosine noise scheduler [36] .Classifier Guidance [13] guides the reverse noising process with priors of an unconditional diffusion model. It requires a classifier p φ (l|m k ), where m k is the sample, l is the class label, and φ is the classification network. Leveraging the connection between diffusion models and score matching [37, 38] , a new noise prediction can be defined as:EQUATIONwhere ᾱk := ∏ k t=1 1β t , β t is the variance of Gaussian noise added to samples at step t. Then DDIM can be performed with the modified noise prediction for conditioned sampling.Algorithm 1 Dynamics-guided DDIM sampling, given a diffusion model ε θ (m k ), design objective F(m k ), and gradient scale s.Input: design objective F(•), gradient scale s m K ← sample from N (0, I) for all k from K to 1 do ε ← ε θ (m k ) -s √ 1 -ᾱk ∇F(m k ) m k-1 ← √ ᾱk-1 m k - √ 1-ᾱk ε √ ᾱk + 1 -ᾱk-1 ε end for return m 0Dynamics Guidance. To guide the design generation towards specified manipulation tasks, we extend classifier guidance to use design objectives constructed from interaction profiles, which we term dynamics guidance. We replace classifier gradients with ∇ m k F(m k ) to guide the DDIM sampling process (Algo. 1). We not only enable guiding unconditional diffusion models with task-specific gradients, but also allow tuning the guidance scale s to trade off diversity and performance. Dhariwal and Nichol [13] showed that when s is larger the distribution becomes sharper and generated samples have higher fidelity, while smaller s leads to more diverse samples, which we also observed in our generated results (Fig. 4 ).We plot the progress of convergence over 40 open-close actions with manipulators generated by baselines and DGDM in Fig. 9 . The Unguided baseline converges the slowest, taking more than 30 steps. The Opt. baseline exhibits an unstable dip at the start and only achieves consistent alignment with the target orientation after 30 steps. Manipulators generated by DGDM not only have larger convergence ranges but also converge faster (with 10 steps) than the baselines. 3 More details on interaction profile Interaction profile. The interaction profile between a manipulator and an object is defined as the distribution of the delta pose change of the object caused by the manipulator-object interaction over all possible initial poses of the object. Here, the manipulators-object interaction is a simple parallel closing action, and the object pose is represented as 2D translation and rotation. For example, in Fig. 10 , when the object's initial pose is (θ , x, y), its delta pose after the interaction is (∆θ , ∆x, ∆y), which provide one data point on the interaction profile. Then, by uniformly sampling all initial poses, we obtain the complete interact profile.In our implementation, we use simulation to obtain the ground truth interaction profile and train a dynamic network to infer the predicted interaction profile given any manipulator-object pair without simulation. Target interaction profile. A target interaction profile is defined for a specific task objective. For example, the target interaction profile for the convergence task is shown in Fig. 11 . This interaction profile indicates the object should rotate in a positive direction when its initial orientation is smaller than the convergence rotation; otherwise, it rotates in a negative direction. Tab. 9 shows examples of other target interaction profiles for different design objectives. Target v.s. predicted interaction profile. The difference between the target interaction profile and the predicted interaction profile (inferred by the dynamics network) provides the gradients for the manipulator design (Fig. 12 ). Since the dynamics network is fully differentiable, we can get its gradient with respect to the input manipulator geometry. List of target interaction profile plots. We provide a comprehensive list of task objectives and their corresponding interaction plots in Tab. 9. For visualization purposes, we simplify the horizontal axes of interaction profile plots to include only initial orientations θ . 
evaluation: 
 Manipulation Tasks & Metrics. We evaluated each approach on held-out objects (8 in 2D, 6 in 3D) and manipulation tasks. Each pair of fingers is mounted to a WSG50 gripper performing a fixed open-close action. We categorize our suite into two difficulty levels: 1) Simple objectives involve single-axis object movements in SE2 space, including shifting up/down/left/right and rotating clockwise/counterclockwise. For each object-manipulator pair, we grid-sampled 360 planar initial object orientations and performed fixed open-close actions. The task is considered as successful if the movement of the object along the specified axis is larger than a predefined threshold (0.03 rad for rotation, 3 mm for x-axis translation, 2 mm for y-axis translation). For example, a manipulator designed for the rotating objective succeeds if it rotates the object larger than 0.03 rad after the first closing action. Then, we report the average success rate over all sampled initial object orientations.2) Complex objectives combine multiple simple objectives to parameterize a broad range of manipulation tasks. For the convergence objective, we report the maximum convergence range ( • ), indicating the broadest range of initial object orientations that can be driven towards a consistent final orientation within a small tolerance (5 • ). Observing continued object movement, we report the metric after the 40th open-close manipulator action. Additionally, we explored rotate clockwise and shift up/left, and rotate either way objectives to showcase DGDM's flexibility in composing conflicting objectives. These tasks were evaluated on average success rates.Comparisons. Removing the dynamics guidance yields the Unguided baseline, which generates task-agnostic manipulators using our geometric diffusion model. Removing our diffusion model yields the GD baseline, which optimizes the manipulator control points using gradients of the design objective ∇F via gradient descent optimization, common for many prior works [5, 7, 8, 19] . We also evaluated a gradient-free optimization baseline -CMA-ES [39] (covariance matrix adaptation evolution strategy) that optimizes finger control points from objectives constructed from the dynamics network. Notably, we ran it with more than ×10 the compute (and ×10 time) of DGDM. To mitigate performance variance due to initialization, we ran each approach 16 times with different initializations per object-task pair and selected the best performance, then averaged among objects. enerating task-specific manipulators designs. DGDM generates tailored fingers for a wide variety of scenarios, surpassing the unguided baseline across all tasks. The advantages of generating custom fingers become more pronounced as the design requirements escalate. For instance, DGDM exhibited a +16.6% improvement over the unguided baseline in 2D simple objectives, a figure that expanded to 20.0% in 2D complex objectives (see Tab. 1). A similar trend was observed when transitioning from 2D to 3D objects (+18.0% over Unguided in 2D, +23.4% over Unguided in 3D, Tab. 1) and from single-object to multi-object designs (+18.0% over Unguided in single-obj 2D, +18.6% over Unguided in multi-obj 2D, Tab. 2). When task complexity, design space, and the target object set grow, a human expert designer would face significantly increased time and effort. DGDM handles progressively complex design requirements by aggregating gradients from individual motion objectives. A new task can be specified as long as users can articulate how each object should move from each initial pose, and can be seamlessly incorporated into the diffusion denoising process.Robust & efficient search with guided diffusion. The GD baseline and DGDM share the same design objective gradients from our dynamics network, differing only in how this gradient information is incorporated. The baseline uses gradient descent, requiring upwards of 18 and 24 minutes to converge in 2D and 3D (for 16 samples), respectively, and is prone to local minima. In contrast, DGDM utilizes classifier-guidance with a diffusion denoising process, which strikes a balance between exploring different modes (by introducing Gaussian noise) and exploiting the current mode (using the gradient of design objective) through a guidance scaling factor (Fig. 4 ). This results in +12.8% and +10.4% higher success rates than the baseline in 2D and 3D simple objectives, respectively. Additionally, our diffusion models prove stable even with diffusion processes as short as 5 timesteps, translating to an average design time of 13 and 54 seconds in 2D and 3D, respectively. Emergent design for convergence. What strategies do our designs employ to achieve convergence from a broader range of initial orientations compared to the unguided baseline (Tab. 1)? We identify two emergent design patterns: 1) Push-and-Catch: One finger features a bulge that pushes the object into the hollow cavity of the other finger (Fig. 5 a, b, h ). This cavity roughly complements the object's shape at the convergence point. 2) Parallel-Align: When objects exhibit symmetric flat edges, our designs utilize two parallel surfaces to align these edges (Fig. 5 c, g ). The generated designs simultaneously exploit object geometry and physics to achieve the most effective convergence.Rotate Either Direction (3D) Specialized or generalized designs for multi-object scenarios. DGDM is able to generate more generic designs for multiple objects (Fig 6 ). In contrast, the unguided baseline lacks task-specific guidance, hindering its ability to guide its generations toward a common design objective for all objects. On the other extreme, the GD baseline often gets stuck in a local minimum. These limitations are reflected in Tab. 2, with our approach achieving +18.6% and +23.4% higher success rates than the unguided baseline, and +14.7% and +17.6% higher success rates than the GD baseline. Naturally, we acknowledge that multi-object finger designs often sacrifice some performance compared to the single-object scenario (-1.5% in 2D, -5.2% in 3D). This balance between generality and Shift Down Target Obj.Shift Right Rotate Counter Convergence task-specific performance is a fundamental trade-off in mechanism design in automation, with the optimal compromise dependent on the specific application.Real-world evaluation with Sim2Real transfer. We show real-world results of all tasks for both 2D and 3D cases by mounting the 3D printed designs on a WSG50 gripper (Fig. 7 ). The material we use for 3D printing objects and fingers is PLA and the molding solution is FDM. In Tab. 3 we show the real-world and simulation quantitative results side by side. For the shifting down/right and rotating counterclockwise tasks, we tested with 10 random initial poses (0 • -360 • orientations and ±5 cm from the center) of the object in the real world and reported the average success rate (%). For the convergence task, the maximum range of initial orientations (°) that can be driven to the target convergence pose is reported. We observe that the performance in the real world is very close and oftentimes better than the simulation result, suggesting a small sim2real gap. his is due to our sensor-less formulation, where we do not need perception or closed-loop control. With the same fixed parallel motion in sim and real, the transferability is only determined by the geometry and contact physics. Our dynamics guidance is conditioned on many initial object poses, enabling the generated designs to be robust to different initial object poses, relying on more prominent physical phenomena that are consistent between sim and real. Moreover, the sensor-less manipulation tasks only require the directions of individual object motions to be accurate but not the magnitudes. We see this effect when we use PLA with a lower friction coefficient in real than in sim, allowing the objects to slide more smoothly but in the same direction, leading to oftentimes better performance in real. 
conclusions: 
 We present Dynamics-Guided Diffusion Model, a versatile framework for the rapid generation of diverse and tailored manipulator geometry designs for unseen tasks. With the ability to generate a new design within 0.8s, this task-agnostic framework lays the groundwork to enable more rapid experimentation and future research. We hope that our framework contributes to the wider adoption of data-driven approaches in robotic mechanism design. 
 
 Appendix: We provide more details about the evaluation manipulation tasks, and introduce more metrics for each task in this section.• Shift up: Shift the object upward (along the negative x-axis in the manipulator frame) under all initial poses. For evaluation, we grid sample 360 initial object orientations with positions in the center of the manipulator, and close the manipulator around the object. The average success rate after one closure S - x ↑ (%) is reported, where success is counted if the object is shifted more than 3 mm along the negative x-axis. We additionally report continuous metrics including average delta object translation along the x-axis after one closure ∆x ↓ (cm), and average final x coordinate of the object after the 40th gripper closure x ↓ (cm). The metrics are averaged among the 360 interaction trails with different initial object orientations. The motion objective is f (o, m, p) = -∆x(o, m, p), then the design objective is aggregated from f (o, m, p) asF(m) = ∑ o ∑ p f (o, m, p).• Shift down: Shift the object downward (along the positive x-axis) under all initial poses. Metrics are average success rate of the object being shifted more than 3 mm along the positive x-axis after one closure S + x ↑ (%), average delta translation along the x-axis ∆x ↑ (cm), and average final x coordinate x ↑ (cm). The motion objective is f (o, m, p) = ∆x(o, m, p).• Shift left: Shift the object leftward (along the negative y-axis) under all initial poses. Metrics are average success rate of the object being shifted more than 2 mm along the negative y-axis after one closure S - y ↑ (%), average delta translation along the y-axis ∆y ↓ (cm), and average final y coordinate y ↓ (cm). The motion objective is f (o, m, p) = -∆y(o, m, p).• Shift right: Shift the object rightward (along the positive y-axis) under all initial poses. Metrics are average success rate of the object being shifted more than 2 mm along the positive y-axis after one closure S + y ↑ (%), average delta translation along the y-axis ∆y ↑ (cm), and average final y coordinate y ↑ (cm). The motion objective is f (o, m, p) = ∆y(o, m, p).• Rotate clockwise: Rotate the object clockwise (negative delta rotation around the z-axis) under all initial poses. Metrics are average success rate of the object being rotated more than 0.03 rad around the negative z-axis after one closure S - θ ↑ (%), average delta rotation around the z-axis ∆θ ↓ (°), and average final orientation θ ↓ (°). The motion objective is f (o, m, p) = -∆θ (o, m, p).• Rotate counterclockwise: Rotate the object counterclockwise (positive delta rotation around the z-axis) under all initial poses. Metrics are average success rate of the object being rotated more than 0.03 rad around the positive z-axis after one closure S + θ ↑ (%), average delta rotation around the z-axis ∆θ ↑ (°), and average final orientation θ ↑ (°). The motion objective is f (o, m, p) = ∆θ (o, m, p). Shift Up (3D)Figure 8 : Results on Simple Objectives. We generate manipulators for simple objectives that involve motion along one dimension in SE2 space. Red and green object masks denote object configurations before and after interaction respectively, overlaid with the image after closing in the manipulator. Compared with baseline methods, finger shapes produced by our method achieve the task much more effectively.• Convergence: Reorient the object towards a fixed final pose under a range of initial poses. The design objective is encouraging the object to rotate in the positive direction when its initial orientation is smaller than the target orientation, or else rotate in the negative direction. The motion objective is:EQUATIONTo determine the best target orientation θ target , we first forward the dynamics network with the manipulator shape initialization, object shape, and sampled initial object poses to get a pseudo interaction profile. We detect the initial object orientation ranges that lead to consecutive positive delta rotations followed by consecutive negative delta rotations as pseudo convergence ranges.Then we select the largest pseudo convergence range's corresponding convergence orientation as θ target . The metric is the maximum convergence range R max c ↑ (°), the largest range of initial orientations leading to θ target within a small tolerance. We report the maximum convergence range within the tolerance of 3°, 5°, and 10°, respectively. 


Decision: **Post-Rebuttal Guidance**

Recommendation: ACCEPT.
Justification: Reviewers unanimously decided to accept. Rebuttal further mitigated any concerns or weaknesses the reviews had


**Pre-Rebuttal Summary**

Summary:
The authors present an approach to generate 2-fingered gripper designs for sensorless manipulation of a unknown part. The method involves a dynamics guided diffusion model, and shows promising results on both sim and real.


Strengths:
- Well written
- Well motivated
- proposed approach is novel, fast and generalizes to new shapes

Weaknesses:
- limited generalization to wider distribution of pose orientations
- lack of comparison to related work, only ablations shown
- interaction profile not well described, questionable justification/explanation for task guidance

Dear authors, looks like the reviewers have diverging opinions on the submission as well as several clarification questions. Hope you could address the reviewer concerns.

Reviews: review: title: title: Interesting idea on using diffusion models and classifier guidance to generate task-specific finger designs for object manipulation.
paper_summary: paper_summary: This paper introduces a novel approach to generate finger designs by leveraging diffusion models and a tailored guidance function. The authors propose to learn a dynamics network/model that uses as input an object representation and its pose (the state), along with the manipulator representation/parameters (the action), an outputs a delta in the objects pose (the next state).  Given a new task, e.g. ”shift the object down”, this model is first used to optimise a design objective/cost function F, by running gradient descent with respect to the manipulator parameters. Noting the problems of using gradient descent for optimizing F, namely convergence to local minima, which leads to very similar designs and task failure, the authors propose to learn an (unconditional?) diffusion model on the manipulator parameters, given data obtained in simulation.  Using this diffusion model and the previously learned dynamics network, the authors employ classifier-guidance to leverage multimodality, design precision and faster inference in comparison to the gradient descent baseline.
main_review: main_review: The paper is clearly written, precise and with enough detail in the main text to be well understood.

I particularly like the idea of using diffusion models as a prior for manipulator design and, as far as I know, this work is novel (although I’m not an expert in this field).

I appreciate that the authors build the paper by first showing the problems with the baseline (gradient descent) and only then introduce the necessity of using diffusion models and classifier guidance. All the properties of diffusion models - multimodality, handling data representations, gradient guidance - are well motivated, and experiments have been done to highlight these properties.

The experiment section shows convincing results that help understand the proposed method. The real-world experiments show that the method transfers well from simulation to reality.

One downside of this work is doing open-loop control. I’m not sure if introducing observations to the model would help in solving some of the failure cases (maybe the authors can comment on that), but I’d expect that for these types of interaction tasks, a closed-loop controller could work better.
questions: questions: Could you clarify how gradient descent is implemented? Are you doing some form of trajectory optimization using the dynamics network (which is a kind of dynamics model)? Are the gradients of the objective function “well behaved”, or would it be worth to try a gradient free optimiser (e.g. CMA-ES)?

It’s unclear to me if you have access to the initial object orientation. You state: “How the object should rotate depends on the initial orientation θ relative to the target orientation θ_target.” Are you running some pose estimation method? In that case, why not use it as a feedback signal and run an open loop? Wouldn’t it be more sensible for this kind of interaction task?

Could you clarify if, for each task description (e.g., shift down, shift right, …), you train a diffusion model (and dynamics network), or is there a single model conditioned on the task description?
limitations: limitations: 3
review_summary: review_summary: The paper idea is very interesting and the method and contributions are clearly explained.  The results show convincing results, in particular the sim2real transfer.
title: title: Review comments
paper_summary: paper_summary: The paper presents a data-driven framework for two-finger gripper design for sensor-free planar object reorientation via parallel closing action. The paper i) learns general (parallel-closing) interaction dynamics between arbitrary finger and object shapes, and ii) uses the learned dynamics model to guide a diffusion process that generates suitable gripper shape.
main_review: main_review: **Strengths**:
- The paper is very clearly written and was a joy to read.
- The work is well-motivated, and the contributions are placed within the context of existing work
- Mos design choices are justified and discussed in sufficient detail
- The proposed method is fast and effective (allowing for rapid prototyping)
- The learned model seems to be agnostic to the object shape
- The experiments include hardware demonstrations

**Weaknesses and questions**:
- The target orientation for the convergence task is a bit confusing to me. Is it specified by the user (I would guess so)? But if yes, why is it different for the proposed method and the unguided baseline in Fig. 5? Was the target orientation randomized in the experiments to test if the proposed approach can generalize in that dimension? Why is the convergence error to the target orientation not reported? Was it always zero for all methods?
- The reason for reporting the best of 16 trails (with varying initial conditions) instead of the statistics over all trails needs to be better discussed.
- It would be interesting to see the task progress over the 40 open-close actions. Do the baselines take more or fewer actions to converge compared to the proposed approach?
- Why was the success criterion defined as a threshold for the simple tasks, instead of a target?
questions: questions: Please address the questions and concerns raised above.
limitations: limitations: 3
review_summary: review_summary: The paper proposes an efficient and effective method for two-finger gripper design that is agnostic to the object shape. Experiments are convincing to me and the paper is easy to follow. My only reservation centers around clarity on a couple of things that I have explained in my comments.
title: title: Good advancement in manipulator design with generative models, lacks some clarity and justifications
paper_summary: paper_summary: Diffusion model based designing of gripper fingers for a specified set of tasks and objects
main_review: main_review: The paper proposes Dynamics guided diffusion model to find optimal gripper finger geometry to perform certain tasks (like rotating and moving) on certain objects without any task-specific training or perception feedback. DGDM is based on a well established approach of that uses a classifier (here, a function of a learned dynamics model) in reverse diffusion to guide the sampling to satisfy an objective. Since the classifier can be flexibly used to specify task information at inference, training can be done in a more general way. Also, as the initial pose of the object is unknown (i.e. no sensors are used), the method does not require any sensory feedback and can generate a robust finger geometry design well suited for the task at hand. The method is evaluated for a variety of objects and tasks to show that DGDM works better than optimization based approaches and unguided diffusion version of itself. Also, DGDM is much faster than optimization based counterparts while achieving considerably higher success rates.

Strengths:

1. The work introduces a new generative model direction to manipulator design and will be a good motivation to explore this interesting direction more in the future.

Weakness:

1. While I understand the importance of the proposed method and application, I consider the work to be very limiting. While the authors agree that the motions are limited to 2D translation and rotation, but the utility of this approach in the shown assembly line situation is also based on lot of assumptions. For example, there is an upper limit on the permissible difference between current and target orientation for the manipulator to solve the task. Also, not all objects must be manipulated similarly, also agreeing on the fact that "different objectives may have conflicting gradient directions".

2. The term interaction profile is not very clearly defined mathematically. While the authors show design objectives within the interaction profiles section and including it in method figure 3, I find their connection and overall formulation very confusing. 

3. The authors have not compared with any related work (if there exists any) but have only shown ablation comparisons. With task information guidance, I will automatically expect DGDM to work better than its unguided alternative without any task information. It is surprising to see that unguided DM is also solving tasks with considerably high success rates. For comparison with gradient descent, the initialization of the manipulator geometry might make some difference. 

I would like to thank the authors for presenting most of the work very clearly.
questions: questions: 1. Clarifying what exactly is an interaction profile. For example, a list of task objectives and their corresponding interaction plots will be very helpful.
2. Clarify figure 3 more. Particularly, is "current interaction profile" showing $\Delta\theta$ vs $\theta$ plot? How do you know $\theta$ as initial pose is unknown? What are "sampled poses"? How do you get "sampled poses" while reverse sampling?
3. Is it possible to compare with any possible related work? Also, if it is possible, initializing the start of optimization baseline with the output of unguided diffusion model might help in understanding the actual contribution of classifier guidance as compared to optimization.
limitations: limitations: 3
review_summary: review_summary: With some more clarifications and justifications, the proposed work might be a good contribution to the community.