Thin-Shell Object Manipulations With Differentiable Physics Simulations
Authors: Yian Wang
Juntian Zheng
Zhehuan Chen
Zhou Xian
Gu Zhang
Chao Liu
Chuang Gan
Venue: ICLR 2024 spotlight (2024)
Keywords: differentiable physics simulation
thin-shell object manipulation
Primary area: applications to robotics, autonomy, planning

Abstract: In this work, we aim to teach robots to manipulate various thin-shell materials. 
Prior works studying thin-shell object manipulation mostly rely on heuristic policies or learn policies from real-world video demonstrations, and only focus on limited material types and tasks (e.g., cloth unfolding). However, these approaches face significant challenges when extended to a wider variety of thin-shell materials and a diverse range of tasks.
On the other hand, while virtual simulations are shown to be effective in diverse robot skill learning and evaluation, prior thin-shell simulation environments only support a subset of thin-shell materials, which also limits their supported range of tasks. 
To fill in this gap, we introduce ThinShellLab - a fully differentiable simulation platform tailored for robotic interactions with diverse thin-shell materials possessing varying material properties, enabling flexible thin-shell manipulation skill learning and evaluation. Building on top of our developed simulation engine, we design a diverse set of manipulation tasks centered around different thin-shell objects. Our experiments suggest that manipulating thin-shell objects presents several unique challenges: 1) thin-shell manipulation relies heavily on frictional forces due to the objects' co-dimensional nature, 2) the materials being manipulated are highly sensitive to minimal variations in interaction actions, and 3) the constant and frequent alteration in contact pairs makes trajectory optimization methods susceptible to local optima, and neither standard reinforcement learning algorithms nor trajectory optimization methods (either gradient-based or gradient-free) are able to solve the tasks alone. To overcome these challenges, we present an optimization scheme that couples sampling-based trajectory optimization and gradient-based optimization, boosting both learning efficiency and converged performance across various proposed tasks. In addition, the differentiable nature of our platform facilitates a smooth sim-to-real transition. By tuning simulation parameters with a minimal set of real-world data, we demonstrate successful deployment of the learned skills to real-robot settings.  ThinShellLab will be publicly available. Video demonstration and more information can be found on the project website https://vis-www.cs.umass.edu/ThinShellLab/.

Published as a conference paper at ICLR 2024
THIN-SHELI OBJECT MANIPULATIONS WITH DIFFER- ENTIABLE PHYSICS SIMULATIONS
Yian Wang"
Juntian Zhenga* Zhchuan Chen
Zhou Xian
Gu Zhang
Chao Liu
Chuang Gan
Umass Amherst SITU MIT MMIT-IMM
Tsinghua University
*peking University
*CMU
ABSTRACT
In this work, we aim teach robots to manipulate various thin-shell materi- als. Prior works studying thin-shell object manipulation mostly rely on heuris- tic policies or learn policies from real-world video demonstrations, and only fo- cus on limited material types and tasks (e.g., cloth unfolding). However, these approaches face significant challenges when extended shell materials and a diverse range of tasks. On the other hand, while virtual simulations are shown to be effective in diverse robot skill learning and evalua- tion, prior thin-shell simulation environments only support subset of thin-shell materials, which also limits their supported range of tasks. To fill in this gap. we introduce 7 T'hiShelllaab robotic interactions with diverse thin-shell materials possessing varying material properties, enabling flexible thin-shell manipulation skill learning and evaluation. Building on top of our developed simulation engine, we design diverse set of manipulation tasks centered around different thin-shell objects. Our experiments suggest that manipulating thin-shell objects presents several unique challenges: thin-shell manipulation relies heavily on frictional forces due to the objects' co-dimensional nature, 2) the materials being manipulated are highly sensitive to minimal variations in interaction actions, and 3) the constant and frequent alter- ation in contact pairs makes trajectory optimization methods susceptible local optima, and neither standard reinforcement learning algorithms nor trajectory op- timization methods (either gradient-based based or gradient-free) are able to solve the tasks alone. To overcome these challenges, we present an optimization scheme that couples sampling-basee based trajectory optimization and gradient-basee optimiza- tion, boosting both learning efficiency and converged performance across various proposed tasks. In addition, the differentiable nature of our platform facilitates smooth sim-to-real transition. By tuning simulation parameters with a minimal set of real-worid data, we demonstrate successful deployment of the learned skills to real-robot settings. Video demonstration and more information can be found on the project website'.
wider variety of thin-
fully differentiable simulation platform tailored for
INTRODUCTION
Manipulating thin-shell materials is complicated due to a diverse range of sophisticated activities involved in the manipulation process. For example, to lift an object using a sheet of paper, we would instinctively create a slight bend or curve in the paper before initiating the lift (Figure (a)). Human beings intuitively learn such thin-shell manipulation skills, such as folding paper to make a crease, drawing out piece of sheet under a bottle. and even complicated card tricks. However, existing robotic systems still struggle in handling these thin-shell objects as flexibly as humans. Compared with manipulating rigid bodies or volumetric materials, manipulating thin-shell materials poses several unique challenges. First, the physical forms of such materials are difficult to handle. For example, picking up a flat sheet intrinsically difficult due to its close-to-zero thickness, preventing any effective grasping from the top. A more effective approach is to use friction force to
'Equal contribution
1 'Corresponding author 'htpps: pettss:/vits wtw.iiswwww..c. cs. umass. edw l.ab/

---

Published as a conference paper at ICLR 2024
(u) Lifting block
(b) Picking up paper
(c) sim-to-real
(d) real experiment
It's common for us to interact with thin-shell materials, such as bending a paper to lift a block (a) or picking up piece of paper (b). Aiming boost versatile robotic skill acquiring for diverse thin-shell materials, we propose ahinShelllab, a fully differentiable simulation platform to- gether with set benchmark tasks ((a) and (b) bottom). Moreover, to bring simulation in line with the realdworld, we adjust physical properties by utilizing the gradient and real-world observations (c). After that, we successfully deploy our policy learned from simulation to the real world (d).
Figure/ I:
bend curve of the sheet before grasping (Figure (b)). Secondly, thin-shell materials are highly sensitive to even minimal variations in actions or contact points. For instance, in the lifting task (Figure (a)), even millimetric movements can lead to drastically different curvatures in the paper. Furthermore, owing to the high complexity of rich contacts, gradient- based optimization methods, which have been shown to be effective and efficient in solving a wide range of deformable object manipulation tasks (Huang et al., 2021; I; Xian et al., 2023), suffers severely from the non-smooth optimization landscape and local optima in thin-shell manipulations.
Many prior works have investigated thin-shell manipulation tasks in real-world. Ha & Song (2021) and Xu et al. (2022b) design action primitives specifically for cloth unfolding. Zhao et al. (2023) trains robots in real-world to learn paper flipping skills. Chi et al. (2022) studies goal conditioned dynamics tasks including cloth placement and rope whipping. Namiki & Yokosawa (2015) studies paper folding with motion primitives. However, all those works designed for solving specific tasks and are hard to scale up. Furthermore, another group of previous works build benchmarks in simulation environment to boost robotic skill learning for thin-shell materials (Lin et al., 2020;
Chen et al., 2023), but they abstract away the physical contact behaviors between manipulators and objects during interacting, and ignore the bending stiffness of thin-shell materials, making them over-simplified for dexterous manipulation tasks. Some previous works also build differentiable simulators for thin-shell materials (Qiao et al., 2020; 2021; Li et al., 2021; Gan et al., 2021; Li al., 2022), but they don't include end-effectors to manipulate thin-shells nor include benchmark tasks for thin-shell object manipulation. Finally, most of these simulation works consider thin-shell materials as cloths, which is only a subset of thin-shell materials, and neglect materials with strong bending stiffness and varying bending plasticity like papers, narrowing the scope of tasks they perform.
In this work, aiming to build a comprehensive benchmark for thin-shell manipulation skill acquiring, we propose ThinShellLab, a fully diffcrentiable simulation platform for developing robotic learning algorithms with thin-shell manipulation tasks. It provides scenarios with various types of thin-shell materials with or without bending plasticity and includes the coupling among thin-shell and volu- metric materials. To build such an environment, we simulate the thin-shell materials and volumetric materials following (Grinspun et al., 2003; Tamstorf & Grinspun, 2013), model the bending plastic- ity of creases by performing plastic deformation on a crease when exceeds the preset tolerance angle, implement frictional contact following (Li et al., 2020), and calculate the analytical gradient through implicit time integration scheme. To make the simulator more suitable for robot learning settings, we implement penalty-ban -baed contact to avoid time-consuming continuous collision detec- tion and to make the gradient more smooth. To enable direct positional control of end-effectors, we apply Stable Neo-Hookean (Smith et al., 2018a) that is robust to inversion. Our engine is developed using Taichi (Hu et al., 2019), which supports massive parallel computation on GPUs or CPUs.
With the differentiable simulator, we evaluate gradient-based planning methods, sampling-base trajectory optimization methods, and ofate-of-the-an reinforcement learning algorithms in our ma- nipulation task settings like block lifting (Figure (a)). As also pointed out by many previous works, in our experiments, we found that reinforcement learning algorithms struggle to solve most of our tasks due to sparse rewards and high dimensions. While pradient-based methods shown to be
2

---

Published as a conference paper at ICLR 2024
effective in previous works (Li et al., 2023; Xian et al., 2023; Huang et al., 2021; Chen et al., 2023), we find it easy to stuck in local optima when working on such thin-shell manipulation due to the constant changes of contact pair, which results in different computational graph for gradient and highly non-smooth optimization landscape. To address this challenge, we propose to combine the sampling-based method and gradient-based trajectory optimization. Specifically, we use the sample- based method to search an overall policy to offer initial contact points and avoid local minima, and then apply the gradient to do refinement upon it, which would solve most of the tasks.
Besides the utilities in policy learning, we also provide inverse design tasks to find suitable material parameters maximize certain goal functions, given fixed action trajectories. Moreover, we employ our gradients to address the sim-to-real gap by creating simulator counterpart to the real world and fine-tuning our system parameters based on real-world observations, as depicted in Figure 1 (c). Following this process, the policy trained in our simulation can be deployed in the real world with greater ease and a reduced simulation-tr -eallll gap (Figure (d)).
We summarize our main contribution follows:
We introduce ThinShellLab, a comprehensive benchmark for robotic learning in thin-shell material manipulation. This benchmark is designed to facilitate the study of complex manipulation skills across a diverse array of thin-shell materials. The benchmark provides diverse collection of manipulation tasks and standardized interface for RI and trajectory optimization algorithms.
At the core of ThinShellLab, we develop fully differentiable simulation engine supporting vol- umetric and thin-shell deformable objects and coupling between them. To the best of our knowl- edge, it is the first simulation environment capable of conducting gradient-basee optimization on thin-shell object manipulation tasks.
We benchmark the performance of wradient-based sampling-based and hybrid trajectory opti- mization methods together with the state-of-the-arr RI. algorithms on our task collection, and show a comprchensive analysis of the challenges and behaviors of each method in different scenarios.
Besides the manipulation tasks, we demonstrate how our differentiable simulator can help solving inverse design problem, and bridge the simulation scenes and the real-world robotics. We show the the simulator's capability of reconstructing real-world physics scene parameters via gradient, and deploying learned skills to real robots.
2 RELATED WORK
Differentiable Simulation. The graphics and robotics community has been increasingly focusing on differentiable simulation in recent years (Todorov et al., 2012; Coumans & Bai, 2016-2021; Xu et al., 2022a). Researchers have developed differentiable simulation engines for deformable objects (Chen et al., 2023; Du et al., 2021; Li et al., 2023; Huang et al., 2021) and fluids (Xian et al., 2023). There are general two approaches of building differentiable simulation. The first approach is to train neural networks predicting forward dynamics (Schenck & Fox, 2018; Li al., 2019; Ummenhofer et al., 2020; Han et al., 2022), which serves as a simulator and is naturally differentiable through the networks. This class of methods often requires collecting a large amount of data, and suffers from problems on generalizability and large sim-to-real gap. The second class explicitly build a simulation system and integrate gradient calculation into the system, through auto-diff for explicit time integration (Hu et al., 2020; Qiao et al., 2020; 2021), or by adjoint methods or implicit function differentiation for implicit time integration (Du et al., 2021; Li et al., 2022).
Thin-Shell Object Manipulation. Robotic manipulation on thin-shell materials is also widely in- vestigated (Chi al., 2022). One popular direction is garment folding or unfolding (Ha & Song, 2021: Xu et al., 2022b; Wu et al., 2023). Most of these previous works are trained in simulators in which the cloth is modeled as mass-spring systems that perform no bending stiffness (Lin et al., 2020) or learned directly from real-world data. Another line of works investigates robotic manipu- lation on paper, which is of greater bending stiffnees compared to cloths. Zhao et al. (2023) learns how to flip bookpages using soft gripper from real-world data. Namiki & Yokosawa (2015) stud- ies folding a sheet of paper in the real-world by extracting motion primitives. While those previous works indeed solve tasks in the realdworld, they are designed for specific tasks and are hard to generalize. Compared with these works, our work aims to build a comprehensive benchmark for thin-shell object manipulation.
3

---

Published as a conference paper at ICLR 2024
Simulator
Integration
Diff
Bending
Bending Plasticity
Coupling
Skill Learning
TDW
Explicit
Softgym
Explicit
DAXBENCH
Explicit
DiffSim
Implicit
DifiCloth
Implicit
C-IPC
Implicit
Ours
Implicit
Table 1: Comparison with previous simulators. we compare our work with other simulators sup- porting thin-shell materials. Here Diff' indicates differentiability, Bending and Bending Plasticity indicates modelling of bending energy and its plasticity behavior, and Coupling refers to the seam- less integration of thin-shell material manipulation with other dynamic objects.
For previous works that are also building simulators and benchmark tasks for thin-shell manipulation (Lin et al., 2020; Chen et al., 2023), they typically model thin-shell materials as cloths with no bending-stiffness by mass-spring model. Additionally, they abstract away the frictional contact in thin-shell material grasping, which could lead to a substantial sim-to-real gap in cases when direct grasping fails. Hence, we aim to build a simulator that can model diverse thin-shell materials with a wide range of bending-stiffnee and bending plasticity, as well as frictional contact between thin- shell materials and end-effectors support various scenarios for robotic skill learning.
3 SIMULATION ENGINES
At the core of ThinShelllaab, Lab, we present fully differentiable simulation engine, implemented using Taichi programming language (Hu et al., 2019; 2020), which enables hardware acceleration on both GPUs and CPUs. The engine models variety of thin-shell and volumetric materials, along with their interactions. In forward simulation, we use an implicit time integration scheme to simulate the dynamics of the whole system and apply Newton's 's method to solve the forward dynamics equation. Table 3 shows a comparison between ThinShellLab and other related simulation engines supporting thin-shell object modelling.
3.1 MATERIAL MODELLING
Following the traditional finite-element methods, ThinShellLab represents both volumetric and thin- shell deformable objects by spatially discretizing the objects into tetrahedral or triangular elements. At each timestep t, the internal interaction of deformable objects is modeled by potential energy Uxx') for each type of material, where x" represents the nodal positions of discretized vertices. We also calculate the derivatives aU/Ox' and 8 (ax)), which correspond to the internal force and force differential required by the implicit Euler solver. For volumetric materials, we maintain the deformation gradient FF 3 relative to the rest configuration at each tetrahedral element e and calculate the Stable Neo-Hookean (Smith et al., 2018b) energy Ua (F²) induced by internal elastic force. For thin-shell materials, we follow the Kirchhoff-Love shell theory to define their potential energy as the sum of stretching energy U. (AA)) and bending energy Up (AO.), which are defined on the stretching ratio A E IR of each triangular element e's and edge lengths, and relative bending angle A0 E R on each edge connecting two adjacent elements, related to
their rest configuration 8- E The energy and forces in thin-shell materials are modeled based on prior work (Grinspun et al., 2003; Tamstorf & Grinspun, 2013). We further model the bending plasticity by setting up yield criterion & and change the rest configuration for the angle on edge if || >
More details of materials and forward simulation are shown in Appendix A.I.
3.2 FRICTIONAL CONTACT MODELLING
We model frictional contact behavior by potential energies defined on the nodal positions. Before every time step f, detect all potential collision pairs using fast spacial hashing and express the signed distance df E R and tangential relative movement uf e R² as functions of vertices in the k-th pair. A negative d' means penetration and therefore should be penalized. The contact potential
4

---

Published as a conference paper at ICLR 2024
energy is the sum of a repulsive penalty potential (xic)) and friction potential Uy(u)):
kr max(d, or, Uz(ui} ps>eso(EEn ||u
(.(d)) 2
The penalty potential U, is quadratic penalty function approximating the barrier d C,, in which er models the thickness of the contact surface and k, served as a stiffness parameter. The friction potential Uy is smooth friction model from Incremental Potential Contact (IPC) (Li et al., 2020), which unifies the sticking and slipping modes by a smooth approximation. Here (4 is the friction coefficient, Åς is the strength of the repulsive force at the last timestep and fo the smoothing kernel proposed by IPC, with a parameter of accuracy tolerance. Benefiting from the robustness of the iterative implicit solver, we can use large k, and tiny cy to achieve high accuracy, while maintaining high numerical stability. The implementation details of the contact model are further explained in Appendix A.3.
3.3 GRADIENT BACK-PROPAGATION
To enable the use of gradient-based optimization methods for our benchmark tasks and parameter optimization, we' ve integrated backward gradient solver into our simulation engine, making it differentiable. Our approach formulates forward simulations energies at each time step (refer to Appendix A. In this framework, gradient paok-propagation treated as implicit function differentiation as outlined in DiffPID (Du al., 2021). Within each time step, gradients of specific objective function from the subsequent system state are propagated backward to the current step's 's state and material parameters. After that, we gather the nodal position gradients and convert it into gradients related to the manipulator trajectory. For more details of the calculations, please refer to Appendix A.2.
optimization problems concerning
3.4 CONSTRAINT SIMULATION
To retain the position of a vertex during forward simulation, we enforce the corresponding dimension of the vector OU /8x and the matrix 0U/(Ox')² to be zero, This ensures invariable solution for the vertex. For simulating a movable rigid body, we move before each timestep and then fix it in place during the subsequent timesteps.
THINSHELLLAB BENCHMARK
Based the proposed differentiable simulation engine, we present robotic skill learning bench- mark providing a diverse collection of thin-shell object manipulation tasks equipped with differen- tiable reward function, and standardized interface for skill learning algorithms. We illustrate the proposed tasks in Figure 2.
4.1 BENCHMARKING INTERFACE
Task formulation In each manipulation task, the robot employs 6-DoF manipulators or 7-DoF paral- lel grippers. These components mimic real-world vision-based tactile sensors (see Section 5.3). We simplify the model by omitting the robot arm's link structure, as arm movement can be derived from end-effector trajectories using inverse kinematics methods. For policy learning, each manipulation task is formalized as the standard S. action space A. reward function R: : A which is determined by forward simulation of the physics engine. We optimize the agent to de- vise policy (a]s)) that samples an action sequence maximizing the expected cumulative return E, |> 7 R( St. az)]. Details of reward setting are listed in Appendix B.
Markov Decision Process (MDP) with state space E. and transition function 7 S
S
State space As mentioned in Section 3. 1, the deformation state of non-rigid objects is characterized by the positions of discretized vertices. To denote relevant quantities, we define N,, as the total number of vertices, N. as the total number bending edges, and N,, as the total degrees of freedom (DoF) for all manipulators. The complete simulation state for the system is represented as S (x, X, r, M), where x, x E RN, x3 denote the positions and velocities non-fi non-fixed vertices, r E N represents the rest configurations, and M E RN,. signifies the pose configurations of manipulators.
5

---

Published as a conference paper at ICLR 2024
Folding
Litting
Pick-Folding
Sliding
Folding-L
Initial
Optimized
Forming
Separating
Following
Card
Bouncing
Figure 2: This figure shows 7 manipulation tasks on the left side and 3 inverse design tasks on the right side. We display the initial and the final position for our manipulation tasks and we show the target transparently Lifting and Forming. For Sliding, we show the final goal with transparency and show the results before and after optimization on the right column. For Bouncing, we show the final state before and after optimization in transparency. We display the behavior of Card in transparent and draw array representing the target moving direction of the card.
Observation To facilitate learning and control algorithms, employ downsampling to reduce the dimension observation space. We select Na points within the scene and stack their positions and velocities, forming a vector of dimensions Na 6. We uniformly sample N, points from each sheet and N₂ points from each volumetric object, and therefore Na = N₂ + N.. Furthermore, we incorporate the manipulators' poses into the final observation vector of dimensions Na 6 + Nm.
Action Space We assume that the elastic manipulators are attached to rigid-body base parts, which have DoF's for single manipulators and 7 DoF's for parallel grippers. The action space then defined as the pose change of all manipulators.
4.2 MANIPULATION TASKS
Lifting This task begins with flat sheet with two manipulators on the bottom, one on the top, and a block positioned on the opposite side. The objective is to raise the block using the sheet and transport different location.
Separating Given a sheet of paper table, cube resting on it, and a parallel gripper gripping the paper, the agent must pull out the paper and separate it from the cube horizontally, striving for maximum separation distance.
Following In a scenario identical Separating, this task involves moving the cube along with the paper towards the right side as far as possible by pulling the paper.
Folding This task revolves around folding a sheet of paper induce plastic deformation. The scene comprises a bending strip on the table with one end fixed and manipulator positioned on it. The objective is to reinforce the crease, either on the upper or lower position of the curve, which split into two tasks: upppeng-upper and Folding-lower (abbreviated as Folding-U and Folding-L).
Pick-Folding this task, the agent operates two manipulators initialized on opposite sides arched table. The objective is to skillfully pick up the paper and fold it,
sheet of paper resting with the specific aim of creating a robust crease in the middle of the paper.
Forming Beginning with curved paper the table, one side fixed, and gripper top, the agent's objective is to manipulate the paper using a single end-effector to achieve a predefined goal shape. This shape is generated by executing random trajectory and saving the final paper position.
4.3 INVERSE DESIGN TASKS
In this set of tasks, we maintain fixed manipulator trajectories and instead optimize system parame- ters to achieve specific objectives.

---

Published as a conference paper at ICLR 2024
Sliding The scene consists of three sheets of paper stacked table, with manipulator pressing and sliding on them. The goal is to optimize the friction coefficient between the sheets to ensure they all move together.
Bouncing We begin with a flat sheet on the table featuring two creases. The sheet initially bounces due the bending force exerted by the creases. The task's objective is to optimize the bending stiffness to achieve higher bounce height.
Card Initialization involves card held by three manipulators, followed by executing card shuf- fling movement to launch the card. The aim in this task to adjust the bending stiffness to project the card as far the right possible.
EXPERIMENT
In this section, we quantitatively evaluate methods including the state-of-the-art RL algorithms, together with campling-based gradient-baser and hybrid trajectory optimization on ThinShellLab's manipulation benchmark tasks, and analyze their behavior and performance on different tasks. We also experiment with inverse design and real-world tasks including real-to-sim system identification and real-world object manipulation.
5.1 METHOD EVALUATION WITH MANIPULATION TASKS
We conduct a comprehensive performance evaluation various methods, including gradient-based trajectory optimization (GD), model-free Reinforcement Learning (RL) algorithms (Soft Actor- Critic SAC (Haarnoja et al., 2018) and Proximal Policy Optimization PPO(Schulman et al., 2017)), and sampling-based trajectory optimization method named Covariance Matrix Adapta- tion Evolution Strategy (CMA-ES) (Hansen & Ostermeier, 2001). To ensure replicability, we use open-source implementations for the RL algorithms (Raffin et al., 2021) and CMA-ES (Hansen et al., 2019). For gradient-based trajectory optimization, we employ the Adam optimizer. Addition- ally, we explore a hybrid approach, initializing optimization with CMA-ES and then transitioning to gradient-basee optimization (CMA-ES + GD). Considering the possibility that both CMA-ES and RL algorithms may require numerous iterations to converge, we report the maximum scores achieved within a specified number of episodes for performance insights. (see Table 2) We delve into task behavior to highlight the inherent challenges, and analyze the strengths and limitations of various methods in addressing these challenges.
Tasks
Lifting
Separating
Following
Folding-U
Folding-I.
Pick-Folding
Forming
PPO
2.22 0.11 6.01 = 0.02 -0.53 + 0.03 1.13 + 0.28 3.32 = 1.19 0.17 + 0.19 2.34 + 0.33
SAC
1.81
0.09
7.73 0.89
2.81 + 0.21
2.43 + 0.02 3.49 0.38
0.85 + 0.84 3.00 + 0.30
CMA-ES
1.42
0.01 8.08
0.18 3.36
0.70
3.22 + 0.04 3.93
0.80
2.14 + 2.14
3.51 + 0.26
GD
4.15
6.80
-0.50
-0.03
0.03
1.69
CMA-ES GD 2.51 0.39 8.24 = 0.27 3.43 + 0.89 3.81 + 0.03 4.68 0.17 4.74 + 2.11 3.63 = 0.14
Tasks
Sliding
Bouncing
Card
Tasks
Sliding
Bouncing
Card
CMA-ES 4.47 = 0.09 0.62 + 0.04 -1.02 + 0.01
GD
4.34 + 0.03 0.75 + 0.01 -1.00 + 0.04
Table 2: We show the maximum reward within a fixed number of episodes and the standard deviation of it. Since GD has randomness in different trials, it has standard deviation.
Task Behavior Our investigation reveals that, despite the structural similarity of the defined Folding tasks, they exhibit markedly divergent behavioral patterns, with Folding-upper displaying notably higher complexity compared to Folding-lower Specifically, in Folding-lower, the agent simply needs to establish contact with or below the upper curve and subsequently push in left-downward direction. Conversely, in the case of Folding-upper, the agent must delicately harness both bending and friction forces to navigate the upper curve towards the right side before proceeding with the folding action. Among the tasks we examined, Pick-Folding poses the greatest challenge. This can be attributed to several factors: 1) the necessity for initial contact points in close proximity to the
7

---

Published as a conference paper at ICLR 2024
CMA-E5
Folding-upper
Folding-lower
Lifting
Forming
Following
Aoarage reeard
Avenge reeard
Averge reward
Avengg reward
200
400
200
400
200 fpisode
400
200 Fpisode
400
700 Fpisode
400
Fpisode
Episode
Separating
Pick-folding
Bouncing
Card
Sliding
Arerage reeang
-1.
ree Burea
-1.
Average
3.4
Avarage
0.2
200 Episode
400
500 Episode
1000
20 Episode
23 Epesode
20
Epesode
Figure 3: Reward curves for all methods. We plot the prefix maximum for the CMA-ES method. We observe that the hybrid method of CMA-ES GD achieves the best performance most tasks.
center; 2) the task's high reliance on precise management of friction forces; 3) the sparse reward structure associated with this task. Separating shows intriguing behavior: the gripper initially grabs the paper towards the block object to impart an initial speed to the block before extracting the paper. Subsequently, we demonstrated the increased efficiency of this approach in real-world scenarios. Additionally, we explored the replacement of paper with a tissue characterized by lower bending stiffness, resulting in observable variations in agent behavior. For more tasks and experiments, include them in Appendix B.
RL Algorithms As depicted in Figure 3, RL algorithms, while not achieving optimal efficiency, ex- hibit reasonable performance in tasks characterized by straightforward hehaviors, such as Folding- lower and Forming. Given their nature of random searching, these methods generally yield higher maximum scores compared to direct GD, which often encounters local optima in most tasks. How- ever, important acknowledge the inherent limitations of RL algorithms. 1) RL algorithms are notably deficient in sample efficiency, requiring considerable number of steps to converge. 2) They demonstrate limited progress in tasks that demand fine ane-grained control, as evident in Lifting, or when tasks become progressively more detailed, such as Forming. 3) RL algorithms face sig- nificant challenges when confronted with demanding tasks, such as Pick-Folding, which features sparse rewards, Folding-upper, characterized by intricate behavioral requirements.
CMA-ES CMA-ES exhibits reasonable performance across most tasks, effectively evading local minima and providing improved initial policies for gradient-based methods. However, it encounters challenges Lifting. Although it manages to acquire the ability to bend the paper to prevent the block from falling, it struggles to advance the block closer to its intended destination. This difficulty can be attributed to the intricate interplay of the three end-effectors, coupled with the sensitivity of the task's dynamics, rendering the sample-based CMA-ES method less suited for this particular challenge. Furthermore, our observations indicate that CMA-ES experiences a decelerated learning curve in later episodes and requires an extended number of episodes achieve convergence. This characteristic, while beneficial for searching for an initial policy, proves less suitable for addressing finer-grained tasks that demand more rapid progress and adaptation.
Gradient-based Trajectory Optimization In all the tasks considered, our gradients effectively guide the policy towards improved performance within the initial time steps. However, note- worthy that this process often becomes ensnared in local optima. This phenomenon is especially evident in the Folding tasks, as illustrated in Figure We attribute this behavior to the limitations of the gradient-based method, which does not account for changes in contact pairs and is sensitive to even minimal alterations in the action sequence.
Hybrid method To address local optima and enhance fine-grained control, we combined the sample- based and gradient-bar based methods. Initially, the sample-base approach identifies contact points and establishes initial policy. Subsequently, the gradient- madient-bbsedd method fine-tunes the trajectory. This hybrid approach excels in most manipulation tasks, except for Lifting, where it eventually converges to similar reward level the gradient-basse based method, albeit less efficiently.
8

---

Published as a conference paper at ICLR 2024
5.2 RESULTS OF INVERSE DESIGN TASKS
For inverse tasks, we evaluate CMA-ES and GD methods. As evident in Figure 3, both the gradient- based and sample-based methods perform effectively in these tasks. However, the gradient-based method exhibits a faster and smoother growth in inverse design tasks, benefiting from its guidance by physical principles. Notably, in the Card and Sliding tasks, the standard deviation of the GD method diminishes over time. This trend indicates that, despite commencing with distinct initial system parameters, our gradient-based method converges these parameters towards a uniform outcome.
While the Bouncing task necessitates a higher bending stiffness and the Sliding task requires an increased friction coefficient, the Card task poses more intricate challenge. In the case of the Card task, the bending stiffness must strike a delicate balance. If it is too high, the card reaches a greater distance but tends to flip out prematurely due limited curvature by the end-effectors, resulting in shorter overall moving distance. Furthermore, altering the bending stiffness can lead to changes in contact pairs, contributing to larger standard deviation in the later time steps of the Card task compared to the other two inverse design tasks.
5.3 REAL-WORLD EXPERIMENT
In this section, we leverage the differentiability of our simulator to fine-tune physics parameters ac- cording to real-world data. We further conduct experiments of transferring skills learned from the simulator to real- world robotic systems, showing the capability of ThinShellLab's learning frame- work to bridge between simulation and real-world scenarios.
Real-to-sim System Identification. In the real-to-sim process, initiate real-world manipulation experiments employing a robotic system equipped with force and tactile sensors. Subsequently, construct corresponding simulation environments that integrate differentiable loss function, quantifying the disparities between simulation outcomes and real-world results. Leveraging the differentiability of our physics engine, we back-propagate gradients adjust material parameters and employ gradient descent methods for optimization, as outlined in Section 5.2. Our real-to-sim system identification tasks encompass Tactile, Friction, and Bending, serving as demonstrations of ThinShellLab's 's capacity to reconstruct real-world scenes from collected data. For comprehensive results and additional details, we refer readers Appendix C.
Real-world Manipulation Experiment. After reconstructing physics parameters, we transferred the Separating manipulation task from simulation to real-world robotic system. We optimize the task in the virtual scene with reconstructed parameters and deploy it to a real scenario with robotic arm and parallel gripper. Our optimized trajectory, as described in Section 5.1, successfully gener- ated an initial velocity on the object to prevent sticking during sheet retraction, proving effective in the real world. In contrast, a straightforward heuristic policy of dragging at a constant speed failed due to insufficient relative speed. For more details, please refer Appendix D.
CONCLUSION AND FUTURE WORK
We introduced ThinShellLab, a comprehensive robotic learning benchmark for thin-shell object ma- nipulation based on a fully differentiable simulation engine, presented with standardized interface for skill learning methods and a set of benchmark tasks. Through these benchmark tasks, we identify challenges like non-continuous optimization landscapes due to complex contacts and high material sensitivity for thin-shell manipulation. To tackle these challenges, we found that a hybrid approach, combining sample-based and gradient-based optimization, performs best across most tasks. We con- ducted real-world experiments, fine-tuning system parameters to match real-world observations, and successfully deployed our learned skills in real-world scenarios.
While we are working on the real-to-sim process to address the simulation-to-real gap, current approach relies only on tactile and force sensors. However, it's important to note that there might still be residual gap due to our contact model and inherent real-world instability, leading random errors. Hence, future research could explore the integration of visual observations and advanced limulation-to-Pe techniques to further minimize this gap.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
We thank Tiantian Liu for the insightful discussions and help with the experiments. Special appreci- ation goes Meshy, where the first author completed an internship and conducted portion of this research. We thank the anonymous reviewers for their helpful suggestions. This work is funded in part by grants from Google, Amazon, Cisco, Toyota Motor North America, and Mitsubishi Electric Research Laboratories.
REFERENCES
Siwei Chen, Yiqing Xu, Cunjun Yu, Linfeng Li, Xiao Ma, Zhongwen Xu, and David Hsu. Daxbench: Benchmarking deformable object manipulation with differentiable physics. In The Eleventh ternational Conference Learning Representations, 2023. URL htttss:/opperreiies net/ met//oomae wttt/forumiiin INAZMo FMnW1.
Cheng Chi, Benjamin Burchfiel, Eric Cousineau, Siyuan Feng, and Shuran Song. Iterative resid- ual policy for goal-conditioned dynamic manipulation of deformable objects. In Proceedings of Robotics: Science and Systems (RSS), 2022.
Erwin Coumans and Yunfei Bai. Pybullet, python module for physics simulation for games, robotics and machine learning- http://pybullet.../
2016-2021.
Tao Du, Kui Wu, Pingchuan Ma, Sebastien Wah, Andrew Spielberg, Daniela Rus, and Wojciech Matusik. Diffpd: Differentiable projective dynamics. ACM Trans. Graph., 41(2), 2021. ISSN 0730-0301. doi: 10. 10.1145/3490168 URL https: //dpi. www.ooog///..oo... 1et.e.111//49111
Chuang Gan, Jeremy Schwartz, Seth Alter, Damian Mrowca, Martin Schrimpf, James Traer, Ju- lian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, Megumi Sano, Kuno Kim, Elias Wang, Michael Lingelbach, Aidan Curtis, Kevin Tyler Feigelis, Daniel Bear, Dan Gutfre- und, David Daniel Cox, Antonio Torralba, James J. DiCarlo, Joshua B. Tenenbaum, Josh Mc- dermott, and Daniel LK Yamins. ThreeDWorld: A platform for interactive multi-modal phys- ical simulation. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021. URL https: wwtp://wweeeeeeeeeeeeeep/.. nee. fftffoormmmdd dbl InWAWW2T.
Eitan Grinspun, Anil N. Hirani, Mathieu Desbrun, and Peter Schröder. Discrete Shells. In D. Breen and M. Lin (eds.), Symposium on Computer Animation. The Eurographics Association, 2003. ISBN 1-58113-659-5 doi: 10.2312/SCA03/002-007.
Huy Ha and Shuran Song. Flingbot: The unreasonable effectiveness of dynamic manipulation for cloth unfolding. In Conference on Robotic Learning (CoRL), 2021.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning, pp. 1861-1870. PMLR, 2018.
Jiaqi Han, Wenbing Huang, Hengbo Ma, Jiachen Li, Joshua B. Tenenbaum, and Chuang Gan. Learn- ing physical dynamics with subequivariant graph neural networks. In Thirty-Sixth Conference on Neural Information Processing Systems, 2022.
Nikolaus Hansen and Andreas Ostermeier. Completely derandomized self-adaptation in evolution strategies. Evolutionary computation, 9(2):159-199) 195, 2001.
Nikolaus Hansen, Youhei Akimoto, and Petr Baudis. Cma-es/pycma on github. zenodo, doi:
10.5281/zenodo. (f5b.34.(fcb. 2019), 2019.
Yuanming Hu, Tzu-Mao Li, Luke Anderson, Jonathan Ragan-Kelley, and Frédo Durand. Taichi: language for pigh-peeformance computation on spatially sparse data structures. ACM Transac- tions on Graphics (TOG), 38(6):201, 2019.
Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan Ragan-Kelley, and Frédo Durand. Difftaichi: Differentiable programming for physical simulation. ICLR, 2020.
10

---

Published as a conference paper at ICLR 2024
Zhiao Huang, Yuanming Hu, Tao Du, Siyuan Zhou, Hao Su, Joshua B. Tenenbaum, and Chuang Gan. Plasticinelab: soft-body manipulation benchmark with differentiable physics. In Interna- tional Conference Learning Representations, 2021. URL forum? d=xCoddREEEDR
nnt//
Zizhou Huang, Davi Colli Tozoni, Arvi Gjoka, Zachary Ferguson, Tesco Schneider, Daniele Panozzo, and Denis Zorin. Differentiable solver for time-dependent deformation problems with contact. arXiv preprint arXiv:2205. 2022.
Minchen Li, Zachary Ferguson, Teseo Schneider, Timothy Langlois, Denis Zorin, Daniele Panozzo, Chenfanfu Jiang, and Danny M. Kaufman. Incremental potential contact: Intersection- and inversion-free large deformation dynamics. ACM Trans. Graph. (SIGGRAPH), 39(4), 2020.
Minchen Li, Danny M. Kaufman, and Chenfanfu Jiang. Codimensional incremental potential tact. ACM Trans. Graph. (SIGGRAPH), 40(4), 2021.
Sizhe Li, Zhiao Huang, Tao Chen, Tao Du, Hao Su, Joshua B. Tenenbaum, and Chuang Gan. Dexdeform: Dexterous deformable object manipulation with human demonstrations and dif- ferentiable physies. In International Conference on Learning Representations, 2023. URL https: http//pppeeeeeeeeeeeeeeeeeeeeeee net http://wwweeeeee//eeeeee forummid=LiV1--11
Yifei Li, Tao Du, Kui Wu, Jie Xu, and Wojeiech Matusik. Diffcloth: Differentiable cloth simulation with dry frictional contact. ACM Trans. Graph., 42(1), oct 2022. ISSN 0730-0301. doi: 10.1 145/ 3527660. URL https: /twii org/ 111.1135152266
Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B. Tenenbaum, and Antonio Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids, 2019.
Xingyu Lin, Yufei Wang. Jake Olkin, and David Held. Softgym: Benchmarking deep reinforcement learning for deformable object manipulation. In Conference on Robot Learning. 2020.
Miguel Angel Zamora Mora, Momchil Peychev, Sehoon Ha, Martin Vechev, and Stelian Coros. Pods: Policy optimization via differentiable simulation. International Conference on Machine Learning, pp. 7805-7817. PMLR, 2021.
Akio Namiki and Shuichi Yokosawa. Robotic origami folding with dynamic motion primitives. In 2015 JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 5623- 5628, 2015. doi:
Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, and Ming C. Lin. Scalable differentiable physics for learning and control. In ICML, 2020.
Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, and Ming C. Lin. Differentiable simulation of soft multi-body systems. In Conference on Neural Information Processing Systems (NeurIPS). 2021.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dor- mann. Stable-baselines3: Reliable reinforcement learning implementations. The Journal of Ma- chine Learning Research, 22(1):12344-122 2021.
Connor Schenck and Dieter Fox. Spnets: Differentiable fluid dynamics for deep neural net- works. ArXiv, abs/1806.06009 2018. URL https: pettps:/aai..ste http:/pp..ssmttticcccchcccccc org/
CorpusID: 49207686.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 2017.
Eftychios Sifakis and Jemnj Barbic. Fem simulation of 3d deformable solids: practitioner's guide to theory, discretization and model reduction. Acm siggraph 2012 courses, pp. 1-50. 2012.
Breannan Smith, Fernando de Goes, and Theodore Kim. Stable neo-hookean flesh simula- tion. ACM Transactions on Graphics (TOG), 37:1 - 15, 2018a. URL https: htpps:///pppi appi. semanti iesanotischoll ar. wtwp:/ooo//ooootaeeecom

---

Published as a conference paper at ICLR 2024
Breannan Smith, Fernando De Goes, and Theodore Kim. Stable neo-hookean flesh simulation. ACM Trans. Graph., 37(2), mar 2018b. ISSN 0730-0301. doi: 10.1145/3180411 URL https:
Rasmus Tamstorf and Eitan Grinspun. Discrete bending forces and their jacobians. Graph. Model., 75:362-370, 2013. URL
9946950.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012. doi: 10.1 10.1109/RROS.011 201..336616.6
Benjamin Ummenhofer, Lukas Prantl, Nils Thuerey, and Vladlen Koltun. Lagrangian fluid simu- lation with continuous convolutions. In International Conference Learning Representations, 2020. URL wwtptttp/////....t. /ttp:s//ppppeeveeeeee
www.ewwwwww..................... net/ forum? d=B1 10s=111Dom DOJSYDH.
Ruihai Wu, Chuanruo Ning, and Hao Dong. Learning foresightful dense visual affordance for deformable object manipulation. In IEEE International Conference on Computer Vision (ICCV), 2023.
Zhou Xian, Bo Zhu, Zhenjia Xu, Hsiao-Yu Yu Tung, Antonio Torralba, Katerina Fragkiadaki, and Chuang Gan. Fluidlab: A differentiable environment for benchmarking complex fluid manipula- tion. In International Conference on Learning Representations, 2023.
Jie Xu, Sangwoon Kim, Tao Chen, Alberto Rodriguez Garcia, Pulkit Agrawal, Wojeiech Matusik, and Shinjiro Sueda. Efficient tactile simulation with differentiability for robotic manipulation. In 6th Annual Conference on Robot Learning, 2022a. URL forum? www.-aiIffffffee..
net//
Zhenjia Xu, Cheng Chi, Benjamin Burchfiel, Eric Cousineau, Siyuan Feng, and Shuran Song. Dex- tairity: Deformable manipulation can be breeze. In Proceedings of Robotics: Science and Systems (RSS), 2022b.
Chao Zhao, Chunli Jiang, Junhao Cai, Michael Wang, Hongyu Yu, and Qifeng Chen. Flipbot: Learning continuous paper flipping via coarse-to-fine exteroceptive-rrorroopptt exploration. pp. 10282-10288, 05 2023. doi: 10.10999CA444911.2........
Shaokun Zheng, Zhiqian Zhou, Xin Chen, Difei Yan. Chuyan Zhang, Yuefeng Geng, Yan Gu, and Kun Xu. Luisarender: A hirf-peeformance rendering framework with layered and unified in- terfaces on stream architectures. ACM Trans. Graph., 41)6), nov 2022. ISSN 0730-0301. doi: 10. 10.11455/355445433555555 URL httt:://.ooo.............1.5555.
A IMPLEMENTATION DETAILS
A.1 FORWARD SIMULATION
We apply implicit time-stepping scheme in our simulation engine. Let n be the total DoF of the whole system and x", v² E Rn be the generalized coordinates and velocities at time-step t. Then the implicit time integration would be formulated as:
x arg min g(x) arg min
II U(x), 1X
x
which yd x + live M and is the mass matrix the system, fent the total external force. This is equivalent to the following gradient form (Du et al., 2021):
0 = Vg(x) = 2か
M(x - yt) (x).
in which fint refers to the internal forces, i.e., vx). Since Vg(x) is in general non-linear, this equation solved iteratively using Newton's method at each time step. To fix the j-th variable in simulation, we only need to set the j-th value of vector Vg(x) to zero and set the j-th row and column of its Hessian matrix to zero in Newton's 's method. Further details of energy calculation will be discussed in Appendix A.4.
12

---

Published as a conference paper at ICLR 2024
A.2 GRADIENT BACK-PROPAGATION
We can back-propagate the gradient through the implicit function Vg(x) = 0. As also mentioned in DiffPD (Du al., 2021), we calculate the derivative between x', and x and x to deliver the gradient of the loss function 8x
aL 0x 0L. उश T + at 8x* Note that we also have to calculate 8x ďx
the loss gradient for 80 in all the time steps, since we are modeling the plasticity of thin-shell materials. To start with, we first rewrite the function as follows:
1 M(x 2x' x-=1 122 'Fent) - fint (x, X(szed: x², si),
- Vg(())
21/2
where xf and x are node positions of 4e and (t. 1)sh timestep, Ø' are undeformed angles for fen timestep, x fized are positions of fixed nodes in the beginning of (t + 1)th timestep, and is the material parameters. Here, B' is actually function of x' and Ø' and the friction force is function of x; and X;-1- We then back-propagate the loss gradient through this implicit function to obtain gradients for each time step. These gradients, applied subset of xized corresponding to the manipulator's 's rigid base part, aggregated to the gradient of manipulator trajectories, as mentioned in Section 4.1.
A.3 COLLISION DETECTION AND CONTACT HANDLING
The collision detection done by fast grid hashing of the contact surfaces. We spatially split the whole 3D space into grids. For each triangular face in object, store its information in grid that the triangle's 's circumcentre is occupying. When querying the contact state an incoming vertex, we search in the nearby 3 x 3 grids and detect collisions with all triangles involved in these grids. In order to ensure the correctness of hashing, we set the edge size of grids to be greater than half of the maximal diameter of all triangles.
Since our contact handling uses quadratic penalty potential instead of strict barrier functions, we design mechanism avoid crashes caused by penetration between vertices and thin-shell objects. Once a collision has been detected, we record the normal direction of contact surface, and then keep unchanged until the collision state removed. In this way, the contacting vertices will not penetrate to the opposite side of the thin-shell object, and therefore make the simulation robust and stable.
While the logarithmic barrier employed in IPC imposes strict non-penetration, it tends to signif- ieantly slow down convergence in Newton's ss method. Moreover, demands continuous collision detection during the solution of forward dynamics and can introduce instability into gradients dur- ing back- propppropag Hence, given that extreme accuracy isn' 't always imperative in robotics tasks, simplify the calculation of the penalty function by retaining the normal direction of contact pairs. This approach remains stable even in scenarios where penetration occurs.
A.4 ENERGY CALCULATION
A.4.1 CONTACT ENERGY
According to Section 3.2, the contact energies U, and U, are defined as
U+(d) ( - 0), = =
where the distance d is signed distance between vertex p and its projection triangle (Pi, P2. Pa). whose normal direction is represented by (p₂ - Pı) x (P3 - Pi). k stands for constant stiffness parameter, y stands for friction coefficient, and À stands for the contact force
13

---

Published as a conference paper at ICLR 2024
along normal direction at last timestep. The signed distance d is then calculated as
(p- pi) (P2 Pi) x (Pa P1)
(P2 - Pi) x (P3 -P1)]
(p- - P1, P3 P1, (8,
(1(p2 pi) x (Pa PI)
dct(p2 P1. Pt. ...P- Pil
(p - -P1) (Ps 10--p -PI)
P2. P3. p| R denote in which the 3D vertices are represented as column vectors. Let [pi, P2. the involved variables, we implemented tool functions in Taichi to calculate
(xx) det(p2 P1. P3 P1.P P1.P-P PPl, C(x) = (Ipp - PP) (Ps PI)ll.
together with the gradients 0D/0x, 0C/0x and Heissan matrices D/ 8C/Ox.. With the help of SymPy library for symbolic computation, we optimized the computation graph and hard coded into Taichi snippets. The repulsive energy U, is then a composition function on (xx), C(x), and therefore we use the chain rule calculate its gradient and Hessian matrix.
For the friction energy Uy( we follow IPC (Li et al., 2020) to calculate the fo ||ul]) func- tion, its derivative fe(e. ||]]) and second derivative f2(Eu, ||ul]). Once we get the /0]uluu and / we transform it back to the tangential space get ,0 and 02 ,
A.4.2 ELASTIC ENERGY
We follow Stable Neo-Hookean (Smith et al., 20) to define the elastic energy as
(F) H (1c - 3) pu -
A 1))
in which F denotes the deformation gradient of one tetrahedral element, Io tr(F"FF), and J det F. The motivation for replacing the original log J term in the Neo-Hookean model with J that 1 is stable even for J < Therefore, even if the whole tetrahedral element is inverted, model still works robustly and can provide gradients and Hessian matrices recover the element (see the original paper of Stable Neo-Hoo (Smith et al., 2018b)). This situation is common in our simulator, since the robot manipulator involves moving boundary and can easily cause inversion when the moving speed is large. We observe that the Stable Neo-Hookean perfectly addresses this problem.
this Stable
The gradient aU./0E can be computed easily. compute the Hessian matrix PU./OFF we first enumerate the differential oF, for single entry and then calculate the correponding force differential /(FU. We follow a well-known tutorial on FEM (Sifakis & Barbic, 2012) to calculate the force differentials, and then concatenate them into the Hessian matrix QPU./OF
A.4.3 THIN-SHELL ENERGY
Following Grinspun et al. (2003), apply the stretching energy U, and the bending energy Us for thin-shell materials. We further divide the stretching energy into U and Ug. representing stretching energy for edges and areas of triangles. The energy formula is follows:
U [ke(1 (1((|1)e/))))
U Σ
K1 - |11/141)3
A
U, Σ Ko(0. - 0-)97e///ell
where e represents all the edges, A represents all the triangles, and 0g represents the angle on edge e. Jel. [4], [he], and O. respectively represent the rest length of the edge, the rest area for the triangle,
14

---

Published as a conference paper at ICLR 2024
one-third of the average height of the two adjacent triangles of the edge, and the rest angle this edge. Ke, Ka, and K2 are the coefficients.
For US, it's simple to calculate the derivatives and Hessian matrix. The derivative of the length |el (0o x ////e The Hessian matrix will require the calculation of aO|cl and &Piel due to symmetry, where To = (0o, bo, co)- Bag Bagübe
to the nodal point positions zo and # is Øeei
For US. it becomes more complex to compute the derivatives and Hessian matrix. Assume the triangle is (Io, (2) and xig = (0n, bn,(,) cn). Similar to the calculation of the contact energy, we use math tools to generate the symbolic formula of the derivative @| and hard code it with Taichi; then, we can calculate all the derivatives due to symmetry. For the Hessian matrix, we use math tools to generate formulas for Ø1 AA øPAl øAAA| then, we can calculate the full matrix using symmetry.
Bug Baoda1 Bag Bag0b,
For Ub. the Hessian matrix is too complex to compute, even when we try to use some math tools. As result, we follow Tamstorf & Grinspun (2013) compute the derivatives and Hessian matrix for bending energy. This paper discusses how to compute the Hessian matrix of bending energy in simplified form.
A.4.4 BACK- PAKKPPROPAGATION
This part has been briefly discussed in Appendix A.2. The equation to be solved in forward simula- tion
Vg(x)= 21
M(x 2x + x'-1 1PM--fees - Line (x, x yined x", H m)) (1)
where x² and - are node positions of trn and (t - 1)th timestep, 8' are undeformed angles for le timestep, x" xiized are positions of fixed nodes in the beginning of (t + 1)th timestep, and v is the material parameters. Here, 0' is actually function of x' and ~' and the friction force is function of x, and x,- DifiPD (Du et al., 2021) discusses the general idea of how to pass gradient in the implicit time integration:
gvg(x) dx Vvg(x) 0x
a Oy
0
θy
dy
dx
1
M(x y) VE(X))
M(x
y)] =
0
Fx
112
Qy
Ty
h
1 M
dx
1
V
M-0
h2
Jy
h2
0x Qy
('gg())
With in hand, we can backpropagate the gradient from xt+1 to y². Given y¹ 2x xt-1 by 122 M feats we can deliver the gradient to both x' and 1-1
Similarly, by substituting y in the above equation with other variables like xyszed or 0" we
calculate the gradient. The remaining challenge is computing the term @Vg(x) in the first line when By changing y into other variables. For these variables, this term will involve the Jacobian matrix for the force, as shown in Equation (1).
For ø' and 17, their relationship to the forces is linear, allowing to calculate the Jacobians directly. Regarding x² fized= the Jacobians are computed during forward simulation, where they are set to zeros. We can apply the same procedure in forward simulation and retain the Jacobians during the backpropagation process.
Concerning x", it primarily influences the friction force since we utilize the pressure force, contact surface, and the original projected point from the last timestep. We observe that Huang et al. (2022) calculates the derivative for friction similarly to our approach. The distinction lies in our choice not to pass the gradient through the surface change during backpropagation.
15

---

Published as a conference paper at ICLR 2024
Task
table
Auble
manipulator Moloth
objoct
bending stiffness
action range
Lifting
5.0
5.0
100
0.001
Separating
0.2
5.0
0.2
100
0.002
Following
0.2
5.0
100
0.002
Folding-U
5.0
5.0
400
0.001
Folding-L
5.0
5.0
400
0.001
Pick-Folding
0.1
5,0
200
0.001
Forming
5.0
5,0
200
0.001
Sliding
0.4
1.0
1000
Bouncing
0.5
Card
1.0
Table 3: Detailed parameters.
A.5 RENDERING
We have customized LuisaRender (Zheng et al., 2022), a ray-tracing rendering framework, to facil- itate the rendering of simulation results. This tailored version of the renderer boasts support for a wide range of material models and possesses the capability to generate highly realistic rendering outcomes.
B DETAILS OF MANIPULATION TASKS
B.1 TASK DETAILS
In RL. algorithms, we employ downsampling of nodal points to construct the observation input.
In all our tasks, we consistently utilize simulation step of 5e-3 seconds, and we establish a max- imum action range specific to each task to ensure system stability. For the majority of our tasks, confine the action range I millimeter per timestep. However, for tasks that inherently de- mand higher speeds, such as the Separate and Following tasks, we extend the action range to millimeters.
In the case of RL and CMA-ES, we employ heuristic approach to truncate the action range. In contrast, for gradient-based trajectory optimization, employ loss function to softly regulate action speed. These adjustments help maintain the stability and effectiveness of our reinforcement learning algorithms in various tasks.
We further list our detailed parameter for each task in Table B.
B.2 REWARD
The positive direction of the x-axis is defined as pointing towards the interior the table for all tasks.
Lifting The reward for successful completion is determined by taking the negative sum of squared distances between each point of the block and its target position. For better distinguish the difference we use -Ig(-reward) as the new reward when plotting and filling the table.
Separating
The reward for this task is computed as the difference between the of the x- coordinates the block and the sum of the x-coordinates of the paper, multiplied by the ratio of the total number of nodal points the block to that on the paper.
Following In the context of this task, the reward is calculated as the negative average x-coordinate of the block.
Folding For Folding-upper, the reward is determined as the sum the upper crease rest angles minus the of lower crease rest angles, whereas for Folding-lower, is the opposite.
16

---

Published as a conference paper at ICLR 2024
Pick-Folding The reward for this task is the sum of crease angles for the edges located in the middle of the sheet.
Forming The reward for this task is the negative sum of the squared distances between each point
on the paper and its respective target position. Similar to Lifting we apply -Ig(-reward, as the new reward.
Sliding The reward for this task is determined as the negative sum of the x-coordinates of the bottom paper.
Bouncing The reward function for this task is defined as the sum of the z-coordinates of the points located on two creases.
Card In this task, the reward is defined as the negative sum of the x-coordinates for every nodal point the card.
The loss is generally negative rewards in those tasks, while for GD method we include another loss to limit the action range.
B.3 MORE ANALYSIS
Although we attempted to guide the learning process in Pick-Folding by introducing small reward for increasing the angle of the middle curve and observing slight improvements in gradient-based methods, the optimization process frequently became trapped in local optima. Furthermore, sample- based trajectory optimization also grappled with finding viable solution, with some trials failing to identify a suitable initial solution, leading to significant variance in the final reward across different trial runs. Nonetheless, it noteworthy that the CMA-ES + GD method demonstrated rapid learning capability when provided with a reasonably well-initializn solution for Pick-Folding. This may be attributed to the task's ' dynamic simplification after the sheet is picked up.
We list the detailed design and analysis of the hybrid method as follows.
Parameters: We opt for population size of 40 in CMA-ES with zero vero-initialized trajec- tories. The initial variance is set to 1.0, corresponding to 0.0003 in each dimension of manipulator movement per timestep. The maximum movement within one timestep is ei- ther 0.001 or 0.002, depending on the task. Generally, we execute CMA-ES for 80 percents of the total episodes, where episodes are defined by the number of rollouts. For the Pick- Folding task, which demands more extensive training, we set the CMA-ES episode count 1000 and supplement with an additional 150 episodes of gradient descent.
To enhance the effectiveness of the CMA-ES method, we filter out noa-feasible states, such as instances where there is no contact between manipulators and thin shell or when the manipulator is experiencing excessive force. In these situations, we directly assign low reward value and save the time for executing the rest actions.
Design:
Some curves in Figure exhibit large variance. Specifically, in Folding-U, Fol- lowing, and Separating, the variance arises from the sensitivity of thin-shell manipulation results to minor changes, as previously discussed in the introduction section. In the case of Pick-Folding, although sensitivity contributes the variance, the primary factor is oc- casional failure of the CMA-EE -EE method. Failures result in rewards near 0, and successful runs yield rewards greater than 6, leading to substantial variance in the outcomes.
nalysis:
B.4 SUCCESS INDICATOR
Defining the level of success in certain tasks proves challenging due to uncertainty regarding the op- timal performance achievable within the current settings using the best policy. Currently, we gauge the success of policy by visually inspecting the results. Nevertheless, insight by presenting the scores human-designed heuristic policies alongside theoretically optimal scores. Subsequently, we can formulate success indicator based on analysis of these scores.
aim to enhance analytical
Human Designed To elucidate the meaning of reward values, we present the scores of human- designed policies for each task in the table below. Notably, generated policies occasionally out-
17

---

Published as a conference paper at ICLR 2024
perform human- counterparts, particularly in tasks requiring meticulous manipulation and where certain policies exhibit unexpected effectiveness (as observed in Separating).
Oracle In addition to human-desipn deeiined policies, we also present the theoretically maximum scores for each task, disregarding certain physical constraints. For Pick-Folding. we calculate this score by assuming the entire middle curve is fully folded. the case of Separating, we assume the paper is affixed to the manipulator, while the block experiences no friction force with the paper. For Fol- lowing, we assume both the manipulator and the block are affixed to the paper. In both Folding-L and Folding-U, U, we assume that the target curve is fully folded while the other one is completely flattened. For tasks involving Lifting and Forming, we assume a perfect match with the target. is important to note that although certain Oracle scores, such those for Pick-Folding and Sepa- rating, are exceptionally high, they are not physically feasible within the current experimental settings. We present
Success Score
To make it further concrete, we design the score of success for each task. Con- cretely, we define the score of success as min(HumanDesign, Oracle/2) for Pick-Folding, Sepa- rating, and Following, and Oracle/2 2 for Folding-U and Folding-L since human-designed policies perform poorly in those two tasks. The successful score of Forming and Lifting is -3 x 10
Success Rate Finally, we calculate the success rate of our CMA-ES GD method according to the mean and standard deviation.
Pick-Folding Separating Following Folding-L Folding- U Forming Lifting
Human Design
5.49
7.23
5.05
1.63
-0.24
0.0
-0,001
Oracle
42.24
20.47
7.35
6.42
6.42
0.0
0.0
Success
5.49
7.23
3.68
3.21
3.21
-0.0003
-0,0003
Success rate
0.36
1.0
0.39
1.0
1.0
0.76
1.0
Table 4: Success Indicator.
C REAL-TO-SIM SYSTEM IDENTIFICATION TASKS
We conduct three real-to-sim system identification tasks Tactile, Friction and Bending to assess ThinShellLab's capability of reconstructing real-world scenes. In the Tactile task, we push a vision- based tactile sensor to flat desktop and then optimize the Young's 's modulus and Poisson's 's ratio of the elastomer using collected data of exerted force and shifting of surface dot matrix. In the Friction task, we push the same tactile sensor to sheet of paper and then drag the paper to create slipping friction between it and the desktop. We optimize the friction coefficient to make the simulated friction force matched with collected data. Finally, the Bending task involves piece of thin-shell sheet object with non-negligible bending stiffness, such as poker card, card paper or metallic ruler, whose stiffines parameter to be identified. We fix the object on the desktop's edge, and then push it using a tactile sensor as shown in Figure 1(c). We optimize the bending stiffness to match the simulated and collected force data. Figure 4 shows the optimization curve of these tasks. Figure shows the dot matrix of tactile sensor after optimizing the Tactile task.
We provide additional insights into the Tactile task, as illustrated in Figure Initially, we perform an action sequence involving Pressing Down and subsequently optimize the Young's modulus and Poisson's 's ratio of the elastomer using collected data on exerted force and the shifting of the surface dot matrix. Following this optimization, coefficient through the execution of Pressing and Sliding. Once these parameters are determined, evaluate the outcomes in novel action sequence involving Pressing and rotating along the x-axis. The results indicate minimal discrepancies in this new action sequence.
further refine the system by optimizing the friction
In the Bending task, have encountered challenges in accurately retrieving small-scale bending stiffness from the real world using tactile sensors. This difficulty arises from the sensitivity of results to the Poisson ratio of the tactile material.
18

---

Published as a conference paper at ICLR 2024
Bending
Bending-cart
0.0175
30-
2.5
0.125
0.0075
1.-
0.0000
0.5
0.000
20
40
60
200
ephoode
optsode
Friction
Tactile
14-
0.20
10
0.10
0.05
20
40
60
80
100
40
60
80
100
opicode
cpsode
Figure 4: Optimization curve of system identification tasks.
300
250
260
280
300
320
340
360
380
Figure 5; Tactile dot matrix after optimization.
D REAL-WORLD MANIPULATION EXPERIMENT
We transfer the Separating and Folding tasks into the real world. In the Folding tasks designed to align with the real-world, we optimize it with rigid manipulator with 3 DoFs (which we deem enough to solve the task). As illustrated in Figure 7, we successfully execute both the Separating and Folding tasks in a real-world setting. Upon comparing the simulation results with the real- world executions, minimal sim-to-real gap. Notably, the Separating task. the model generates an initial velocity, showcasing the successful approximation of bending stiffness in real-to-sim process, where the
observe highly similar behaviors, albeit not precisely identical, indicating
19

---

Published as a conference paper at ICLR 2024
Before optimizution
After optimization
Optimization Curve
Pressing down
Pressing along x-axis
Before optimization
After optimization
Optimization Curve
Pressing and Sliding
Figure 6: Real-to-sim tactile task. We show the final stage of the tactile marker before and after optimization. The red points are the 2d marker position in real world and the blue points are in simulator.
Separating
DENE
Sim
Real
Real-heuristic
Folding-L
Folding-U
Real
Real
Figure Real-world manipulation results.
bending stiffness is estimated be sufficiently large impart velocity to the object. The success in the Folding task further demonstrates the model's awareness of the dynamic properties of the paper, a result attributed to the effectiveness of our real-to-sim process.
E SPEED PERFORMANCE
In ThinShellLab, both the forward and backward passes are parallelized single GPU, The major bottlenecks the running time are 1) linear equation solving at each iteration of Newton's method and 2) Hessian matrices building and SPD projecting. For Hessian matrices, the Taichi programming language to process the submatrices for independent elements in parallel. We also implement the QR iteration in Taichi to project each submatrix manifold of SPD
20

---

Published as a conference paper at ICLR 2024
(symmetric positive definite) matrices. For linear equation solving, cuSPARSE library, which supports solving sparse matrices using Cholesky Factorization on GPU. The remaining parts of the simulation (e.g.. line search and collision detection) are fast enough and therefore can be negligible. Collision detection is efficiently solved by space hashing using Taichi's sparse data structure.
Python binding of the
To benchmark the running speed of the simulation, we test our simulator on the Forming task, which contains typical scene with 1698 Dol's involving both volumetric and thin-shell deformable bodies. The test is done on a single RTX 4090. On average, the forward/backwer s respectively for 100 timesteps, i.e.. framerates of 2.74 fps 21.41 fps. In comparison, DiffCloth reports framerates at 0.73 fps 2.72 fps using the same Newton-Cholest Cholesky method and 7.14 fps / 17.54 fps using the fast PD acceleration, both under scene with 2469 DoFs. C-IPC reports framerate at 0.024 fps for forward simulation of thin-shell scene with 18.3K DoFs.
passes take 36.41 s 4.67
In practice, takes about hours train a policy for one task.
While we run our experiments on single GPU, it is also possible to run multiple environments on different GPUs parallel. We didn't 't do this because gradient- trajectory optimization methods do not benefit from parallel environments.
F FUTURE DIRECTIONS
Through the evaluation of various methods in our benchmark tasks of thin-shell manipulation, have concluded that reinforcement learning algorithms exhibit limited data efficiency. This inef- ficiency arises from the sensitivity of thin-shell materials and the heightened complexity of their states. Simultaneously, gradient-based methods face challenges in reliability within contact-rich scenarios. While sample-based methods, such CMA-ES, generally perform well, they suffer from reduced data efficiency later episodes and intricate settings. Consequently, we propose a hybrid approach that combines the strengths of CMA-ES and gradient-based methods to achieve superior results.
In terms of future work, recommend exploring closed-loop control under varied parameters, initializations, and tasks. Currently, our focus has primarily been on open-loop methods like CMA- ES and gradient-based methods. We posit that an avenue worth exploring involves leveraging the synergy between reinforcement learning and gradient methods (Mora et al., 2021) for improved data efficiency in closed-loop control. Additionally, an alternative strategy could involve collecting successful trajectories under diverse settings using open-loop methods and subsequently employing a neural network for imitation learning. This approach aims address generalization challenges rather than optimization problems.
21


Reviews:
[{"review_id": "SZQ8HmKXnI", "reviewer": "Reviewer_Vd5A", "summary": "The paper introduces a differentiable simulator designed to handle thin shell materials such as sheets of paper and cloth to explore manipulation of such objects. To this end a set of tasks are defined to challenge various methods. Finally, an approach to tackle the proposed tasks using combining sampling and optimization is proposed.", "questions": "- What solution quality / success rate do the various rewards correspond to?\n- How fast does the simulator run, is it real-time, faster, slower?\n- What is the limit of object complexity that can be simulated?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Development of a simulator for specific kind of object geometries\n- Definition of a set of tasks to challenge manipulation methods on thin objects\n- Overall easy to understand and follow", "weaknesses": "The idea of the paper are clear and the writing is easy to follow. Some of that, however, stems from a lack of detail regarding the simulator and the tasks. The description of the simulator, one of the core contributions, is barely a page long and mostly covers high-level concepts, physical quantities being modeled, and methods used to implement the simulator. I would have expected to see greater detail about this as currently it's hard to assess if this is a straight forward task of writing up a few equations or involved required complex derivations and developments. What also would be good is a discussion regarding the chosen way of modeling thin objects, as the approach selected is likely not the only one and a discussion of the pros and cons of modeling the interactions one way or another would be a valuable contribution.\n\nAnother aspect where more detail would be expected are the task descriptions. The tasks are easy to understand at a high level. However, their actual definition is omitted which makes it unclear what objective the various methods are required to optimize for later on. As there are many ways to formulate the described tasks it is important to have this information present.\n\nOne aspect of the experiments that could be improved is conveying what the goal is. The only information is that the goal is to evaluate the performance and behavior of the simulator and methods. Being more concrete and actually laying out the things to be investigated would improve the readability of the entire section.\n\nThe presentation of the results is at times incomplete. For example, Table 2 shows numbers with +/- values but there is no mention of what those values are. One can assume that it is mean +/- standard deviation, however, this should be stated clearly. In that table there is also a single row which has no +/- values, why? There are also various cells in the table that are lacking values, which is not explained. The results list score values for each of the tasks, but without knowing level of \"success\" these scores correspond to it is impossible to know whether the differences in scores are significant or not.\n\nThe presentation of the real-world experiments does not convey any information and fails to provide information. This section should either be improved to add actual experimental data, as opposed to referring the reader to an appendix with a handful of plots, and provide a discussion.\n\nThe paper describes a hybrid approach, though details are extremely limited. Due to the lack of detail of that approach and the focus being about the simulator, it is unclear why this was added nor that it adds anything. The idea of combining global search with local refinement is sensible. However, I would expect this to be common knowledge in the RL community given the (by default) unguided nature of RL exploration and the fact that RL tends to perform well once it finds a solution to a problem. There are also odd behaviors of the hybrid method that are not discussed. In some of the curves shown in Figure 3 the variance in the scores are excessively large, why does this occur?\n\nAnother question that should be addressed is the benchmark aspect. The paper states that the goal is for this to be used as a benchmark. Is the idea to use this similar to gym and atari game setups, or more like the benchmarks used in the computer vision community with tasks on withheld data to counteract overfitting on a set of tasks?\n\nThe core aspects of the paper, i.e. the simulator and task descriptions, are good yet could benefit from additional detail. The experiments are ok, though need to be improved to more clearly convey their information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JcwMZ7MmTM", "reviewer": "Reviewer_3BVm", "summary": "The paper introduces ThinShellLab, a differentiable simulation platform tailored for robotic interactions with diverse thin-shell materials possessing varying material properties, enabling flexible thin-shell manipulation skill learning and evaluation. The experiments highlight unique challenges in thin-shell manipulation, such as reliance on frictional forces, sensitivity to minimal variations in interactions, and frequent contact pair changes. The authors also study a hybrid optimization scheme that combines sampling-based trajectory optimization and gradient-based optimization. They also showcase the proposed simulation platform allows for a smooth transition from simulation to real-world robot deployment.", "questions": "1. In Sec 4.1, it seems that the observation includes a set of points. Does the order of points matter? How do the learning algorithms (SAC, PPO) used in the paper handle the observation if the input observations (points) are unordered?\n2. In Sec 4.1, does the 6DoF stand for the pose change in SE(3), and does the 7DoF include one more DoF for gripper closeness (mimic behaviors for two finger tips)?\n3. In Sec 5.1, the authors mentioned that they reported \"the maximum scores achieved within a specified number of timesteps\". Does it mean that the benchmark currently focuses on \"optimization\" rather than \"generalization\"? Besides, how do the authors count \"timesteps\" for CMA-ES and GD methods?\n4. What's the meaning of \"episode\" (x-axis) for GD and CMA-ES in Figure 3?\n5. In Sec 5.2, for \"our gradient-based method converges these parameters towards a uniform outcome.\", do the authors mean that the method converges to similar solutions despite different initialization?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed simulator can model diverse thin-shell materials with a wide range of bending-stiffness and bending plasticity, as well as frictional contact between thin-shell materials and end-effectors, which is hardly modeled in prior benchmarks.", "weaknesses": "1. The writing is not clear enough. Please see the *Questions* section for details.\n2. Figure 2 can be improved. The images are not well aligned. There is not a clear boundary between \"left\" and \"right\" mentioned in the caption, which makes it hard to distinguish 7 manipulation tasks and 3 inverse design tasks.\n3. It is better to have some success indicators for tasks in the benchmark. Otherwise, it is hard for readers to understand difference in performance between methods, if only rewards are provided (e.g., Table 2).\n\nMinor typo: In B.1, the table is linked to the section B.1 instead of Table 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "32QgZYRv5g", "reviewer": "Reviewer_j6Cr", "summary": "The paper shows a comprehensive benchmark framework for thin-shell object manipulation. A simulation environment is provided that expands upon previous work by including bending-stiffness and frictional contact. Since this simulation is differentiable, the authors show how gradient-based optimization can be used to enhance existing gradient-free methods for finding better performing policies for various robotics tasks. Real-world validation is shown on a specific subset of tasks for sim-to-real accuracy.", "questions": "How well does the simulation environment run in practice, is it achieving something close to real-time, or how long does simulation take? When parallelizing the simulations on GPU, does that only benefit CMA-ES and RL, or is GD also gaining some advantage from the parallelized environments?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "A much needed framework for robotic manipulation of general thin-shell objects is proposed by the authors, which will enable the robotics community to tackle more challenging problems in the future. Although the separate parts in the simulation is not a novelty in itself, the whole framework and its validation in benchmarks is a valuable addition to the field. The paper describes the tasks clearly, and the accompanying website is easy to understand and shows the core contributions well. A lot of analysis was provided as well to all the tasks, and on how and why the hybrid optimization method helps in solving the given problems.", "weaknesses": "The presentation in the paper does not feel quite as refined as the website, especially the figures, feel like they could use some work. For example, Figure 2 does not have the image bounding boxes align, and within the Sliding subfigure you can see how the edges of the images don't match up. A similar small issue is present in Figure 3 with a typo in \"Separatering\". Unfortunately, most of the presented tasks are dynamic, hence images don't capture them all that well, and hence the animations on the website do a more convincing job on why this particular application case matters.\nAnother point is that a lot of information is pushed to the Appendix, even though it could be beneficial to have in the main text. For example, the description of the hybrid method, which is a quite large engineering contribution of the paper to show working policies, is described on a high-level. A more detailed description would help here, and mentioning how much work it was to tune/balance the gradient-free and gradient-based steps could help the community. Another example is how the real-to-sim system identification is performed, how the disparity between simulation and reality is computed, these details should be included.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4ewyeC2z65", "reviewer": "Reviewer_BtTD", "summary": "This study focuses on teaching robots to manipulate a variety of thin-shell materials. It includes the introduction of ThinShellLab, a fully differentiable simulation platform that is specifically designed for robotic interactions with an array of thin-shell materials. The researchers conduct numerous experiments using reinforcement learning (RL) algorithms, in conjunction with sampling-based, gradient-based, and hybrid trajectory optimization methods, across 10 different thin-shell manipulation tasks.", "questions": "Interested to know what is the speed of the simulation? Is multiple GPU parallelization supported by the simulation?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The ThinShellLab is unique, standing as the first simulator of its kind to support a diverse range of thin-shell materials while also maintaining differentiability.\n\n2. Unlike existing simulators, it has the capability to support Bending Plasticity.\n\n3. This innovative simulator offers a benchmark for understanding prevalent methods associated with thin-shell objects, accommodating various types of approaches such as RL, and trajectory optimization.", "weaknesses": "1. The overall presentation is good. However, there could be improvements in clearly labeling the three system identification inverse tasks - Sliding, Bouncing, and Card - in both Table 2 and Figure 3. Their current placement together without clear labels within the table or graph is confusing. \n\n2. Furthermore, it would enhance the paper if some clear conclusions were drawn, along with providing some directions or suggestions to guide readers to explore each type of method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xGyMxZtuu7", "reviewer": "Reviewer_k5Lf", "summary": "This work develops ThinShellLab, a simulator and benchmark for robotic learning in thin-shell material manipulation.\nThe simulator models volumetric and thin-shell materials with finite-element methods with tetrahedral and triangular elements, and is differentiable.\nThe robotic manipulation benchmark defines tasks including lifting, separating, following, folding, pick-folding, and forming. The benchmark is used to evaluate manipulation methods including sampling-based, gradient-based, and hybrid trajectory optimization, as well as reinforcement learning methods.\nThe simulator is also shown to support system identification, which allows transferring certainly simulated policies to be deployed in the real world.", "questions": "- The contact model uses quadratic energy instead of barrier. How does the system correctly identify that 2 triangles are penetrating? Based on the paper, it seems it is based on normal direction, however, this does not seem correct since the normal would flip if one of the contacting shell is flipped.\n- The paper describes the system as simulating tetrahedral and triangular elements, how is the rigid-body action space achieved?\n- Newton's method requires solving a large linear system. How is this implement on the GPU? Is it solved through sparse matrix algorithms, conjugate gradient methods, or some other method?\n- What is the speed/performance of this simulator? What are the bottlenecks?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- This work is the first to properly implement a differentiable thin-shell object simulator for robotic manipulation.\n- The paper is overall written clearly.\n- The manipulation tasks are well-designed to reflect properties of thin-shell materials. The benchmarked methods have good coverage.", "weaknesses": "- Some details of the simulator are not clear (see Questions section).\n- Some useful simulation features are not shown in this paper.\n  - It seems the system does not fully support/couple with rigid-body simulation, which is important for modeling robots.\n  - The paper does not show any tasks involving constraint simulation (e.g., cloth with one end fixed).\n- Real-world manipulation only contains a single case. It is not convincing that in general the described system identification method will improve sim-to-real policy deployment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)