Differentiable Discrete Elastic Rods for Real-Time Modeling of Deformable Linear Objects
Authors: Yizhou Chen
Yiting Zhang
Zachary Brei
Tiancheng Zhang
Yuzhen Chen
Julie Wu
Ram Vasudevan
Venue: CoRL 2024 (2024)
Keywords: Deformable Linear Objects Modeling
Physics-Informed Learning
Differentiable Simulation
TL;DR: This paper proposes a novel framework that combines a differentiable physics-based model with learning framework to model DLOs accurately and in real-time.

Abstract: This paper addresses the task of modeling Deformable Linear Objects (DLOs), such as ropes and cables, during dynamic motion over long time horizons. This task presents significant challenges due to the complex dynamics of DLOs. To address these challenges, this paper proposes differentiable Discrete Elastic Rods For deformable linear Objects with Real-time Modeling (DEFORM),  a novel framework that combines a differentiable physics-based model with a learning framework to model DLOs accurately and in real-time.  The performance of DEFORM is evaluated in an experimental setup involving two industrial robots and a variety of sensors. A comprehensive series of experiments demonstrate the efficacy of DEFORM in terms of accuracy, computational speed, and generalizability when compared to state-of-the-art alternatives. To further demonstrate the utility of DEFORM, this paper integrates it into a perception pipeline and illustrates its superior performance when compared to the state-of-the-art methods while tracking a DLO even in the presence of occlusions.  Finally, this paper illustrates the superior performance of DEFORM when compared to state-of-the-art methods when it is applied to perform autonomous planning and control of DLOs.

Differentiable Discrete Elastic Rods for Real-Time Modeling of Deformable Linear Objects
Yizhou Chen
Yiting Zhang Zachary Brei Tiancheng Zhang
Yuzhen Chen Julie Wu Ram Vasudevan
Department of Robotics, University of Michigan, Ann Arbor, MI 48109, United States {yizhouch, yitzhang, breizach, zhangte, yuzhench, jwuxx, @unvijumichh chh edu https: htte://rrahmlapp www..aaa.....ha...a........ github..hh......gg. i0/DEFORM/
Abstract:
This paper addresses the task of modeling Deformable Linear Objects (DLOs), such ropes and cables, during dynamic motion over long time hori- zons. This task presents significant challenges due to the complex dynamics of DLOs. To address these challenges, this paper proposes differentiable Discrete Elastic Rods For deformable linear Objects with Real-time Modeling (DEFORM), a novel framework that combines a differentiable physics-based model with a learning framework to model DLOs accurately in real-time. The performance of DEFORM evaluated in an experimental setup involving two industrial robots and variety of sensors. comprehensive series of experiments demonstrate the efficacy of DEFORM in terms of aceuracy, computational speed, and general- izability when compared to state-of-the-arr alternatives. To further demonstrate the utility of DEFORM, this paper integrates it into a perception pipeline and il- lustrates its superior performance when compared to the state-of-the-art methods while tracking DLO even in the presence of occlusions. Finally, this paper il- lustrates the superior performance DEFORM when compared to state-of-the-art methods when is applied perform autonomous planning and control of DLOs.
Keywords: Deformable Linear Objects Modeling, Physics-Infooe Informed Learning. Differentiable Simulation
88
This paper introduces DEFORM, novel framework that combines a diffcrentiable physics-based model with learning framework to model and predict dynamic DLO behavior accurately in real-time. The figure shows DEFORM's predicted states (yellow) and the actual states (red) for DLO over 4.5 seconds 100 Hz. Note that the prediction is performed recursively, without requiring during process. video related experiments can be found the supplementary material.
Figure 1:
ground truth perception
Introduction
Robotic manipulation tasks such surgical suturing vehicle wire harness assembly [1, 2, 3] are challenging problems because they require accurately and dynamically manipulating DLOs over
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

a)
DER
b)
Numerical Integration with Residual Learning
<)
Enforce Inextensibility with Momentum Conservation
bend
twist
+ (?), (8)
(9), (12), (13)
(1)
2
3
Figure 2: Overview of DEFORM contributions (green), a) DER models discretize DLOs into vertices, segment them into clastic rods, and model their dynamic propagation. DEFORM reformulates DER into Differentiable DER (DDER) which deseribes how to compute gradients from the prediction loss, enabling efficient system identification and incorporation into deep lcarning pipelines. b) To compensate for the error from DER's 's nu- merical integration, DEFORM introduces residual learning via DNNs. c) 1 + 2: DER enforces inextensibility, but this does not satisfy classical conservation principles. + 3: DEFORM enforces inextensibility with momentum conservation, which allows dynamic modeling while maintaining simulation stability.
long time horizons (> 1s) [4]. Nonlinear dynamics make manipulating DLOs challenging because computationally expensive models are required for accurate prediction. Further, practical applica- tions such as vehicle wire harness assembly often result in DLO occlusions, meaning perception systems cannot robustly estimate the entire DLO configuration. For robotic manipulators to suc- cessfully manipulate DLOS, a novel modeling method capable of quick and accurate predictions of dynamic DLO behavior, even in the presence of occlusions, is needed.
To model DI.Os, physics-based approaches can be used, but they suffer a tradeoff between costly computation and low accuracy when predicting dynamic movements. State-of-the-ar learning- based modeling approaches have used deep neural networks (DNNs) to model interaction propa- gation in DLOs [5, 6, 7]. Despite their success, these methods require large datasets of real-world manipulation data, which can be impractical to collect and are not robust to variations in DLO prop- erties. Alternatively, using a physics-based model as a prior for DNNs is promising due to increased sampling efficiency and generalizability this approach has not been applied to predict behavior of DL.Ss during dynamic motions.
[8. 9, 10, 11, 12]. However, to the best of our knowledge,
A recent promising physics- based modeling method is Diserete Elastic Rods (DER). This approach qualitatively and efficiently describes the 3-D behavior of DLOs [13, 14). Despite its potential for thysics-informe learning, applying DER theory to model DLOs in the real-world is challenging for three reasons. First, tuning DER model parameters to match a real-world DLO is difficult. Second, computing analytical gradients for DER models challenging. Third, numerical errors arise from discrete numerical integration of DER models and from enforcing DI.O inextensibility. These drawbacks affect the accuracy of the DER model, limiting the utility of embedding a DER model in learning-bass based framework. In particular, lack of analytical gradients prevent learning over multiple steps, which reduces the accuracy of long horizon predictions.
To address the above challenges, this paper introduces Differentiable Discrete Elastic Rods For Deformable Linear Objects with Real-time Modeling (DEFORM), a novel framework that uses a differentiable physics-based learning model, with real-time inference speed, to accurately predict dynamic DLO behavior over long time horizons. As illustrated in Figure 2, the main contribu- tions of DEFORM are four-fold. First, novel DLO model is presented, called the Differentiable Discrete Elastic Rod (DDER) model that makes DER models differentiable with respect to any vari- ables. Second, a numerical integration error compensation method that uses DNNs is developed to improve accuracy when numerically integrating DER models. Third, a novel inextensibility en- forcement algorithm is presented for DER models that guarantees momentum conservation, thereby enabling accurate prediction of dynamic DLO behavior. Fourth, comprehensive set of experiments are presented, which illustrate that DEFORM has superior performance in terms of accuracy, com- putational speed, and sampling efficiency. We also demonstrate the utility of DEFORM for robust DLO tracking and precise shape-matching manipulation tasks.

---

2 Related Work
Learning-Band Bastd Modeling. Latent dynamies modeling approaches, such as graph neural networks (GNN) [15) and Bidirectional LSTM (Bi-LSTM) [6], reason over vertices spatially formulate latent dynamics of DLOs for predicting their behavior in 2-D. Unfortunately, we illustrate in this paper through real-world experiments, these methods are unable to accurate describe the behavior of DL.Ss during dynamic motions in 3D.
Physics-Based Modeling. Traditionally, DLOs have been modeled using physics- basee approaches. Examples include mass-spring system models [16, 17, 18], Position-Based Dynamics (PBD) mod- els [19, 20, 21]. Discrete Elastic Rod (DER) models [13, 14, 22, 23], Finite Element Method (FEM) models [24, 25, 26]. Although easy to implement, the mass mass-spring system imposes unneces- sary stiffness to enforce inextensibility. resulting in instability during simulation. FEM models ac- curately modeling DLO behavior, but are impractical for real-time prediction due to their prohibitive computational costs. PBD and DER methods have been applied in planning and control frameworks for manipulation tasks due to their computational efficiency [20, 21, 27, 28, 29]. However, PBD models are sensitive to parameter selection and assume quasi-static behavior. As illustrated in this paper, this results in low accuracy during dynamic manipulation. Though DER models have been qualitatively validated to describe the 3D behavior of DLOs, we illustrate that their quantitative performance while describing dynamic motion is lacking.
Differentiable Modeling Framework. Differentiable models are computational frameworks where every operation differentiable with respect to the input parameters. Importantly. differentiable model enables integration into learning framework to improve learning and can enable efficient pa- rameter auto-tuning via gradient-based optimization during training. There are differentiable models for deformable objects, such as DiffCloth [16, 17] and XPBD [21]. but these models tend to suffer from numerical instability. This makes it challenging to apply these modeling frameworks while trying to perform parameter identification for real-world DL.O manipulation. To address these lim- itations, this paper proposes a differentiable DER model, called DDER, which is incorporated into a learning framework to achieve numerical stability. This results in better predictive performance for dynamic DLO behavior when compared state-of-the-art methods and enables auto-tuning of DLO parameters using limited real-world data.
3 Preliminaries
Deformable Linear Object (DLO) Model. To model DLO, we separate the DLO into an indexed set of n vertices at each time instance. The ground truth locations of vertex at time is denoted by x E TR3 and the set of all n vertices is denoted by X, Let M E RR33 denote the mass matrix of vertex i. Our objective is model the dynamic behavior of a DLO that is being controlled by robot with following assumption: the first and last edge of the DLO are each held by distinct end effeecor. Note this assumption is commonly made during DLO manipulation [30, 20, 19, 7, 31]. Using this assumption, let the orientation and position of the two end effectors at time t be denoted by u, E 12 which we refer to as the input. The velocity of the vertices of the DLO can be denoted as V = (X, X,1)/Ar At. Note that V, is an approximation of the actual velocity. Finally, note that we distinguish between ground truth elements and predicted elements by using the eircumflex symbol (e.g., X, is the ground truth set of vertices at time and X, is the predicted set of vertices at time t).
Modeling DLOs with DER for Dual Manipulation. DLOSs primarily exhibit three distinct de- formation modes: bending, twisting, and stretching [13]. This paper focuses on DLOs that are inextensible (e.g., cables and wires), and therefore do not experience stretching. DER models intro- duce a single scalar 0; E IR to represent bending and twisting in R This parameter describes the potential energy P(X,, u., 0r, ax) of the DLO that arises due to bending or twisting, where a denotes the vector of the material properties of each vertex of the DLO (i.e., their mass, bending modulus, and twisting modulus) [13, Section 4.2). Notably, DER models assume that the DLO reaches an equilibrium state between each simulation time step obtain 0;
3

---

0 (X,, a:) = arg min P(X,, a) 0,
(1)
Once the potential energy expression is derived, then the restorative force experienced during defor- mation is set equal to the negative of the gradient of the potential energy expression with respect to the vertices. Therefore, the equations of motion of the DLO are:
aP(X, u. (x, ox) 0x. After solving (1), numerically integrate this formula to predict the velocity and position by applying the Semi-Implicit Euler method:
MX.
(2)
V++ = V A.M aP(X,, (X, a))
(3)
ax,
X = X A, 6+1-
(4)
(3)-(4) can be recursively solved to simulate the behavior of the DER model of a DLO.
Inextensibility Enforcement. During simulation, the DLO's length may change due to numerical errors, but this undesirable when modeling inextensible DLOs. To ensure constant length, DER methods project a computed solution onto constraint that enforces inextensibility(32), How- ever, this projection operation because it does not ensure that momentum is conserved. This paper adopts constraint solver based on a PBD model [33], which is described in Section 4.3, to enforce inextensiblity while conserving momentum. This improves prediction accuracy compared to classical methods.
introduce instabilities during simulation of dynamic behavior
Methodology
This section first introduces DDER, formulation of DER to ensure differentia- bility, which enables efficient DER model parameter estimation using real-world data. Next, this section describes how to incorporate a learning-assisted integra-
Algorithm One Step Prediction with DEFORM
Inputs: Xt, Vr, u,
0, (X,) arg ming, P(X,, O2, a) -1 aP(X. o0 (X..a)..a)
D Sectionn.411
2: Ve+1 = A.M
>> (3)
2x.
D. i+ + DNN 4: Inextensibility Enforcement on X
3: X = A.
>> Section.4.2
tion method to improve the speed of DLO simulation and accuracy. Finally, this section describes position-based straint that enforces inextensibility while preserving momentum, which results in improved numerical stability during simulation. These results are combined in a prediction algo- rithm that is described in Algorithm 1. This algorithm can be run over multiple steps to generate an accurate prediction of the state of the DLO over long time horizon.
& Section.4.3
5: Ve+1= = =(X++ X.)/A, D Velocity Update
return X VinI
4.1 Differentiable Discrete Elastic Rods (DDER)
We first develop reformulation of DER such that the DER model is differentiable with respect to any model variables (i.e., material properties, and DNN weights). We call this extension Differen- tiable Discrete Elastic Rods (DDER). To do this, we implement DDER in PyTorch [34] by taking advantage its automatic differentiation functionality. Unfortunately, simulating the DDER model to predict X using (3) and (4) also requires solving an optimization problem (1). As result, to minimize the prediction loss, we need to solve bi-level optimization problem at each time For instance, if we train learning model to identify the material parameters by minimizing prediction loss between experimentally collected ground truth data X,+1 and model prediction X,41 for each time i, then the following bi-level optimization problem needs to be solved:
min
X X (0))
(5)
0: = arg min P(X,, u. 0, a.)
(6)
0.

---

Note that X(+1 depends on the previously computed 0 due to (3) and (4). To compute the gradient of this optimization problem with respect to a, one can perform implicit differentiation, which we implement using Theseus [35].
4.2 Integration Method with Residual Learning
To compensate for numerical emrors that arise from discrete integration methods and to improve the numerical stability of the algorithm that we use to simulate the DDER model, we incorporate learning-based framework into the integration method as follows:
V++ = V. - A,M- ( aP(X,, 0g (Xe, a), σ)) + DN)(X...)):
Ox,
X,+1 = X, + A (VetI + DNN(X, V.,a)).
(8)
where DNN denotes deep neural network. We utilize a structure similar to residual learning [36], with major difference: we define shortcut connection that is equal to the integration method that is derived from the DDER physics-basel modeling. Unlike pure learning methods, the short cut connection grounds the DNN in physical laws while also leveraging data-driven insights. As shown in the ablation study results, this approach significantly improves modeling performance. Additioanl discussion of section can be found Appendix B.5. Implementation details of the DNN structure can be found in Appendix B.I. Note that this contribution can also be applied standard DER models, but does not improve performance as much as it does when combined with our DDER modeling approach as we illustrate in this paper.
4.3 Improved Inextensiblity Enforcement with PBD
To enforce inextensibility and conserve momentum after simulating the DLO forward for a single time step (8), we enforce an additional constraint on the DLO in a manner similar to the PBD method [33]. First, we define a constraint function between spatially successive vertices at each time step: ((
[++
*++1
||e/121
(9)
where é denotes the edge between vertices and + when the DLO is undeformed. Note that this constraint is non-zero if the distance between successive vertices of the DLO changes. The PBD method iteratively modifies the positions of each of the vertex using a pair of correction terms, Ax and Ax until the following constraint is satisfied:
C(x + A&, X,+1 + A& <,
-++1
(10)
where >> 0 is a user-specified threshold. To calculate the correction terms, PBD typically applies a Taylor Approximation of the constraint and sets this equal to zero (i.e., the correction to the vertex locations are chosen to ensure that the constraint is satisfied). The correction term is then given by:
AR
C(
1+1
(11)
&411l2
6+1
Notably, this correction the vertex location does not ensure that the línear and angular momen- tum is preserved. To ensure that linear and angular momentum is conserved after the correction is applied, one needs to ensure that the following pair of equations are satisfied:
n-2
2
Σ M' AS = O and Σ r' x (M' A =
(12)
where r' is the vector between 2,41 and common rotation center. Note in particular that if the mass matrix associated with each vertex is not identical, then the correction proposed in (11) leads to violation of (12). To address this issue, we modify the correction proposed in (11) by multiplying it by 3 E R that is inversely correlated with the weight ratio between successive vertices (i.e.,
Mr let B (|A| (1 and gi+1 (IM* [|M*||=||M++]]]] Then, if we let the correction between
M++1
successive steps be equal 3 A one can ensure that (12) satisfied. Verification proof can be found in Appendix B.2. Pseudo code to enforce inextensibility while preserving momentum can be found in Appendix B.3.
5

---

SERORM
97
33
53
7um
0-4.57M
DLO robot-human manipulation
(b) DLO 1. dual-rrbot manipulation
B4LSTW
2UNN
(c) robot--uuune munipulationn
manipalation
Figure 3: Visualization of the performance of DEFORM and Bi-LSTM in predicting trajectorics for DLOs and 2. The ground truth vertices of the DLOs are marked with red hollow circles. The predicted vertiees marked with orange solid circles for DEFORM and blue solid circles for the Bi-LSTM model, respectively,
5 Experiments and Results
This section covers the real-world evaluation of DEFORM, including: I) a quantitative comparison of modeling accuracy with state-of-the-art methods, 2) computational speed assessment, 3) an abla- tion study on parameter auto-tuning, residual learning, and multi-step training, 4) using DEFORM track DLOs with RGB-D camera, and 5) shape matching manipulation demonstration. Code and data will be provided upon final paper acceptance. Additional information is in Appendix C.
5.1 Experiment Setup:
Hardware Setup. Three distinct cables and distinct ropes were constructed validate the modeling accuracy of DEFORM. Spherical markers were attached to the DLOs and used to create training and evaluation datasets. We set the number of vertices equal to the number of MoCap markers. We use an Opti motion capture (MoCap) system to obtain the ground truth vertex locations. Because the markers are incorporated during training and evaluation, their effect on the dynamics is incorporated into all subsequent evaluations. Note that for real-world applications, the markers would be needed, A Franka Emika Research robot and Kinova Gen3 robot are used for dual manipulation.
Dataset Collection and Training. For each DLO, we collect 350 seconds of dynamic trajectory data in the real-world using the motion capture system at a frequency of 100 Hz. The model for each of the five DLOs trained separately using the trajectory data collected from our real-world experiments. The dataset is split with 80% for training and 20% for evaluation. In training, models are trained using the same 100 steps (1s) of dynamic motion, which included slow and fast DLO motions. The L.1 loss over the 100 step prediction horizon is used for training. For evaluation, the trained model is used to predict DLO behavior for 500 steps without accessing ground truth.
Baseline Comparisons. Physics-Based Modeling: The XPBD [21] approach constructed us- ing PBD, formulating constraints quasi-statically to model stretching and bending behavior. The DRM [19] method is rooted in the concept of directional diminished rigidity such that the dynamics of a vertex is dependent on its configuration relative the gripper's configuration. The original DER [13] modeling approach is included baseline to demonstrate its performance when com- pared to other analytical methods and to DEFORM. Baarning-Basee Modeling: Yan et al. [6] and

---

Wang et al. [15] use Bi-LSTM and GNN to represent DLO and reason about interactions between the vertices respectively. Physics-based models with enhancements: To further demonstrate the capabilities DEFORM, we enhanced XPBD and DRM with residual learning and material prop- erty parameter tuning, which we denote as XPBD-NN and DRM-NN, respectively.
5.2 Modeling Results
Table (Left) summarizes the results of using DEFORM when compared to baselines using the average 1.1 loss. Performance is evaluated when predicting the state of the wire over 5s predic- tion horizon. Note that DEFORM outperforms all baselines for each of the five evaluated cables. An illustration of the performance of these methods can be found in Figure Next, compare computational speed for one step inference. Table (Right) indicates that Bi-LSTM has the fastest speed. Note that DEFORM is able to generate single step predictions at a 100Hz frequency.
Table 1: Left: Baseline comparison results for modeling real-world DLO. Right: Computational Speed.
Modeling Accuracy (10 mm)
Computational Speed (10 ?s)
Method
2
4
3
5
XPBD[21]
4,00
3.85
3.80
3.62
4.35
1.45
1.38
1.33
1.31
1.36
DRM[19]
3.64
4.16
3.21
4.02
4.19
0.45
0.44
0.43
0.44
0.41
[6-
1.98
1.75
1.10
1.75
1.88
0.04
0.03
0.03
0.03
0.03
GNN[I5]
3.41
2.23
2.15
2.49
2.68
1.69
1.67
1.59
1.60
1.55
DER[13]
1.96
1.91
1.86
1.50
1.65
1.81
1.77
1.76
1.79
1.73
NN-XPBD
2.50
2.39
2.18
2.62
2.73
1.99
1.85
1.91
1.86
1.83
NN-DRM
2.91
3.22
2.64
2.99
3.31
0.92
0.82
0.81
0.84
0.79
DEFORM
1.01
0.97
0.77
0.85
0.99
0.97
0.92
0.94
0.95
0.92
Table 2: Ablation Study.
Ablation Study. Results are summarized in Ta- ble and show the effect of each DEFORM contribution. Note that a standard DER method included as baseline and does not include any of the DEFORM contributions. The results of the study show that residual learning has the biggest improvement the baseline per- formance, but the auto-tuning of material prop- erties has non-trivial affect as well. Further, the improved inextensibility enforcement makes large improvement for more compliant DLOs, such as DLO1. This is because the more compliant DLOs swing more during dynamic behavior, so ensuring momentum is conserved greatly improves prediction accuracy. The results for DL.O4 and DLO5, well an additional ablation study, are in Appendix C.3 and C.4, respectively. Note that the stiffnesses of the tested DLOs can be found in Appendix Table 4.
Accuracy (10 (00-2m)
Method
1
2
3
DER
1.96
1.91
1.86
W/O Residual Learning
1.54
1.77
1.42
W/O System ID
1.21
1.32
1.09
Original Inextensibility
1.65
1.23
1.00
DEFORM
1.01
0.97
0.77
State Estimation. We next evaluate the performance of DEFORM when it is used in concert with sensor observations for state estimation. These experi- ments included occlusions of the DLO that arose from the manipulators or humans moving in front of the sensor. this experiment, compare how well the state estima- tion framework described in Appendix D does when ing the DEFORM model versus the Bi-LSTM model. Bi- LSTM was chosen because it was the second best model in terms model accuracy. We also compare how well each these methods do with varying frequencies of sen- sor measurement updates. As in the earlier experiment, evaluate the LI norm of the prediction over a 5s long prediction horizon. Experimental results shown in Figure 4. Unsurprisingly, each method improves when
0.020
0-LSTM
0.005
0.000
Percestion FPS
Figure 4: The average L1 loss (m) while per- forming state estimation.

---

Figure lapse comparison DEFORM model and the Bi-LSTM model when each is used to perform a shape matching task. The goal configuration is green whereas the start configuration is pink. The DEFORM model enables the planning algorithm to successfully complete the task whereas the planning algo- rithm using the Bi-LSTM model fails complete the task.
given access to more frequent sensor updates. This illustrates that the state estimation algorithm described in Appendix D.2 behaves appropriately. Notably, the DEFORM model with no state es- timation does better than the Bi-LSTM model even when the Bi-LSTM model is given access to fps updates. The method using the DEFORM model with access to 30 fps sensor measurements is the most effective.
3-D Shape Matching Manipulation. To demonstrate the practical applicability of DEFORM, illustrate its use for DLO 3-D shape matching tasks in both simulation and the real world.
This experiment uses ARMOUR [37], receding-horizon tra- jectory planner and tracking controller framework, manipu- late DI.Os. ARMOUR uses either DEFORM or Bi-L-TTM to predict the manipulated DLO state and tries to match the fi- nal DLO configuration with a desired configuration. Details of combining ARMOUR with the DLO simulators are in Ap- pendix C.6. We compare the performance of each method on 100 random cases in PyBullet simulation and 20 random cases in the real world. If the Euclidean distance between each individual pair of ground truth and desired DLO vertices is less than 0.05 meters, then the trial is success. ARMOUR is given 30 seconds to complete the task, otherwise the trial is marked as a failure. The results are summarized in Table 3 and Figure shows one of the real-world results. A video of these experiments can be found in the supplementary material.
Table 3: Shape matching manipulation success rate
Success Rate
Method
Real
Sim
Bi-LSTM DEFORM
10/20
78/100
17/20
90/100
Conclusion and Limitations
This paper presents DEFORM, method that embeds residual learning in novel differentiable De- formable Linear Object (DLO) simulator. This enables accurate modeling and real-time prediction of dynamic DLO behavior over long time horizons during dynamic manipulation. To demonstrate DEFORM's efficacy, this paper performs comprehensive set of experiments that evaluate accuracy, computational speed, and its usefulness for manipulation and perception tasks. When compared to the state-of-the-art, DEFORM is more accurate while being sample efficient without sacrificing computational speed. This paper also illustrates the practicality of applying DEFORM for state estimation track an occluded DLO during dynamic manipulation. Furthermore, DEFORM is combined with manipulation planning and control framework for 3D shape matching of DLO in both simulations and real-world scenarios, achieving a higher task success rate compared to the closest-performing baseline.
Limitations. We recognize several opportunities to enhance DEFORM. Currently, DEFORM could be improved by adding contact modeling to accurately predict interactions between different com- ponents under load [38]. Additionally, DEFORM does not model multi-branched DL.Os, which common in wire-harnes assembly [39]. Furthermore, DEFORM assumes that the ends of the DLO are grasped, which could restrict manipulation dexterity. We look forward to addressing these aspects in our future work to improve DEFORM model's 's accuracy and applicability.

---

Acknowledgments
The authors would like to gratefully thank reviewers for giving useful comments. This work is supported by the Ford Motor Company and the Air Force Office of Scientific Research under the Award No: MURI FA9550-23-1-0400.
References
[1] Y. Adagolodjo, L. Goffin, M. De Mathelin, and H. Courtecuisse. Robotic insertion of flexible needle in deformable structures using inverse finite-element simulation. IEEE Transactions Robotics, 35(3):697-708, 2019.
[2] L. Cao, X. Li, T. Phan, A. M. H. Tiong, J. Liu, and S. J. Phee. A novel robotic suturing system for flexible endoscopic surgery. In 2019 International Conference Roboties and Automation (ICRA), pages 1514-1520. IEEE, 2019.
[3] N. Lv, J. Liu, and Y. Jia. Dynamic modeling and control deformable linear objects for single-arm and dual-arm robot manipulations. IEEE Transactions Robotics, 38(4):2341- 2353, 2022.
[4] M. Saha and Isto. Manipulation planning for deformable linear objects. IEEE Transactions on Robotics,
2007.
[5] W. Ford. Graph neural networks: A review of methods and applications. Al Open, 1:57-
81, 2015. www.htppww/oloaom////////////0/0 iriencedireeco http://ww..........c.eccceaaaa/a
URL https// //ww.
[6] M. Yan, Y. Zhu, N. Jin, and J. Bohg. Self-supervised learning of state estimation for manipu- lating deformable linear objects, 2020.
[7] Y. Yang, J. Stork, and T. Stoyanov. Learning to propagate interaction effects for modeling deformable linear objects dynamics. pages 1950-1957, 05 2021. doi: 109////999/. 0021.9561636
[8] T. Tang, Y. Fan, H.-C. Lin, and M. Tomizuka. deep learning framework for solving forward and inverse problems involving nonlinear par- tial differential equations. Journal of Computational Physics, 378:2427-2433, 2017. doi: (W00//ROS.001..8000000
informed neural networks: A
[9] T. Bikmukhametov and J. Jäschke. Combining machine learning and process engineering physics towards enhanced accuracy and explainability of data-driven models. Computers Chemical Engineering, 138: 106834, 2020. ISSN 0098-1354. doi.https://doi.rrg/// 1016/j. compchemeng.2022 2020. 106834. URL https://wwwt sci wwwp:./ieeeneeeree.............. www.eeccttcccceeeeeeeeeeeeeeeeee science/ence/ arriclleppa/$000..33411133
[10] A. Zeng, S. Song, J. Lee, A. Rodriguez, and T. Funkhouser. Tossingbot: Learning to throw arbitrary objects with residual physics. 2019.
[11] B. Bianchini, M. Halm, and M. Posa. Simultaneous learning of contact and continuous dynam- ics. In 7th Annual Conference on Robot Learning, 2023. URL https: openreview. net / Eorum? wuuuummiiiiee -id-- 1166_D666600
[12] Gao, M. Y. Michelis, A. Spielherg, and R. K. Katzschmann. Sim-to-real of soft robots with learned residual physics, 2024.
[13] M. Bergou, M. Wardetzky, S. Robinson, Audoly, and Grinspun. Discrete elastic rods. In ACM SIGGRAPHI 2008 Papers, SIGGRAPH '08, New York, NY, USA, 2008. Association for Computing Machinery. ISBN 9781450301121. 1tt.ii/ogg1o..1155////111 1360666.
111.1.11///11900044 191199999999990196222 URL https:

---

[14] M. Bergou, B. Audoly, E. Vouga, M. Wardetzky, and E. Grinspun. Discrete viscous threads. ACM Trans. Graph., 29(4), jul 2010. ISSN 0730-0301. 10011...15/1777765.177877 URL
https: /tt///d.../ org/ /tgg/10.
1778853.
[15] C. Wang, Y. Zhang, X. Zhang, Wu, X. Zhu, S. Jin, Tang, and M. Tomizuka. Offline- online learning of deformation model for cable manipulation with graph neural networks. IEEE Robotics and Automation Letters, (00055544-5511 Apr. 2022. ISSN 2377-3774. 10i:10.1109) Ira.2022.318337. URL http: dx doi. org/oo/g///..o /10. 110111199//1 1L.09/LR.... 3158377..11
[16] D. Baraff and A. Witkin. Large steps in cloth simulation. In Proceedings of the 25th Annual Conference on Computer Graphies and Interactive Techniques, SIGGRAPH '98, page 43-54, New York, NY, USA, 1998. Association for Computing Machinery. ISBN 0897919998. doi: 241145/220814.288822 URL https:: /dts::///do.. oegpoi..rgg/11 111.11....1111111 280821.
[17] J. Liang, M. Lin, and V. Koltun. Differentiable cloth simulation for inverse problems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, edi- tors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: httpss:/proceeedngggg wewerieeeesppeeeeee wwwpccpppppppeeeeeeeeeeeeeeeeeee wel_ffieesppppee fiiees/paerr/211 pdf.
[18] T. Liu, A. W. Bargteil, F. O'Brien, and L. Kavan. Fast simulation of mass-spring sys- tems. ACM Transactions on Graphics, (2(6))20 Nov. 2013. URL http: //cg./c.///// cis. upenn. edu caluupublicann wwwpppulliiinnnaooac Liii FMS. Proceedings ACM SIGGRAPH Asia 2013, Hong Kong.
[19] M. Ruan, D. McConachie, and D. Berenson. Accounting for directional rigidity and con- straints in control for manipulation of deformable objects without physical simulation. In 2018 IEEF/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 512-519,
2018. doi: 109/TROO 2018 8594520..
[20] D. Berenson. Manipulation of deformable objects without modeling and simulating deforma- tion. In 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 4525 4532, 2013. 109.0..11///RO..000000 001.0..021..600000
[21] Liu, E. Su, Lu, M. Li, and M. C. Yip. Differentiable robotic manipulation deformable rope-like objects using compliant position-ber based dynamics, 2022.
[22] D. Tong, A. Choi, I. Qin, W. Huang, J. Joo, and M. K. Jawed. Sim2real neural controllers for physics-based robotic deployment of deformable linear objects, 2023.
[23] N. Lv, J. Liu, and Y. Jia. Dynamic modeling and control deformable linear objects for single-arm and dual-arm robot manipulations. IEEE Transactions on Robotics, 38(4):2341- 2353, 2022.
[24] M. Gareía, C. Mendoza, L. Pustor, and A. Rodríguez. Optimized linear fem for modeling deformable objects. Computer Animation and Virtual Worlds, 17(3-4):393-400 2006.
[25] S. Sin, D. Schroeder, and Barbič. Vega: non-linear fem deformable object simulator. In Computer Graphics Forum, volume 32, pages 48. Wiley Online Library, 2013.
[26] A. Koessler, N. R. Filella, B.-C. Bouzgarrou, L. Lequièvre, and J.-A. C. Ramon. An efficient approach to closed-loop shape control of deformable objects using finite element models. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 1637-1643. IEEE, 2021.
[27] A. Sintov, Macenski, A. Borum, and T. Bretl. Motion planning for dual-arm manipulation of elastic rods. IEEE Robotics and Automation Letters, PP:1-1, 07 2020. doi: 10.1 109/LRA. 2020.301 120 13352.
10

---

[28] T. Bretl and Z. McCarthy. Quasi-static manipulation of a kirchhoff elastic rod based on geometric analysis of equilibrium configurations. The International Journal of Robotics Re- search, 33:48 68, 2014. URL https:/ / www.ntsccchoolllr.ccooooo wtt.a//ooo.....ooooooooo...t.... CorpusID: 856279.
[29] D. Tong, A. Choi, L. Qin, W. Huang, J. Joo, and M. K. Jawed. Sim2real neural controllers for physics-based robotic deployment of deformable linear objects, 2023.
[30] C. Chi, B. Burchfiel, E. Cousineau, S. Feng, and S. Song. Iterative residual policy for goal- conditioned dynamic manipulation of deformable objects. In Proceedings of Robotics: Science and Systems (RSS), 2022.
[31] Y. She, Wang, S. Dong, N. Sunil, Rodriguez, and Adelson. Cable manipulation with tactile-reactive gripper, 2020.
[32] R. Goldenthal, D. Harmon, R. Fattal, M. Bercovier, and E. Grinspun. Efficient simulation
of inextensible cloth. ACM Trans. Graph., 26(3):49-es es, jul 2007. ISSN 0730-0301. doi: .17.1177666 4438.
10. 10.1145/1276377 1276433.. URL https:://// /twii org/oo.g///...ooo.oooooooo
[33] M. Müller, Heidelberger, M. Henmix, and J. Ratcliff. Position based dynamics. J. Vis. Comun. Image Represent., 18(2):009-118, apr 2007. ISSN 1047-3203. doi:10. 1016/..vvii.. 2007.01.005. URL https:: /tt.///oo.iioo org/ / 10. 1016 6 jvcir. 2007.000.00 00007.01.05
[34] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. Deto M. Raison, A. Te- jani, S. Chilamkurthy, B. Steiner, L. Fang. J. Bai, and S. Chintala. Pytorch: An imperative style, tigh-performance deep learning library, 2019.
[35] L. Pineda, T. Fan, M. Monge, S. Venkataraman, Sodhi, R. Q. Chen, J. Ortiz, D. DeTone, A. Wang, Anderson, J. Dong, B. Amos, and M. Mukadam. Theseus: A library for differen- tiable nonlinear optimization, 2023.
[36] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition, 2015.
[37] J. Michaux, P. Holmes, B. Zhang, C. Chen, B. Wang, S. Sahgal, T. Zhang, S. Dey, S. Kousik, and R. Vasudevan. Can't 't touch this: Real-time, safe motion planning and control for manipu- lators under uncertainty, 2023.
[38] A. Choi, D. Tong, M. Jawed, and J. Joo. Implicit Contact Model for Discrete Elastic Rods in Knot Tying. Journal of Applied Mechanics, 88(5):050010, 03 2021. ISSN 0021-8936. 10i1001115/1.400000
URL https: //wi/iiio////
1tt.//...111111..111111111 4050238.
[39] O. Salunkhe, J. Stahre, D. Romero, D. Li, and B. Johansson. Specifying task allocation in automotive wire harness assembly stations for human-robot collahoration. Computers In- dustrial Engineering, 184:109572, 2023. ISSN 0360-8352. doi.htps://oo..or///..01666 cie.2023.109572. URL https: http:://www..iiieeee www.sscienceireee..ooo wwt./co..cccaeeeee. com//ecience/arriilee pi1/S036083522300333
[40] R. Bishop. There is more than way to frame curve. The American Mathematical Monthly, 82:Mathematical Association of America-, 03 1975. doi:10.2307/2311944
[41] T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks, 2017.
[42] M. Fey and J. E. Lenssen. Fast graph representation learning with pytorch geometric, 2019.
[43] J. Przemieniecki. Theory of Matrix Structural Analysis. McGraw-Hie 1968. ISBN
9780070509047. URL https: /books.oks///////
11

---

[44] S. Mamedov, A. R. Geist, J. Swevers, and S. Trimpe. Pseudo-rigid body networks: learning interpretable deformable object dynamics from partial observations, 2024. URL https:
//axxii.oggabs/22000 077955
[45] S. Mamedov, A. R. Geist, R. Viljoen, S. Trimpe, and J. Swevers. Learning deformable linear object dynamics from a single trajectory, 2024. URL https: httppp://aaxxi..r///// http://iiwrgga/////aaaaaaa 2b4/ 03476.
[46] A. Myronenko, X. Song, and M. Carreira-Perpinan, Non-rigid point registration: Coherent point drift. Advances in neural information processing systems, 19, 2006.
[47] Tang, Y. Fan, H.-C. Lin, and M. Tomizuka. State estimation for deformable objects by point registration and dynamic simulation. In 2017 IEEE/RSJ International Conference on Intelli- gent Robots and Systems (IROS), pages 2427-2433, 2017. doi:10. 109.0...11/RRRSS... 201.//000.217.2200055555
[48] S. Ge, G. Fan, and M. Ding. Non-rigid point set registration with global-local topology preser- vation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 245-251, 2014.
[49] C. Chi and D. Berenson. Occlusion- robuss deformable object tracking without physics simu- lation. In 2019 JEEE/RS International Conference on Intelligent Robots and Systems (IROS). IEEE, Nov. 2019. doi: 1a.1110..11111 L00//ir044087.2209998888 2977797777. UtRhttpp:/ / dxx doi. www..do/i......... 1109/ IROS 2019 9.1...4 8967827.
[50] Y. Wang. D. McConachie, and D. Berenson. Tracking partially-oclludd deformable objects while enforcing geometric constraints, 2020.
[51] T. Tang and M. Tomizuka. Track deformable objects from point clouds with structure preserved registration. The International Journal of Robotics Research, 41(6):599 6144, 2022.
[52] Tang, Y. Fan, H.-C. Lin, and Tomizuka. State estimation for deformable objects by point registration and dynamic simulation. In 2017 IEEE/RSJ International Conference on Intelli- gent Robots and Systems (IROS), pages 2427-2433, 2017. doi: 100/1110.201...2000002
[53] T. Tang, Y. Fan, H.-C Lin, and M. Tomizuka. State estimation for deformable objects by point registration and dynamic simulation. In 2017 IEEE/RSJ International Conference on Intelli- gent Robots and Systems (IROS), pages 2427-2433, 2017. 1tt::0..1109/ROSS.010111
[54] Y. Yang, J. A. Stork, and T. Stoyanov. Particle filters in latent space for robust deformable linear object tracking. IEEE Robotics and Automation Letters, 7(4): 1257-12577-22584 2022.
[55] K. Lv, M. YU, Y. Pu, X. Jiang, G. Huang, and X. Li. Learning to estimate 3-d states of de- formable linear objects from single-frame occluded point clouds. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 7119-7125. IEEE, 2023.
12

---

A
Additional DER Background
x2
x"
m
u1
1
x²
x n-
0
4
m2
Figure 6: An illustration of DER coordinate frames.
Bishop Frame. To accurately approximate the potential energy of the DLO that arises due to bend- ing or twisting, DER first assigns Bishop frames {t', u', v') as reference frame onto edge i. t R represents the unit tangent vector along edge i. The vector u' R is defined iteratively through rotation operation u' R u'-1 This rotation is governed by rotation matrix R E R3x3 which satisfies following conditions: t² = R t' and t' x t' = R (t'-1 x t'). v', orthogonal to t' and u', is defined by v¹ t' x u'. This transportation results in the least amount of twist, making ideal for adapting the Bishop frame as the most natural reference frame. Further details of the Bishop frame can found [40].
Material Frame. DER further introduces scalar 0 to measure the rotation of material frames (t', m m$) relative to the Bishop frame along t', m m are defined as following
m = cos 6¹ u' sin 0¹ .v,, m sin 0* u' + cos 8²
(13)
The material frames are used in (1) to obtain potential energy. An illustration of Bishop frame and material frame can be found in Figure 6. More relevant details can be found in [13, Section 4].
B Algorithmic Details
B.1 DNN Structure
We built our DNN using a Graph Neural Network (GNN) [5]. This is motivated in part by re- cent work that has illustrated how to successfully apply GNNs to model the spatial relationship between vertices in DLO [15]. We adapt the default implementation of graph convolution net- work (GCN) [41] found in the PyTorch Geomeric library [42]. GCNs have demonstrated their ability to reduce computational costs while effectively extracting informative latent representations from graph-structured data. X, and V are feature-wise concatenated as (X,, V) E Rax6 and are the input of the GCN, We set the feature dimension to 32 for message passing, allowing each node to receive information from its local neighborhood. We aggregate each node's neighbors' features using summation. The outputs of the GCN are flattened and decoded by MLP constructed with two linear layers with a Rectified Linear Unit (ReLU) in the middle.
B.2 Verification of Improved Inextensiblity Enforcement with PBD
First, note that (9) applies only to successive vertices. This simplifies the conservation of momentum equation(12) (1) so that it only involves successive vertices, as shown below:
M A& + M A
(14)
x M A& + ++ x (M++1
A 0
(15)
The complete proposed solution, as detailed in Section 4.3, can be expressed as:
M1+1
x x
A&
C(
sit1
(16)
(|IM || + | |
|
Xtt1ll2
13

---

M
++1 x(+1)
-(xx+1 sft
x141)
(17)
(II l|a + || 112)
||
82+1112
Expanding (14) with (16) and (17):
M M
x
M'. A& M
Axtt1
C(8
(I)M
112
||
112)
|| 2111l2
(18)
M
M
x+1
8
s++1
++1
M + (1+))
|
8141l12
Since DLOs are linear and symmetric about their center axis, their mass matrices are also symmetric [43]. Consequently, M M .M' -M+L allowing us to cancel both terms in (18), resulting in
zero linear momentum change. A key observation here is that (16) and (17) collinear along the
edge i. Therefore, AR; and Ax x++
t+1 will not change orientation of edge i, naturally satisfies (15).
B.3 Summary and Discussion of Improved Inextensiblity Enforcement with PBD
Algorithm summarizes our proposed method to enforce inextensibility while preserving mo- mentum. Note that in practice, the while loop in Algorithm 2 typically converges after two it- erations for 0.05. Once have the output from Algorithm 2, we update the velocities of the vertices to reflect the new vertex locations
Algorithm Enforcing Inextensibility with Mo-
mentum Preserving PBD
Require:
X and 0
1: while any C(X 23+1
X1+1
do
2:
for i= 2 to n do
3:
K X B
A
(1I)
4: end for
(i.e., Line 5 of Algorithm 1). Further, as shown Algorithm and relying solely on DNNs to capture the in- extensibility of DLOs can lead simulation instability. This instability arises because modeling stiff behavior, such the inextensibility of DLOs, makes the learning process sensitive to inputs and hinders effective gradient propagation. A key issue with using neural network predictions in position- position-based dynamics (PBD) is that the dynamics/data lie on low-dimensional manifold within a figh-ddmenniionl space. Over several open-loop predictions, the network's 's prediction errors can push the system state away from this manifold, leading to deterioration in the network's 's predictions. This occurs because the network has never encountered these "off-manifold" points during training. Algorithm designed to ensure that model inputs remain on the manifold on which the training data was collected, thereby mitigating these issues.
5: end while
in Table 6, skipping Algorithm
6: return: X,+1
> Updated Vertices
B.4 Training Setup
Let U1T-- denote the set of inputs applied between times t = 1 and = T 1, and let X (X1, X Xr} E Txnx3 denote the associated ground truth trajectory of the DLOO from times 1 = T. With known X,, V1 and U1:1-1. Algorithm can be applied recursively to generate predicted associated trajectory X2.T. Let & denote parameters of DNN. The objective of training is to solve the following optimization problem:
Σ 1
min α
(19)
By taking advantage of DEFORM's differentiabilityy T can be set to values greater than 2 to capture long-term behavior. This multi-step training pipeline results in higher prediction accuracy than the single-step training pipeline, as shown in Table 6.
B.5 Discussion of Neural Network Utilization in (7)
This section provides additional discussion of using neural networks in (7) and (8). Incorpo- rating GNN enhances prediction accuracy by addressing inaccuracies arising from spatial and
14

---

temporal discretization. This network can effectively identify and adjust for these imperiections, as demonstrated in the ablation study. As the training involves a multi-step process, the GNN al- lows DEFORM to leverage gradients backpropaged from future prediction losses, which enables better informed and proactive corrections. GNN can implicitly capture variations (e.g., shearing) in residual energy that are not directly modeled by the traditional DER framework.
However, the integration of neural network also introduces several challenges. Primarily, it in- creases the computational load. Moreover, the addition of neural network, particularly in capturing and correcting impulse vectors, introduces complexity in ensuring that the network's 's outputs remain physically plausible and consistent with the DER's s modeling output.
B.6 Comparison of DER other DLO dynamics formulations
Originally derived from computer graphics, the DER approaches uses Lagrangian mechanics to model the dynamics of a chain of point masses, employing a unique choice of coordinates that ef- fectively captures bending and twisting dynamics. This method contrasts with recent studies that apply Newtonian multi-body dynamics using Featherstone's ' algorithms [44] and [45]. For instance, [45] models DL.O dynamics using chain of rigid bodies defined minimal coordinates like angles between body elements, which simplifies the computation by making forward dynamics explicit and eliminating the need for implicit optimization additional constraint enforcement algorithms. While both Newtonian and Lagrangian dynamics conserve energy, their methods for deriving con- servative forces differ, necessitating deeper discussion on the advantages and limitations of each approach.
From Lagrangian perspective, DER's innovative coordinate choice and post-integration con- straint enforcement allow for the modeling of complex, stiff behaviors through various optimiza- tion solvers, enhancing stability and accuracy crucial for DLO simulations. The accurate and stable model offers high-fidelity prior for learning residual behavior, significantly enhancing sampling efficiency and performance as shown in the ablation study. However, this can introduce signifi- cant computational complexity and burden due to the need to solve implicit optimization problems and enforce constraints. In contrast, Newtonian dynamics offer simpler, faster computations due to their explicit nature and minimal coordinate system, facilitating the integration of complex neural networks. However, this can sometimes result in simulation instabilities, particularly when mod- eling stiff materials, and may struggle to capture complex deformable behaviors without extensive modifications. The pros and cons of each method illustrate clear trade-off between computational simplicity and the ability to handle complex behaviors. Overall, there are certainly benefits to using Newtonian mudti-body dynamics that warrant further exploration.
C Experimental Details
C.1 Hardware Parameters
We use an OptiTrack motion capture (Mocap) system to obtain the ground truth vertex locations for DLOs as depicted in Figure 7. Spherical markers with a diameter 7.9 mm and weight 0.4 g, are attached to the DLOs using the OptiTrack tracking system. Ten Flex3 cameras capture the motion of the markers at a frequency of 100Hz, with positional error of less than 0.3mm. For perception with an RGB-D camera,
an Azure Kinect DK with resolution of 720 1080 and frequency constructed shown in Figure 7. The physical properties of each wire are outlined in Table 4. These properties include the length, weight, and stiffness of each DLO, as well as the number of Mocap markers attached to it. The stiffness of each DLO is ranked on a relative scale.
of 30 Hz. Three distinct cables and two distinct ropes
C.2 Software Implementation:
All experiments conducted in Python, an Ubuntu 20.04 machine equipped with AMD Ryzen PRO 5995WX CPU, 256GB RAM, and 128 cores. All training pipelines were built within
15

---

DLO1
Mocap
RGB-D Camera
Kinova Gen3
Robot
Franka Emika Panda Robot
Markers
Mocap
DLO
DLOS
Figure 7: Left: An illustration of the experimental setup. Right: An illustration of the DLOs that are used to evaluate and compare the performance of DEFORM with various state-of-ther of-the-rrt DLO modeling methods.
Table 4; Material Properties
# Mocap Markers
Name
Length [m]
Weight [g]
Stiffness
DLO
1.152
34.5
3
13
DLO
0.996
65.2
4
12
DLO 3
0.998
96.8
5
12
DLO
0.973
22.0
1
12
DLO 5
0.988
19.2
2
12
the PyTorch training framework. We implement DEFORM with PyTorch and use the Levenberg- Marquardt algorithm as the solver for Theseus. We utilize Py Torch for training and Numpy for non- batched prediction. We initialize the length and mass parameters of the DLO according to Table 4. The other material properties are initialized randomly and learned during the training process. We use SGD optimizer with 10 -4 learning rate for training.
C.3 Ablation Study: DLO4 and DLO5
As supplement for Table 2, the ablation study of DLO4 and DLO5 is shown in Table 5.
C.4 Ablation Study: Inextensibility Enforcement with Learning and Single-step Training
We conducted further ablation study on training DEFORM without the improved inextensibility enforcement and training DEFORM using only a single-step prediction shown in Table 6. When DEFORM is trained without enforcing inextensibility, simulation instability results in very high prediction loss, highlighting the importance of properly enforcing inextensibility over long time horizons. Additionally, training DEFORM using only a single-step prediction results in lower long- term prediction accuracy compared training DEFORM using 100-step prediction, demonstrating the importance of DEFORM's differentiability for accurate long-term predictions.
Table 5: Ablation Study with DLO and DLO
Accuracy (10 2m)
Method
5
DER W/O Residual Learning
1.50
1.65
1.33
1.26
W/O System ID
1.02
1.10
Original Inextensibility
1.29
1.58
DEFORM
0.850
0.987
16

---

Table 6: Additional Ablation Study.
Accuracy (10 (10-2m
Method
2
W/O Enforcing Inextensibility with PBD
7.7 x 109
260 x 10
310 x 10
Single Step Prediction Training
1.82
1.74
1.79
DEFORM
1.01
0.97
0.77
C.5 Effeet of Number of Vertices: Accuracy and Computational Efficiency
To further investigate the impact of vertices number on accuracy and computational efficiency, we conducted additional experiments using DLO 1. Figure 8 illustrates that increasing the number of vertices enhances prediction accuracy by allowing DEFORM to capture finer details due improved resolution. However, this increase in detail is at the cost of reduced computational efficiency. In our case we chose the number of vertices to achieve optimal prediction accuracy while ensuring real- time inference capabilities at a 100Hz frequency.
Accuracy and Computational Time Vertices Number
0.022
0.020
0.09
Accuracy (t)
0.018
0,016
0.07
0.014
0.06
0.010
Vertices Number
Figure 8: Accuracy and Compuational Time Vertices Number
C.6 ARMOUR
To perform a shape matching task, we rely upon ARMOUR[37], an optimization- motion plan- ning and control framework. The goal of the shape matching task use robot arm to manipulate the DLO from initial configuration a predefined target configuration. To accomplish this goal, ARMOUR performs planning in receding horizon fashion. During each planning iteration, AR- MOUR selects trajectory to follow by solving optimization problem. ARMOUR represents joint trajectories of the robot as polynomials, where the coefficients are considered as the decision variables of the optimization. One end of the rope is rigidly attached to the end effector of the robot, whose configuration can be computed by forward kinematics. By plugging the end effector configuration at the end of the trajectories into DEFORM, we are able get prediction of the rope configuration the end of the robot motion. The cost function for the optimization problem to minimize is then defined the norm distance between the predicted rope configuration and the target rope configuration. Joint position, velocity, and torque limits of the arm are considered as constraints of the optimization problem in ARMOUR's 's formulation, which ensures that the robot motion is feasible. The solution to this optimization problem is then the joint trajectories of the robot that manipulate the rope to approach the desired configuration. To command the robot to track the solution trajectories, we use the controller proposed ARMOUR. This process is conducted re- peatedly until the desired configuration is reached, or a planning iteration limit is reached resulting in failure. More details about ARMOUR's 's trajectory parameterization and associated closed loop controller can be found in [37, Section IX]. The cost function minimizes the distance between the
17

---

predicted DLO configuration given chosen robot trajectory and target DLO configuration. The predicted state of the DLO is computed via DL.O modeling technique.
D DLO Tracking with Modeling
This section first discusses the difficulties of incorporating existing framework with modeling for long-time DLO tracking under occlusion. It then proposes a novel perception pipeline, which is utilized in Section 5.2.
D.I DLO Tracking Review
If DLO fully observable, then current state-of-the- oott----tttt-tttt-- methods estimate the location of the DLO's vertices by applying Gaussian Mixture Model (GMM), performing clustering, and then using [46, 47, 48, 49, 50, 51, 52, 53]. The output of this GMM algorithm is the mean locations in RR3 of each of the Gaussians, which is then set equal to the vertex locations of the DLO. However, in practical applications, occlusion of DLOs during manipulation often leads to perception challenges, which complicates accurate prediction. In particular, due to occlusions, one cannot simply set the number of mixtures in the GMM equal to the number of vertices of DLO. Doing so results in the vertices being incorrectly distributed only the unoceluded parts of the DLO. Some of the aforementioned methods[49, 50] have been applied perform tracking of DLOs under occlusion. Typically, this is done by leveraging short time horizon prediction with geometric regularization using prior observations. [49, 50, 51, 52, 53] Though powerful, work accurately, these methods require frequent measurement updates and can struggle in the presence of occlusions for long time horizons. Recent research has explored partiele filtering within lower- dimensional latent space embedding and applied tearning-basee techniques for shape estimation under occlusion [54, 55]. Each of these methods rely upon different models for DLOOs that tend to have numerical instabilities when used for prediction. As result, these perception methods require high frequency sensor measurement updates to behave accurately. This paper illustrates that our proposed model allows to adapt DEFORM's long time horizon prediction capability to relax the frequency of sensor updates, which reduces the overall computational cost of tracking DLOs in the presence of occlusions.
D.2 DLO Tracking with DEFORM
This section describes how we can use DEFORM to perform robust DLO tracking. Notably, DE- FORM enables to deal with occlusions without requiring frame-by-frame sensor update of the state of the DLO. In particular, this is possible due the DLO over long time horizons. As result, utilize GMM model because it is independent of time. The state estimation approach is outlined in Algorithm 3. Additionally, Algorithm 4 sum- marizes the perception pipeline that describes the tracking of DLOs with state estimation and initial state estimation using DEFORM.
model accurately predicting the state of
D.3 State Estimation in the Presence of Occlusions
Suppose that we are given access to an RGB-D sensor observing our DLO during manipulation and suppose that we translate these measurements at time step into a point cloud which we denote by RMx
S. = ("
To address the limitations of the algorithms discussed in Section 2, we leverage our predictions generated by applying Algorithm 1 as follows. We first filter the point cloud at time using our predicted data. If any element of the point cloud beyond some distance from our prediction, X,, then we remove it. Let S, denote the remaining points the point cloud. Next, we detect which of the vertices of the DLO unoccluded from the RGB-D sensor by checking if their predicted depth close to their observed depth in the sensor. Suppose the number of vertices that are unoccluded is ñ.
18

---

Algorithm State Estimation with Presence of Occlusion
Inputs: X++1
> Algorithm.
1: ST+1 RGB-D Camera
2: S1+1 filter filtterXet11 X S1+1)
Unoccluded X f Depth Matching(Sr+1. X
4: n-+ groups DBSCAN((1+1)
5: for each group do
6: GMM Number + Match( Unoccluded X group)
7: Mixture Center +- GMM(group.)
8: end for
9:
for each X..I do
10:
if Observed then
11:
Vertex + Associated Mixture Center
12: else
13:
Vertex + Predicted Vertex
14: end if
15:
end for
16:
return: Vertices
Algorithm& Tracking DLO with Modeling
Initial State Estimation
Require: S < RGB-D Camera
1: DLO Initial Guess + S
2: while DLO not Static do
3: Execute DEFORM
> Algorithm.
4: end while
5: Return X,, V1 0
Tracking DLO under Manipulation
1: while Tracking DLO do
2: while Predicting DLO do
3: Execute DEFORM
Algorithm.
4: end while
5: State Estimation and Correction
6:
end while
Once this done, we apply DBSCAN to group Š, into ñ + groups. Next, we associate each of the unoccluded vertices with one of the groups by finding the group to which its predicted location is smallest. We then take each group and perform GMM on the group with a number mixtures equal the number of vertices that were associated with that group. Note, each mixture is associated with unoccluded vertex of the DLO. The predicted location of the vertex generated by DEFORM updated by setting the new vertex location equal to the mean of its associated mixture. If particular vertex was occluded, then its predicted location is left equal to its output from Algorithm 1.
D.4 Initial State Estimation under Occlusion
Note that in many instances, it can be difficult to estimate the location of the vertices when the sensor turns on and the DI.O occluded. Unfortunately, Algorithm and the state estimation algorithm from the previous subsection require access full initial state. Fortunately, we can address this problem by making the following assumption: When the sensor measurements begin, the DLO is static and is only subject to gravity and/or the manipulators which are holding its ends. Under the above assumption, we initialize the DLO using initial guess that aligns with observed vertices. This initial guess is then forward simulated using Algorithm repeatedly until the DLO reaches static state. This steady state is then used as the predicted state for all the vertices, including the occluded vertices.
19


Reviews:
[{"review_id": "V1RlyCSEwP", "reviewer": "Reviewer_rJLv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces DEFORM, a method that integrates residual learning into a novel differentiable simulator designed for Deformable Linear Objects (DLOs). The framework enables precise modeling and real-time prediction of DLO behavior over extended time periods during dynamic manipulation. The efficacy of DEFORM is demonstrated through extensive experiments assessing its accuracy, computational speed, and applicability in manipulation and perception tasks. Compared to existing state-of-the-art methods, DEFORM achieves higher accuracy and sample efficiency, without compromising computational speed. Additionally, DEFORM proves practical for state estimation, particularly in tracking DLOs in occlusion-rich environments. The paper further showcases DEFORM's utility by combining it with a manipulation planning and control framework to achieve 3D shape matching of DLOs in both simulated and real-world settings, resulting in a higher task success rate than the nearest baseline.", "summary_of_recommendation": "In summary, the paper on DEFORM presents a novel approach that combines physics-based modeling with learning techniques for real-time modeling of Deformable Linear Objects (DLOs). The experiments conducted demonstrate the effectiveness of DEFORM in terms of accuracy, computational speed, and practical applications in manipulation tasks. While the findings show promising results, it is acknowledged that there are some limitations to be addressed in future work. Therefore, it is recommended that further research be conducted to strengthen the acceptance of DEFORM by addressing these limitations and enhancing its capabilities, particularly in contact modeling and multi-branched DLO modeling. By addressing these areas, DEFORM can be further improved to increase its accuracy and applicability in dynamic DLO modeling.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1, Could you clarify the computational efficiency of DEFORM, specifically how it scales with the complexity of the DLOs?\n\n2. Could you provide more details on the manipulation planning and control framework integrated with DEFORM for 3D shape matching?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6aOIkXfguc", "reviewer": "Reviewer_jfrQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces DEFORM, a novel framework combining differentiable physics-based models with learning frameworks to accurately model and predict the dynamic behavior of Deformable Linear Objects (DLOs) such as ropes and cables. The framework promises real-time performance and addresses challenges like numerical integration errors and inextensibility enforcement. The authors present comprehensive experiments to demonstrate DEFORM's superior accuracy, computational speed, and generalizability compared to state-of-the-art methods", "summary_of_recommendation": "This is a decent paper for CoRL", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 2, "quality_of_limitations": 3, "questions_for_rebuttal": "How to decide the number of vertices required for one DLO?\n\nCan DEFORM handle twists? It is in the physic model but the demonstrate tasks do not have twist?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vrxX7u5PGN", "reviewer": "Reviewer_ktjq", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose a discrete-time DLO dynamics model that combines a differntiable discrete elastic rod (DER) dynamics model with neural network regression. The model is derived in three steps:  1. Implement DER dynamics in PyTorch such that it is differentiable wrt. the potential parameter $\\theta$ (resorting to implicit differentiation via Theseus) 2. An MLP is added to the DER dynamics model that learns the residual in the impulses and the residual of the numerical integration on the position-level 3. Add constraint enforcing algorithm to ensure inextensibility of the DLO model during integration (similar to [33])", "summary_of_recommendation": "The proposed combination of DER dynamics models with neural network regression forms an important contribution towards understanding how to most effectively combine DLO dynamics models with neural networks for planning and control.  The discussion of DER and why the authors chose this particular DLO dynamics model compared to other models needs to be improved.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "**Question - C1:** What are the pros and cons of the particular choice of input/outputs to the residual network? To which extent does the used GNN incorporate the structural knowledge underlying the DER model?\n\n**Question - C2:** With the goal in mind of learning the errors of DLO dynamics with neural networks, what are the pros and cons of formulating dynamics in terms of a potential compared to resorting to Newtonian dynamics algorithms as in [2]?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper addresses the task of modeling Deformable Linear Objects (DLOs), such as ropes and cables, during dynamic motion over long time horizons. It proposes differentiable Discrete Elastic Rods For deformable linear Objects with Real-time Modeling (DEFORM), a framework that combines a differentiable physics-based model with a learning framework to model DLOs accurately and in real-time. Compared to existing state-of-the-art methods, DEFORM achieves higher accuracy and sample efficiency, without compromising computational speed. 
Overall, this seems like a good paper. However, issues raised by the reviewers, such as the writing, metrics, implementation details, need to be fixed during the rebuttal, in addition to clarifying certain statements indicated by the reviewers.
Decision: Accept