Differentiable Discrete Elastic Rods for Real-Time Modeling of Deformable Linear Objects

Authors: yizhou  chen
yiting  zhang
zachary  brei
tiancheng  zhang
yuzhen  chen
julie  wu
ram  vasudevan
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-05 15:00:13

Abstract: This paper addresses the task of modeling Deformable Linear Objects (DLOs), such as ropes and cables, during dynamic motion over long time horizons. This task presents significant challenges due to the complex dynamics of DLOs. To address these challenges, this paper proposes differentiable Discrete Elastic Rods For deformable linear Objects with Real-time Modeling (DEFORM),  a novel framework that combines a differentiable physics-based model with a learning framework to model DLOs accurately and in real-time.  The performance of DEFORM is evaluated in an experimental setup involving two industrial robots and a variety of sensors. A comprehensive series of experiments demonstrate the efficacy of DEFORM in terms of accuracy, computational speed, and generalizability when compared to state-of-the-art alternatives. To further demonstrate the utility of DEFORM, this paper integrates it into a perception pipeline and illustrates its superior performance when compared to the state-of-the-art methods while tracking a DLO even in the presence of occlusions.  Finally, this paper illustrates the superior performance of DEFORM when compared to state-of-the-art methods when it is applied to perform autonomous planning and control of DLOs.

Main paper: 
 introduction: 
 Robotic manipulation tasks such as surgical suturing or vehicle wire harness assembly [1, 2, 3] are challenging problems because they require accurately and dynamically manipulating DLOs over 8th Conference on Robot Learning (CoRL 2024), Munich, Germany. long time horizons (> 1s) [4] . Nonlinear dynamics make manipulating DLOs challenging because computationally expensive models are required for accurate prediction. Further, practical applications such as vehicle wire harness assembly often result in DLO occlusions, meaning perception systems cannot robustly estimate the entire DLO configuration. For robotic manipulators to successfully manipulate DLOs, a novel modeling method capable of quick and accurate predictions of dynamic DLO behavior, even in the presence of occlusions, is needed.To model DLOs, physics-based approaches can be used, but they suffer a tradeoff between costly computation and low accuracy when predicting dynamic movements. State-of-the-art learningbased modeling approaches have used deep neural networks (DNNs) to model interaction propagation in DLOs [5, 6, 7] . Despite their success, these methods require large datasets of real-world manipulation data, which can be impractical to collect and are not robust to variations in DLO properties. Alternatively, using a physics-based model as a prior for DNNs is promising due to increased sampling efficiency and generalizability [8, 9, 10, 11, 12] . However, to the best of our knowledge, this approach has not been applied to predict behavior of DLOs during dynamic motions.A recent promising physics-based modeling method is Discrete Elastic Rods (DER). This approach qualitatively and efficiently describes the 3-D behavior of DLOs [13, 14] . Despite its potential for physics-informed learning, applying DER theory to model DLOs in the real-world is challenging for three reasons. First, tuning DER model parameters to match a real-world DLO is difficult. Second, computing analytical gradients for DER models is challenging. Third, numerical errors arise from discrete numerical integration of DER models and from enforcing DLO inextensibility. These drawbacks affect the accuracy of the DER model, limiting the utility of embedding a DER model in a learning-based framework. In particular, a lack of analytical gradients prevent learning over multiple steps, which reduces the accuracy of long horizon predictions.To address the above challenges, this paper introduces Differentiable Discrete Elastic Rods For Deformable Linear Objects with Real-time Modeling (DEFORM), a novel framework that uses a differentiable physics-based learning model, with real-time inference speed, to accurately predict dynamic DLO behavior over long time horizons. As illustrated in Figure 2 , the main contributions of DEFORM are four-fold. First, a novel DLO model is presented, called the Differentiable Discrete Elastic Rod (DDER) model that makes DER models differentiable with respect to any variables. Second, a numerical integration error compensation method that uses DNNs is developed to improve accuracy when numerically integrating DER models. Third, a novel inextensibility enforcement algorithm is presented for DER models that guarantees momentum conservation, thereby enabling accurate prediction of dynamic DLO behavior. Fourth, a comprehensive set of experiments are presented, which illustrate that DEFORM has superior performance in terms of accuracy, computational speed, and sampling efficiency. We also demonstrate the utility of DEFORM for robust DLO tracking and precise shape-matching manipulation tasks. 
related work: 
 Learning-Based Modeling. Latent dynamics modeling approaches, such as graph neural networks (GNN) [15] and Bidirectional LSTM (Bi-LSTM) [6] , reason over vertices spatially to formulate latent dynamics of DLOs for predicting their behavior in 2-D. Unfortunately, as we illustrate in this paper through real-world experiments, these methods are unable to accurate describe the behavior of DLOs during dynamic motions in 3D.Physics-Based Modeling. Traditionally, DLOs have been modeled using physics-based approaches. Examples include mass-spring system models [16, 17, 18] , Position-Based Dynamics (PBD) models [19, 20, 21] , Discrete Elastic Rod (DER) models [13, 14, 22, 23] , or Finite Element Method (FEM) models [24, 25, 26] . Although easy to implement, the mass-spring system imposes unnecessary stiffness to enforce inextensibility, resulting in instability during simulation. FEM models accurately modeling DLO behavior, but are impractical for real-time prediction due to their prohibitive computational costs. PBD and DER methods have been applied in planning and control frameworks for manipulation tasks due to their computational efficiency [20, 21, 27, 28, 29] . However, PBD models are sensitive to parameter selection and assume quasi-static behavior. As illustrated in this paper, this results in low accuracy during dynamic manipulation. Though DER models have been qualitatively validated to describe the 3D behavior of DLOs, we illustrate that their quantitative performance while describing dynamic motion is lacking.Differentiable Modeling Framework. Differentiable models are computational frameworks where every operation is differentiable with respect to the input parameters. Importantly, a differentiable model enables integration into a learning framework to improve learning and can enable efficient parameter auto-tuning via gradient-based optimization during training. There are differentiable models for deformable objects, such as DiffCloth [16, 17] and XPBD [21] , but these models tend to suffer from numerical instability. This makes it challenging to apply these modeling frameworks while trying to perform parameter identification for real-world DLO manipulation. To address these limitations, this paper proposes a differentiable DER model, called DDER, which is incorporated into a learning framework to achieve numerical stability. This results in better predictive performance for dynamic DLO behavior when compared to state-of-the-art methods and enables auto-tuning of DLO parameters using limited real-world data. 
preliminaries: 
 Deformable Linear Object (DLO) Model. To model a DLO, we separate the DLO into an indexed set of n vertices at each time instance. The ground truth locations of vertex i at time t is denoted by x i t ∈ R 3 and the set of all n vertices is denoted by X t . Let M i ∈ R 3×3 denote the mass matrix of vertex i. Our objective is to model the dynamic behavior of a DLO that is being controlled by a robot with following assumption: the first and last edge of the DLO are each held by a distinct end effector. Note this assumption is commonly made during DLO manipulation [30, 20, 19, 7, 31] . Using this assumption, let the orientation and position of the two end effectors at time t be denoted by u t ∈ R 12 , which we refer to as the input. The velocity of the vertices of the DLO can be denoted as V t = (X t -X t-1 )/∆t. Note that V t is an approximation of the actual velocity. Finally, note that we distinguish between ground truth elements and predicted elements by using the circumflex symbol (e.g., X t is the ground truth set of vertices at time t and Xt is the predicted set of vertices at time t).Modeling DLOs with DER for Dual Manipulation. DLOs primarily exhibit three distinct deformation modes: bending, twisting, and stretching [13] . This paper focuses on DLOs that are inextensible (e.g., cables and wires), and therefore do not experience stretching. DER models introduce a single scalar θ i t ∈ R to represent bending and twisting in R 3 . This parameter describes the potential energy P (X t , u t , θ t , α) of the DLO that arises due to bending or twisting, where α denotes the vector of the material properties of each vertex of the DLO (i.e., their mass, bending modulus, and twisting modulus) [13, Section 4.2] . Notably, DER models assume that the DLO reaches an equilibrium state between each simulation time step to obtain θ * t :EQUATIONOnce the potential energy expression is derived, then the restorative force experienced during deformation is set equal to the negative of the gradient of the potential energy expression with respect to the vertices. Therefore, the equations of motion of the DLO are:EQUATIONAfter solving (1) , one can numerically integrate this formula to predict the velocity and position by applying the Semi-Implicit Euler method:EQUATIONEQUATION(3)-( 4) can be recursively solved to simulate the behavior of the DER model of a DLO.Inextensibility Enforcement. During simulation, the DLO's length may change due to numerical errors, but this is undesirable when modeling inextensible DLOs. To ensure constant length, DER methods project a computed solution onto a constraint set that enforces inextensibility [32] . However, this projection operation can introduce instabilities during simulation of dynamic behavior because it does not ensure that momentum is conserved. This paper adopts a constraint solver based on a PBD model [33] , which is described in Section 4.3, to enforce inextensiblity while conserving momentum. This improves prediction accuracy compared to classical methods. This section first introduces DDER, a reformulation of DER to ensure differentiability, which enables efficient DER model parameter estimation using real-world data. Next, this section describes how to incorporate a learning-assisted integration method to improve the speed of DLO simulation and accuracy. Finally, this section describes a position-based constraint that enforces inextensibility while preserving momentum, which results in improved numerical stability during simulation. These results are combined in a prediction algorithm that is described in Algorithm 1. This algorithm can be run over multiple steps to generate an accurate prediction of the state of the DLO over a long time horizon. 
differentiable discrete elastic rods (dder): 
 We first develop a reformulation of DER such that the DER model is differentiable with respect to any model variables (i.e., material properties, and DNN weights). We call this extension Differentiable Discrete Elastic Rods (DDER). To do this, we implement DDER in PyTorch [34] by taking advantage of its automatic differentiation functionality. Unfortunately, simulating the DDER model to predict Xt+1 using (3) and ( 4) also requires solving an optimization problem (1) . As a result, to minimize the prediction loss, we need to solve a bi-level optimization problem at each time t. For instance, if we train a learning model to identify the material parameters by minimizing a prediction loss between experimentally collected ground truth data X t+1 and our model prediction Xt+1 for each time t, then the following bi-level optimization problem needs to be solved:EQUATIONEQUATIONNote that Xt+1 depends on the previously computed θ * t due to (3) and (4). To compute the gradient of this optimization problem with respect to α, one can perform implicit differentiation, which we implement using Theseus [35] .To compensate for numerical errors that arise from discrete integration methods and to improve the numerical stability of the algorithm that we use to simulate the DDER model, we incorporate a learning-based framework into the integration method as follows:EQUATIONEQUATIONwhere DNN denotes a deep neural network. We utilize a structure similar to residual learning [36] , with one major difference: we define a shortcut connection that is equal to the integration method that is derived from the DDER physics-based modeling. Unlike pure learning methods, the short cut connection grounds the DNN in physical laws while also leveraging data-driven insights. As shown in the ablation study results, this approach significantly improves modeling performance. Additioanl discussion of section can be found in Appendix B.5. Implementation details of the DNN structure can be found in Appendix B.1. Note that this contribution can also be applied to standard DER models, but does not improve performance as much as it does when combined with our DDER modeling approach as we illustrate in this paper.To enforce inextensibility and conserve momentum after simulating the DLO forward for a single time step (8) , we enforce an additional constraint on the DLO in a manner similar to the PBD method [33] . First, we define a constraint function between spatially successive vertices at each time step:EQUATIONwhere e denotes the edge between vertices i and i + 1 when the DLO is undeformed. Note that this constraint is non-zero if the distance between successive vertices of the DLO changes. The PBD method iteratively modifies the positions of each of the vertex using a pair of correction terms, ∆x i t+1 and ∆x i+1 t+1 , until the following constraint is satisfied:EQUATIONwhere ϵ > 0 is a user-specified threshold. To calculate the correction terms, PBD typically applies a Taylor Approximation of the constraint and sets this equal to zero (i.e., the correction to the vertex locations are chosen to ensure that the constraint is satisfied). The correction term is then given by:EQUATIONNotably, this correction to the vertex location does not ensure that the linear and angular momentum is preserved. To ensure that linear and angular momentum is conserved after the correction is applied, one needs to ensure that the following pair of equations are satisfied:n-2 i=3 M i • ∆x i t+1 = 0 and n-2 i=3 r i × (M i • ∆x i t+1 ) = 0 (12)where r i is the vector between xi t+1 and a common rotation center. Note in particular that if the mass matrix associated with each vertex is not identical, then the correction proposed in (11) leads to a violation of (12) . To address this issue, we modify the correction proposed in (11) by multiplying it by a β ∈ R that is inversely correlated with the weight ratio between successive vertices (i.e., let β i = M i+1 (||M i ||2+||M i+1 ||2) and β i+1 = M i (||M i ||2+||M i+1 ||2) 
experiments and results: 
 This section covers the real-world evaluation of DEFORM, including: 1) a quantitative comparison of modeling accuracy with state-of-the-art methods, 2) computational speed assessment, 3) an ablation study on parameter auto-tuning, residual learning, and multi-step training, 4) using DEFORM to track DLOs with an RGB-D camera, and 5) a shape matching manipulation demonstration. Code and data will be provided upon final paper acceptance. Additional information is in Appendix C.Hardware Setup. Three distinct cables and two distinct ropes were constructed to validate the modeling accuracy of DEFORM. Spherical markers were attached to the DLOs and are used to create training and evaluation datasets. We set the number of vertices equal to the number of MoCap markers. We use an OptiTrack motion capture (MoCap) system to obtain the ground truth vertex locations. Because the markers are incorporated during training and evaluation, their effect on the dynamics is incorporated into all subsequent evaluations. Note that for real-world applications, the markers would not be needed. A Franka Emika Research 3 robot and a Kinova Gen3 robot are used for dual manipulation.Dataset Collection and Training. For each DLO, we collect 350 seconds of dynamic trajectory data in the real-world using the motion capture system at a frequency of 100 Hz. The model for each of the five DLOs is trained separately using the trajectory data collected from our real-world experiments. The dataset is split with 80% for training and 20% for evaluation. In training, models are trained using the same 100 steps (1s) of dynamic motion, which included slow and fast DLO motions. The L1 loss over the 100 step prediction horizon is used for training. For evaluation, the trained model is used to predict DLO behavior for 500 steps without accessing ground truth.Baseline Comparisons. Physics-Based Modeling: The XPBD [21] approach is constructed using PBD, formulating constraints quasi-statically to model stretching and bending behavior. The DRM [19] method is rooted in the concept of directional diminished rigidity such that the dynamics of a vertex is dependent on its configuration relative to the gripper's configuration. The original DER [13] modeling approach is included as a baseline to demonstrate its performance when compared to other analytical methods and to DEFORM. Learning-Based Modeling: Yan et al. [6] and Wang et al. [15] use Bi-LSTM and GNN to represent a DLO and reason about interactions between the vertices respectively. Physics-based models with enhancements: To further demonstrate the capabilities of DEFORM, we enhanced XPBD and DRM with residual learning and material property parameter tuning, which we denote as XPBD-NN and DRM-NN, respectively.Table 1 (Left) summarizes the results of using DEFORM when compared to baselines using the average L1 loss. Performance is evaluated when predicting the state of the wire over a 5s prediction horizon. Note that DEFORM outperforms all baselines for each of the five evaluated cables. An illustration of the performance of these methods can be found in Figure 3 . Next, we compare computational speed for one step inference. 4 .State Estimation. We next evaluate the performance of DEFORM when it is used in concert with sensor observations for state estimation. These experiments included occlusions of the DLO that arose from the manipulators or humans moving in front of the sensor. In this experiment, we compare how well the state estimation framework described in Appendix D does when using the DEFORM model versus the Bi-LSTM model. Bi-LSTM was chosen because it was the second best model in terms of model accuracy. We also compare how well each of these methods do with varying frequencies of sensor measurement updates. As in the earlier experiment, we evaluate the L1 norm of the prediction over a 5s long prediction horizon. Experimental results are shown in Figure 4 . Unsurprisingly, each method improves when given access to more frequent sensor updates. This illustrates that the state estimation algorithm described in Appendix D.2 behaves appropriately. Notably, the DEFORM model with no state estimation does better than the Bi-LSTM model even when the Bi-LSTM model is given access to 1 fps updates. The method using the DEFORM model with access to 30 fps sensor measurements is the most effective.To demonstrate the practical applicability of DEFORM, we illustrate its use for DLO 3-D shape matching tasks in both simulation and the real world. This experiment uses ARMOUR [37] , a receding-horizon trajectory planner and tracking controller framework, to manipulate DLOs. ARMOUR uses either DEFORM or Bi-LSTM to predict the manipulated DLO state and tries to match the final DLO configuration with a desired configuration. Details of combining ARMOUR with the DLO simulators are in Appendix C.6. We compare the performance of each method on 100 random cases in a PyBullet simulation and 20 random cases in the real world. If the Euclidean distance between each individual pair of ground truth and desired DLO vertices is less than 0.05 meters, then the trial is a success. ARMOUR is given 30 seconds to complete the task, otherwise the trial is marked as a failure. The results are summarized in Table 3 and Figure 5 shows one of the real-world results. A video of these experiments can be found in the supplementary material. 
conclusion and limitations: 
 This paper presents DEFORM, a method that embeds residual learning in a novel differentiable Deformable Linear Object (DLO) simulator. This enables accurate modeling and real-time prediction of dynamic DLO behavior over long time horizons during dynamic manipulation. To demonstrate DEFORM's efficacy, this paper performs a comprehensive set of experiments that evaluate accuracy, computational speed, and its usefulness for manipulation and perception tasks. When compared to the state-of-the-art, DEFORM is more accurate while being sample efficient without sacrificing computational speed. This paper also illustrates the practicality of applying DEFORM for state estimation to track an occluded DLO during dynamic manipulation. Furthermore, DEFORM is combined with a manipulation planning and control framework for 3D shape matching of a DLO in both simulations and real-world scenarios, achieving a higher task success rate compared to the closest-performing baseline.Limitations. We recognize several opportunities to enhance DEFORM. Currently, DEFORM could be improved by adding contact modeling to accurately predict interactions between different components under load [38] . Additionally, DEFORM does not model multi-branched DLOs, which are common in wire-harness assembly [39] . Furthermore, DEFORM assumes that the ends of the DLO are grasped, which could restrict manipulation dexterity. We look forward to addressing these aspects in our future work to improve DEFORM model's accuracy and applicability. Bishop Frame. To accurately approximate the potential energy of the DLO that arises due to bending or twisting, DER first assigns Bishop frames {t i , u i , v i } as reference frame onto edge i. t i ∈ R 3 represents the unit tangent vector along edge i. The vector u i ∈ R 3 is defined iteratively through a rotation operation 
: 
 Hz. Note that the prediction is performed recursively, without requiring access to ground truth or perception during the process. A video of related experiments can be found in the supplementary material. 
 
 Appendix: u i = R i • u i-1. This rotation is governed by a rotation matrix R i ∈ R 3×3 , which satisfies following conditions:t i = R i • t i-1 and t i-1 × t i = R i • (t i-1 × t i). v i , orthogonal to t i and u i , is defined by v i = t i × u i . This transportation results in the least amount of twist, making it ideal for adapting the Bishop frame as the most natural reference frame. Further details of the Bishop frame can be found in [40] .Material Frame. DER further introduces a scalar θ i to measure the rotation of material frames {t i , m i 1 , m i 2 } relative to the Bishop frame along t i . m i 1 , m i 2 are defined as followingEQUATIONThe material frames are used in (1) to obtain potential energy. An illustration of Bishop frame and material frame can be found in Figure 6 . More relevant details can be found in [13, Section 4 ].We built our DNN using a Graph Neural Network (GNN) [5] . This is motivated in part by recent work that has illustrated how to successfully apply GNNs to model the spatial relationship between vertices in a DLO [15] . We adapt the default implementation of a graph convolution network (GCN) [41] found in the PyTorch Geomeric library [42] . GCNs have demonstrated their ability to reduce computational costs while effectively extracting informative latent representations from graph-structured data. Xt and V are feature-wise concatenated as ( Xt , V) ∈ R n×6 and are the input of the GCN. We set the feature dimension to 32 for message passing, allowing each node to receive information from its local neighborhood. We aggregate each node's neighbors' features using summation. The outputs of the GCN are flattened and decoded by a MLP constructed with two linear layers with a Rectified Linear Unit (ReLU) in the middle.First, note that (9) applies only to successive vertices. This simplifies the conservation of momentum equation( 12) so that it only involves successive vertices, as shown below:EQUATION)r i × (M i • ∆x i t+1 ) + r i+1 × (M i+1 • ∆x i+1 t+1 ) = 0 (15)The complete proposed solution, as detailed in Section 4.3, can be expressed as:EQUATION)EQUATIONExpanding ( 14) with ( 16) and ( 17):EQUATIONSince DLOs are linear and symmetric about their center axis, their mass matrices are also symmetric [43] . Consequently,M i+1 • M i = M i • M i+1, allowing us to cancel both terms in (18) , resulting in zero linear momentum change. A key observation here is that ( 16) and ( 17) are collinear along the edge i. Therefore, ∆x i t+1 and ∆x i+1 t+1 will not change orientation of edge i, naturally satisfies (15) . for i = 2 to n -1 do 3:B.EQUATIONend for 5: end while 6: return : Xt+1 ▷ Updated Vertices Algorithm 2 summarizes our proposed method to enforce inextensibility while preserving momentum. Note that in practice, the while loop in Algorithm 2 typically converges after two iterations for ϵ = 0.05. Once we have the output from Algorithm 2, we update the velocities of the vertices to reflect the new vertex locations (i.e., Line 5 of Algorithm 1). Further, as shown in Table 6 , skipping Algorithm 2 in Algorithm 1 and relying solely on DNNs to capture the inextensibility of DLOs can lead to simulation instability. This instability arises because modeling stiff behavior, such as the inextensibility of DLOs, makes the learning process sensitive to inputs and hinders effective gradient propagation. A key issue with using neural network predictions in position-based dynamics (PBD) is that the dynamics/data lie on a low-dimensional manifold within a high-dimensional space. Over several open-loop predictions, the network's prediction errors can push the system state away from this manifold, leading to deterioration in the network's predictions. This occurs because the network has never encountered these "off-manifold" points during training. Algorithm 2 is designed to ensure that model inputs remain on the manifold on which the training data was collected, thereby mitigating these issues.Let U 1:T-1 denote the set of inputs applied between times t = 1 and t = T -1, and let X 1:T = {X 1 , X 2 , . . . , X T } ∈ R T×n×3 denote the associated ground truth trajectory of the DLO from times t = 1 to t = T. With known X 1 , V 1 and U 1:T-1 , Algorithm 1 can be applied recursively to generate predicted associated trajectory X2:T . Let ϕ denote parameters of DNN. The objective of training is to solve the following optimization problem:EQUATIONBy taking advantage of DEFORM's differentiability, T can be set to values greater than 2 to capture long-term behavior. This multi-step training pipeline results in higher prediction accuracy than the single-step training pipeline, as shown in Table 6 .B.5 Discussion of Neural Network Utilization in (7) This section provides an additional discussion of using neural networks in (7) and (8) . Incorporating a GNN enhances prediction accuracy by addressing inaccuracies arising from spatial and temporal discretization. This network can effectively identify and adjust for these imperfections, as demonstrated in the ablation study. As the training involves a multi-step process, the GNN allows DEFORM to leverage gradients backpropaged from future prediction losses, which enables better informed and proactive corrections. GNN can implicitly capture variations (e.g., shearing) in residual energy that are not directly modeled by the traditional DER framework.However, the integration of a neural network also introduces several challenges. Primarily, it increases the computational load. Moreover, the addition of a neural network, particularly in capturing and correcting impulse vectors, introduces complexity in ensuring that the network's outputs remain physically plausible and consistent with the DER's modeling output.Originally derived from computer graphics, the DER approaches uses Lagrangian mechanics to model the dynamics of a chain of point masses, employing a unique choice of coordinates that effectively bending and twisting dynamics. This method contrasts with recent studies that apply Newtonian multi-body dynamics using Featherstone's algorithms [44] and [45] . For instance, [45] models DLO dynamics using a chain of rigid bodies defined in minimal coordinates like angles between body elements, which simplifies the computation by making forward dynamics explicit and eliminating the need for implicit optimization or additional constraint enforcement algorithms. While both Newtonian and Lagrangian dynamics conserve energy, their methods for deriving conservative forces differ, necessitating a deeper discussion on the advantages and limitations of each approach.From a Lagrangian perspective, DER's innovative coordinate choice and post-integration constraint enforcement allow for the modeling of complex, stiff behaviors through various optimization solvers, enhancing stability and accuracy crucial for DLO simulations. The accurate and stable model offers a high-fidelity prior for learning residual behavior, significantly enhancing sampling efficiency and performance as shown in the ablation study. However, this can introduce significant computational complexity and burden due to the need to solve implicit optimization problems and enforce constraints. In contrast, Newtonian dynamics offer simpler, faster computations due to their explicit nature and minimal coordinate system, facilitating the integration of complex neural networks. However, this can sometimes result in simulation instabilities, particularly when modeling stiff materials, and may struggle to capture complex deformable behaviors without extensive modifications. The pros and cons of each method illustrate a clear trade-off between computational simplicity and the ability to handle complex behaviors. Overall, there are certainly benefits to using Newtonian multi-body dynamics that warrant further exploration.We use an OptiTrack motion capture (Mocap) system to obtain the ground truth vertex locations for DLOs as depicted in Figure 7 . Spherical markers with a diameter of 7.9 mm and weight of 0.4 g, are attached to the DLOs using the OptiTrack tracking system. Ten Flex3 cameras capture the motion of the markers at a frequency of 100Hz, with a positional error of less than 0.3mm. For perception with an RGB-D camera, we use an Azure Kinect DK with a resolution of 720 x 1080 and a frequency of 30 Hz. Three distinct cables and two distinct ropes were constructed as shown in Figure 7 . The physical properties of each wire are outlined in Table 4 . These properties include the length, weight, and stiffness of each DLO, as well as the number of Mocap markers attached to it. The stiffness of each DLO is ranked on a relative scale.All experiments were conducted in Python, on an Ubuntu 20.04 machine equipped with an AMD Ryzen PRO 5995WX CPU, 256GB RAM, and 128 cores. All training pipelines were built within As a supplement for Table 2 , the ablation study of DLO4 and DLO5 is shown in Table 5 .We conducted a further ablation study on training DEFORM without the improved inextensibility enforcement and training DEFORM using only a single-step prediction as shown in Table 6 . When DEFORM is trained without enforcing inextensibility, simulation instability results in very high prediction loss, highlighting the importance of properly enforcing inextensibility over long time horizons. Additionally, training DEFORM using only a single-step prediction results in lower longterm prediction accuracy compared to training DEFORM using a 100-step prediction, demonstrating the importance of DEFORM's differentiability for accurate long-term predictions. To further investigate the impact of vertices number on accuracy and computational efficiency, we conducted additional experiments using DLO 1. Figure 8 illustrates that increasing the of vertices enhances prediction accuracy by allowing DEFORM to capture finer details due to improved resolution. However, this increase in detail is at the cost of reduced computational efficiency. In our case we chose the number of vertices to achieve optimal prediction accuracy while ensuring realtime inference capabilities at a 100Hz frequency. To perform a shape matching task, we rely upon ARMOUR [37] , an optimization-based motion planning and control framework. The goal of the shape matching task is to use a robot arm to manipulate the DLO from an initial configuration to a predefined target configuration. To accomplish this goal, ARMOUR performs planning in a receding horizon fashion. During each planning iteration, AR-MOUR selects a trajectory to follow by solving an optimization problem. ARMOUR represents joint trajectories of the robot as polynomials, where the coefficients are considered as the decision variables of the optimization. One end of the rope is rigidly attached to the end effector of the robot, whose configuration can be computed by forward kinematics. By plugging the end effector configuration at the end of the trajectories into DEFORM, we are able to get a prediction of the rope configuration at the end of the robot motion. The cost function for the optimization problem to minimize is then defined as the norm distance between the predicted rope configuration and the target rope configuration. Joint position, velocity, and torque limits of the arm are considered as constraints of the optimization problem in ARMOUR's formulation, which ensures that the robot motion is feasible. The solution to this optimization problem is then the joint trajectories of the robot that manipulate the rope to approach the desired configuration. To command the robot to track the solution trajectories, we use the controller proposed in ARMOUR. This process is conducted repeatedly until the desired configuration is reached, or a planning iteration limit is reached resulting in a failure. More details about ARMOUR's trajectory parameterization and associated closed loop controller can be found in [37, Section IX]. The cost function minimizes the distance between the predicted DLO configuration given a chosen robot trajectory and a target DLO configuration. The predicted state of the DLO is computed via a DLO modeling technique.This section first discusses the difficulties of incorporating existing framework with modeling for long-time DLO tracking under occlusion. It then proposes a novel perception pipeline, which is utilized in Section 5.2.If a DLO is fully observable, then current state-of-the-art methods estimate the location of the DLO's vertices by applying a Gaussian Mixture Model (GMM), performing clustering, and then using Expectation-Maximization [46, 47, 48, 49, 50, 51, 52, 53] . The output of this GMM algorithm is the mean locations in R 3 of each of the Gaussians, which is then set equal to the vertex locations of the DLO. However, in practical applications, occlusion of DLOs during manipulation often leads to perception challenges, which complicates accurate prediction. In particular, due to occlusions, one cannot simply set the number of mixtures in the GMM equal to the number of vertices of the DLO. Doing so results in the vertices being incorrectly distributed only to the unoccluded parts of the DLO. Some of the aforementioned methods [49, 50] have been applied to perform tracking of DLOs under occlusion. Typically, this is done by leveraging short time horizon prediction with geometric regularization using prior observations. [49, 50, 51, 52, 53] Though powerful, to work accurately, these methods require frequent measurement updates and can struggle in the presence of occlusions for long time horizons. Recent research has explored particle filtering within a lowerdimensional latent space embedding and applied learning-based techniques for shape estimation under occlusion [54, 55] . Each of these methods rely upon different models for DLOs that tend to have numerical instabilities when used for prediction. As a result, these perception methods require high frequency sensor measurement updates to behave accurately. This paper illustrates that our proposed model allows us to adapt DEFORM's long time horizon prediction capability to relax the frequency of sensor updates, which reduces the overall computational cost of tracking DLOs in the presence of occlusions.This section describes how we can use DEFORM to perform robust DLO tracking. Notably, DE-FORM enables us to deal with occlusions without requiring a frame-by-frame sensor update of the state of the DLO. In particular, this is possible due to our model accurately predicting the state of the DLO over long time horizons. As a result, we utilize a GMM model because it is independent of time. The state estimation approach is outlined in Algorithm 3. Additionally, Algorithm 4 summarizes the perception pipeline that describes the tracking of DLOs with state estimation and initial state estimation using DEFORM.Suppose that we are given access to an RGB-D sensor observing our DLO during manipulation and suppose that we translate these measurements at time step t into a point cloud which we denote by S t = {s 1 t , s m t , . . . , s M t } ∈ R M×3 . To address the limitations of the algorithms discussed in Section 2, we leverage our predictions generated by applying Algorithm 1 as follows. We first filter the point cloud at time t using our predicted data. If any element of the point cloud is beyond some distance from our prediction, Xt , then we remove it. Let St denote the remaining points in the point cloud. Next, we detect which of the vertices of the DLO are unoccluded from the RGB-D sensor by checking if their predicted depth is close to their observed depth in the sensor. Suppose the number of vertices that are unoccluded is ñ. State Estimation and Correction 6: end while Once this is done, we apply DBSCAN to group St into ñ + 1 groups. Next, we associate each of the unoccluded vertices with one of the groups by finding the group to which its predicted location is smallest. We then take each group and perform GMM on the group with a number of mixtures equal to the number of vertices j that were associated with that group. Note, each mixture is associated with an unoccluded vertex of the DLO. The new predicted location of the vertex generated by DEFORM is updated by setting the new vertex location equal to the mean of its associated mixture. If a particular vertex was occluded, then its predicted location is left equal to its output from Algorithm 1.Note that in many instances, it can be difficult to estimate the location of the vertices when the sensor turns on and the DLO is occluded. Unfortunately, Algorithm 1 and the state estimation algorithm from the previous subsection require access to a full initial state. Fortunately, we can address this problem by making the following assumption: When the sensor measurements begin, the DLO is static and is only subject to gravity and/or the manipulators which are holding its ends. Under the above assumption, we initialize the DLO using an initial guess that aligns with observed vertices. This initial guess is then forward simulated using Algorithm 1 repeatedly until the DLO reaches a static state. This steady state is then used as the predicted state for all the vertices, including the occluded vertices. 


Decision: This paper addresses the task of modeling Deformable Linear Objects (DLOs), such as ropes and cables, during dynamic motion over long time horizons. It proposes differentiable Discrete Elastic Rods For deformable linear Objects with Real-time Modeling (DEFORM), a framework that combines a differentiable physics-based model with a learning framework to model DLOs accurately and in real-time. Compared to existing state-of-the-art methods, DEFORM achieves higher accuracy and sample efficiency, without compromising computational speed. 
Overall, this seems like a good paper. However, issues raised by the reviewers, such as the writing, metrics, implementation details, need to be fixed during the rebuttal, in addition to clarifying certain statements indicated by the reviewers.

Reviews: review: title: title: Quick Overview and Initial Impressions of DEFORM Framework for Real-time DLO Modeling
paper_summary: paper_summary: This paper introduces DEFORM, a method that integrates residual learning into a novel differentiable simulator designed for Deformable Linear Objects (DLOs). The framework enables precise modeling and real-time prediction of DLO behavior over extended time periods during dynamic manipulation. The efficacy of DEFORM is demonstrated through extensive experiments assessing its accuracy, computational speed, and applicability in manipulation and perception tasks. Compared to existing state-of-the-art methods, DEFORM achieves higher accuracy and sample efficiency, without compromising computational speed. Additionally, DEFORM proves practical for state estimation, particularly in tracking DLOs in occlusion-rich environments. The paper further showcases DEFORM's utility by combining it with a manipulation planning and control framework to achieve 3D shape matching of DLOs in both simulated and real-world settings, resulting in a higher task success rate than the nearest baseline.
main_review: main_review: ### Quality
The quality of the paper is high, with a well-organized structure. The writing is generally clear and technically sound, although some sentences could be broken down for better readability.

### Clarity
The clarity of the paper is commendable. Key concepts and methodologies are explained in a straightforward manner.

### Originality
The originality of the paper is significant. The introduction of DEFORM, a framework that combines differentiable physics-based modeling with residual learning for real-time DLO prediction, represents a novel approach in the field.

### Strengths:
1.  The integration of residual learning into a differentiable simulator for DLOs is a novel concept that addresses existing limitations in real-time modeling and long-term prediction.
2.  The extensive set of experiments validates the framework's efficacy in terms of accuracy, computational speed, and generalizability.

### Weaknesses:
1.  Some sentences are lengthy and complex, which may hinder readability.
2.  Some specific metrics need to be explained.
questions: questions: 1, Could you clarify the computational efficiency of DEFORM, specifically how it scales with the complexity of the DLOs?

2. Could you provide more details on the manipulation planning and control framework integrated with DEFORM for 3D shape matching?
limitations: limitations: 3
review_summary: review_summary: In summary, the paper on DEFORM presents a novel approach that combines physics-based modeling with learning techniques for real-time modeling of Deformable Linear Objects (DLOs). The experiments conducted demonstrate the effectiveness of DEFORM in terms of accuracy, computational speed, and practical applications in manipulation tasks. While the findings show promising results, it is acknowledged that there are some limitations to be addressed in future work. Therefore, it is recommended that further research be conducted to strengthen the acceptance of DEFORM by addressing these limitations and enhancing its capabilities, particularly in contact modeling and multi-branched DLO modeling. By addressing these areas, DEFORM can be further improved to increase its accuracy and applicability in dynamic DLO modeling.
title: title: Good paper with extensive validation in experiments
paper_summary: paper_summary: The paper introduces DEFORM, a novel framework combining differentiable physics-based models with learning frameworks to accurately model and predict the dynamic behavior of Deformable Linear Objects (DLOs) such as ropes and cables. The framework promises real-time performance and addresses challenges like numerical integration errors and inextensibility enforcement. The authors present comprehensive experiments to demonstrate DEFORM's superior accuracy, computational speed, and generalizability compared to state-of-the-art methods
main_review: main_review: The paper presents an advancement in the modelling of deformable linear objects by introducing a hybrid framework that combines the precision of physics-based models with the adaptability of learning-based approaches. The results demonstrate the potential of DEFORM for real-time, accurate, and robust DLO modelling, making it a valuable contribution to the field. 

The integration of differentiable Discrete Elastic Rods (DER) with deep learning that leverages the strengths of both physics-based and data-driven models is not new to the field. But the results are good. The framework's ability to provide real-time predictions is significant for practical applications. 

My only scepticism about this paper is that it is not a robotics paper, robot are there just to move the DLO around. But of course, this work is beneficial to works that manipulate DLOs. Therefore I would encourage the authors to opensource the code once the paper is accepted

'These objects are often categorized as Deformable Linear Objects (DLOs) because they have complex dynamics due to bending and twisting' No, it is because they are linear and also deformable.
questions: questions: How to decide the number of vertices required for one DLO?

Can DEFORM handle twists? It is in the physic model but the demonstrate tasks do not have twist?
limitations: limitations: 3
review_summary: review_summary: This is a decent paper for CoRL
title: title: Nice paper with convincing experiments
paper_summary: paper_summary: The authors propose a discrete-time DLO dynamics model that combines a differntiable discrete elastic rod (DER) dynamics model with neural network regression. The model is derived in three steps:  1. Implement DER dynamics in PyTorch such that it is differentiable wrt. the potential parameter $\theta$ (resorting to implicit differentiation via Theseus) 2. An MLP is added to the DER dynamics model that learns the residual in the impulses and the residual of the numerical integration on the position-level 3. Add constraint enforcing algorithm to ensure inextensibility of the DLO model during integration (similar to [33])
main_review: main_review: I enjoyed reading this paper.

**Strength:**
- Proposes a novel method to learn DLO dynamics with DER
- Decently written
- Convincing experiments with various DLOs

**Weaknesses:**
- The introduction of the DER model could be improved, e.g. by adding an illustration of the involved coordinate frames
- **C1: Pros and cons of the choice of MLP inputs/outputs?** In Equation (7), the authors add to the DER model a neural network to learn the error on the impulse vector. The network receives as input and predicts the impulse in Cartesian coordinates. Clearly, this is an obvious way on how to add a network to Equation (7), but I would appreciate if the authors discuss the pros/cons of placing a network at this particular place in the model. After all, the DER model defines the potential function solely in terms of neighboring edge/frame coordinates, 

- **C2: Comparison of DER to other DLO dynamics formulations:** Initially emerging from the field of computer graphics, the DER implements Lagrangian dynamics of a chain of point masses with a nifty choice of coordinates. In comparison, the recent work [1] and [2] (see below) learn DLO dynamics by resorting to Newtonian multi-body dynamics following Featherstone's multi-body algorithms. In particular, [2] implements DLO dynamics as a chain of rigid bodies which are described in minimal coordinates (angles between body elements). As the dynamics are described in minimal coordinates, the forward dynamics are explicit such that they do not require solving an implicit optimization problem and one does not need to enforce implicit constraints with an additional algorithm. Also note that both Newtonian and Lagrangian dynamics equations respect energy conservation, they only differ by how conservative impulses/forces are being derived. In this regard, the pros and cons of resorting to Lagrangian mechanics and position-based dynamics requires further discussion.

- **C3: Algorithm for Inextensiblity Enforcement with PBD:** The following statement requires clarification: *"Further, as shown in Table 6, skipping Algorithm 2 in Algorithm 1 and relying solely on DNNs to capture the inextensibility of DLOs can lead to simulation instability. This instability arises because modeling stiff behavior, such as the inextensibility of DLOs, makes the learning process highly sensitive to inputs and hinders effective gradient propagation [8, 11]. "* - A general problem with adding neural network predictions to position-based dynamics (PBD) is that your dynamics/data lie on a low-dimensional manifold inside a high-dimensional space. In turn, over the cause of several open-loop predictions, the network's prediction errors will push the system state away from the manifold on which the data has been collected on. In turn, the network predictions deteriorate as the network has never seen these "off-manifold" points during training. Through Algorithm 2, the authors ensure that model inputs remain on the manifold on which training data has been collected on.

- **C4: Ensuring momentum conservation:** In line 187, the authors propose to weight the state correction term using the inertia matrices of neighboring nodes. I do not see, why this correction term enforces momentum conservation when applying the correction term. Either improve the explanation or plot the change in system momentum before/after the constraint enforcement is being applied with/without using the momentum correction term.


**Minor comments:**
- Line 98: Please emphasize that the velocity $V_t = X_{t} - X_{t-1}/ \Delta t$ is only an approximation of the actual velocity
- Typos, e.g. Appendix B - "Experiemental details"

*[1] -  "Pseudo-rigid body networks: learning interpretable deformable object dynamics from partial observations", Mamedov et al., 2024 - https://arxiv.org/abs/2307.07975*

*[2] - "Learning deformable linear object dynamics from a single trajectory", Mamedov et al., 2024 - https://arxiv.org/abs/2407.03476*
questions: questions: **Question - C1:** What are the pros and cons of the particular choice of input/outputs to the residual network? To which extent does the used GNN incorporate the structural knowledge underlying the DER model?

**Question - C2:** With the goal in mind of learning the errors of DLO dynamics with neural networks, what are the pros and cons of formulating dynamics in terms of a potential compared to resorting to Newtonian dynamics algorithms as in [2]?
limitations: limitations: 3
review_summary: review_summary: The proposed combination of DER dynamics models with neural network regression forms an important contribution towards understanding how to most effectively combine DLO dynamics models with neural networks for planning and control.  The discussion of DER and why the authors chose this particular DLO dynamics model compared to other models needs to be improved.