Cloth-Splatting: 3D Cloth State Estimation from RGB Supervision

Authors: alberta  longhini
marcel b üsching
bardienus p duisterhof
jens  lundell
jeffrey  ichnowski
mårten  björkman
danica  kragic
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 04:22:33

Abstract: We introduce Cloth-Splatting, a method for estimating 3D states of cloth from RGB images through a prediction-update framework. Cloth-Splatting leverages an action-conditioned dynamics model for predicting future states and uses 3D Gaussian Splatting to update the predicted states. Our key insight is that coupling a 3D mesh-based representation with Gaussian Splatting allows us to define a differentiable map between the cloth's state space and the image space. This enables the use of gradient-based optimization techniques to refine inaccurate state estimates using only RGB supervision. Our experiments demonstrate that Cloth-Splatting not only improves state estimation accuracy over current baselines but also reduces convergence time by $\sim 85$ \%.

Main paper: 
 introduction: 
 Teaching robots to fold, drape, or manipulate deformable objects such as cloths is fundamental to unlock a variety of applications ranging from healthcare to domestic and industrial environments [1] .While considerable progress has been made in rigid-object manipulation, manipulating deformables poses unique challenges, including infinite-dimensional state spaces, complex physical dynamics, and state estimation of self-occluded configurations [2] . Specifically, the problem of state estimation has led existing works on visual manipulation to either rely exclusively on 2D images, overlooking the cloth's 3D structure [3, 4, 5] , or to use 3D representations that neglect valuable information in RGB observations [6, 7, 8] .Prior work on cloth state estimation often relies on 3D particle-based representations derived from depth sensors, including graphs [9, 10] and point clouds [11] . While point clouds effectively capture the object's observable state, they lack comprehensive structural information [6] . Alternative approaches rely on Graph Neural Networks (GNNs) trained on simulated data to predict graph representations using action-conditioned cloth dynamics [7, 8, 12] . However, these representations require online fine-tuning to bridge the sim-to-real gap. Using depth information to supervise the refinement of the cloth's 3D state estimates involves time-consuming simulation-based optimization techniques, as it lacks visual clues such as texture [13] . How to leverage RGB observations, which contain such information, remains an open problem.We propose Cloth-Splatting, a method to estimate 3D states of cloth from RGB supervision by combining GS [14] with an action-conditioned dynamics model. The key idea of our method is to represent the 3D state of the cloth as a mesh and create a differentiable mapping between the cloth state space and the observation space using GS. This is achieved by populating the mesh faces with 3D Gaussians and expressing their positions relative to the mesh vertices. Given this, we can address the problem of estimating the 3D state of the cloth using a prediction-update framework akin to Bayesian filtering. Starting with a previous state estimate and a known robotic action, Cloth-Splatting predicts the next state using a learned dynamics model of the cloth. This prediction is then updated using RGB observations, leveraging the rendering loss provided by GS, allowing the refinement of the state estimate using visual clues such as texture and geometry.We experimentally evaluate the 3D state estimation of cloth in simulated and real-world environments. Results show that Cloth-Splatting outperforms all 2D and 3D baseline tracking techniques, being 57 % more accurate and ∼ 85 % faster than the best-performing baseline. We further showcase how refining mesh-based representations enables closed-loop manipulation of cloths. Together, the results suggest that tracking deformable objects using only RGB observations is feasible, efficient, and accurate. The contributions of this work are 1) a mesh-constrained object-centric extension of GS; 2) a vision-supervised 3D state estimation for cloth with action-conditioned dynamics priors, and 3) experiments that suggest successful sim-to-real transfer. 
related work: 
 Cloth Manipulation. Manipulating cloth in robotics involves tasks like folding, smoothing, lifting, and inserting objects into deformable bags [15, 3, 4, 5] . Manipulation approaches can be categorized into model-free methods, mapping cloth states or sensory data to actions [16, 17, 18] , and modelbased methods, utilizing cloth models to devise manipulation strategies [19, 13, 20] . Model-based approaches often rely on 3D representations focusing on optimizing pick-and-place actions. However, optimizing manipulation between pick and place in a closed-loop manner is crucial for handling planning errors and object variability [21, 22, 23, 24, 6] , but remains underexplored, largely due to the challenges of real-world cloth state estimation [13] . Our work addresses this gap, enabling feedback-loop manipulations using graph representations.Cloth State Estimation. Cloth state estimation is a well-studied area in robotics, computer vision, and computer graphics [25, 26] . While vision-based methods for 2D and 3D cloth state estimation have been developed for static observations of cloths lying flat on a surface [27, 28, 29, 30] , tracking these estimates over time is challenging due to the complex non-linear dynamics of deformables. A first solution to this problem proposes a self-supervised method that leverages an action-conditioned dynamics model of the cloth and test-time optimization to refine 3D state representations from point cloud observation [13] . However, this method requires aligning real-world states with simulated ones and disregards informative feedback provided by RGB observations. Unlike previous methods, our work refines the 3D cloth states using RGB feedback. This is made possible by a differentiable GS map linking 3D states to image observations.Vision-based tracking. Tracking points in 2D image space over time is a widely studied problem [31, 32, 33, 34, 35] . However, 3D visual trackers [36, 37] have gained popularity more recently, as they better address the occlusion challenges inherent in 2D tracking. Extensions of NeRF [38] to non-static scenes [39, 40, 41, 42] , already allow to track the 3D motion of dynamic scene content, but often lack applicability to real-world scenarios [43] . 3D Gaussian Splatting [14] based methods, like Dynamic 3D Gaussians (DynaGS) [44] , track by explicitly modeling the position and covariance of each Gaussian over time. In contrast, 4DGS [45] learns a continuous deformation field to track Gaussian displacements. MD-Splatting [46] builds on 4DGS for cloth tracking using a shadow network and physics-inspired regularization. DeformGS [47] extends on MD-Splatting by learning object-centric masks, and simplifying the regularization terms. However, these methods require dense observations (at least 50 cameras for MD-Splatting and DeformGS) and involve computationally costly per-scene optimizations (see Fig. 4 ).Mesh-constrained GS. Similar to Cloth-Splatting, several methods combine meshes and 3D Gaussians for deformable object representation [48, 49, 50] . However, like PhysGaussian [51] , they rely on explicitly modeled dynamics and lack refinement with visual observations, limiting their tracking capabilities. 
problem formulation: 
 The problem addressed in this paper is estimating the 3D state of a cloth M t+1 at time t + 1 given the observations Y 1:t+1 and the agent's actions a 1:t . The state of the cloth is represented as an augmented mesh M t = (V t , Vt , E t ), where V t ∈ R N×3 are the vertices positions, Vt ∈ R N×3 are the velocity of the vertices, and E t ∈ Z L×2 + the edges. Only V t and Vt are estimated while the edges E t , that form a triangular mesh structure for connectivity, remain constant, i.e.,E 0 = E 1 = • • • = E t .The observations Y t+1 = I 0 t+1 , . . . , I K t+1 are a set of K multi-view RGB images I k t+1 ∈ R w×h×3 with unique camera matrices P = {P 0 , . . . , P K } where P k ∈ R 4×4 . The actions a t ∈ R 3 are Cartesian end-effector velocities.We reframe this estimation problem as a Bayesian filtering problem, where the goal is to infer the joint posterior distribution p(M t+1 |Y 1:t+1 , a 1:t ) recursively over time with a general predictionupdate framework. The prediction step is assumed to have access to a transition probability function p(M t+1 |M t , a t ), which allows to compute a prior p(M t+1 |Y 1:t , a 1:t ) of the state of the system at time t + 1:EQUATIONprovided the history of observations Y 1:t+1 and the agent's actions a 1:t . Given a new observation Y t+1 , the update of the prior estimate is proportional to the product of the measurement likelihood p(Y t+1 |M t+1 ) and the predicted state. Thus, the joint posterior distribution is obtained as follows:EQUATIONwhere η is a normalization constant, ensuring that the posterior distribution integrates to 1. The state estimation problem then reduces to solving (2). 
method: 
 Solving the problem of cloth 3D state estimation through a prediction-update framework requires modeling two key components: the transition probability function p(M t+1 |M t , a t ) for the prediction step and the measurement likelihood p(Y t+1 |M t+1 ) for the update step.To handle the non-trivial deformable objects dynamics, we approximate the transition probability function with a deterministic GNN f θ θ θ parameterized by θ θ θ, which is described in more detail in Section 4.1. This GNN allows us to predict the state of the cloth at time t + 1, denoted as Mt+1 .For the update step, we model the measurement likelihood p(Y t+1 |M t+1 ) using a measurement model h(Y t+1 |M t+1 ), which maps the predicted state Mt+1 to the predicted observation Ŷt+1 . The likelihood is then expressed as a function of the measurement error ||Y t+1 -Ŷt+1 || 2 2 . Still, modeling the highly non-linear measurement model h that maps from the state space of the cloth to the image space is challenging. The key insight of our work is to approximate the measurement model using GS, such that Y t+1 ≈ h GS (M t+1 , P). Details on h GS are provided in Section 4.2. By combining the transition model f θ θ θ and the measurement model h GS , we create a differentiable mapping between the cloth's state space and the observation space. This allows us to obtain the updated state Mt+1 via gradient-based optimization by minimizing the measurement error in the observation space:EQUATIONas described in Section 4.3.An overview of the full method is visualized in Fig 2 , showing the prediction-update framework proposed in this work.We model the cloth's action-conditioned transition function f θ θ θ using a GNN parameterized by θ θ θ that is designed to effectively handle the mesh structure and complex cloth dynamics [9] . The input to the GNN consists of the last m states of the mesh M t-m:t and the external robot action a t . To condition the GNN to the action, we assume the grasped particle is rigidly attached to the gripper, directly updating its position and velocity, similar to Wang et al. [52] . This constraint allows f θ θ θ to dynamically predict the next state of the cloth Mt+1 = f θ θ θ (M t-m:t , a t ) by propagating the effects of actions across the graph structure. Specifically, the network predicts the acceleration of each node, which we integrate using a forward-Euler integrator to compute the next velocity and position of each vertex. We train f θ θ θ on a one-step Mean Squared Error (MSE) loss between the predicted and ground-truth meshes.We utilize GS as the measurement model Ŷt+1 = h GS ( Mt+1 , P), where h GS is a differentiable map between the mesh representation and the image space.Background. GS synthesises images by projecting the centers µ µ µ and covariances Σ of 3D Gaussians to the image plane and then uses α-blending to aggregate their colors c to pixel colors. To enforce the covariance matrix Σ to be positive and semi-definite, GS decompose it into a rotation R and scale S for each Gaussian:EQUATIONGiven the camera matrix P, the covariance matrix can be projected into image space as:EQUATION)where J is the Jacobian of the affine approximation of the projective transformation. During rendering, the color c p of a pixel is computed by blending N ordered Gaussians overlapping the pixel:EQUATIONwhere c i is the color of each Gaussian and α i is given by evaluating the projected 2D Gaussian with covariance multiplied with the learned opacity of each Gaussian.Mesh-constrained GS. To connect GS to the mesh representation, we assign each Gaussian to a face of the initial mesh. The mean position µ µ µ of each Gaussian is then defined in linear barycentric coordinates:EQUATIONwhere v 1 , v 2 , v 3 ∈ V are the vertices of the assigned face andb 1 + b 2 + b 3 = 1.The rotation R of the Gaussian is then expressed relative to the assigned face. During a mesh deformation, the parameters of the Gaussians and the barycentric coordinates remain constant and represent the cloth's time-invariant appearance.The goal of the state update is to improve the state prediction Mt+1 based on the observation Y t+1 such that the updated state Mt+1 minimizes the measurement error (Eq. 3).One option to refine the state would be to directly optimize the transition function f θ θ θ so it yields Mt+1 as an improved state estimate. However, f θ θ θ requires recursive roll-out for estimating future states. Coupled with the internal propagations of the GNN, this can lead to vanishing gradients.To circumvent the previous issues, we instead define the updated state as the sum:EQUATIONwhere δ Mt+1 = u ψ ψ ψ (t + 1) is a learned residual state update parameterized by ψ ψ ψ. The residual map u ψ ψ ψ is modeled as a Multi-layer Perceptron (MLP), mapping the time to an offset from the state prediction, and is optimized using gradient descent on L obs . Intuitively, this approach keeps the prior estimate from the GNN fixed, refining the cloth state by minimizing the discrepancy between the observed image and the rendered cloth state. The residual map is optimized per scene with the initial output set close to zero. We also add the regularization loss L reg = L SSIM + L iso + L magn , where L SSIM is the Structural Similarity Index Measure (SSIM) loss [53] , L iso ensures an As-Rigid-As-Possible (ARAP) behavior [54, 13] in the cloth, and L magn minimizes overall motion for numerical stability. Details on the regularization and the mesh-constrained GS can be found in the Appendix. 
experimental evaluation: 
 The primary objective of the experiments is to evaluate how fast and accurate Cloth-Splatting can estimate 3D cloth states from RGB images compared to other baselines. We further ablate different components of our method, provide qualitative results, and demonstrate the feasibility of using Cloth-Splatting for robotic cloth manipulation in both simulated and real-world scenarios.Dataset Generation. Due to the lack of ground-truth states in real-world scenarios, we generated a synthetic dataset composed of 75 different scenes to conduct quantitative evaluations. We included three different object categories: TOWEL, SHORTS, and TSHIRT with five mesh variations of each object and five different trajectories per variation. The mesh variations were generated following Lips et al. [55] . Each trajectory consists of random interactions with the cloths, performed using the NVIDIA Flex simulator [56, 57] . We recorded RGB-D images from 4 different camera views for each scene and time step, where the depth is only used for the baseline methods. We used Blender [58] to render photo-realistic scenes. We further record robot actions and ground-truth cloth states.Action-conditioned learned dynamics and mesh initialization. We implemented the actionconditioned models using the Graph Network Simulator (GNS) architecture proposed in [9] , which consists of a GNN with a node encoder, a processor, and a decoder. The input to the model is a mesh reconstructed from a 3D point cloud of the cloth at t = 0. We use Delaunay triangulation to reconstruct the mesh [59] . The point cloud can be obtained from either depth observations or multi-view stereo reconstructions [60] . We trained the model exclusively on the TOWEL object and split the dataset into a training, validation, and test set.Gaussian Splatting and Residual Model. For initialization, two randomly parameterized Gaussians are positioned on each face of the mesh, by sampling their barycentric coordinates from the distribution N (1/3, 0.05) and normalizing them so that they add up to 1. To model the appearance of the cloth, we first optimize the Gaussians using observations from t = 0 and the measurement loss L obs without regularization for the first 1.5k iterations. Subsequently, we jointly optimize the Gaussians and the residual network for an additional 5.5k -6.5k iterations for learning the residual deformation and previously unseen parts of the cloth, such as a differently colored backside.Update. We implement two different update procedures. The ITERATIVE update predicts one step ahead, refines the prediction with GS, and then uses the refined state as input to the GNN for the next prediction. In contrast, ROLLOUT predicts future states by unrolling the GNN over the entire trajectory and then refines all states simultaneously with GS, resulting in a faster overall runtime. We apply ROLLOUT in the tracking experiments and ITERATIVE in the manipulation experiments. A runtime and accuracy comparison between the approaches is provided in the appendix.To quantify the tracking performance, we use: median trajectory error (MTE) [61] , which measures the distance between the estimated cloth and ground-truth tracks; position accuracy (δ) [33] , which measures the percentage of tracks within the pre-defined distance thresholds 10, 20, 40, 80, and 160 mm to the ground truth; and the survival rate, which assesses the average number of frames until the tracking error exceeds a predefined threshold [61] , which we set to 50 mm.We compare Cloth-Splatting to the following baselines: Dynamic 3D Gaussians (DynaGS) [44] , which separately models the positions and rotations of each Gaussian; and MD-Splatting [46] , which extends GS by projecting non-metric Gaussians into a metric space to track deformable objects better. We also include the unrefined GNN predictions to establish a performance baseline, as well as a 2D tracking method obtained by evaluating RAFT [62] on all views and then reporting only the trajectories from the view with the lowest MTE. We denote this as RAFT-Oracle as it has access to the ground-truth cloth state to select the best-performing view.We assess the tracking accuracy of Cloth-Splatting across the 75 synthetic scenes against the baselines. The results in Table 1 show Cloth-Splatting outperforms the next best baseline MD-Splatting by 9.66 % in MTE on average, while also having the best overall position accuracy δ avg and survival rate. All methods show a varying performance between the different scenes, with all methods consistently being the worst on SHORTS which contain the largest, self-occluded deformation of the cloth with an example shown in Fig. 3 . Analysing the unrefined GNN predictions shows a competitive performance, even on the shapes which are out-of-distribution from the training data, indicating that the GNN generalizes across different geometries.We report ablation studies on the mesh-constrained component of Cloth-Splatting in Table 2 on a SHORTS scene. In the following, we refer to each ablation by the corresponding identifier, eq. (A1). Using the state predictions of the GNN without the GS update (A1) results in a high tracking error, likely due to the previously unobserved shape of the cloth. Similarly, using Cloth-Splatting without the GNN (A2) and relying only on the initial mesh state for initialization leads to poor tracking accuracy, as the model must learn the entire deformation just from visual observations. Additionally, removing the regularization terms (A3) significantly reduces accuracy as it fails to enforce cloth structural properties. Consequently, parts of the cloth stretch into unnatural states to satisfy the visual observations rather than maintaining the underlying shape. Interestingly, reducing the number of camera views (A6, A7) does not degrade the performance as much as other ablations, although we experience a larger drop in performance for a monocular setup (A5).To quantify how the cloth model prior affects convergence time, we compare the MTE of Cloth-Splatting to the best-performing baseline throughout their training durations. The experiments were performed on an NVIDIA 4090 GPU and an Intel i9-14900K processor. Fig. 4 shows that Cloth-Splatting achieves faster convergence times than the MD-Splatting and overall lower tracking error. The efficiency of Cloth-Splatting not only reduces computational costs but also enables faster adaptation to new scenarios, making it a more practical choice for real-world online applications.We present simulated qualitative rendering and tracking results in Fig. 3 and compare these to MD-Splatting. Both methods capture the underlying dynamics of the scene. Nevertheless, the tracking results of MD-Splatting are negatively influenced by occasional tracking of visual artifacts. Although high-quality renderings are not the goal of Cloth-Splatting, the results show that the general visual appearance of the observed cloth is reproduced faithfully.To showcase the transferability of Cloth-Splatting to the real world, we qualitatively evaluate the tracking of a real cloth folded by a 7-DOF Franka Emika Panda robot, observed by 3 extrinsically calibrated RealSense d435 from which we only collect RGB images. Fig. 1 shows qualitative real world results for Cloth-Splatting and the GNN predictions. While the GNN predictions partially predict the correct mesh structure, they suffer from compounding errors from the roll-out predictions. Subsequently, Cloth-Splatting successfully updates the mesh to better model the cloth shape. We further showcase that our state estimation process enables closed-loop optimization of folding trajectories using graph state representations. We focus on the half-folding task from Garcia-Camacho et al. [63] , optimizing the trajectory between predefined pick-and-place positions. Our method refines the cloth's state estimate during each re-planning step, using model-predictive control (MPC) for planning and Cloth-Splatting for refinements (MPC-CS). We compare against a baseline using a predefined linear trajectory (Fixed), an openloop (MPC-OL) baseline, and an oracle baseline (OL-ORACLE) with ground-truth state access. A detailed planning framework is presented in Algorithm 2 in the Appendix. The results in Table 3 show that our method outperforms all baselines, achieving results comparable to those of the oracle, underscoring the effectiveness of our method in refining state estimates for model-based closed-loop manipulation. Qualitative results for simulation and real-world manipulations can be found in Appendix F. 
conclusions: 
 We introduced Cloth-Splatting, an approach for 3D state estimation of cloths using RGB supervision. Cloth-Splatting integrates an action-conditioned cloth dynamics model with Gaussian Splatting to enhance the accuracy of 3D state representations based solely on RGB feedback using sparse (3-4) camera views. We experimentally validated the quality of these estimates and showcased the computational efficiency of our framework. These advancements position Cloth-Splatting as a more practical choice for real-world applications compared to existing baselines.Limitations. While Cloth-Splatting shows significant improvements over the baselines, its speed is still insufficient for real-time applications. Additionally, the need for a calibrated multi-camera setup may present challenges in real-world scenarios, although scalable computer vision techniques for calibration, such as those in Dust3r [60] , can help mitigate this issue. Since Cloth-Splatting assumes a static visual appearance, dynamic appearance changes, such as shadows or changing lighting conditions, can occasionally lead to the tracking of visual artifacts. Furthermore, the initial mesh initialization requires occlusion-free observations of the cloth. To address this limitation, recent advances in template-based reconstruction [64] of crumpled cloth present a promising direction for future research. 
: 
 Figure 1 : Cloth-Splatting state estimation of a real-world cloth. We introduce Cloth-Splatting, a 3D state estimation method for cloth from sparse RGB observations that combines Graph Neural Network (GNN) with Gaussian Splatting (GS). Given an initial sequence of 3D mesh predictions from the GNN (left), Cloth-Splatting (right) updates it using RGB observations, achieving state-of-the-art 3D tracking performance for deformables. 
 
 Appendix: The action-conditioned dynamics model builds on the GNS architecture [9] , which consists of three parts: encoder, processor, and decoder. The encoder consists of two MLPs, ϕ p and ϕ e , which map vertices and edge features into latent embeddings h i and g jk respectively. The processor comprises L = 15 Graph Network (GN) blocks with residual connections that propagate the information throughout the mesh. Each GN block includes an edge update MLP, a vertice update MLP, and a global update MLP. The decoder is an MLP ψ that outputs acceleration for each point: ẍi = ψ(h L i ), which we use to update the position of each vertice of the cloth mesh via Euler integration.The input vertice features consist of past k = 3 velocities and the vertice type. The vertice type is a binary flag used to distinguish grasped vertices from non-grasped vertices. The edge features include the distance vector (v jv k ) and its norm ∥v jv k ∥. To condition the model on the actions of the robot, we update the velocity of the pick point based on the robot's action before giving the state of the cloth in input to the network. This facilitates the propagation of the actions throughout the GNS to predict future states.We train the action-conditioned dynamics on towel objects, using the mean-squared error between predicted and simulator-obtained accelerations for 200 epochs using Adam [65] .Orientation estimation The orientation of the Gaussian R depends on the orientation of the associated face R F and the static orientation of the Gaussian on the face R ′ , resulting in R = R F R ′ . We estimate the face orientation R F in a deformed mesh via a vector registration between the initial positions of the face vertices v i 0 and the positions in the deformed state v i t . This can be formulated as an optimization problem minEQUATION)which we solve using the RoMa toolbox [66] . During the optimization of the barycentric coordinates, we permit them to take on negative values, which would locate the Gaussian outside of the assigned face. This approach enables us to detect, when a Gaussian should get assigned to a different face, with the barycentric coordinates then being relative to the vertices of that new face.Optimization For the mesh-constrained Gaussian Splatting, we build on the original Gaussian Splatting procedure, with the main modification that we constrain the Gaussian positions on the surface of a pre-defined mesh as described in the 4.2. Details of Gaussian Splatting, such as the pruning, densification, and regular resetting of opacities, remain unchanged. Nevertheless, in order to keep the number of 3D Gaussians low, we increase the required opacity for Gaussians to not be pruned, since we can assume that there are no transparent parts on the reconstructed cloth. Therefore, a normal reconstruction of the appearance of cloth only requires about 4k Gaussians.We observe that when the Gaussians are optimized over the whole range of training, the visual appearance and the tracking degrades. For example, the Gaussian position on the mesh starts to fit the deformed appearance instead of the residual dynamics model learning the proper offset. Therefore, the learning rates of the Gaussians' attributes (color, position, scale, . . . ) are annealed over the first 6k iterations and afterward frozen so only the residual dynamics model is optimized.We implement the residual dynamics model as a 3-layer ReLU MLP with a width of 256. The input to the MLP is a scalar value in the range [0, 1], corresponding to the normalized time step, which is encoded with the sinusoidal frequency encoding also used in NeRF [38] , using 6 frequencies. The output size is 3 × N , with N being the number of vertices in the mesh.We randomly initialize weights and biases of the output layer with a zero-centered normal distribution with a covariance of 0.0001, to start with a residual close to zero.As discussed in Section 4.3, we learned the state updated by adding the following regularization losses: L reg = L SSIM + L iso + L magn , where L SSIM is the SSIM loss [53] , L iso ensures neighboring vertices in the cloth maintain a constant distance, and L magn minimizes overall motion.The isometric loss:EQUATIONensures that the neighbouring vertices N (v t,i ) of v t,i maintain a constant distance from time t to t + 1.The structural similarity index measure loss (SSIM) [67] is estimated for windows of the images and goes beyond the purely per-pixel color loss and also considers color gradients within the pixels local neighborhood. The loss between two windows w an v can be estimated with:EQUATION)where µ is the mean color of each window, σ 2 the color (co-)variances, and c 1 and c 2 are constants to stabilize the loss.The motion loss:EQUATIONencourages to learn a solution with the smallest possible motion per vertice, which we found necessary to prevent instabilities during training.The synthetic dataset consists of meshes representing three types of cloth objects: TSHIRT, SHORTS, and TOWEL. We procedurally generate meshes with random configurations, sizes, and overall shapes for each category based on the methods detailed in [55] . Post-generation, the meshes are deformed using NVIDIA Flex [56, 57] with random manipulation trajectories.The manipulation trajectories are constructed using quadratic Bézier curves with three control points. Specifically, the pick and place locations represent the primary control points, which we randomly selected on the cloth particles. The third control point, positioned midway between the pick and place points, was set to a random height within the range [0.05, 0.15] cm. Additionally, this control point was randomly tilted between [-π/4, π/4] rad around the axis formed by the pick and place points to add variability in the manipulation trajectories. We finally discretized the manipulation trajectory into a series of small displacements depending on the gripper velocity, ∆x 1 , . . . , ∆x T , ensuring:x pick + T i=1 ∆x i = x place ,randomly sampling the gripper velocity in the interval [0.5, 2] cm/s.To bridge the simulation-to-reality gap, we rendered the complete manipulation trajectory using Blender [58] . The real-world set-up is shown in Fig. 6 . We used 3 calibrated RealSense d435 cameras to collect RGB observations of the environment. We utilized one rectangular cloth for the experiments, also visualized in Fig. 6 . The robot used for the experiments was a Franka-Emika Panda robot. We employed a Cartesian position controller to execute a folding trajectory, which was randomly generated using the same procedure as the simulated data. We assumed prior knowledge of the pick and place locations and that the cloth was already in a grasped configuration.We recorded RGB observations from all three cameras throughout the manipulation process. Depth observations were additionally captured at t = 0 to initialize the cloth mesh for dynamics predictions. At each timestep, segmentation and video tracking modules pre-trained on Grounding-DINO [68] and Segment Anything (SAM) [69] were used to generate masks of the cloth and the gripper, respectively, using the prompts "cloth" and "robot gripper." These masks were subsequently tracked over time using the video tracker XMEM [70] .For the tracking part application of our method, we predict future states by unrolling the GNN for the full trajectory length and then refine all states via GS. In the following, we refer to this version of our method as ROLLOUT. A second option is to predict H-steps ahead, refine these steps with GS, and then use the latest refined states to update the input of the GNN for the next H predictions. We will refer to this option as ITERATIVE. While ROLLOUT has the potential to be faster than ITERATIVE, it suffers from a larger error accumulation during the rollout of the GNN, which may negatively affect overall tracking performance. This experiment aims to evaluate the trade-off between iteratively updating the GNN input and the overall time of execution.In Table 4 , we present the tracking and execution time results over a single TOWEL scene. For the ITERATIVE method, we report the results for predicting 1, 2, 4, or 8 steps ahead before updating the GNN input with the refined states. ROLLOUT predicts and updates the full 16 timesteps of the scene jointly. Training parameters are the same for both methods, with the difference that we update the predictions ITERATIVE for 1000 iterations per prediction-update step, while we update the predictions from ROLLOUT directly 6000 iterations.We can observe that doubling the number of prediction steps approximately halves the total time of ITERATIVE. The results clearly show the trade-off between tracking quality and time. Since one of our objectives was to improve the speed of computation, we chose to present in the paper the tracking results of the ROLLOUT method, as it offers a significant speed advantage while maintaining sufficient accuracy. One of the assumptions of our method is to have access to an initial estimate of the mesh of the cloth at time t = 0. However, this estimate is prone to error due to sensor noise or approximation errors.To assess the robustness of our method to errors in mesh initialization, we selected a simulated towel and applied several augmentations to its initial mesh: , and additive random noise sampled from a multivariate Gaussian distribution with zero mean and 0.005 variance. These augmentations were applied to the initial mesh before being used to unroll the GNN. We then refined the predictions with our method and evaluated the tracking performance as in the previous section.Building on this initial analysis, we expanded our evaluation by introducing a variation of the experiment that tests our method's ability to refine mesh initialization based on observations of the cloth's initial state. In this extended approach, we first refined the augmented mesh with our method using observations at time t = 0. Following this, we unrolled the GNN using the refined initial mesh. Lastly, we refined the predictions with our method and evaluated the tracking performance.The results of these experiments are presented in Table 5 . While the tracking error is the lowest on the error-free initialization, our method is still able to achieve comparable tracking accuracy despite the inaccurate mesh initialization. Refining the initial mesh state resulted in a modest improvement overall. However, it was crucial in preventing two initializations with ROT augmentations from producing tracking performance so poor that the evaluation script could not generate valid metrics (highlighted with † in the Table. 5).This brings us to the conclusion that, while an error on the initialization has a negative influence on the tracking quality, Cloth-Splatting remains capable of sufficiently tracking the cloth and is therefore not significantly limited by this factor. In this experiment, we demonstrate both in simulation and in the real world that our state estimation process enables closed-loop optimization of folding trajectories using graph state representations. We focus on the benchmarking half-folding task introduced in [63] , where the objective is to fold a cloth in half. We assume pick-and-place positions to be given a-priori, and we aim to optimize the folding trajectory between these two. This task is particularly challenging with only one gripper, as a predefined linear trajectory between the pick and place positions does not result in an accurate fold. We illustrate this in Figure 7 (right -Fixed), where we show that folding by executing a predefined fixed trajectory between the pick and the place locations results in a poor fold. We refer to this manipulation as Fixed baseline. This scenario, common in model-based cloth manipulation, underscores the importance of optimizing the folding trajectory beyond the pick-and-place locations to achieve the desired goal. To address this, we optimize the trajectory in a closed-loop manner, planning from a sequence of random actions using our prediction-update framework along with model-predictive control (MPC). During each re-planning step, we obtain the refined state estimate of the cloth using our method. The algorithm describing the state refinement is presented in Alg. 1, while an overview of the planning algorithm is presented in 2. We refer to this manipulation approach as (MPC-CS). Simulation: We consider one TOWEL and one TSHIRT to test our method on variations of cloth types. The success is evaluated by calculating the mean-squared error (MSE) between the desired goal state and the final state of the cloth. We compare MPC-CS against the following baselines: the FIXED baseline previously introduced, an open-loop (MPC-OL) baseline that plans the best actions in an open-loop fashion, and an oracle baseline that has access to the ground truth state of the cloth at each time step (OL-ORACLE). We repeat the folding task 10 times per method and report the results in Table 6 . Our method outperforms all baselines, achieving results comparable to those of the oracle. Additionally, we present qualitative results of the final fold achieved by each method in Fig. 7 and Fig. 8 for the towel and the t-shirt respectively. These results underscore the effectiveness of our method in refining state estimates for model-based closed-loop manipulation. Although the planning is not fast enough to be executed in real-time, this is the first approach to showcase closed-loop manipulation with graph representations, underscoring the relevance of our proposed state estimation method.Real world: We evaluate our method, MPC-CS, on real-world half-folding, comparing it against two baselines: FIXED and the open-loop MPC-OL. Unlike the simulation experiments, we lack groundtruth meshes for this evaluation. Figure 9 presents qualitative results for each method, with the final outcome depicted at Time 4. It is evident that the FIXED baseline, which does not optimize the folding trajectory, and the MPC-OL baseline, which operates in an open-loop manner, both fail to produce satisfactory folds. In contrast, our method successfully refines the mesh estimate and replans the fold, resulting in a superior final outcome. This experiment demonstrates the applicability of our state estimation method in manipulation tasks.  


Decision: Summarizing the strengths and weaknesses pointed out by the reviewers:

Strengths:
- Using GS for closed-loop error correction of 3D state is novel
- Results outperform existing methods
- Good ablation analysis
- Generalization to real world (at least for a small sample size)

Weaknesses:
- Experiment sample size is fairly small, especially given it's in sim
- Limited to only state estimation, but no manipulation
- Missing some citations to other papers relating to cloth
- Method requires occlusion free initial state of cloth

The reviewers did have differing opinions on a couple points. 1 of the reviewers pointed out some clarity issues in the writing, while another thought it was well-written and clear. This suggests that at least some improvement in the clarity can be made. Another point is at least one reviewer pointed out the nicheness of the problem, while another said it is highly applicable to robotics. The authors might consider better justifying why robotics should care about 3D cloth state estimation.

Overall, all 3 reviewers recommend acceptance, although weakly. The authors should pay special attention to the specific concerns brought up by each reviewer to increase the likelihood the paper is accepted.

**Update after rebuttal**: The authors' rebuttal was quite extensive. They directly addressed many of the reviewers' concerns, and even ran several more experiments. All 3 reviewers responded to the rebuttal, 2 deciding to keep their weak accept recommendation and 1 reviewer upgrading to strong accept. I think this is a clear indication that this paper is a significant contribution to the field, and should be accepted.

Reviews: review: title: title: Nice idea well executed, but could use more experiments
paper_summary: paper_summary: This paper presents a way of using gaussian splatting to correct for errors of a graph neural network representing a cloth simulation
main_review: main_review: Pros:
+ I like the idea of using gaussian splatting as a way of correcting for errors based on visual observation. It was novel to me and seems like a good fit for the problem.
+ Incorporating closed loop solutions to this problem seems like a clear benefit over previous approaches
+ Results outperform existing methods on the experiments
+ Ablation analysis shows the benefit of several proposed methods
+ Results generalize to real world examples (in a small sample size)

Cons:
- The experiments were limited to 5 cloth examples with 2 sequences each. This is a pretty small sample size for simulation which could provide infinite samples.
- The methods section was a bit hard to follow at times, but many of these issues could be corrected with some rewrites.
- The problem scope itself is fairly limited to the niche problem of estimating (but not manipulating) cloth. I would have liked to see a method use this estimation to advance some end task performance.

Quality: The research seems well thought-out and backed by mathematical foundations and experiments in sim and real world.
Clarity: I think the writing quality was quite high in general, but certain parts of the method were unclear to me or arranged oddly (see below).
Originality: This method builds on prior work but has interesting novel improvements on those methods.
Significance: I think this work lacks broad significance due to its nature of being applied to the niche problem of cloth pose estimation. Additionally, as the authors mention, the runtime being slower than real-time means this method is probably not useful in actual robotic manipulation tasks involving cloths. However I do think the advancement over previous methods is significant enough to merit interest.

Clarity:
I feel certain rearranging of information would have made this paper easier to understand on a first read-through. Specifically, the placement of equations 1 and 2 before their "english language" explanations made me as a reader do more parsing and rereading than necessary. I would put the explanation first and then end with the equations. Similarly equation 4 takes argmin of psi, but never defines psi until 168 which was confusing to read. Another example is line 121 referencing figure 2 which appears an entire page earlier. As a rule, I would avoid putting a figure well before it is referenced or it will be either read out of context or ignored. 

Small typo: Line 52 says 57% times and 85% times. Is it 57% (as I suspect) or 57 times (which would be much much larger).
questions: questions: Certain things in this paper were mentioned but never defined in a satisfactory way. I would like the authors to answer some questions about the definitions in their methods section.
- The action space is mentioned several times but never defined. How is it parameterized? Motor torques? What is the dimensionality?
- How is the process initialized? Does it always assume a uniform flat cloth to start? It seems like this method updates existing predictions but wouldn't be able to give an initial prediction. Having just one initialization would severely limit the usefulness of the method.
- Over time, does the GNN incorporate the output of the GN or does it just adjust on a per-timestep basis? The video provided makes it look like the GNN is not changed so over time it accumulates more and more error. 
- The normalization constant was given a very long formula but never referenced. Does it matter or is it factored out by way of the minimization performed? 
- What range is the integral over in the normalization constant and equation 2? dM_t doesn't mean much to me (or could mean several things). 

Additionally I have some questions about the experiments:
- Can the authors explain why RAFT-Oracle does so poorly when it has access to extra information? I would have expected it to outperform other methods, though I'm not entirely sure what RAFT-Oracle actually is. However it seems to severely underperform.
- Why were there so few cloth examples and so few experimental trials? Given the experiments are in simulation, there should be an abundance of examples the authors could use. This in my opinion is the biggest flaw in the paper. It's hard to draw strong conclusions from 10 trials.
limitations: limitations: 3
review_summary: review_summary: The paper has strong foundations, but the lack of sufficient experiments means it could be of limited effectiveness outside the examples tried.
title: title: Good use of GS, needs some refinement
paper_summary: paper_summary: Cloth-splatting addresses the crucial challenge of state estimation in deformable object manipulation, a problem that has traditionally been difficult to solve due to the complex and dynamic nature of deformable materials. Key contributions include mesh-constrained Gaussian splatting for deformables, and an image-based 3D state estimation framework to accurately track deformables like cloth.
main_review: main_review: The paper presents a well-motivated approach to deformable object state estimation. This work holds significant implications for applications in robotics, where accurate and efficient cloth state estimation is necessary to manipulate them. The experiment on real-world cloth manipulation is particularly encouraging.

The following is some feedback and areas for improvement, including questions to be addressed in the rebuttal.

1. As part of the motivation, you should cite works that currently use visual manipulation of cloth [1][2][3]. Since there isn't much established work on cloth state estimation in the real world.
1. The method requires a complete and occlusion-free prior mesh of the cloth at the initial time step($M0 = (V0, \dot(V0), E0)$), limiting its applicability to scenarios where such a mesh is readily available. You should add this to the limitations.
1. With regards to your stated limitation that regular GS is static. You know you’re using such a GS for trajectories and motions in time. Why not use other GS methods, like those with deformation fields that can account for motion?
1. MD-splatting optim times (line 86-88): What is long per-scene optimization time? What is your per-scene optimization time? Can you include a table with comparisons of these?
1. Line 178: You say that $L_{iso}$ ensures neighboring vertices in cloth maintain a constant distance: Can you explain this? I thought the cloth can stretch or compress, since it is deformable. So triangle edges in the mesh can enlarge or become smaller.
1. Line 220 - 224: When you measure distance between ground truth and predicted cloth meshes, what distance metric do you use?
1. Your choice of a pre-trained dynamics model as the structural graph prior raises questions about its necessity and potential alternatives.
    1. @sim experiments: You have access to a sim, and provide the same inputs to your method as you do to the sim. Using the simulator itself as the structural prior could simplify the approach. So the sim results are good, but not as interesting. The real benefit of your method will show in the real world, when the simulator will fail to appropriately model the cloth deformation (sim2real gap).
    1. @real world: Even for real world expts, you currently use the sim to create dynamics data to train a dynamics model. And during execution, you need a full cloth mesh at time t=0. And robot actions at every timestep. You could just give this data to the simulator and use that as a structured prior. That way you save time and effort by not needing to train the dynamics model.
1. Additionally, since the real world is where the method can truly show its benefits, it would be great to have more expts in the real world with cloth manipulation. In the review, I have marked 'Robotics Focus' as 'Sufficient demonstration on hardware', but at least one more deformable expt would be helpful.

*Minor Points*

1. Typo: $\thickapprox 85$% should be $\sim 85$%, if you mean to say “approx 85%”
1. Typo line 52: What do you mean by 57% times? And ~85% times?
1. Typo line 66: Similarly to our work
1. Typo line 78: NeRF, not NerF
1. Section 3: It’s usually common to use symbol S for state (not M), and symbol O for observation (not Y).
1. Eq 4: Define $\psi$ earlier – it’s used in Eq (4) but defined in line 168
1. Typo line 145: ‘evaluating a the projected 2D Gaussian’
1. Line 238-250: When you say (1), (2) etc – are you referring to Eq. (1), Eq. (2)? At first I thought it was a numbered list. Please clarify.
1. The term "action-conditioned dynamics model" might be unnecessarily verbose, as most dynamics models in robotics and related fields inherently consider states and actions as inputs.

Update: Additional clarifications and real world experiments during the rebuttal are good. Thank you for your contribution.

*References:*

[1] Avigal, Y., Berscheid, L., Asfour, T., Kröger, T., & Goldberg, K. (2022, October). Speedfolding: Learning efficient bimanual folding of garments. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 1-8). IEEE.

[2] Salhotra, G., Liu, I. C. A., Dominguez-Kuhne, M., & Sukhatme, G. S. (2022). Learning deformable object manipulation from expert demonstrations. IEEE Robotics and Automation Letters, 7(4), 8775-8782.

[3] Xu, Z., Chi, C., Burchfiel, B., Cousineau, E., Feng, S., & Song, S. (2022). Dextairity: Deformable manipulation can be a breeze. arXiv preprint arXiv:2203.01197.
questions: questions: Please see main review for the feedback and questions that need to be addressed.
limitations: limitations: 2
review_summary: review_summary: Cloth-splatting presents a promising solution to the challenging problem of state estimation for deformable objects. The use of gaussian splatting is a great potential benefit to updating cloth meshes for real world experiments. However, addressing the feedback above and providing clarification on certain technical aspects would strengthen the paper's impact and adoption.
title: title: Initial Review
paper_summary: paper_summary: The paper proposes Cloth-Splatting, a deformable object state estimation framework based on a graph neural network (GNN) and a constrained Gaussian splatting (GS). In an iterative process, the state of the object is predicted by the GNN first. Then it’s corrected using the measurement with the constrained GS. The proposed method is evaluated quantitatively on a cloth tracking dataset and qualitatively on a real robot experiment. The cloth tracking experiment shows the proposed method outperforms the baselines.
main_review: main_review: Strength
1. The paper is well written and easy to follow.
2. The constrained GS is an interesting idea. By restricting the GS on the meshes, the proposed method is able to update the states using the differentiable map.
3. The ablation study clearly demonstrates the benefits of each module/loss in the framework. 
4. The proposed method outperforms the baselines on the synthetic dataset. 

Weakness
1. Since the GS are constrained on the meshes, the performance of the proposed method might be sensitive to the mesh initialization. 

2. The explanation on the iterative update is a bit unclear. Is the update parameterized by an MLP with time as the only input (Ln 168?)? Or is it a function of $M$ and $P$? However, in (3) and (4) $\delta M$ is obtained from an optimization? In addition, (3) and (4) should belong to section 4.3, not in the problem statement. 

3. Although the proposed method shows great performance using the metrics provided in the paper, the current metrics do not capture the “missing mesh.” For example, in Figure 1 one can clearly see that some part of the cloth is not covered by the mesh. If a method is initialized conservatively, say, only in the middle of the cloth, it can end up having better results because the large deformation is excluded from the evaluation.
questions: questions: 1. Can the GNN be overfitted to a specific motion? If we commend a motion that GNN has not seen before, will it still have a decent prediction?
2. How sensitive is the proposed method to the initialization of the meshes? Since the GS are constrained on the meshes, I imagine if the meshes are initialized incorrectly, the proposed method might not converge well?
3. How sensitive is the proposed method to the type of motion? Currently in the experiments there is only limited vertical movement of the clothes. Does it still track the clothes well if they are moved vertically?
4. How are $b_1,b_2,b_3$ determined in (8)?
5. What is the predefined distance set for the position accuracy metric?
6. How are the nodes initialized? From the provided video, it looks like there are some empty (transparent) spaces in the cloth. Is it an effect of missing nodes or is it because the GS does not capture the correct color?
7.  In addition, is there a way to prevent incorrect mesh initialization from happening?
limitations: limitations: 3
review_summary: review_summary: This paper proposes an interesting framework for deformable object state estimation. The constrained GS formulation is novel, and the performance on the synthetic dataset outperforms the baselines. However, the presentation on the state update can be improved, and there remain some questions to be answered. As a result, I’m recommending a weak accept.