Cloth-Splatting: 3D Cloth State Estimation from RGB Supervision
Authors: Alberta Longhini
Marcel Büsching
Bardienus Pieter Duisterhof
Jens Lundell
Jeffrey Ichnowski
Mårten Björkman
Danica Kragic
Venue: CoRL 2024 (2024)
Keywords: 3D State Representations
Gaussian Splatting
Deformable Objects
Vision-based Tracking
TL;DR: We present Cloth-Splatting, a method that integrates a pre-trained action-conditioned dynamics model with Gaussian Splatting for 3D state estimation of cloth-like deformable objects from RGB observations.

Abstract: We introduce Cloth-Splatting, a method for estimating 3D states of cloth from RGB images through a prediction-update framework. Cloth-Splatting leverages an action-conditioned dynamics model for predicting future states and uses 3D Gaussian Splatting to update the predicted states. Our key insight is that coupling a 3D mesh-based representation with Gaussian Splatting allows us to define a differentiable map between the cloth's state space and the image space. This enables the use of gradient-based optimization techniques to refine inaccurate state estimates using only RGB supervision. Our experiments demonstrate that Cloth-Splatting not only improves state estimation accuracy over current baselines but also reduces convergence time by $\sim 85$ \%.

Cloth-Splatting: 3D Cloth State Estimation from RGB Supervision
Alberta Longhini*!, Marcel Büisching*) Bardienus P. Duisterhof Jens Lundell', Jeffrey Ichnowski Mårten Björkman', Danica Kragic' KTTH Royal Institute of Technology "Carnegie Mellon University
Abstract: We introduce Cloth-Splatting, method for estimating 3D states of cloth from RGB images through prediction-update framework. Cloth-Splatting leverages an action-conditioned dynamics model for predicting future states and uses 3D) Gaussian Splatting to update the predicted states. Our key insight is that coupling a 3D mesh-based representation with Gaussian Splatting allows us to define differentiable map between the cloth's state space and the image space. This enables the of gradient-based optimization techniques refine inaccurate state estimates using only RGB supervision. Our experiments demonstrate that Cloth-Splatting not only improves state estimation accuracy over current baselines but also reduces convergence time by ~ 85 %. Code and videos available at: kth-
Keywords: 3D State Estimation, Gaussian Splatting, Vision-based Tracking, De- formable Objects
(a) 1=
(b)
(c) t=14
Figure 1: Cloth-Splatting state estimation of a real-world cloth. We introduce Cloth-Splatting a 3D state estimation method for cloth from sparse RGB observations that combines Graph Neural Network (GNN) with Gaussian Splatting (GS). Given initial sequence of 3D mesh predictions from the GNN (left), Cloth-Splatting (right) updates using RGB observations, achieving state-of-the-a tracking performance for deformables.
1 Introduction
Teaching robots to fold, drape, or manipulate deformable objects such as cloths is fundamental to unlock a variety of applications ranging from healthcare to domestic and industrial environments [1]. While considerable progress has been made rigid-object manipulation, manipulating deformables poses unique challenges, including diinite-dimensiona state spaces, complex physical dynamics, and state estimation of self-occluded configurations [2]. Specifically, the problem of state estimation has led existing works visual manipulation to either rely exclusively on 2D images, overlooking the cloth's 's 3D structure [3, 4, 5]. or to use 3D representations that neglect valuable information in RGB observations [6, 7, 8).
Prior work on cloth state estimation often relies on particle-based representations derived from depth sensors, including graphs [9, 10] and point clouds [11]. While point clouds effectively cap- ture the object's observable state, they lack comprehensive structural information [6]. Alternative
Equal contribution. Correspondence to (albertal, (asching)ekth...
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

approaches rely on Graph Neural Networks (GNNs) trained on simulated data to predict graph rep- resentations using action-conditioned cloth dynamics [7, 8, 12]. However, these representations require online fine-tuning to bridge the sim-to-real gap. Using depth information to supervise the refinement of the cloth's 3D state estimates involves time-consuming simulation- optimization techniques, as lacks visual clues such as texture [13]. How to leverage RGB observations, which contain such information, remains an open problem.
We propose Cloth- -oort-Spla,,nng. a method to estimate 3D states of cloth from RGB supervision by combining GS [14] with an action-conditioned dynamics model. The key idea of our method is to represent the 3D state of the cloth as a mesh and create a differentiable mapping between the cloth state space and the observation space using GS. This is achieved by populating the mesh faces with 3D Gaussians and expressing their positions relative the mesh vertices. Given this, we can address the problem of estimating the 3D state of the cloth using prediction-update framework akin to Bayesian filtering, Starting with previous state estimate and a known robotic action, Cloth- Splatting predicts the next state using a learned dynamics model of the cloth. This prediction is then updated using RGB observations, leveraging the rendering loss provided by GS, allowing the refinement of the state estimate using visual clues such as texture and geometry.
We experimentally evaluate the 3D state estimation of cloth in simulated and real-world environ- ments. Results show that Cloth-Splatting outperforms all 2D and 3D bascline tracking techniques, being 57 % more accurate and ~ 85 % faster than the bert-performing baseline. We further showcase how refining mesh-bases representations enables closed-loop manipulation of cloths. Together, the results suggest that tracking deformable objects using only RGB observations is feasible, efficient, and accurate. The contributions of this work are 1) a mesh-constrained object-centric extension of GS; 2) a vision- susionsspeeriiee 3D state estimation for cloth with centon-conditioned dynamies priors, and 3) experiments that suggest successful sim-to-real transfer.
2 Related Work
Cloth Manipulation. Manipulating cloth in robotics involves tasks like folding, smoothing, lifting, and inserting objects into deformable bags [15, 3, 4, 5]. Manipulation approaches can be categorized into model-free methods, mapping cloth states or sensory data to actions [16, 17, 18), and model- based methods, utilizing cloth models to devise manipulation strategies [19, 13, 20J. Model-based approaches often rely on 3D representations focusing on optimizing pick-and-place actions. How- ever, optimizing manipulation between pick and place in closed-loop manner is crucial for han- dling planning errors and object variability [21, 22, 23, 24, 6], but remains underexplored, largely due to the challenges of real-world cloth state estimation [13]. Our work addresses this gap, enabling feedback-loop manipulations using graph representations.
Cloth State Estimation. Cloth state estimation is a well-studied area in robotics, computer vision, and computer graphics [25, 26]. While vision- based methods for 2D and 3D cloth state estimation have been developed for static observations of cloths lying flat surface [27, 28, 29, 30], tracking these estimates over time is challenging due to the complex non-linear dynamics of deformables. A first solution to this problem proposes a self-supervised method that leverages an action-conditioned dynamics model of the cloth and test-time optimization to refine 3D state representations from point cloud observation [13]. However, this method requires aligning real-world states with simulated ones and disregards informative feedback provided by RGB observations. Unlike previous methods, our work refines the 3D cloth states using RGB feedback. This is made possible by differentiable GS map linking 3D states to image observations.
Vision-based tracking. Tracking points in 2D image space over time is a widely studied problem [31, 32, 33, 34, 35]. However, 3D visual trackers [36, 37] have gained popularity more recently, as they better address the occlusion challenges inherent in 2D tracking. Extensions of NeRF [38] to non-static scenes [39, 40, 41, 42], already allow to track the 3D motion of dynamic scene con- tent, but often lack applicability to real-world scenarios [43]. 3D Gaussian Splatting [14] based methods, like Dynamic 3D Gaussians (DynaGS) [44], track by explicitly modeling the position and

---

covariance of each Gaussian over time. In contrast, 4DGS [45] learns continuous deformation field to track Gaussian displacements. MDD-Splatting [46] builds on 4DGS for cloth tracking using a shadow network and physics-inspired regularization. DeformGS [47] extends on MD-Splatting by learning object-centric masks, and simplifying the regularization terms. However, these meth- ods require dense observations (at least 50 cameras for MD-Splatting and DeformGS) and involve computationally costly per-scene optimizations (see Fig. 4).
Mesh-constraingg GS. Similar to Cloth -otth-Spaatinn, several methods combine meshes and 3D Gaus- sians for deformable object representation [48, 49, 50). However, like PhysGaussian [51], they rely on explicitly modeled dynamics and lack refinement with visual observations, limiting their tracking capabilities.
Problem Formulation
The problem addressed in this paper estimating the 3D state of cloth M4+1 at time + given the observations 114+1 and the agent's actions a13- The state of the cloth is represented mented mesh M, = (Vr, Ve, E,), where V, E RNx3 are the vertices positions. V, E Ng3 are the velocity of the vertices, and E, E Æt.x² the edges. Only V, and V, are estimated while the edges E,, that form a triangular mesh structure for connectivity, remain constant, i.c., Eg Ej = =...= E,. The observations Y++1= = {Io
aug-
IK_ are set of multi-view RGB images I E Rwxhhx with unique camera matrices = (P", Pk) where pk E R4x4, The actions a, RR are Cartesian end-effector velocities.
We reframe this estimation problem Bayesian filtering problem, where the goal is to infer the joint posterior distribution p(M,+1| Ya a1:)) recursively over time with a general prediction- update framework. The prediction step is assumed to have access to transition probability function p MM, at), which allows to compute a prior p(M++1) Yia, an;) of the state of the system at time +
p Y lat, a1:) = p(M MM, (M) Y a12-))dMM
provided the history of observations Y1+++1 and the agent's actions a1:-- Given new observation Y the update of the prior estimate is proportional to the product of the measurement likelihood (0Y41/MMr and the predicted state. Thus, the joint posterior distribution is obtained as follows:
an:)) \ Y
pYY M++1))(MM++ Yı:,, a1:),
1)
where 7j is a normalization constant, ensuring that the posterior distribution integrates to 1. The state estimation problem then reduces solving (2).
Method
Solving the problem of cloth 3D state estimation through prediction-update framework requires modeling two key components: the transition probability function p(M,+ MM,, a.) for the predic- tion step and the measurement likelihood ((+++1l (M+11) for the update step.
To handle the non-trivial deformable objects dynamics, we approximate the transition probability function with a deterministic GNN fe parameterized by 0, which is described in more detail in Section 4.1. This GNN allows us to predict the state of the cloth at time t + 1, denoted as M
For the update step, model the measurement likelihood p(Y(+1 e+1 MM+1) using measurement model h(Y.41 r+ MM+1), which maps the predicted state M++1 to the predicted observation Y The likelihood then expressed as function of the measurement error Y|Y Ye+1lle. Still, modeling the highly non-linear measurement model h that maps from the state space of the cloth to the image space is challenging. The key insight of our work is to approximate the measurement model using GS, such that Y x has(M,.. 4+1 P). Details on has provided in Section 4.2.
3

---

Prediction
U Update
Final State
Ranliv
M
M
GNN
has
Labe
Yau
Ye.
M,
M
M
Yu
Mesh-constrained GS
Ma, Garnaar
GS
M
M
M
M M
Figure 2: Overview of Cloth-Splatting. We estimate the 3D state of the cloth through prediction-update framework. use GNN to approximate the transition function that estimates the next state M,+1 given an action a, and the state M,. We then employ our proposed mesh-constrained GS as mcasurement model has to ohtain an estimate of observation Y441 given the current refined state M M + 8M4+1 The differentiability of GS allows us iteratively optimize the update dM of the state estimate via the photometric consistency loss between the ground-truth observation Ye+1 and the rendered state Ye+1-
By combining the transition model and the measurement model has, create differentiable mapping between the cloth's state space and the observation space. This allows us to obtain the updated state M441 via gradient-based optimization by minimizing the measurement error in the observation space:
Lobs
Ye41
has (M,( P)
as described in Section 4.3.
An overview of the full method visualized in Fig 2, showing the prediction-update framework proposed in this work.
4.1 State Prediction
We model the cloth's 's action-conditioned transition function fe using a GNN parameterized by that is designed to effectively handle the mesh structure and complex cloth dynamics [9]. The input to the GNN consists of the last m states of the mesh M,-mst and the external robot action a, To condition the GNN to the action, we assume the grasped particle is rigidly attached to the gripper, directly updating its position and velocity, similar to Wang et al. [52]. This constraint allows fo to dynamically predict the next state of the cloth M = fe (M,-mct, ax) by propagating the effects of actions across the graph structure. Specifically, the network predicts the acceleration of each node, which we integrate using forward-Eule integrator to compute the next velocity and position of each vertex. We train fe on one-step Mean Squared Error (MSE) loss between the predicted and ground-truth meshes.
4.2 Mesh-constrained GS
We utilize GS as the measurement model Y.+1 has (M+11, where liss is differentiable map between the mesh representation and the image space.
Background. GS synthesises images by projecting the centers and covariances of 3D Gaussians the image plane and then uses a-blending to aggregate their colors e to pixel colors. To enforce the covariance matrix E to be positive and semi-definite, GS decompose into rotation R and scale S for each Gaussian:
RT.
(4)
Given the camera matrix the covariance matrix can be projected into image space as:
E' = J'=PPT"
(5)

---

where J is the Jacobian of the affine approximation of the projective transformation. During render- ing, the color cº of a pixel is computed by blending N ordered Gaussians overlapping the pixel:
i-1 (5))))
C
(6)
iEN
J=1
where c; is the color of each Gaussian and og is given by evaluating the projected 2D) Gaussian with covariance multiplied with the learned opacity of each Gaussian.
Mesh-constrained GS. To connect GS the mesh representation, we assign each Gaussian face of the initial mesh. The mean position μ of each Gaussian is then defined in linear barycentric coordinates:
= biv + byv2 + bgv"
where v v2,v?, v3 E V are the vertices of the assigned face and b + b2 - bj 1. The rotation R of the Gaussian then expressed relative to the assigned face. During a mesh deformation, the parameters of the Gaussians and the barycentric coordinates remain constant and represent the cloth's invarinvariant appearance.
4.3 State Update
The goal of the state update is to improve the state prediction M++1 based on the observation Y(41 such that the updated state M minimizes the measurement error (Eq. 3).
One option to refine the state would be to directly optimize the transition function fo so it yields M4+1 as an improved state estimate. However, fe requires recursive roll-out for estimating future states. Coupled with the internal propagations of the GNN, this can lead vanishing gradients.
To circumvent the previous issues, we instead define the updated state the sum:
M4+1 M + 5M
(8)
where MM++ = (t 1) is a learned residual state update parameterized by sb. The residual map Mg is modeled as a Multi-layer Perceptron (MLP), mapping the time to an offset from the state prediction, and optimized using gradient descent on Lohs- Intuitively, this approach keeps the prior estimate from the GNN fixed, refining the cloth state by minimizing the discrepancy between the observed image and the rendered cloth state. The residual map is optimized per scene with the initial output set close zero. We also add the regularization loss Lug LSSIM L + L where LSSIM is the Structural Similarity Index Measure (SSIM) loss [53], Liso ensures an As-Rigid-As- Possible (ARAP) behavior [54, 13] in the cloth, and Lmagn minimizes overall motion for numerical stability. Details on the regularization and the mesh-constrained GS can be found in the Appendix.
Experimental Evaluation
The primary objective of the experiments to evaluate how fast and accurate Cloth-Splatting can estimate 3D cloth states from RGB images compared to other baselines. We further ablate different components of our method, provide qualitative results, and demonstrate the feasibility of using Cloth-Splatting for robotic cloth manipulation in both simulated and real-world scenarios.
5.1 Experimental Set-up
Dataset Generation. Due to the lack of ground-truth states in real-world scenarios, we generated synthetic dataset composed of 75 different scenes to conduct quantitative evaluations. We included three different object categories: TOWEL, SHORTS, and TSHIRT with five mesh variations of each object and five different trajectories per variation. The mesh variations were generated following Lips al. [55]. Each trajectory consists of random interactions with the cloths, performed using the NVIDIA Flex simulator [56, 57]. We recorded RGB-D images from 4 different camera views for each scene and time step, where the depth is only used for the baseline methods. We used
5

---

Blender [58] to render photo-realistic scenes. We further record robot actions and ground-truth cloth states.
Action-conditioned learned dynamics and mesh initialization. We implemented the action- conditioned models using the Graph Network Simulator (GNS) architecture proposed in [9], which consists of GNN with node encoder, processor, and decoder. The input to the model is mesh reconstructed from a 3D point cloud of the cloth at t = O. We use Delaunay triangulation to reconstruct the mesh [59]. The point cloud can be obtained from either depth observations or multi-view stereo reconstructions [60]. We trained the model exclusively on the TOWEL object and split the dataset into a training, validation, and test set.
Gaussian Splatting and Residual Model. For initialization, two randomly parameterized Gaus- sians are positioned each face of the mesh, by sampling their barycentric coordinates from the distribution (11/3 0.05) and normalizing them so that they add up to 1. To model the appearance of the cloth, we first optimize the Gaussians using observations from = 0 and the measurement loss Lobs without regularization for the first 1.5k iterations. Subsequently, we jointly optimize the Gaussians and the residual network for an additional 5.5k 6.5k iterations for learning the residual deformation and previously unseen parts of the cloth, such as a differently colored backside.
Update. We implement two different update procedures. The ITERATIVE update predicts one step ahead, refines the prediction with GS, and then uses the refined state as input to the GNN for the next prediction. In contrast, ROLLOUT predicts future states by unrolling the GNN over the entire trajectory and then refines all states simultancously with GS, resulting in a faster overall runtime. We apply ROLLOUT in the tracking experiments and ITERATIVE in the manipulation experiments. A runtime and accuracy comparison between the approaches is provided in the appendix.
5.2 Quantitative Results
To quantify the tracking performance, we use: median trajectory error (MTE) [61], which measures the distance between the estimated cloth and ground-truth tracks; position accuracy (6) [33], which
measures the percentage of tracks within the pre-defined distance thresholds 10, 20, 40, 80, and 160 mm to the ground truth; and the survival rate, which assesses the average number of frames until the tracking error exceeds a predefined threshold [61], which we set to m0mm.
We compare Cloth-Splattigg to the following baselines: Dynamic 3D Gaussians (DynaGS) [44], which separately models the positions and rotations of each Gaussian; and MD-Splatting [46], which extends GS by projecting non non-metric Gaussians into a metric space to track deformable objects better. We also include the unrefined GNN predictions establish performance baseline, well as 2D tracking method obtained by evaluating RAFT [62] on all views and then reporting only the trajectories from the view with the lowest MTE. We denote this as RAFT-Oracle as it has access to the ground-truth cloth state to select the hest-performing view.
5.2.1 Comparative Tracking Evaluation
We assess the tracking accuracy of Cloth-Splatting across the 75 synthetic scenes against the base- lines. The results in Table show Cloth-Splatting outperforms the next best baseline MD-Splatting by 9.66 % in MTE on average, while also having the best overall position accuracy davg and sur- vival rate. All methods show a varying performance between the different scenes, with all methods consistently being the worst on SHORTS which contain the largest, self-occluded deformation of the cloth with an example shown in Fig. 3. Analysing the unrefined GNN predictions shows competi- tive performance, even on the shapes which are out-of-disttibution from the training data, indicating that the GNN generalizes across different geometries.
5.2.2 Model Ablation
We report ablation studies on the mesh-constrained component of Cloth Soott-Splatting in Table 2 on SHORTS scene. In the following, we refer each ablation by the corresponding identifier, eq. (A1).

---

Table 1: Quantitative evaluation. Comparison of Cloth-Splatting and the baselines in tracking quality. We report the mean and the standard deviation per metric (1 σ) and mark the best result bold.
Metric
Method
SHORTS
TOWEL
TSHIRT
Mean
RAFT-Oracle [35]
26.511 + 23.854
17.951
8,062
14,90
18.324 23821
3D MTE
DynaGS [44]
15.987 +: 17.972
7.352
3.265
9,574
4,304
10.924 = 11.246
+ [m]]
MD-Splatting [46]
7.043 + 9.265
1.755 + 2.711
2.109
2,907
3.635 = 6.235
GNN
17.388 +: 15.824
9.157 + 7.292
14,826
12.295
13.853
= 12:64
Cloth-Splatting
4.928 + 5,240
1.703
1,213
3.159
2.818
3,284
3.722
RAFT-Oracle [35]
0.660 0.166
0,661 0.114
0,744 = 0.133
0,683 0.143
3D 6avg
DynaGS [44]
0.773 0.141
0.829
0.808
0,083
0,804 0.105
MD-Splatting [46]
0.816 + 0.098
0.870 = 0.059
0.855 0.082
0.847 D.0K3
GNN
0.720 =: 0.128
0.791 = 0.076
0.731
0.141
0.747 = 0.121
Cloth-Splatting
0.851 + 0.075
0.879 0.057
0.858
0,080
0,862 0,072
RAFT Oracle [35]
0.666 = 0.170
0.734 = 0.119
0.752
0.182
0.715 = 0.145
Survial rate
DynaGS [44]
0.795 = 0.154
0,860 = 0.072
0,850
0.092
0.835 0.113
MD-Splatting [46]
0.869 4 0.085
0,910
0,059
0.881
0,082
0,887 0,077
GNN
0.771 +
0.860
0.770
0.138
0,800
Clott-Splatting
0.917 + 0.067
0.927 # 0,059
0.888
0.084
0,910 0.072
Using the state predictions of the GNN without the GS update (AI) results in high tracking error, likely due to the previously unob- served shape of the cloth. Similarly, using Cloth-Splatting without the GNN (A2) and relying only on the initial mesh state for initial- ization leads to poor tracking accuracy, as the model must learn the
Table 2: Model ablation.
3D MTE [mm]
Ablation
(A1) Only GNN
21.552
entire deformation just from visual observations. Additionally, re-
(A2) No GNN
16.135
(A3) No fireg
15.772
moving the regularization terms (A3) significantly reduces accuracy
(A4) No R₁
10.799
as it fails to enforce cloth structural properties. Consequently, parts of
(A5) view
16.525
the cloth stretch into unnatural states to satisfy the visual observations
(A6) views
9.535
rather than maintaining the underlying shape. Interestingly, reducing the number of camera views (A6, A7) does not degrade the perfor- mance as much as other ablations, although we experience a larger
(A7) views
9.004
Full (4 views)
8.923
drop in performance for a monocular setup (A5).
5.2.3 Time Ablation
To quantify how the cloth model prior affects convergence time, we compare the MTE of Cloth- Splatting to the best-performing baseline throughout their training durations. The experiments were performed NVIDIA 4090 GPU and Intel i9- 14900K processor. Fig. shows that Cloth- Splatting achieves faster convergence times than the MD- MD-Splatting and overall lower tracking error. The efficiency of Cloth-Splatting not only reduces computational costs but also enables faster adap- tation to new scenarios, making it a more practical choice for real-world online applications.
5.3 Qualitative Results
We present simulated qualitative rendering and tracking results in Fig. and compare these to MD- Splatting. Both methods capture the underlying dynamics of the scene. Nevertheless, the tracking results of MD-Splatingg are negatively influenced by occasional tracking of visual artifacts. Al- though high-quality renderings are not the goal of Cloth-Splatting. the results show that the general visual appearance of the observed cloth is reproduced faithfully.
To showcase the transferability of Cloth-Splating to the real world, we qualitatively evaluate the tracking of a real cloth folded by a 7-DOF Franka Emika Panda robot, observed by 3 extrinsically calibrated RealSense d435 from which we only collect RGB images. Fig. shows qualitative real world results for Cloth-Splatting and the GNN predictions. While the GNN predictions partially pre- dict the correct mesh structure, they suffer from compounding errors from the roll-out predictions. Subsequently, Cloth Cloth-Splatting successfully updates the mesh to better model the cloth shape.
7

---

24
Splatting
22
Splatting
MTE [mmm]
16.42 (627 83%%
16
14
12
10
6
8
10
16
18
20
Time [min]
Figure 4: Time ablation. Convergence times of Cloth-Splatting and MD-Splatting on SHORTS scene. The offset to the y-axis arises from GNN predictions (1.36 s) and the 3D Gaussian initialization (2
(a) Cloth Soth-Splatting
(b) AD-Sppatting
Figure 3: Qualitative results (sim). Tracking and rendering results on SHIRT (top) and SHORTS (bottom) scenes.
5.4 Robotic Manipulation Use Case
We further showcase that our state estimation process en- ables closed-loop optimization of folding trajectories using graph state representations. We focus on the half-folding task from Carcia-Camacho et [63], optimizing
Table 3: Manipulation results We port the mean standard deviation of the MSE computed between the fi- nal cloth state and the goal state. Re- sults presented in units of 10
trajectory be- tween predefined pick-and-place positions. Our method re- fines the cloth's state estimate during each re-planning step, using model-predictivy control (MPC) for planning and Cloth- Splatting for refinements (MPC-CS). We compare against baseline using a predefined linear trajectory (Fixed), an open- loop (MPC-OL) baseline, and an oracle baseline (OL-ORACLE) with ground-truth state access. detailed planning framework presented in Algorithm the Appendix. The results in Table 3 show that our method outper- forms all baselines, achieving results comparable to those of the oracle, underscoring the effective- ness of our method in refining state estimates for model-based closed- manipulation. Qualitative results for simulation and real-world manipulations be found in Appendix F.
Method
TOWEL
TSHIRT
FIXED MPC-OL
2.2 0.
2.4 +: 0.
1.8 #:1
7.3 5
MPC-CS (us)
0.6 266
1.2 o
MPC-ORACLE 0.4 0:02
0.8 o5
6 Conclusions
We introduced Cloth-Splatting, approach for 3D state estimation of cloths using RGB supervi- sion. Cloth-Splatting integrates an action-conditioned cloth dynamics model with Gaussian Splat- ting to enhance the accuracy of 3D state representations based solely on RGB feedback using sparse (3-4) camera views. We experimentally validated the quality these estimates and showcased the computational efficiency of our framework. These advancements position Cloth Splatting as more practical choice for real-world applications compared existing baselines.
Limitations. While Cloth-Splatting shows significant improvements over the baselines, its speed is still insufficient for real-time applications. Additiinnlly, the need for calibrated multi-camera set- up may present challenges in real-world scenarios, although scalable computer vision techniques for calibration, such as those in Dust3r [60]. can help mitigate this issue. Since Cloth- Cooth-Splattin assumes a static visual appearance, dynamic appearance changes, such as shadows or changing lighting conditions, occasionally lead to the tracking of visual artifacts. Furthermore, the initial mesh initialization requires occlusion-free observations of the cloth. To address this limitation, cent advances in template-based reconstruction [64] of crumpled cloth present promising direction for future research.

---

Acknowledgments
This work was supported by the Swedish Research Council; the Wallenberg Artificial Intelligence, Autonomous Systems and Software Program (WASP) funded by Knut and Alice Wallenberg Foun- dation; the European Research Council (ERC-884807); and the Center for Machine Learning and Health (CMLH). The computations were enabled by the Pittsburgh Supercomputing Center and by the Berzelius resource provided by the Knut and Alice Wallenberg Foundation at the Swedish Na- tional Supercomputer Centre.
References
[1] A. Longhini, Y. Wang, I. Garcia-Camacho, D. Blanco-Mulero, M. Moletta, M. Welle, G. Alenyå, H. Yin, Z. Erickson, D. Held, et al. Unfolding the literature: A review of robotic cloth manipulation. arXiv preprint arXiv:2407.01366 2024.
[2] J. Zhu, A. Cherubini, C. Dune, D. Navarro-Alarcon Alambeigi, D. Berenson, F. Ficuciello,
K. Harada, J. Kober, Li, al. Challenges and outlook in robotic manipulation of deformable objects. Robotics & Automation Magazine, 29(3)667-77, 2022.
[3] Y. Avigal, L. Berscheid, T. Asfour, T. Kröger, and K. Goldberg. Speedfolding: Learning
efficient bimanual folding of garments. In IEEE/RSJ IROS, pages 1-8, 2022.
[4] G. Salhotra, I.-C A. Liu, M. Bratinuez-Kuhnne and G. S. Sukhatme. Learning deformable object manipulation from expert demonstrations. RA-L, 7(4): 2022.
[5] 7. Xu, C. Chi, B. Burchfiel, E. Cousineau, S. Feng, and S. Song. Dextairity: Deformable manipulation can be a breeze. RSS, 2022.
[6] A. Longhini, M. C. Welle, Z. Erickson, and D. Kragic. AdaFold: Adapting folding trajectories of cloths via feedback-loop manipulation. arXiv preprint arXiv:2403.06211, 2024.
[7] X. Lin, Y. Wang, Huang, and D. Held. Learning visible connectivity dynamics for cloth smoothing. In CoRL, pages 256 266. PMLR, 2022.
[8] Huang. Lin, and D. Held. Mesh-based dynamics with occlusion reasoning for cloth manipulation. RSS, 2022.
[9] A. Sanchez-Gonzalez, J. Godwin, T. Pfaff, R. Ying, Leskovec, and P. Battaglia. Learning to simulate complex physics with graph networks. In ICML, pages 8459-8468. PMLR, 2020.
[10] A. Longhini, M. Moletta, A. Reichlin, M. C. Welle, D. Held, Z. Erickson, and D. Kragic. Edo- net: Learning elastic properties of deformable objects from graph dynamics. In IEEE ICRA, pages 3875-3881, 2023.
[11] P. Sundaresan, R. Antonova, and J. Bohgl. Diffcloud: Real-to-sim from point clouds with differentiable simulation and rendering of deformable objects. In 2022 IEEE/RSJ IROS, pages 10828-10835. IEEE, 2022.
[12] X. Ma. D. Hsu, and W. S. Lee. Learning latent graph dynamics for visual manipulation of deformable objects. In JEEE ICRA, pages 9266-82733 2022.
[13] Z. Huang, X. Lin, and D. Held. Self-supervised cloth reconstruction via action-conditioned cloth tracking. In IEEE ICRA, pages 7111-7118, 2023.
[14] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis. 3D Gaussian Splatting for real-time radiance field rendering. ACM ToG, 42(4), July 2023.
[15] D. Seita, A. Ganapathi, Hoque, M. Hwang, E. Cen, A. K. Tanwani, A. Balakrishna,
B. Thananjeyan, J. Ichnowski, N. Jamali, et al. Deep imitation learning of sequential fabric smoothing from algorithmic supervisor. In JEEE/RSJ IROS, pages 9651-9651 2020.

---

[16] Y. Wu, W. Yan, T. Kurutach, L. Pinto, and P. Abbeel. Learning to manipulate deformable objects without demonstrations. RSS, 2020.
[17] Y. Wang. Z. Sun. Z. Erickson, and D. Held. Learning to dress people with diverse poses and garments. RSS, 2023.
[18] Matas, S. James, and A. J. Davison. Sim-to-real reinforcement learning for deformable object manipulation. In CoRL, pages 734-743. PMLR, 2018.
[19] K. Mo, Y. Deng, C. Xia, and X. Wang. Learning language-conditioned deformable object manipulation with graph dynamics, arXiv preprint arXiv:2303.01310, 2023.
[20] R. Hoque, D. Seita, A. Balakrishna, A. Ganapathi, A. K. Tanwani, N. Jamali, K. Yamane,
S. Iba, and K. Goldberg. Visuospatial foresight for physical sequential fabric manipulation. Autonomous Robots, pages 1-25, 2022.
[21] Y. Li, Yue, D. Xu, E. Grinspun, and P. K. Allen, Folding deformable objects using predictive simulation and trajectory optimization. In IEEE/RSJ IROS, pages 6000-6006, 2015.
[22] I. Garcia-Camacho, A. Longhini, M. Welle, G. Alenya, D. Kragic, and J. Borras. Standard- ization of cloth objects and its relevance in robotic manipulation. IEEE ICRA, pages 8298- 8304. IEEE, 2024.
[23] V. Petrík and V. Kyrki. Feedhack-based fabric strip folding. In IEEE/RSJ IROS, pages 773- 778, 2019.
[24] J. Hietala, D. Blanco-Mulero, G. Alcan, and V. Kyrki. Learning visual feedback control for dynamic cloth folding. In IEEE/RSJ IROS, pages 1455-1462, 2022.
[25] S. Saito, J. Yang, Q. Ma, and M. J. Black. Scanimate: Weakly supervised learning of skinned clothed avatar networks. In IEEE/CVF CVPR, pages 2886-288 2897, 2021.
[26] Z. Su, Yu, Y. Wang, and Y. Liu. Deepeloth: Neural garment representation for shape and style editing. IEEE TPAMI, 45(2):1581-1599) 2022.
[27] R. Danëfek, E. Dibra, C. Öztireli, R. Ziegler, and M. Gross. Deepgarment: 3d garment shape estimation from a single image. In Computer Graphics Forum, volume 36, pages 269-280. Wiley Online Library, 2017.
[28] B. Jiang, J. Zhang, Y. Hong, J. Luo, L. Liu, and H. Bao. BCNet: Learning body and cloth shape from single image. ECCV, pages 18-35, 2020.
[29] C. Chi and Song. Garmentnets: Category-level pose estimation for garments via canonical space shape completion. In IEEE/CVF ICCV, pages 3324-3333 2021.
[30] Y. Li, Y. Wang, M. Case, S.-F. Chang, and P. K. Allen. Real-time pose estimation of deformable objects using a volumetric approach. In JEEE/RSJ IROS, pages 1046-1052. IEEE, 2014.
[31] N. Karaev, I. Rocco, B. Graham, N. Neverova, A. Vedaldi, and C. Rupprecht. CoTracker: It is better to track together. arXiv preprint arXiv:2307.07635, 2023.
[32] X. Shi, Z. Huang, W. Bian, D. Li, M. Zhang, K. C. Cheung, S. See, H. Qin, J. Dai, and H. Li. VideoFlow: Exploiting temporal cues for multi-frame optical flow estimation. In JEEE/CVF ICCV, pages 12435-12446 2023.
[33] C. Doersch, A. Gupta, L. Markeeva, A. Recasens, L. Smaira, Y. Aytar, J. Carreira, A. Zis- serman, and Y. Yang. Tap-vid: A benchmark for tracking any point in video. Advances in Neural Information Processing Systems, 35:13610- 13626, 2022.
[34] Q. Wang, Y.-.. Chang, R. Cai. Z. Li, B. Hariharan, A. Holynski, and N. Snavely. Tracking everything everywhere all at once. In IEEE/CVF ICCV, pages 19738-19749, 2023.
10

---

[35] Z. Teed and J. Deng. Raft: Recurrent all all-pairs field transforms for optical flow. In ECCV, pages 402-419. Springer, 2020.
[36] Z. Teed and J. Deng. RAFT-3D: Scene flow using rigid-motion embeddings. In IEEE/VVF CVPR, pages 8375-8384, 2021.
[37] W. Yuan, Z. Lv, T. Schmidt, and S. Lovegrove. STaR: Self-suuer tracking and reconstruc- tion of rigid objects in motion with neural rendering. In IEEE/CVF CVPR. pages 14152,, 2021.
[38] NeRF: Representing scenes as neural radiance fields for view synthesis. In ECCV, pages 405- 421, 2020.
[39] A. Pumarola, E. Corona, G. Pons-Moll, and F. Moreno-Noguer, D-NeRF: Neural Radiance Fields for Dynamic Scenes, In JEEE/CVF CVPR, pages 10313-10322, 2021.
[40] K. Park, U. Sinha, J. T. Barron, S. Bouaziz, B. Goldman, S. M. Seitz, and R. Martin-Brualla. Nerfies: Deformable Neural Radiance Fields. In IEEE/CVF ICCV, pages 5845-5854, 2021.
[41] Z. Li, S. Niklaus, N. Snavely, and O. Wang. Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes. In IEEE/CVF CVPR, pages 6494-6504, 2021.
[42] Li, Q. Wang, Cole, R. Tucker, and Snavely. DynlBaR: Neural Dynamic Image-Basee Based Rendering. In IEEE/CVF CVPR, pages 4273 4284, 2023.
[43] H. Gao, R. Li, S. Tulsiani, B. Russell, and A. Kanazawa. Monocular Dynamic View Synthesis: A Reality Check. In NeurlPS, pages
2022.
[44] Luiten, G. Kopanas, B. Leibe, and D. Ramanan. Dynamic 3D Gaussians: Tracking by persistent dynamic view synthesis. In International Conference on 3D Vision (3DV), pages 800-809, 2024.
[45] G. Wu, T. Yi, Fang, L. Xie, X. Zhang, W. Wei, W. Liu, Q. Tian, and X. Wang. 4D Gaussian Splatting for real-time dynamic scene rendering, In IEEE/CVF CVPR, pages 20310-20320, 2024.
[46] B. Duisterhof, Mandi, Yao, J.-W. Liu, M. Shou, S. Song, and J. Ichnowski. MD- Splatting: Learning metric deformation from 4D gaussians in highly deformable scenes. arXiv preprint arXiv:2312.00588 2023.
[47] B. P. Duisterhof, Z. Mandi, Y. Yao, J.-W. Liu, J. Seidenschwarz, M. Z. Shou, R. Deva, S. Song, S. Birchfield, B. Wen, and Ichnowski. DeformGS: Scene flow in highly deformable scenes for deformable object manipulation. WAFR, 2024.
[48] J. Waczyáska, P. Borycki, S. Tadeja, J. Tabor, and P. Spurek. GaMeS: Mesh-based adapting and modification of gaussian splatting, arXiv preprint arXiv:2402.014550 2024.
[49] L. Gao, J. Yang, B.-T. Zhang, J.-M. Sun, Y.J. Yuan, H. Fu, and Y.-K. Lai. Mesh-based gaussian splatting for real-time large-scale deformation. arXiv preprint arXiv:2402.04996 2024.
[50] Y. Jiang, C. Yu, T. Xie, X. Li, Y. Feng, H. Wang, M. Li, H. Lau, F. Gao, Y. Yang, and C. Jiang- VR-GS: A physical dynamics-awar interactive gaussian splatting system in virtual reality. In ACM SIGGRAPH, number 23, 2024.
[51] T. Xie, Z. Zong, Y. Qiu, X. Li, Feng, Y. Yang, and C. Jiang. PhysGaussian: Physics- integrated gaussians for generative dynamics. In IEEE/CVE CVPR, pages 4389-4398, 2024.
[52] Y. Wang, D. Held, and Z. Erickson. Visual haptic reasoning: Estimating contact forces by observing deformable object interactions. RA-L, 14)11146-111,,,,, 2022.

---

[53] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. 7IP, 2004.
[54] O. Sorkine and M. Alexa. As-rigid-as-possibee surface modeling. In Symposium on Geometry processing, volume 4, pages 109-116. Citeseer, 2007.
[55] Lips, V.-L.. De Gusseme, et al. Learning keypoints for robotic cloth manipulation using synthetic data, arXiv preprint arXiv:2401.01734 2024.
[56] M. Macklin, M. Müller, N. Chentanez, and T.-Y. Kim. Unified particle physics for real-time applications. ACM Transactions on Graphics (TOG), 33(4):1-12, 2014.
[57] X. Lin, Y. Wang, J. Olkin, and D. Held. Softgym: Benchmarking deep reinforcement learning for deformable object manipulation. In CoRL, pages 432-448. PMLR, 2021.
[58] B. O. Community. Blender a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. http:tt:::/wwbbleeteeet blender. org.
[59] B. Delaunay et al. Sur la sphere vide. Izv. Akad. Nauk SSSR, Otdelenie Matematicheskii i Estestvennyka Nauk, 7(793-800)1-2,, 1934.
[60] S. Wang, V. Leroy, Y. Cabon, B. Chidlovskii, and J. Revaud. Dust3r: Geometric 3d vision made easy, arXiv preprint arXiv:2312.14133 2023.
[61] Y. Zheng, A. W. Harley, Shen, G. Wetzstein, and L. J. Guibas. PointOdyssey: large-scale synthetic dataset for long-term point tracking. In CVPR, pages 19855-19865, 2023.
[62] Z. Teed and J. Deng. RAFT: Recurrent All-Pairs Field Transforms for Optical Flow. In ECCV, pages 402-419, 2020.
[63] I. Garcia-Camacho M. Lippi, M. C. Welle, H. Yin, R. Antonova, A. Varava, J. Borras, C. Tor- ras, A. Marino, G. Alenya, et al. Benchmarking bimanual cloth manipulation. RA-L, 5(2):
1111-1118, 2020.
[64] W. Wang, G. Li, M. Zamora, and S. Coros. Trtm: Template-based reconstruction and target- oriented manipulation of crumpled cloths. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 12522-12528. IEEE, 2024.
[65] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[66] R. Brégier. Deep regression on manifolds: A 3D rotation case study. International Confer- ence on 3D Vision (3DV), pages 166-174, 2021.
[67] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from enror visibility to structural similarity, IEEE Transactions on Image Processing, 13(4):600- 612, 2004.
[68] S. Liu, Z. Zeng. T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhu, et al. Ground- ing DINO: Marrying DINO with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.0544 2023.
[69] A. Kirillov, E. Mintun, N. Ravi, H. Mao, Rolland, Gustafson, T. Xiao. S. Whitehead, A. C. Berg, W.-Y. Lo, al. Segment anything. In IEEE/CVF CVPR, pages 3992-4003, 2023.
[70] H. K. Cheng and A. G. Schwing. XMem: Long-term video object segmentation with an atkinson-shifee memory model. ECCV, pages 640-658. Springer, 2022.
12

---

A Cloth-Splatting Implementation
A.1 Action-Conditonned Dynamics Architecture and Training
The action-conditioned dynamics model builds on the GNS architecture [9], which consists of three parts: encoder, processor, and decoder. The encoder consists of two MLPs, Φp and de, which map vertices and edge features into latent embeddings h2 and タ5* respectively. The processor comprises L 15 Graph Network (GN) blocks with residual connections that propagate the information throughout the mesh. Each GN block includes an edge update MLP, a vertice update MLP, and global update MLP. The decoder is an MLP ei that outputs acceleration for each point: ž; = (h("), which we use to update the position of each vertice of the cloth mesh via Euler integration.
The input vertice features consist of past & 3 velocities and the vertice type. The vertice type is binary flag used to distinguish grasped vertices from non-grasped vertices. The edge features include the distance vector (u,- uba) and its norm |0y 2k |l. To condition the model on the actions of the robot, we update the velocity of the pick point based on the robot's 's action before giving the state of the cloth in input to the network. This facilitates the propagation of the actions throughout the GNS to predict future states.
We train the action-conditionnd dynamics on towel objects, using the mean-squared error between predicted and comulltor-obtainge accelerations for 200 epochs using Adam [65].
A.2 Mesh-constrained Gaussian Splatting
Orientation estimation The orientation of the Gaussian R depends on the orientation of the as- sociated face R. and the static orientation of the Gaussian on the face R', resulting in R. = R.''. We estimate the face orientation R deformed mesh via a vector registration between the initial positions of the face vertices va and the positions the deformed state v/. This can be formulated optimization problem
Σ |R vill?. (=11233)
min R
which we solve using the RoMa toolbox [66]. During the optimization of the barycentric coordi- nates, we permit them to take on negative values, which would locate the Gaussian outside of the assigned face. This approach enables us to detect, when a Gaussian should get assigned to a different face, with the barycentric coordinates then being relative the vertices that new face.
Optimization For the mesh-constrained Gaussian Splatting, we build the original Gaussian Splatting procedure, with the main modification that we constrain the Gaussian positions the surface of pre-defined mesh as described in the 4.2. Details of Gaussian Splatting, such as the pruning, densification, and regular resetting of opacities, remain unchanged. Nevertheless, in order to keep the number of 3D Gaussians low, we increase the required opacity for Gaussians to not be pruned, since we can assume that there are no transparent parts on the reconstructed cloth. Therefore, a normal reconstruction of the appearance of cloth only requires about 4k Gaussians.
We observe that when the Gaussians optimized over the whole range of training, the visual ap- pearance and the tracking degrades. For example, the Gaussian position on the mesh starts to fit the deformed appearance instead of the residual dynamics model learning the proper offset. Therefore, the learning rates of the Gaussians' attributes (color, position, scale,... are annealed over the first 6k iterations and afterward frozen so only the residual dynamics model is optimized.
A.3 Residual dynamics model
We implement the residual dynamies model as 3-layer ReLU MLP with width of 256. The input the MLP is scalar value the range [0, 1J. corresponding to the normalized time step, which is encoded with the sinusoidal frequency encoding also used in NeRF [38], using 6 frequencies. The output size is 3 x N, with N being the number of vertices in the mesh.
13

---

We randomly initialize weights and biases of the output layer with zero- centered normal distribu- tion with covariance of 0.0001, to start with residual close to zero
A.4 Regularization
As discussed in Section 4.3, we learned the state updated by adding the following regularization losses: Ligg LSSI Liso + Lmagn. where ESSI is the SSIM loss [53], Liso ensures neighboring vertices in the cloth maintain constant distance, and Lmagn minimizes overall motion.
The isometric loss:
T-IN-1 Liso ΣΣ
Σ
(d(ve, ves) d(v 2(41,)l
(10)
1=
(Vva.)
ensures that the neighbouring vertices (V,,,) of Ut maintain a constant distance from time t to t ++1.
The structural similarity index measure loss (SSIM) [67] estimated for windows of the images and goes beyond the purely per-pixel color loss and also considers color gradients within the pixels local neighborhood. The loss between two windows an can be estimated with:
(2 + (2) + ch) (0] + 22 + ca) + o + 02)
LSSIM (v. w)
(11)
where y is the mean color of each window, o² the color (co-)variances, and ei and 02 are constants stabilize the loss.
The motion loss:
T-IN-1 ΣΣ Σ t=
Lmaggn
|| - U
(12)
encourages to learn solution with the smallest possible motion per vertice, which we found neces- sary to prevent instabilities during training.
B Synthetic Data
The synthetic dataset consists of meshes representing three types of cloth objects: TSHIRT, SHORTS, and TOWEL. We procedurally generate meshes with random configurations, sizes, and overall shapes for each category based the methods detailed in [55]. Post-generation, the meshes are deformed using NVIDIA Flex [56, 57] with random manipulation trajectories.
The manipulation trajectories are constructed using quadratic Bézier curves with three control points. Specifically, the pick and place locations represent the primary control points, which we ran- domly selected the cloth particles. The third control point, positioned midway between the pick and place points, was set to random height within the range (0.05, 0.15) cm. Additionally, this con- trol point was randomly tilted between |-T/4, T/ /4] rad around the axis formed by the pick and place points to add variability in the manipulation trajectories. We finally discretized the manipulation trajectory into a series of small displacements depending on the gripper velocity, Az ArT, ensuring:
Dpick
Az
Pplace
randomly sampling the gripper velocity in the interval [0.5, 2) cm/s.
To bridge the simulation-to-realle gap, rendered the complete manipulation trajectory using Blender [58].
14

---

(a) Towel: flat.
(b) Shorts: flat.
(c) T-shirt: flat.
(d) Towel: deformed.
(e) Shorts: deformed.
(n) T-shirt: deformed.
Figure Example of synthetic images generated for the objects considered our experiments (towel, shorts, (-shirt). For each object, we show the flat (top row) and the deformed (bottom row) states, rendered with Blender.
C Real-world Set-up and Data Collection
The real-world set-up shown in Fig. We used 3 calibrated RealSense d435 cameras collect RGB observations of the environment. We utilized one rectangular cloth for the experiments, also visualized in Fig. 6. The robot used for the experiments was a Franka-Emika Panda robot. We em- ployed Cartesian position controller to execute folding trajectory, which was randomly generated using the same procedure as the simulated data. We assumed prior knowledge of the pick and place locations and that the cloth was already in a grasped configuration.
We recorded RGB observations from all three cameras throughout the manipulation process. Depth observations were additionally captured at
to initialize the cloth mesh for dynamics pre- dictions. At each timestep, segmentation and video tracking modules pre-trained on Grounding- DINO [68] and Segment Anything (SAM) [69] were used to generate masks of the cloth and the gripper, respectively, using the prompts "cloth" and "robot gripper." These masks were subsequently tracked over time using the video tracker XMEM [70].
D Time-comparison between ITERATIVE and ROLLOUT updates
For the tracking part application of our method, we predict future states by unrolling the GNN for the full trajectory length and then refine all states via GS. In the following, we refer this version of method as ROLLOUT. second option is to predict H-steps ahead, refine these steps with GS, and then use the latest refined states to update the input of the GNN for the next H predictions. We will refer to this option ITERATIVE. While ROLLOUT has the potential to be faster than ITERATIVE, it suffers from larger error accumulation during the rollout of the GNN, which may negatively affect overall tracking performance. This experiment aims to evaluate the trade-off between iteratively updating the GNN input and the overall time of execution.
In Table 4, present the tracking and execution time results single TOWEL scene. For the ITERATIVE method, we report the results for predicting 1, 2, 4, or 8 steps ahead before updating
15

---

Figure 6: Overview of experimental set-up.
the GNN input with the refined states. ROLLOUT predicts and updates the full 16 timesteps of the scene jointly. Training parameters are the same for both methods, with the difference that we up- date the predictions ITERATIVE for 1000 iterations per prediction-updatt step, while we update the predictions from ROLLOUT directly 6000 iterations.
We can observe that doubling the number of prediction steps approximately halves the total time of ITERATIVE. The results clearly show the trade-off between tracking quality and time. Since one of our objectives was to improve the speed of computation, we chose to present in the paper the tracking results of the ROLLOUT method, as it offers significant speed advantage while maintaining sufficient accuracy.
Table 4: Comparison of ITERATIVE and ROLLOUT versions of Cloth Sooth-Splattnng..
ITERATIVE
ROLLOUT
prediction steps
2
16
3D MTE [mm]
0.767
0.890
0.819
1.170
5.328
3D davg
0.893
0.891
0.893
0.867
0.753
Survival rate
t
0.937
0.950
0.947
0.907
0.776
Time [min]
4 53:35
30:14
15:53
9:32
2:12
E Robustness Against Initialization Error
One of the assumptions of our method is to have access to an initial estimate of the mesh of the cloth at time = 0. However, this estimate is prone to eиror due to sensor noise or approximation errors. To assess the robustness of our method errors in mesh initialization, we selected a simulated towel and applied several augmentations to its initial mesh: TRANS, ROT, SCALING, NOISE and all combined TRSN. For each augmentation, randomly sampled 10 different values. Given cloth size of 0.2 x 0.2[m²), we sampled and translations within [-0.05, 0.05), translations
16

---

within [-0.003, 0.003), yaw rotations within [-30, +30] degrees, random scaling coefficients within [0.8, 1.2), and additive random noise sampled from a multivariate Gaussian distribution with zero mean and 0.005 variance. These augmentations were applied to the initial mesh before being used to unroll the GNN. We then refined the predictions with our method and evaluated the tracking performance as in the previous section.
Building on this initial analysis, we expanded our evaluation by introducing a variation of the exper- iment that tests our method's ability to refine mesh initialization based on observations of the cloth's s initial state. In this extended approach, we first refined the augmented mesh with our method using observations at time = Following this, we unrolled the GNN using the refined initial mesh. Lastly, we refined the predictions with our method and evaluated the tracking performance.
The results of these experiments are presented in Table 5. While the tracking error is the lowest on the error-free initialization, our method is still able to achieve comparable tracking accuracy despite the inaccurate mesh initialization. Refining the initial mesh state resulted in a modest improvement overall. However, it was crucial in preventing two initializations with ROT augmentations from producing tracking performance so poor that the evaluation script could not generate valid metrics (highlighted with f in the Table. 5).
This brings us to the conclusion that, while an emor on the initialization has a negative influence on the tracking quality, Cloth-Splatting remains capable of sufficiently tracking the cloth and is therefore not significantly limited by this factor.
Table 5: Tracking with initialization error: We report the tracking metrics as mean per augmentation type and include the error free initialization as Ref.
Augmentation
Metric
Refined
Ref
TRANS
ROT
SCALING
NOISE
TRSN
Mean
3D MTE
No
3.346
2.598
2.998
2.482
3.595
2.961
2.193
+ [mm]
Yes
3.123
2.887
2.763
2.859
3.394
2.953
3D 8avg
No 0.819
0.825
0.823
0.826
0.815
0.822
0.835
Yes
0.824
0.826
0.828
0.825
0.820
0.825
Survival
No
0.865
0.871
0.867
0.869
0.864
0.867
0.887
Yes
0.871
0.872
0.875
0.870
0.866
0.872
1 Two augmentations had to be excluded since their tracking performance was too insufficient to estimate the metrics.
F Manipulation Experiment Beyond State Estimation
In this experiment, we demonstrate both in simulation and in the real world that our state estimation process enables closed-loop optimization of folding trajectories using graph state representations. We focus on the benchmarking half-folding task introduced in [63], where the objective is to fold cloth in half. We assume pick-and-plac positions to be given a-priori, and we aim to optimize the folding trajectory between these two. This task is particularly challenging with only one gripper, as a predefined linear trajectory between the pick and place positions does not result in an accu- rate fold. We illustrate this in Figure 7 (right Fixed), where we show that folding by executing predefined fixed trajectory between the pick and the place locations results in poor fold. We refer to this manipulation as Fixed baseline. This scenario, common in model-based cloth manip- ulation, underscores the importance of optimizing the folding trajectory beyond the pick-and-place locations to achieve the desired goal. To address this, we optimize the trajectory in a closed-loop manner, planning from a sequence of random actions using our prediction- with moree-predietive control (MPC). During each re-planning step, we obtain the refined state es- timate of the cloth using our method. The algorithm describing the state refinement is presented in Alg. 1, while overview of the planning algorithm is presented in 2. We refer to this manipulation approach (MPC-CS).
framework along
17

---

Algorithm 1: Cloth-Splatting. State refinement.
Result:
Refined state M
Input: Estimated state: M4+1, New observation: Yt+1, Measurement model: has, Camera matrices: P, Epochs: E.
MM++1 f 0
M1+1 + M,+1 + 5M++1
3 for E - E do
Y141 -- has( (M, P)
Lobs +- Y Y 1+11/2
D Eq. (3)
8M++1 x VLohs
M441 + M + 8M++1
8 end
We consider one TOWEL and one TSHIRT our method variations of cloth types. The success is evaluated by calculating the mean-squared enror (MSE) between the desired goal state and the final state of the cloth. We compare MPC-CS against the following baselines: the FIXED baseline previously introduced, an open-loop MLC baseline that plans the best actions in an open-loop fashion, and an oracle baseline that has access to the ground truth state of the cloth at each time step (OL-ORACLE). We repeat the folding task 10 times per method and report the re- sults in Table Our method outperforms all baselines, achieving results comparable those of the oracle. Additionally, we present qualitative results of the final fold achieved by each method in Fig. 7 and Fig. 8 for the towel and the t-shirt respectively. These results underscore the effective- ness of method in refining state estimates for model-based closed-loop manipulation. Although the planning is not fast enough to be executed in real-time, this is the first approach to showcase closed-loop manipulation with graph representations, underscoring the relevance of our proposed state estimation method.
Simulation:
We evaluate our method, MPC-CS, on real-world half-folding, comparing it against two baselines: FIXED and the open-loop MPC-OL. Unlike the simulation experiments, we lack ground- truth meshes for this evaluation. Figure 9 presents qualitative results for each method, with the final outcome depicted at Time 4. It is evident that the FIXED baseline, which does not optimize the folding trajectory, and the MPC-OL baseline, which operates in an open-loop manner, both fail to produce satisfactory folds. In contrast, our method successfully refines the mesh estimate and re- plans the fold, resulting in superior final outcome. This experiment demonstrates the applicability of our state estimation method in manipulation tasks.
Real world:
18

---

Algorithm 2: Closed-loop manipulation with Cloth- Kloth-Splatting iterative update.
Result:
Optimized folding actions To.r.
Input: Initial observation: Yo. Initial cloth point cloud: PCo. Camera matrices P. Pick and
place positions: {Trick, Iplace } Goal state: Mg. Transition function: fo. Measurement model: hgs. Planning Horizon: Prediction Horizon: H, Number of action candidates: N, Initial control sequence ap:H, Control variance: Σ
Mo f- Mesh Initialization ((PCa))
2 Mo + Mo
Initialize Gaussian baricenters has(Mo; P)
for f +---1 1 to T do
for n 1 to do
H < -N(a Σ) M, M,
>> Sample candidates
Refined state as input
for h t to H do
M1+1 t fM ah) Ja(a)i) + || - M
>> Model rollout
10
Cost function
11
end
J (a" (14H) + Eh=t Ja(
12
13
end
G arg Ja nE(I...,N)
15
Execute a;
16
Y++1 + New observation
17
M441 + CS(M YGs,,
b> Cloth- Mhoth-Splatting- AI).
18 end
Table 6: Comparison of different manipulation strategies. Each method is tested 10 times, and report the and standard deviation of the MSE computed between the final state and the goal state. Results presented in units of 10
Metric
Object
FIXED
MPC-OL
MPC-CS (us)
MPC-ORACLE
MSE
TOWEL
2.2 12::0.4 0.4
1.8 + 2.1
0.6 + 0.6
0.4 + 0.2
MSE
TSHIRT
2.4
7.3 5.2
1.2 + 0.8
0.8 0.5
19

---

START
(a) FIXED
(b) MPC-OL
(c) MPC-CS
(d) MPC-ORACLE
GOAL
Figure 7: Qualitative results the half folding for different manipulation strategies.
START
(a) FIXED
(b) MPC-OL
(c) MPC-CS
(d) MPC-ORACLE
GOAL
Figure Qualitative results the half folding for different manipulation strategies.
20

---

Time 1
Time 2
Time 3
Time 4
FIXED
(b) MPC-OL
(c) MPC-CS
Figure Qualitative results of the manipulation outcomes the methods (a) PIXED, (b) MPC-OL, and (c) MPC-CS are presented. Each method is illustrated at four distinct time points during execution, with the final fold shown at Time 4. Best viewed with zoom.
21


Reviews:
[{"review_id": "MqZVFVz48R", "reviewer": "Reviewer_eUkr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a way of using gaussian splatting to correct for errors of a graph neural network representing a cloth simulation", "summary_of_recommendation": "The paper has strong foundations, but the lack of sufficient experiments means it could be of limited effectiveness outside the examples tried.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Certain things in this paper were mentioned but never defined in a satisfactory way. I would like the authors to answer some questions about the definitions in their methods section.\n- The action space is mentioned several times but never defined. How is it parameterized? Motor torques? What is the dimensionality?\n- How is the process initialized? Does it always assume a uniform flat cloth to start? It seems like this method updates existing predictions but wouldn't be able to give an initial prediction. Having just one initialization would severely limit the usefulness of the method.\n- Over time, does the GNN incorporate the output of the GN or does it just adjust on a per-timestep basis? The video provided makes it look like the GNN is not changed so over time it accumulates more and more error. \n- The normalization constant was given a very long formula but never referenced. Does it matter or is it factored out by way of the minimization performed? \n- What range is the integral over in the normalization constant and equation 2? dM_t doesn't mean much to me (or could mean several things). \n\nAdditionally I have some questions about the experiments:\n- Can the authors explain why RAFT-Oracle does so poorly when it has access to extra information? I would have expected it to outperform other methods, though I'm not entirely sure what RAFT-Oracle actually is. However it seems to severely underperform.\n- Why were there so few cloth examples and so few experimental trials? Given the experiments are in simulation, there should be an abundance of examples the authors could use. This in my opinion is the biggest flaw in the paper. It's hard to draw strong conclusions from 10 trials.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "coRISsheJd", "reviewer": "Reviewer_FCbJ", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Cloth-splatting addresses the crucial challenge of state estimation in deformable object manipulation, a problem that has traditionally been difficult to solve due to the complex and dynamic nature of deformable materials. Key contributions include mesh-constrained Gaussian splatting for deformables, and an image-based 3D state estimation framework to accurately track deformables like cloth.", "summary_of_recommendation": "Cloth-splatting presents a promising solution to the challenging problem of state estimation for deformable objects. The use of gaussian splatting is a great potential benefit to updating cloth meshes for real world experiments. However, addressing the feedback above and providing clarification on certain technical aspects would strengthen the paper's impact and adoption.", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "Please see main review for the feedback and questions that need to be addressed.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vhsKWMOGrK", "reviewer": "Reviewer_wB9e", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes Cloth-Splatting, a deformable object state estimation framework based on a graph neural network (GNN) and a constrained Gaussian splatting (GS). In an iterative process, the state of the object is predicted by the GNN first. Then it’s corrected using the measurement with the constrained GS. The proposed method is evaluated quantitatively on a cloth tracking dataset and qualitatively on a real robot experiment. The cloth tracking experiment shows the proposed method outperforms the baselines.", "summary_of_recommendation": "This paper proposes an interesting framework for deformable object state estimation. The constrained GS formulation is novel, and the performance on the synthetic dataset outperforms the baselines. However, the presentation on the state update can be improved, and there remain some questions to be answered. As a result, I’m recommending a weak accept.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. Can the GNN be overfitted to a specific motion? If we commend a motion that GNN has not seen before, will it still have a decent prediction?\n2. How sensitive is the proposed method to the initialization of the meshes? Since the GS are constrained on the meshes, I imagine if the meshes are initialized incorrectly, the proposed method might not converge well?\n3. How sensitive is the proposed method to the type of motion? Currently in the experiments there is only limited vertical movement of the clothes. Does it still track the clothes well if they are moved vertically?\n4. How are $b_1,b_2,b_3$ determined in (8)?\n5. What is the predefined distance set for the position accuracy metric?\n6. How are the nodes initialized? From the provided video, it looks like there are some empty (transparent) spaces in the cloth. Is it an effect of missing nodes or is it because the GS does not capture the correct color?\n7.  In addition, is there a way to prevent incorrect mesh initialization from happening?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Summarizing the strengths and weaknesses pointed out by the reviewers:

Strengths:
- Using GS for closed-loop error correction of 3D state is novel
- Results outperform existing methods
- Good ablation analysis
- Generalization to real world (at least for a small sample size)

Weaknesses:
- Experiment sample size is fairly small, especially given it's in sim
- Limited to only state estimation, but no manipulation
- Missing some citations to other papers relating to cloth
- Method requires occlusion free initial state of cloth

The reviewers did have differing opinions on a couple points. 1 of the reviewers pointed out some clarity issues in the writing, while another thought it was well-written and clear. This suggests that at least some improvement in the clarity can be made. Another point is at least one reviewer pointed out the nicheness of the problem, while another said it is highly applicable to robotics. The authors might consider better justifying why robotics should care about 3D cloth state estimation.

Overall, all 3 reviewers recommend acceptance, although weakly. The authors should pay special attention to the specific concerns brought up by each reviewer to increase the likelihood the paper is accepted.

**Update after rebuttal**: The authors' rebuttal was quite extensive. They directly addressed many of the reviewers' concerns, and even ran several more experiments. All 3 reviewers responded to the rebuttal, 2 deciding to keep their weak accept recommendation and 1 reviewer upgrading to strong accept. I think this is a clear indication that this paper is a significant contribution to the field, and should be accepted.
Decision: Accept