CloDS: Visual-Only Unsupervised Cloth Dynamics Learning in Unknown Conditions
Authors: Yu-Liang Zhan
Jian Li
Wenbing Huang
Yang Liu
Hao Sun
Venue: ICLR 2026 Poster (2026)
Keywords: Neural Dynamic Simulation; Visual Dynamics Grounding; Unsupervised Learning
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: Deep learning has demonstrated remarkable capabilities in simulating complex dynamic systems. However, existing methods require known physical properties as supervision or inputs, limiting their applicability under unknown conditions. To explore this challenge, we introduce Cloth Dynamics Grounding (CDG), a novel scenario for unsupervised learning of cloth dynamics from multi-view visual observations. We further propose Cloth Dynamics Splatting (CloDS), an unsupervised dynamic learning framework designed for CDG. CloDS adopts a three-stage pipeline that first performs video-to-geometry grounding and then trains a dynamics model on the grounded meshes. To cope with large non-linear deformations and severe self-occlusions during grounding, we introduce a dual-position opacity modulation that supports bidirectional mapping between 2D observations and 3D geometry via mesh-based Gaussian splatting in video-to-geometry grounding stage. It jointly considers the absolute and relative position of Gaussian components. Comprehensive experimental evaluations demonstrate that CloDS effectively learns cloth dynamics from visual data while maintaining strong generalization capabilities for unseen configurations. Our code is available at https://github.com/whynot-zyl/CloDS. Visualization results are available at https://github.com/whynot-zyl/CloDS_video.

Published as a conference paper at ICLR 2026
CLODS: VISUAL-ONLY UNSUPERVISED CLOTH DY- NAMICS LEARNING IN UNKNOWN CONDITIONS
Yuliang Zhan' Jian Li Wenbing Huang', Yang Liu", Hao Sun
'Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China SScoool of Engineering Science University of Chinese Academy Sciences, Beijing, China Emails: zhanyul angeruc edu. (Y.Z.;; hacsun ruc edu. (H.S.)
ABSTRACT
Deep learning has demonstrated remarkable capabilities in simulating complex dynamic systems. However, existing methods require known physical properties supervision or inputs, limiting their applicability under unknown conditions. To explore this challenge, we introduce Cloth Dynamics Grounding (CDG), a novel scenario for unsupervised learning of cloth dynamics from multi-view vi- sual observations. We further propose Cloth Dynamics Splatting (CloDS), an unsupervised dynamic learning framework designed for CDG. CloDS adopts a three-stage pipeline that first performs trains dynamics model on the grounded meshes. To cope with large non- linear deformations and severe self-occlusions during grounding, we introduce a dual-position opacity modulation that supports bidirectional mapping between 2D observations and 3D geometry via mesh-based Gaussian splatting in video-to- geometry grounding stage. It jointly considers the absolute and relative position of Gaussian components. Comprehensive experimental evaluations demonstrate that CloDS effectively learns cloth dynamics from visual data while maintaining strong generalization capabilities for unseen configurations. Our code is avail- able at https: //:::///tttttt git..b.. com/ whynot - 2y1 /Cl oDS. Visualization results are available ht : / wht.t..bbt..u...gggggogg.g.....t
grounding and then
thub. wwwpwhynottwoooooooooooo 2yl/cl/l/cocccoo /ccooosssviee sevvveve...
INTRODUCION
Accurately modeling complex dynamical systems is fundamental to forecasting (Si Chen, 2025; Li et al., 2025a), control (Wei et al., 2025; Liu et al., 2025), and optimization (Zhou et al., 2024; Tang et al., 2025) across diverse natural and engineered domains. Recently, deep learning has revo- lutionized the simulation of various dynamic systems, including fluid (Zeng et al., 2025), cloth (Li et al., 2025b), and multi-body dynamics (Carleo Troyer, 2017). However, current approaches are highly dependent supervision derived from physics and environmental properties in known ditions (e.g.. particles, meshes) (Pfaff et al., 2020; Wang et al., 2024c). This dependency hinders the feasibility of unsupervised dynamics learning from visual data in robotics (Finn et al., 2016; Lee et al., 2018) and computer vision (Shi et al., 2015; Wang et al., 2017; Li et al., 2025c), where physical properties under an unknown environmental condition are inaccessible.
To address this challenge, intuitive physics approaches inspired by human perception have emerged. These approaches enable unsupervised dynamics learning directly from visual data (Wu et al.; Li et al., 2025a). Existing intuitive physics ap- proaches have made notable progress modeling rigid-body interactions (e.g.., collisions and forces). However, they still fall short in de- formable continuum mechanics, particularly cloth dynamics. This paper explores a novel scenario intuitive physics: Cloth Dynamics Grounding (CDG), aimed at unsupervised learning cloth dy- namics from a series of multi-view videos. CDG is challenging due to the infinieediimeniioall state
Table 1: Capabilities of different vision-only methods.
Method
3D Dynamic Learning
Video Prediction
Dynamic Scene
Novel Synthesis
3D reconstruction
X
Video prediction
CloDS (ours)
Corresponding author
Equally contributed

---

Published as a conference paper at ICLR 2026
(a) Dynamic Learning
(b) Prediction
(c) Dynemic Scenes Novel View Syntheeis
Interpolation
Extrapolation
Train
Test
views
Novel views
CLR
Predict
Predict
Generalization
novel state
ATO
Generalization state
grantyy
Gonerditzation
gravity
Arr
novel sTate
Figure 1: CloDS has the following capabilities: (a). Learning underlying cloth dynamics. (b). Video prediction through the forward process of DVC. (c). Novel view synthesis in dynamic scenes.
spaces of the cloth, the complex physical dynamics, and the strong self-occlusion. Given these chal- lenges, existing visual-only methods struggle to handle CDG effectively. Dynamic scene synthesis methods often fail to generalize beyond observed frames (Hu et al., 2025a; Duan et al., 2024; Zhao et al., 2024; Wang et al., 2025). In contrast, video prediction approaches have predictive capac- ity, but fall short in CDG due to their difficulty in maintaining temporal consistency amid frequent self-occlusions, and their inability reason effectively about underlying geometric structures (Gao et al., 2022; Hu et al., 2023). Table 1 presents a comparison of vision-only model capabilities.
Differentiable Visual Computing (DVC) pro- vides a promising solution to CDG by con- necting visual observations with their underly- ing physical representations (Spielberg et al., 2023). As shown in Figure 2. DVC performs -eeommmtgyy grounding by establishing differentiable mappings between observations and their geometric structures, which enables effective dynamics learning. Existing DVC ap- proaches analogous to CDG include fluid dy- namics grounding (Guan et al., 2022) and discrete element learner (Wang et al., 2024b), which model fluids and solids with particles. However, particle- basee representations unsuitable for cloth due to its thin structure and small relative position variations. Addressing CDG thus hinges on selecting an appropriate geometrie representation and designing an efficient 3D>2D mapping. Given the thin structure and pronounced deformability, we adopt mesh-based representations anchored with Gaus- sian components (Waczyńska et al., 2024). This configuration effectively establishes differentiable mappings between 3D geometry and 2D observations. Furthermore, complex self-occlusion patterns during cloth motion can induce perspective distortion artifacts in traditional neural rendering (Kerbl et al., 2023). Our framework resolves these challenges through dual- dual-position opacity modulation: simultaneously conditioning Gaussian opacity on world-space (relative) coordinates and mesh-space (absolute) coordinates of Gaussian components.
ICLR
ICLR
Goometric Representation
Dynamical Evolution
Imsage Syathesis
Forward
....r Backward
Figure 2: Overview of Differentiable Visual Com- puting framework in Cloth Dynamics Reasoning.
Therefore, in this paper, we introduce Cloth Dynamic Splatting (CloDS), the first known unsuper- vised visual-only method for learning cloth dynamics under unknown conditions. To achieve this goal, we represent the cloth as meshes and design a corresponding Spatial Mapping Gaussian Splat- ting module with nuul-position opacity modulation to establish differentiable mapping between 2D observations and 3D geometry. Moreover, we develop a three-stage training framework for CloDS to enable unsupervised dynamics learning. The learned dynamics can be applied to downstream animations of various garments (SHAO et al., 2024). Our contributions are summarized as follows:
We introduce and explore Cloth Dynamies Grounding (CDG), novel intuitive physical problem.
We propose Cloth Dynamic Splatting (CloDS) for unsupervised CDG from multi-view videos. CloDS also supports predict video and generate novel view models of dynamic scenes (Figure l).
The videos synthesized by CloDS significantly outperform current state-of of-the-ar video prediction models, and it can also generalize to unseen configuration.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Mesh-based cloth simulation. Cloth simulation long-standing research field in graphics and physics Lu et al. (2025); Li et al. (2022); Volino et al. (2009); Tiwari & Bhowmick (2023). Mesh- based methods are widely adopted for their geometric accuracy and efficiency Baraff & Witkin (2023); D. Li et al. (2022); Peng et al. (2023); Tiwari et al. (2023). Recent Data-driven approaches are introduced to further enhance simulation efficiency Wandel et al. (2025); Deng et al. (2024); Santesteban et al. (2021); Shao et al. (2023). However, they rely on physics- based supervision from numerical simulators in known environments (c.g., known material parameters, pre-trained GNNs Rong al. (2025)), and unsupervised dynamics learning in unknown conditions remains an open challenge. Distinct from existing methods, we propose CloDS, a visual-only approach that learns cloth dynamics in unsupervised manner under unknown conditions.
Mesh gaussian splatting. Gaussian splatting has achieved remarkable results in areas such as 3D reconstruction Kerbl et al. (2023); Guédon & Lepetit (2024); Wu et al. (2024b) and Dynamic scenes Novel View Synthesis. While existing work focuses primarily on rigid bodies or minimally de- formed objects, scenes with substantial deformations remain understudied Jiang et al. (2024). Cur- rent approaches address this problem by applying mesh-based constraints, anchoring Gaussian com- ponents mesh faces and adjusting them the mesh deforms Waczyńska et al. (2024); Waczynska et al. (2024) Duan et al. (2024); Gao et al. (2024). In CDG, large deformations and self-occlusion cause rendering artifacts (e.g., perspective) if Gaussians remain statically bound. To address this, CloDS dynamically controls opacity using world-space and mesh-space coordinates.
3 PROBLEM FORMULATION
In this paper, we aim to build a model for CDG, targeting unsupervised learning of cloth dynamics from multi-view videos. At time i, the mesh is denoted as M, = (x:"" .rA" F.). where xt M K are the world-space and mesh-space coordinates of K nodes, and their connectivity. The corresponding multi-view images are denoted by Y, kE(LN)) where T E wgxxhx3
In CDG, the model aims to infer p(M++1| (M)) recursively over time. Given only multi-view videos Y1.++1. the model needs learn p(M [Mt) through the estimation of (P(++1((Y.i)) If the mapping between pixel space and 3D space ((M,)Y11) is accessible, p(Y (Y1s) can be expressed as:
P(Y+i = p(Y M (MMM+M+ [M:) Yia-)),
where p(M, (Y, Y14-1) can be framed Bayesian filtering problem (Longhini et al., 2025b). Thus, the joint posterior distribution is obtained as follows:
p( (Y, Yi 1) up ()
where 77 is normalization constant ensuring the posterior distribution integrates to 1. If the transi- tion probability function (MM-1lYY-- can be learned, then (M.-)) can be expressed as:
pl (.M.Yaa)). = D(MAMM--)p(MM--14
dM,-1-
Therefore, to iteratively learn P(Y++1(Y1:), it is essential to model the dynamic transition p(M++1|AMs and spatial mappings pMY, pY.e) These models are jointly trained to ap- proximate p(Y++1/Vı=)) (see Appendix A for details). After learning p(M++1|M,) the model can simulate cloth dynamics. Simultaneously, using p( M,), it maps the 3D underlying representation of the cloth to the pixel space, forming a continuous video to implement DVC forward process.
4 METHOD
In this section, we describe the Cloth Dynamics Splatting framework. We first delineate the overall methodology. then detail the neural simulator and the Spatial Mapping Gaussian Splatting module, and finally describe the training procedure.
3

---

Published as a conference paper at ICLR 2026
b
M2
Spatial Mapping
M₁
Gaussian Splatting
mode
Dynamies Learninx GNN
Ma+1
Dynamies Leurning GNN
M4+1
Forward
Backrward
Mesh Gaussian Splatting
Mc
Am!
Dynamical
Views I
Evolution
Vicas
ICLR
ICLR
L++1
d
noteston
DVC Fonward
Mesh
Pranddation
Goussion
14p+-1
ICLA
CCUR
geommetirg
Updating
Updating
T
Backrward
Figure 3: The overview of Cloth Dynamics Splatting (CloDS) for Cloth Dynamics Grounding, (a). Overall model architecture. (b). The forward and backward processes of Spatial Mapping Gaussian Splatting (SMGS) and DVC forward process.
(c). The detail of SMGS forward process. (d). The Mesh Gaussian Splatting in SMGS. SMGS can obtain the mesh of cloth M4+1 based on Mr or M, through backpropagation. In the forward stage of DVC, Azt W is typically set O. However, when extracting the mesh Mc+1 from images It+1, the learnable Az is used to make Ie-1 close to 4++1-
4.1 OVERVIEW
Most existing methods rely on physics-based supervision in known environments, while unsuper- vised dynamics learning under unkonwn condition remains a key challenge. Unlike existing meth- ods, we directly learn cloth dynamics from visual observations. To achieve this goal, we design Spatial Mapping Gaussian Splatting (Section 4.3), differentiable 2D-to-3D mapping module that grounds video frames into geometric representations. This mapping permits the subsequent Dynam- ics Learning GNN (Section 4.2) to learn cloth dynamics directly from pixel-space supervision. An overview our approach is depicted in Figure
4.2 GRAPH NEURAL DYNAMICS LEARNER
Due to the thin thin-plane and self-occlusion properties of cloth, model it as meshes. The dy- namic learner pl MM++1 (M,) recursively estimates mesh MaT from initial state Mg. Conventional mesh-based prediction methods typically employ Graph Neural Networks (GNNs) to model dy- namics. Our framework supports various GNN architectures. Without loss of generality, we select MGN (Pfaff et al., 2020) as neural simulator. MGN encodes world-space coordinates W mesh- space coordinates 2.M and relative positions, then applies message passing to model interactions among nodes. x.!h denotes the 3D spatial coordinates, and z;M denotes the 2D UV coordinates. The aggregated node features are decoded to the next positions _ As mesh targets are not available in CDG, an effective mapping function is required to extract meshes from the pixel space.
4.3 SPATIAL MAPPING GAUSSIAN SPLATTING
Gaussian Splatting (GS) represents the radiance field with anisotropic 3D Gaussians. Each Gaussian component is characterized by center μ and covariance matrix Σ which is decomposed into ro- tation R and a scale S. The color of each component determined by a spherical harmonic function G. Pixel color is obtained by compositing N components as Σie aa;; [ (1 ay), where as the opacity of components. Gaussian Splatting enables 3D-to-pixe mapping. However, CDG re- quires temporally consistent mappings to learn dynamics from the pixel space. To address this chal-
4

---

Published as a conference paper at ICLR 2026
lenge, we develop Spatial Mapping Gaussian Splatting (SMGS) based on the GaMes (Waczyńska et al., 2024) which leverage meshes to controllably preserve the Gaussian correspondence.
We first establish static mapping by anchoring Gaussian components on the mesh faces. Each Gaussian center μt is computed via barycentric interpolation: μr = 3 + 82X12 + 3 where 31 - - + B3 = 1 and V = [X,,, x,5 is the face. As the mesh deforms, Gaussians are updated by the same 3 with mapping. The rotation matrix of each Gaussian, R = r2,
vertex positions X!W AXW" maintain temporal consistency F2, r3]. must also remain consistent with the rotation of its associated face. The first vertex of R is defined by normal vector:1 norm (orm((X) ((x x x (X1,3 X(1)), where x is the cross product. The second vertex is r2 norm (X1)2 x)) The third is obtained as single step in the 2000): r3 = norm (orth ( X r1. F2)).
process (Strang,
The scale diag(s), $2, 83), where 81 $2
||x,% X and 83 = (X
XW r3). However, in CDG, strong self- occlusions and large deformations cause ex- isting mesh-based GS methods suffering from perspective distortions and color errors (Fig-
(d) SMGS (wio The result of rendering. Videos are available in Part 2 at URL.
(o) Ground Truth (b) (ours)
Figure 4:
ure 4c). To address this, we modulate the Gaussian opacity through dual-position opacity modu- lation based on the world-space μ² and mesh-space coordinates 2.' of their centers μ (Figure 3d). The ausl-position opacity modulation is defined as:
Ogit Solu W pM.)
AM represent the world-space (rel- where So is a Multilayer Perceptron with parameters 0, and pr..1 Mi ative positions) and mesh-space coordinates (absolute positions) of ith Gaussian center. Detailed explanations of absolute and relative coordinates are provided in Appendix B. Without world-space coordinates ,,", the dual-position opacity modulation reduces to standard GaMeS, Relative po- sitions reduce perspective errors (Figure 4c) and absolute positions yuM prevent the cloth from be- coming transparent when it moves into previously unseen regions (Figure 4d). This mapping enables 3D-to-2D projection in CDG, allowing CloDS to complete the DVC forward rendering (Figure 3c). After establishing temporal Gaussian correspondence, we recover 3D labels for the neural simulator by mapping back from 2D. Inspired by 3D reconstruction (Worchel et al., 2022), we backpropagate W Az to adjust mesh nodes from x. to wW 1 aligning the rendered image with Y141 (Figure 3b);
arg min Cgeometry (MG) 20 Y(41), AzW
(S)
where Lyeometry is the reconstruction loss (detailed in Section 4.4), and J.W S + An.. Gaussian components are then updated and re-rendered until convergence, establishing the 2D-to- 3D mapping and enabling recursive 3D label generation for training the Dynamies Learning GNN.
4.4 DYNAMICS-AWARE UNSUPERVISED TRAINING FRAMEWORK
To solve CDG, we propose a three-step training framework. First, we estimate SMGS M,). Then, SMGS maps images to 3D space via backpropagation to obtain p(M, Yi:), recursively re- constructing M..T Finally, the dynamics simulator is trained with M to model (M++1lM, The first two stages adopt single-step training without temporal losses, while the third stage applics rollout strategy for temporal learning. The overall algorithm is summarized in Algorithm S.I.
First stage: Gaussian component construction. The mesh and multi-view images of the first frame are used to build the cloth's Gaussian component representation via SMGS, optimized with the standard 3D Gaussian splatting loss (Kerbl et al., 2023):
Crender (1 A)2,
XLD-ssim (Y, Y)),
where À is a constant (typically 0.2), Ye is the SMGS rendering result at time t, and Y, is the ground truth. Equation 6 is also used as part of the second stage loss, so we use Y instead of Y,. Unlike the warm-up phase in NeuroFluid (Guan et al., 2022), which requires particle representations across
5

---

Published as a conference paper at ICLR 2026
multiple time steps to train the rendering model, CloDS only needs the mesh from the first frame to construct the 3D cloth representation.
Second stage: Extracting mesh from image space. By iteratively optimizing Art in SMGS P(Y++1|M) recursively obtain the mesh M1.1. An edge loss is introduced during training to preserve cloth shape and prevent excessive deformation by maintaining relative node distances:
Geeometry = LSMGS (EEMGS(EM)) 1), Y(+1) + +y
N-1
Ledge
Σ
d( (. - d(< x 33,0
(8)
1=0 jW]EE
where 7 is constant, 20 is the world-space coordinate of the nodes in the first frame, N is the number of nodes and xWW [+1 = 2, + Arr the node coordinates obtained through SMGS mapping.
Third stage: dynamics simulator training. Finally, the meshes M.T obtained from the 2D pixel- space serve as supervision for training the GNN, allowing it to learn cloth dynamics under unknown conditions. We adopt the loss function from (Pfaff et al., 2020) with rollout strategy:
Lodde
MSR(E W , ,,,),
where T is the rollout length (set to 8 in our experiments) and a. = GGN(..(GNN(2))) ))..). Af- ter training, GNN learns the cloth dynamics. Combined with SMGS, it enables dynamical evolution and image synthesis (Algorithm S.2), completing the DVC forward process shown in Figure 1.
EXPERIMENT
In this section, evaluate the performance of CloDS and its components: Dynamics Learning GNN and SMGS. We first describe the experimental setup. We then assess CloDS's CDG performance, demonstrate SMGS effectiveness in dynamic scene novel view synthesis, and evaluate CloDS in the DVC forward process. Additionally, we explore CloDS's generalization and real-world potential, and finally, we further analyze the impact of neural simulator selection and SMGS components.
5.1 EXPERIMENTAL. SETUP
Dataset and metrics. To evaluate whether the model has Iearned the underlying dynamics, we fol- low existing vidco-based based dynamics learning works (Wang et al., 2024b; Zhu al., 2024a) to generate the dataset. We generate the benchmark using Blender (Community, 2018) on the FLAGSIMPLE dataset (Pfaff et al., 2020) by rendering multi-view cloth videos. FLAGSIMPLE training dataset consists 1000 trajectories (400 steps each), with unique initial states. We render multi-view images for 120 trajectories: 100 for training (viewed) and 20 for testing (unviewed). Detailed dataset infor- mation is provided in Appendix D. CloDS is evaluated Cloth Dynamics Grounding, Dynamic Scene Novel View Synthesis, and DVC forward modeling, with splits detailed in Appendix D.
For consistent comparisons, we use task task-specific metrics. Following NeuroFluid (Guan al., 2022), CDG uses rollout RMSE between the predicted and ground truth node positions. Dynamic Scene Novel View Synthesis adopts PSNR, SSIM (Wang et al., 2004), and LPIPS (Zhang al., 2018) following 3DGS (Kerbl et al., 2023). The DVC forward process reports PSNR, SSIM, LPIPS, and RMSE following video prediction works (Zhong et al., 2023). Metric details are in Appendix D.2.
Baseline models. In CDG, fully fair comparison with mesh-based training methods is challenging, as CloDS targets unsupervised cloth dynamics learning. We adopt MGN (Pfaff et al., 2020) as the Dynamics Learning GNN and use it as a baseline, comparing CloDS trained on video data with MGN trained on mesh data to evaluate whether CloDS can effectively learn cloth dynamics from visual observations. For Dynamic Scene Novel View Synthesis, compare SMGS with GaMeS (Waczyńska et al., 2024), as well as dynamic scene novel view synthesis models: 4DGS (Wu al., 2024a), MSTH (Wang et al., 2024a) and MSD-GS (Hu et al., 2025b). For the DVC forward process, we compare CloDS with video prediction models: SimVP (Gao et al., 2022), MAU (Chang al., 2021), MMVP (Zhong et al., 2023), and TAU (Tan al., 2023). See Appendix D.3 for details.

---

Published as a conference paper at ICLR 2026
Viewed
Unviewed
t- 50
300
400
50 t
400
Referenes
MGN
CDDS
Figure 5: Visualization of the cloth prediction results. - 2 denotes the first predicted frame,
t = 300 is the last frame of interpolation, and 400 is the last frame of extrapolation. =!" and M
it represent the 3D and 2D mesh positions, respectively. Errors are visualized in red, with deeper color indicating larger errors. All methods share the same error bars. "Viewed" and "Unviewed" denote whether the cloth's 's initial state was seen during training.
Table 2: Average RMSE between predicted mesh nodes and ground truth (details in Appendix D.2). "Viewed" and "Unviewed" refer to trajectories 1 1-50 the training set and all trajectories in the test set, respectively. Mean and standard deviation are reported.
Viewed
Unviewed
Method
Interpolation desauo
Extrapolation de=4no
Interpolation
Extrapolation de=
A99 1g300
Avg.
JAVG
Avg.
423300
AWG
demsun
Avg.
300-c1ga00
Avg.
MGN
0,1293
0.1279
0,1286
0.1304
0,1277
0,1291
0,1357
0.1359
0.1358
0.1329
0.1299
0.1314
0004
#0002
0.006
0.03
0.655
0.009
0.0644
0.082
0.1394
0.1366
0,1380
0,1386
0,1390
0,1388
0,1426
0,1493
0,1460
0.1394
0.1329
MGN
0.1362
1:0071
CloDS
0,1313
0.1305
0,1309
0,1312
0.1336
0.1324
0,1437
0,1418
0,1428
0.1381
0.1411
0,19
0.07
0.045
4:
CloDS
0,1302
0,1285
0.1294
0.1320
0.1293
0.1307
0.1395
0.1381
0.1388
0.1340
0.1310
0.1325
0.1320
0.1322
0.1321
0,1349
0.1338
0.1344
0.1408
0.1390
0.1399
0.1353
0.1324
CloDS
0.1339
8065
0.077
0.033
4: 0.088
8:086
0.259
3. 0.064
3. 0.068
1. 0.066
4: 0.019
1 0.083
5.2 CLODS ENABLES UNSUPERVISED CLOTH DYNAMICS LEARNING FROM VIDEOS
We train models on different subsets of mesh and video data. We use all training mesh trajectories and the first 50 trajectories to obtain MGN and MGN respectively. We then fine-tuned MGN on the remaining 50 training videos to obtain CloDS. Finally, we trained CloDS on the first 50 videos and CloDS#* on all training videos. We conduct interpolation and extrapolation experiments on both training and test sets. In NeuroFluid (Guan et al., 2022), interpolation of viewed states is termed dynamic grounding, extrapolation of viewed states is termed dynamic prediction and prediction of unviewed states is termed novel scene generalization. The results are shown in Table 2.
First, CloDS consistently outperforms MGN* on both viewed and unviewed trajectories. As final frame pre- dictions accumulate rollout errors, this highlights the superiority of CloDS in capturing cloth dynamics. Fig- ure 5 visualizes the final-frame predictions. CloDS aligns more closely with the ground truth, particularly in challenging regions such as cloth edges, demonstrat- ing that our approach effectively enables unsupervised dynamics learning. Secondly, CloDS* achieves better
Table 3: Performance Dynamic Scenes Novel View Synthesis.
Model
PSNR (dBy)
SSIM
3DGS
39.6263-3117
9.986:0.07
2.53:0020
1DGS
23.2089=1.86
9,718+0066
15.82=1.27
MSTH
23.1353:
9.82240.88
16.53=1.32
MSD-GG
29.3478-2.35
9.73110
12.971104
G&M&S
33.02/991.54
9.937+0.12
5.21-0442
SMGS (Ouns
36,2368: 1.17
9.95910.06
3.53-0.21
performance on viewed trajectories than CloDS but worse on unviewed ones. It is attributed to overfitting on seen trajectories and limited ability to generalize to unseen trajectories because of the insufficient training data. However, it confirms that our approach can effectively learn dynamics from videos in an unsupervised manner. Finally, CloDS** achieves performance close to MGN, which relies on full mesh supervision. This demonstrates that with sufficient data, CloDS enables near- optrmal cloth dynamics learning (i.e.. MGN trained on all mesh trajectories). We further ana- lyze the impact of initial frame errors on CloDS's ability learn cloth dynamics in Appendix E.
7

---

Published as a conference paper at ICLR 2026
t-296
[+340
t-3
t+400
Reference
MMVP
(Ours)
Figure 6: Visualization of DVC process and video prediction results. Videos are Part at URL.
5.3 SMGS IMPROVES RENDERING UNDER DEFORMATION AND OCCLUSION
The results of dynamic scene novel view synthesis are shown in Table 3. 3DGS, trained separately per frame, serves as the upper bound. It is evident that SMGS achieves superior performance in novel view synthesis. This is primarily due to the large deformation and strong self-occlusion caused by cloth motion, which existing models fail to handle adequately. GaMeS does not adjust the opacity of the Gaussian components during cloth motion, leading to incorrect weight distribution when com- ponents overlap in occluded cloth areas. This causes perspective and rendering errors (Figure 4c). In contrast, SMGS assigns opacity using both 3D world-space and mesh-space coordinates: the for- mer preserves relative positioning to prevent perspective errors, while the latter constrains opacity avoid transparency in unseen regions (Further ablation study on SMGS detailed in Section 5.8).
5.4 CLODS OUTPERFORMS ON THE FORWARD PROCES OF DVC
Video prediction models learn physi-
Table 4: Performance cloth dynamic video prediction.
cal dynamics from videos by generat- ing future frames from current observa-
Model
PSNR
SSTM.wff
I.PIPS
RMSE
tions without reasoning about 3D struc- tures. CloDS also trains on videos. As
MAU
23.7249
9.745_0.06
1.548.0.11
6.78.0 S4
TAU
23.9968-1.88 188
9.781=0.06
1.43==0.10
6.65+0.51
23.9678-1
1.335:0.10
6.59.0.50
shown in Figure 3b, it predicts 3D mesh states, from which we render cloth- motion videos using SMGS. Table 4 compares CloDS with existing video prediction models, where CloDS achieves significantly higher video quality. To investigate this result, we present visualized outputs in Figure 6 (video demonstrations are available in Part at URL). These visualizations reveal that video prediction models accumulate errors over time at cloth edges due to strong self-occlusion, which hampers their ability to maintain temporal consistency. In contrast, CloDS directly models the cloth in 3D space, thereby ensuring better consistency in self-occlusion regions. We further analyze the performance of CloDS and video prediction models the multi tulti-trajectorries in Appendix H.
SimVP
25.4770-2.04
9.801=0.07
1.020=0.09
5.57+0-45
CloDS (ours)
26.6207
0.39
9.817-0.03
0.899.0.04
4.78.0.21
5.5 FUTURE ANALYSIS
CloDS Generalizes to New Shapes and Textures. The ability of a neural simulator to generalize to unseen cloth shapes reflects its effectiveness in learning cloth dynamics. We evaluate this by testing our video-supervised dynamic simulator on a cylindrical cloth (Figure 7a). The results show accurate predictions on the new shape, indicating that CloDS successfully learns cloth dynamics from videos in unsupervised manner. To assess texture robustness, we retrain CloDS with modified cloth textures. As shown in Figure 7b, CloDS maintains strong performance despite texture changes. Videos are available in Part 3 and 4 at URL. Additional results on learning cloth dynamics under complex lighting conditions are presented Appendix H.5.
CloDS has the potential for application to real-world datasets. This paper investigates the feasi- bility of addressing CDG. Synthetic data facilitates training and evaluation with ground truth meshes. Most existing methods for unsupervised dynamics learning from videos rely simulated rather
8

---

Published as a conference paper at ICLR 2026
Generalization Shape and Texture
Real-world Experiments
1-20
th40
to60
pag
D-10
1=20
B-30
tm40
Real-world Video
WNT
VW
CloDs (Ours)
b
Background
Referance
CloDS
CloDs (Ours)
Figure 7: Generalization capabilities of CloDS. (a). Cloth shape generalization. (b). Cloth texture robustness. (c). Woald data exploration. Videos are available Part 3,4 and URL.
0.16
0.6
0.15
RMSE
0.4
0.1
02
0.13
0:12
BSMS-GNN
00
50
250 250
Cidhs
Method
Time step
Mathod
Base model
CloDS
CloDs
GaMes /wra μ*)
CloDS (wob n²
Figure 8: Ablation study. (a). Impact of neural dynamic simulator replacement. (b). Impact of dual-position opacity modulation in SMGS.
I-2
1-12
1-22
B432
+442
1-300
[+2
1-12
D-22
1=32
p-42
14300
Ground Truth
GT
CloDS (Ours)
ClaDS (Ours)
(Ours)
GT
LPP
(Ours)
Figure 9: Visualization of result. (a). Cloth prediction results. "NAN" is no valid output. (b). Object-cloth collision. (c). Real-world garment. Videos are available in Part 8, and 10 URL.
than real data (Wang et al., 2024b; Guan et al., 2022). However, we acknowledge that real-world exploration is meaningful and challenging (e.g., ensuring experiment consistency and dataset con- struction). this end, we make our best efforts for CloDS real-world settings. We first capture multi-view videos of cloth. To focus CloDS on learning cloth dynamics, we extract the cloth regions by SAM (Kirillov et al., 2023) and then train CloDS with the extracted videos. Demo videos are available in Part URL. We observe that CloDS is able to ground cloth dynamics from real-wer world data, although artifacts remain. This may be attributed to camera frame rate limitations and complex real-world lighting conditions. We leave further improvements to future work.

---

Published as a conference paper at ICLR 2026
5.6 CLODS PROVIDES ADVANTAGES OVER EXISTING GEOMETRY-AWARE APPROACHES
Since CDG fundamentally relies on 3D-aware modeling, it is essential to compare against geometry- aware and unsupervised methods. we adapt LIP Zhu et al. (2024b), which uses point clouds geo- metric representation, to CDG setting. We also adapt CS Longhiin et al. (2025a), which represents cloth using meshes. important to note that neither LIP nor CS fully unsupervised. For LIP, we pretrain Particle Posterior Estimator and the Probabilistic Particle Simulator on cloth point-cloud data. For CS, we pretrain GNN on cloth-mesh data. Both LIP and CS are then trained on videos. The predicted 3D geometry is visualized in Figure 9a. We observe that our method substantially outperforms existing geometry-aware approaches. LIP employs point clouds as its geometric rep- resentation, and the lack of mesh-based topological constraints makes it difficult to preserve stable relative positions during inference, leading to a rapid collapse of the cloth shape. In contrast, CS benefits from the mesh topology, which enforces more coherent and physically plausible deforma- tions. As a result, it performs noticeably better than LIP, though still falling short of CloDS.
5.7 CLODS EXCELS UNDER COMPLEX FORCES AND WORLDWORLD GARMENT
CloDS Demonstrates Robust in Object-Cloth Collision Dynamics. CloDS is designed to learn cloth dynamics in an unsupervised manner under unknown environments. In this section, investi- gate its applicability to more complex scenarios containing multiple interacting objects. Inspired by VGPL Li et al. (2020), we assign each object a rigidity attribute, which constrains the allowable rela- tive motion among its nodes. This additional attribute enables the model to handle diverse materials and their interactions. To systematically train and evaluate, we construct an Object- -Coth Collision dataset, where a piece of cloth falls under gravity and interacts with a moving rigid sphere. We then train CloDS on this dataset. As shown Figure 9b, CloDS successfully learns the underlying object-cloth collision dynamics, demonstrating strong generalization to multi-object scenarios.
CloDS Generalizes Well to Real Garment Dynamics. In Section 5.5, we demonstrated the poten- tial of CloDS on real-world datasets. Here, we further evaluate its generalization to real garments. We construct a Real-Garment dataset by performing physics-based simulations Narain et al. (2012) on high-quality garment meshes from the DeepFashion3D V2 dataset Heming et al. (2020). CloDS is then trained and evaluated on this datasct. As shown in Figure 9c, CloDS learns reliable dynamics even realistic garment geometries, indicating strong generalization to complex real-worke shapes.
5.8 ABLATION STUDY
Impact of neural simulator selection. The neural simulator is designed to learn cloth dynamics. Without loss of generality, we adopt MGN as the backbone. To further demonstrate the robustness of CloDS, we also use HCMT (Yu et al., 2024), DHMP (Deng et al., 2024), and BSMS-GNN (Cao al., 2023) as backhones. As shown in Figure 8a, all based models trained only nodes show performance improvement after further training with videos, confirming the robustness of CloDS.
Impact of SMGS components. To validate the necessity of introducing w.W and MM in SMGS, conduct ablation study during the mesh extraction stage (Second Stage). As shown in Figure 8b, errors accumulate over time without W and nd eventually making the meshes unusable. This is because, without μ and nM, perspective errors and transparency issues in unseen regions occur, respectively. Visualizations of the errors are available in Part 2 at URL.
6 CONCLUSION
This paper explores a novel problem: Cloth Dynamics Grounding, where cloth dynamics are learned from visual observations under unknown conditions without physical supervision. To address this challenge, we propose CloDS together with three-stage training framework. CloDS utilizes Spatial Mapping Gaussian Splatting to establish a mapping between the 2D pixel space and 3D space, en- abling the dynamic simulator to learn cloth dynamics directly from videos. SMGS handles large deformations and severe self-occlusion by using both relative and absolute positions of the Gaus- sian components. This design ensures accurate mapping between the 2D and 3D spaces during rendering. Extensive experiments confirm the necessity of this design. They also show that CloDS effectively addresses CDG and outperforms existing models in dynamic novel view synthesis and video prediction tasks. In future work, we aim to explore visual learning of the dynamics of multiple objects in complex scenes under unknown conditions.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
We confirm that all authors have read and adhered to the ICLR Code of Ethics. This work complies with the ethical principles outlined therein, and no part of this study violates ethical standards.
REPRODUCIBILITY STATEMENT
We have made every effort to ensure the reproducibility of our results. A complete description of our model architecture, training procedure and evaluation metrics is our paper, with additional implementation details in Appendix D. Our source code. including data preprocessing scripts and configuration files, is available at https: / / thub. com/ 2y1.nnt0.. This repository enables the reproduction of all experiments reported in the paper.
THE USE OF LARGE LANGUAGE MODELS
In preparing this manuscript, we used Large Language Model (ChatGPT, GPT-5) solely for lan- guage refinement and grammar editing. All research ideas, experimental designs, analyses, and interpretations were conceived and executed by the authors. The model did not contribute re- search ideation, data collection, analysis, or the generation of novel scientific content. All scientific claims, interpretations, and conclusions are entirely our own, and the authors take full responsibility for the contents of this paper.
ACKNOWLEDGMENTS
The work is supported by the National Natural Science Foundation of China (No. 62276269 and No. 92270118). the Beijing Natural Science Foundation (No. 1232009), and the Strategic Priority Research Program of the Chinese Academy of Sciences (No. XDB0620103).
REFERENCES
David Baraff and Andrew Witkin. Large steps in cloth simulation. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pp. 767-778. 2023.
Yadi Cao, Menglei Chai, Minchen Li, and Chenfanfu Jiang. Efficient learning of mesh-based phys- ical simulation with bi-stride multi-scale graph neural network. In International conference on machine learning, pp. 3541-3558. PMLR, 2023.
Giuseppe Carlco and Matthias Troyer. Solving the quantum many-body problem with artificial neural networks. Science, 355(6325):602 606, 2017.
Zheng Chang, Xinfeng Zhang, Shanshe Wang. Siwei Ma, Yan Ye, Xiang Xinguang, and Wen Gao. Mau: A motion-aware unit for video prediction and beyond. Advances in Neural Information Processing Systems, 34:26950-26962, 2021.
Blender Online Community, Blender 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. URL http: http::///www..// blender
D. Li, Min Tang. Yun Yang, Zi Huang, R.F. Tong, Shuang Cai Yang, Yao Li, and Dinesh Manocha. N-cloth: Predicting 3d cloth deformation with mesh-based networks. In Computer Graphics Forum, volume 41, pp. 547-558. Wiley Online Library, 2022.
Huayu Deng, Xiangming Zhu, Yunbo Wang, and Xiaokang Yang. Discovering message passing hierarchies for bash- physics simulation. arXiv e-prints, pp. arXiv-2410, 2024.
Yuanxing Duan, Fangyin Wei, Qiyu Dai, Yuhang He, Wenzheng Chen, and Baoquan Chen. 4d- rotor gaussian splatting: towards efficient novel view synthesis for dynamic scenes. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-11, 2024.

---

Published as a conference paper at ICLR 2026
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. Advances in neural information processing systems, 29, 2016.
Lin Gao, Jie Yang, Bo-Tao Zhang, Jia-Mu Sun, Yu-Jie Yuan, Hongbo Fu, and Yu-Kun Lai. Real- time large-scale deformation of gaussian splatting. ACM Transactions on Graphics (TOG), 43(6): 1-17, 2024.
Zhangyang Gao, Cheng Tan, Lirong Wu, and Stan Z Li. Simvp: Simpler yet better video prediction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp- 2022.
Shanyan Guan, Huayu Deng, Yunbo Wang, and Xiaokang Yang. Neurofluid: Fluid dynamics grounding with particle-driven neural radiance fields. In International Conference on Machine Learning. pp. 7919 7929. PMLR, 2022.
Antoine Guédon and Vincent Lepetit. Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and high-quality mesh rendering. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 5354-5363, 2024.
Zhu Heming, Cao Yu, Jin Hang, Chen Weikai, Du Dong, Wang Zhangye, Cui Shuguang, and Han Xiaoguang. Deep fashion3d: A dataset and benchmark for 3d garment reconstruction from single images. In Computer Vision - ECCV 2020, pp. 111-530. Springer International Publishing, 2020. ISBN 978-3-030-58452-3
Bingbing Hu, Yanyan Li, Rui Xie, Bo Xu, Haoye Dong, Junfeng Yao, and Gim Hee Lee. Learnable infinite taylor gaussian for dynamic view rendering. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 26844-26854, 2025a.
Xiao Hu, Libo Long, and Jochen Lang. Motion decoupled 3d gaussian splatting for dynamic object representation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 3590-3598, 2025b.
Xiaotao Hu, Zhewei Huang, Ailin Huang, Jun Xu, and Shuchang Zhou. A dynamic multi-scale voxel flow network for video prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6121-6131, 2023.
Binbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2d gaussian splatting for geometrically accurate radiance fields. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-11, 2024.
Ying Jiang, Chang Yu, Tianyi Xie, Xuan Li, Yutao Feng, Huamin Wang, Minchen Li, Henry Lau, Feng Gao, Yin Yang, al. Vr-gs: A physical dynamics-aware interactive gaussian splatting system in virtual reality. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-2,
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Trans. Graph., 129-)113- 2023.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan-Yen Lo, et al. Segment anything. Proceed- ings of the IEEE/CVF international conference on computer vision, pp. 4015-4026 2023.
Alex X Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint arXiv:1804.01523. 2018.
Jian Li, Han Wan. Ning Lin, Yu-Liang Zhan, Ruizhi Chengze, Haining Wang. Yi Zhang, Hongsheng
Liu, Zidong Wang, Fan Yu, al. Slotpi: Physics-informed object-centric reasoning models. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pp. 1376-1387, 2025a.
Xueting Li, Ye Yuan, Shalini De Mello, Gilles Daviet, Jonathan Leaf, Miles Macklin, Jan Kautz, and Umar lgbal. Simavatar: Simulation- avatars with layered hair and clothing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 26320 26330, June 2025b.
12

---

Published as a conference paper at ICLR 2026
Yifan Li, Yukai Gu, Yingqian Min, Zikang Liu, Yifan Du, Kun Zhou, Min Yang, Wayne Xin Zhao, and Minghui Qiu. Viper: Process- aware evaluation for generative video reasoning. arXiv preprint arXiv:2512.24952 2025c.
Yifei Li, Tao Du, Kui Wu, Jie Xu, and Wojeiech Matusik. Diffcloth: Differentiable cloth simulation with dry frictional contact. ACM Transactions Gruphics (TOG), 42(1):1-20, 2022.
Yunzhu Li, Toru Lin, Kexin Yi, Daniel Bear, Daniel Yamins, Jiajun Wu, Joshua Tenenbaum, and Antonio Torralba. Visual grounding of learned physical models. In International conference on machine learning, pp. 5927-5936. PMLR, 2020.
Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jianfeng Gao, et al. Sora: A review on background, technology, limitations, and opportunities of large vision models. arXiv preprint arXiv:2402.17177 2024.
Yurou Liu, Jiahao Chen, Rui Jiao, Jiangmeng Li, Wenbing Huang, and Bing Su. Denoisevae: Learn- ing molecule adaptive noise distributions for denoising based 3d molecular pre-training In The Thirteenth International Conference on Learning Representations, 2025.
Alberta Longhini, Marcel Büsching, Bardienus P Duisterhof, Jens Lundell, Jeffrey Ichnowski, Marten Björkman, and Danica Kragic. Cloth-splatting: 3d cloth state estimation from rgb su- pervision. arXiv preprini axXiv:2501.01715, 2025a.
Alberta Longhini, Marcel Büsching, Bardienus Pieter Duisterhof, Jens Lundell, Jeffrey Ichnowski, Marten Björkman, and Danica Kragic. Cloth-splatting: 3d cloth state estimation from rgb super- vision. In 8th Annual Conference on Robot Learning. 2025b.
Zixuan Lu, Ziheng Liu, Lei Lan, Huamin Wang, Yuko Ishiwaka, Chenfanfu Jiang, Kui Wu, and Yin Yang. hegh-performance epu cloth simulation using domain-decomposee projective dynamics. ACM Transactions on Graphics (TOG), 44(4):1-17, 2025.
Saman Motamed, Laura Culp, Kevin Swersky, Priyank Jaini, and Robert Geirhos. Do generative video models learn physical principles from watching videos? arXiv preprint MXXi:25501.09038, 2025.
Rahul Narain, Armin Samii, and James F O'brien. Adaptive anisotropic remeshing for cloth simu- lation. ACM transactions graphics (TOG), 31(6):1-10, 2012.
Tao Peng. Wenjie Wu, Junping Liu, Li Li, Jiazhe Miao, Xinrong Hu, Ruhan He, and Lijun Li. Pgn- cloth: Physics-based graph network model for 3d cloth animation. Displays, 80: 10:102434, 2023.
Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez and Peter Battaglia. Learning mesh-based simulation with graph networks. In International Conference on Learning Representations, 2020.
Boxiang Rong, Artur Grigorev, Wenbo Wang. Michael Black, Bernhard Thomaszewski, Christina Tsalicogiou, and Otmar Hilliges. Gaussian garments: Reconstructing simulation-reat ready clothing with photorealistic appearance from multi-view video. In 2025 International Conference on 3D Vision (3DV), pp. 1054-1063. IEEE, 2025.
Igor Santesteban, Nils Thuerey, Miguel A Otaduy, and Dan Casas. Self-supervised collision han- dling via generative 3d garment models for virtual try-on. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 11763-11773, 2021.
Yidi Shao, Chen Change Loy, and Bo Dai. Towards multi-layered 3d garments animation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 14361-14370, 2023.
YIDI SHAO, Chen Change Loy, and Bo Dai. Learning 3d garment animation from trajectories of piece of cloth. Advances in Neural Information Processing Systems, 37:41803-41825,2000
Xingjian Shi, Zhourong Chen, Hao Wang, Dit- Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional Istm network: A machine learning approach for precipitation nowcasting. Ad- vances in neural information processing systems, 28, 2015.
13

---

Published as a conference paper at ICLR 2026
Phillip Si and Peng Chen. Latent-ensf: A latent ensemble score filter for high- ditensiiensi data similation with sparse observation data. In The Thirteenth International Conference on Learning Representations, 2025.
Andrew Spielberg, Fangcheng Zhong, Konstantinos Rematas, Krishna Murthy Jatavallabhula, Cen- giz Oztireli, Tzu-Mao Li, and Derek Nowrouzezahrai. Differentiable visual computing for inverse problems and machine learning. Nature Machine Intelligence, 5(11):1199-11) 2023.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video rep- resentations using Istms. In International conference machine learning, pp. 843-852 PMLR, 2015.
Gilbert Strang, Linear algebra and its applications. 2000.
Cheng Tan, Zhangyang Gao, Lirong Wu, Yongje Xu, Jun Xia, Siyuan Li, and Stan Z Li. Tem- poral attention unit: Towards efficient spatiotemporal predictive learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 18770-18782, 2023.
Song Tang, Chuang Li, Pu Zhang, and RongNian Tang. Swinlstm: Improving spatiotemporal pre- diction accuracy using swin transformer and Istm. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 33470-13479 2023.
Xinyu Tang, Yuliang Zhan, Zhixun Li, Wayne Xin Zhao, Zhenduo Zhang, Zujie Wen, Zhiqiang Zhang, and Jun Zhou. Rethinking sample polarity in reinforcement learning with verifiable wards. arXiv preprint 1rXiv:251222162,,222 2025.
Lokender Tiwari and Brojeshwar Bhowmick. Garsim: Particle based neural garment simulator. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 4472- 4481, 2023.
Lokender Tiwari, Brojeshwar Bhowmick, and Sanjana Sinha. Gensim: Unsupervised generic gar- ment simulator. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 4169-4178.
2023. 2023.
Pascal Volino, Nadia Magnenat-Thane Thalmann, and Francois Faure. simple approach to nonlinear tensile stiffness for accurate cloth simulation. ACM Transactions Graphics, Artilee,t 2009.
Joanna Waczynska, Piotr Borycki, Joanna Kaleta, Slawomir Tadeja, and Przemystaw Spurek. D- miso: Editing dynamic 3d scenes using multi-gaussians soup. Advances in Neural Information Processing Systems, 37:107865-107889 2024.
Joanna Waczyúska, Piotr Borycki, Slawomir Tadeja, Jacek Tabor, and Przemyslaw Spurek. Games: Mesh-based adapting and modification of gaussian splatting. arXiv preprint urXiv2240.001559
2024.
Nils Wandel, Stefan Schulz, and Reinhard Klein. Metamizer: A versatile neural optimizer for fast and accurate physics simulations. In The Thirleenth International Conference on Learning Representations, 2025.
Feng Wang, Zilong Chen, Guokang Wang, Yafei Song, and Huaping Liu. Masked space-time hash encoding for efficient dynamic scene reconstruction. Advances in Neural Information Processing Systems, 36, 2024a.
Jiaxu Wang, Jingkai SUN, Ziyi Zhang, Junhao He, Qiang Zhang, Mingyuan Sun, and Renjing Xu. DEL: Discrete element learner for learning 3d particle dynamics with neural rendering. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b. URL https: wpwpeeeeeeeee.eeeeeeee net forum? wtw.:mmmmiddddddddooomm 2awkDDDSOOOO.
Qi Wang, Pu Ren, Hao Zhou, Xin- Yang Liu, Zhiwen Deng, Yi Zhang, Ruizhi Chengze, Hongsheng Liu, Zidong Wang, Jian-Xun Wang. et al. p2 net: Pde-preserved coarse correction network for efficient prediction of spatiotemporal dynamics. In The Thirty-eighth Annual Conference on Newral Information Processing Systems, 2024c.
14

---

Published as a conference paper at ICLR 2026
Yuhao Wang, Ruiyang Ren, Yucheng Wang, Jing Liu, Wayne Xin Zhao, Hua Wu, and Haifeng Wang. Bee-rag: Balanced entropy engineering for retrieval -uggmente generation. arXiv preprint arXiv:2508.05100, 2025.
Yunbo Wang, Mingsheng Long, Jianmin Wang, Zhifeng Gao, and Philip S Yu. Predmn: Recurrent neural networks for predictive learning using spatiotemporal Istms. Advances in neural informa- lion processing systems, 30, 2017.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from enror visibility structural similarity. IEEE transactions image processing, 13(4):600- 612, 2004.
Long Wei, Haodong Feng, Yuchen Yang, Ruiqi Feng, Peiyan Hu, Xiang Zheng, Tao Zhang. Dixia Fan, and Tailin Wu. Cl-diffphyeon: Closed-loop diffusion control of complex physical systems. In The Thirleenth International Conference on Leurning Representations, 2025.
Markus Worchel, Rodrigo Diaz, Weiwen Hu, Oliver Schreer, Ingo Feldmann, and Peter Eisert. Multi-view mesh reconstruction with neural deferred shading. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6187-6197, June 2022.
Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20310-20320, 2024a.
Tong Wu, Yu-Jie Yuan, Ling-Xiao Zhang, Jie Yang, Yan-Pei Cao, Ling-Qi Yan, and Lin Gao. Recent advances in 3d gaussian splatting. Computational Visual Media, 10(4):613-642, 2024b.
Ziyi Wu, Nikita Dvorikk Klaus Greff, Thomas Kipf, and Animesh Garg. Slotformer: Unsupervised visual dynamics simulation with object-centric models. The Eleventh International Conference on Learning Representations.
Youn-Yeo Yu, Jeongwhan Choi, Woojin Cho, Kookjin Lee, Nayong Kim, Kiseok Chang, ChangSe- ung Woo, Ilho Kim, SeokWoo Lee, Joon- Young Yang, et al. Learning flexible body collision dynamics with hierarchical contact mesh transformer. In ICLR, 2024.
Bocheng Zeng, Qi Wang, Mengtao Yan, Yang Liu, Ruizhi Chengze, Yi Zhang, Hongsheng Liu, Zidong Wang, and Hao Sun. Phympgn: Physics-encoded message passing graph network for spatiotemporal pde systems. In The Thirteenth International Conference on Learning Represen- tations, 2025.
Junbo Zhang, Yu Zheng, and Dekang Qi. Deep spatio-temporal residual networks for citywide crowd flows prediction. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018
Boming Zhao, Yuan Li, Ziyu Sun, Lin Zeng, Yujun Shen, Rui Ma, Yinda Zhang, Hujun Bao, and Zhaopeng Cui. Gaussianprediction: Dynamic 3d gaussian prediction for motion extrapolation and free view synthesis. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-12, 2024.
Yiqi Zhong, Luming Liang, llya Zharkov, and Ulrich Neumann. Mmvp: prediction. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp- 4273 4283, 2023.
masri video
Qian- Yi Zhou, Jaesik Park, and Vladlen Koltun. Open3d: A modern library for 3d data processing. arXiv preprint arXiv: /001.09877, 2018
Wen Zhou, Shuichiro Miwa, and Koji Okamoto. Advancing fluid dynamics simulations: A com- prehensive approach to optimizing informed neural networks. Physics of Fluids, 36(1), 2024.
15

---

Published as a conference paper at ICLR 2026
Xiangming Zhu, Huayu Deng, Haochen Yuan, Yunbo Wang. and Xiaokang Yang. Latent intuitive physics: Learning to transfer hidden physics from a 3d video. In ICLR, 2024a.
Xiangming Zhu, Huayu Deng, Haochen Yuan, Yunbo Wang, and Xiaokang Yang. Latent intuitive physics: Learning to transfer hidden physics from 3d video. In The Twelfth International Con- ference on Learning Representations, 2024b.
16

---

Published as a conference paper at ICLR 2026
APPENDIX
A PROBLEM FORMULATION DETAILS
Based on Eq. 2 and Eq. 3, the problem can be defined as solving the following equation:
MWw.:///AwAAMMMMMMMMMMMMMAAMAMAA DDM/MM-DDMM-- dM,-
P( ryp(Y++ IM.Y=
(S.1)
We can observe that by solving this equation, the model can learn the joint posterior distribution p(M[+1|MM))
From Eq. S.I. can that in order to Iearn p(Y++1|Yi by iterative method, the key is to model the dynamics learning function p(M (Mc) and the spatial mapping function p(Y{ ((M))aa The dynamics learning function p(M++1| primarily serves to learn cloth dynamics, predicting the next mesh state Mc+1 based on the current mesh state Mr. The spatial mapping function
and p(M, (Y1a) establish a differentiable mapping between 3D real-word space and 2D pixel space, bridging the gap between the underlying representation of cloth and real-world observations.
B DETAILED EXPLANATIONS OF ABSOLUTE AND RELATIVE COORDINATES
In this paper, the distinction between absolute and relative is determined by whether a node's po- sition varies in space over time. The position of node changes in world space, so world-space coordinate x W is relative. The position of node does not change in mesh space, so mesh-space coordinate z is absolute.
We build the Gaussian representation of cloth from the first frame. During dynamics learning, the cloth then moves to poses. If
only X and not X in Equation 4, the opacity of the Gaussian components does not change with the relative position during the deformation of cloth. This reduces standard GaMeS (Waczyńíka al., 2024). Fixed opacity causes perspective errors during rendering (Figure 4c). Adding X W allows the opacity to change according to the relative position of the components and avoids perspective problems. If we use only X XW and not XMM in Equation 4, the Gaussian components obtain incorrect opacity in regions not covered in the first frame. This results in components not being visible the observations (Figure 4d). Adding X as a fixed variable maintains robustness to new 3D positions during rendering.
C CLOTH DYNAMICS SPLATTING ALGORITHM
We train CloDS using shree-stage training framework (Section 4.4), as detailed in Algorithm 1. Upon completion of training, the learned Gaussian representation and the neural simulator GNN are employed to predict mesh dynamics, thereby realizing the forward process of DVC. The full generation pipeline for this forward process is illustrated in Algorithm S.2.
D EXPERIMENT DETAILS
D.1 DATASET DETAILS
The FLAGSIMPLE dataset was simulated using AreSim (Narain al., 2012) with linear elements. Its training set contains 1,000 trajectories, 100 validation, and 100 test trajectories. The initial state of each trajectory is unique, and the cloth meshes are regular which means all edges having similar length. Each trajectory represents the dynamic evolution of the same shaped cloth in the same environment over 400 time steps. The simulation time step 0.02s. The cloth meshes are triangular and regular which means all edges have similar lengths. We used Blender to render all time steps of the first 120 trajectories from the FLAGSIMPLE dataset training set to generate the training and testing images with resolution of 800 800 in this paper. During Blender rendering, the first frame of the first trajectory to extract the cloth's 's UV coordinates, and applied the same UV coordinates the mesh for all time steps, ensuring consistency in the texture of all cloths.
used

---

Published as a conference paper at ICLR 2026
Algorithm S.1 Dynamics-Aware Unsupervised Training Framework.
Input: Sequences of image from N different viewpoints Y1.4+1 Wiittheen.y Mesh is denoted as
M (w" ax E).
1: Extracting initial Mesh M1 from the initial frame 7.N via 2DD Gaussian Splatting Huang al. (2024).
repeat
Gaussian Component Construction
3: I1 SMGS (i)W)
4: Compute render loss Lecometry between il-N and I/'N using Eq. equation 6.
5: Backward Crender.
until SMGS converges
7:
Initialize Mr M1, M
for each Yr Y1.T+1 do
Extracting Mesh from Image Space
9: repeat
10:
=W I141 2t + Áw!
11:
MM+1 (I
-M E441,
12:
T141 < (Me41, ( View
13:
Compute geometry loss Lecometry between 71NN and 1if1 using Eq. equation 8.
14:
Backward Lycometry.
until Ax;" converges
16:
S + $ Ax!"
17: Mc+1 + (# 5M
2t+1, F)
18:
Append M to M.
19: end for
20: for each M: in MI-T-1 do
Train Dynamics Simulator
21:
Mc+1 < GNN(M,)
(#+41, ZM E) &- -MM++1
23: Compute node loss Gnode between 2.+1 and lett using Eq. equation
24:
Backward Lnode-
25: end for
26:
return 2D-3D mapping SMGS, newrdlssttimmn
Algorithm S.2 The forward process of DVC in CloDS.
Input: Sequences different viewpoints View,, initial mesh state Mo = (ar W x E), neural dynamic simulator GNN and neural render SMGS.
I: Initialize I <- 0.
2: for
T do Mr < G
4: I - SMGS(M,, View,)
5: Append 4, to
6: end for
7: return video 7
2

---

Published as a conference paper at ICLR 2026
Generally, we consider the first 300 time steps of the first 100 trajectories as the training set, and the last 100 time steps of the first 100 trajectories and all time steps of the remaining 20 trajectories as the test set. In different tasks, there are slight variations in the training and testing data.
In the Cloth Dynamics Grounding task, the training data for MGN and MGN consist of meshes from the first 300 time steps of the 1-50 trajectories and the first 300 time steps of the 1-100 trajec- tories, respectively. The training data for CloDS* and CloDS** consists of videos from the first 300 time steps of the 1-50 trajectories and the first 300 time steps of the 1-100 trajectories, respectively. CloDS uses videos from the 51-100 trajectories and meshes information from the 1-50 trajectories as training data. During testing. meshes from the 51-100 trajectories and all time steps of the test set are used as the test data. Since the imitial states of the 51-100 trajectories have been seen by the model during training in most models (Except for MGN they are referred to as the 'Viewed' trajectories, whereas the trajectories in the test set represent cloth poses and positions that the model has never seen, and thus are referred to as the "Unviewed' trajectories.
When assessing the performance of SMGS, use multi-view images from the first 200 time steps of 5 trajectories as both training and testing data. SMGS generates Gaussian component represen- tations of the cloth from the first frame of each trajectory and then generates multi-view images for 200 time steps by changing the mesh. The results from the 5 trajectories are averaged to obtain the final experimental result in table 3.
When evaluating CloDS's performance on the forward of DVC, for the baseline, since it cannot utilize multi-view videos, we use fixed-view images from the first 300 time steps of 5 trajectories for training, with the following 100 time steps as the test data. For CloDS, we use the corresponding multi-view videos traning and test data. The final experimental result in tabel is the average from 5 trajectories.
D.2 EVALUATION METRICS
For different tasks, we use various evaluation metries. Metrics such as Peak Signal-to-Noise Ratio (PSNR), LPIPS, and SSIM are commonly used and will not be elaborated on further in this paper. In this section, we mainly focus on the Rollout RMSE in the Cloth Dynamics Grounding task and the RMSE metric in Video Prediction task.
In the Cloth Dynamics Grounding task, the model predicts the mesh at each time step by performing a rollout from the first frame's 's mesh input. Therefore, Rollout RMSE is used to evaluate whether the model has learned correct cloth dynamics. Rollout RMSE is computed as the Root Mean Squared Error (RMSE) of position in Lagrangian systems and momentum in Eulerian systems. It averages the error across all spatial coordinates, mesh nodes, time steps in each trajectory, and all trajectories in the test data. The formula for Rollout RMSE is follows:
N
T
K
AVG IST
(S.2)
N
(=1
t=1
j=1
N
400
K
1 K
AVG
1ST
N
400
T
W
(S.3)
where d. represents the predicted mesh node world-space coordinates, N is the number of trajec- tories and K is the number of nodes. The specific steps for obtaining this are:
F = GNN(GNNN....GNNNC )..)).
(S.4)
Specifically, the rollout method predicts the first frame based on the ground truth, and the subsequent time step predictions are obtained recursively to derive the system trajectory.
Similar to the Rollout RMSE, the RMSE in Table 4 is calculated as the average RMSE between the predicted image I and the ground truth image L, across all time steps of the rollout. The formula is as follows:
1 RMSEvp T
RMSE( Lt).
(S.5)
3

---

Published as a conference paper at ICLR 2026
Table S.I: Performance of CDR under different noise conditions. "+ Gaussian noise" refers to adding Gaussian noise with O mean and 0.001 variance to each node. Gaussian noise separately along X, and Z axes and averages errors. "+ Scaling noise" applies random scaling factor in [0.95, 1.05] to the cloth.
"+ Translation noise" adds
Viewed
Unviewed
Method
Interpolation
Extrapolation JAYG +5300
Interpolation
Extrapolation
dAvG
d1=300
dc=100
dAYG
di=300
dAGG
d1=100
0.1313
0.1305
0.1312
0.1336
0.1437
0.1418
0.1381
0.1411
CloDS*
= 0/466
+ 00089
+ 0.037
+: 0.048
+ 0.029
0.045
= 0.046
0.048
0. 0.1362
0.1347
0.1359
0.1367
0.1481
0.1463
0. 0.1402
0. 1423
+ Gaussian noise
= 07033
+ 0.000
0.065
+ 0.097
+ 0.084
+ 0.062
= 0.075
= 0,056
0.1353
0.1332
0.1363
0.1378
0.1453
0. 0.1479
0.1397
0. 1434
Translation noise
0.098
+ 0,060
+ 0.079
0.092
0.077
=
0.1434
0.1455
0.1451
0.1474
0.1495
0.1471
0.1423
0. 1.1477
+ Scaling noise
= 0.062
= 0.065
+ 0.077
+ 0.089
+ 0.064
+ 0.071
= 0.053
0.088
D.3 BASELINE MODELS
To evaluate the performance of CloDS, we compared it with different baseline methods. The detailed descriptions of each model are presented below.
In Cloth Dynamics Grounding, unlike existing methods that learn cloth dynamics using mesh-based labels, CloDS can learn the underlying physical dynamics of cloth from sparse viewpoint videos without requiring the use of physical representations as labels. Therefore, a fair comparison with cur- rent models that use mesh-based labels is challenging. In the CDG task, we compare MGN trained with different data and training methods. The specific data splits are presented in Appendix D.I.
MGN (Pfaff et al., 2020): MGN is a mesh-based general method that can model various physical systems accurately and effectively. It has good generalization ability and can be extended during inference. It uses a message-passingg neural network (MPNN) with an Prooode-rrcess--eee chitecture to predict properties such acceleration for dynamic systems, and then the mesh state is obtained through an integrator.
ar-
In the dynamic scene novel view synthesis task, need to compare SMGS with relevant baselines. The descriptions of the related baselines are as follows:
4DGS (Wu et al., 2024a): A new explicit representation method is proposed, which simultane- ously contains three-dimensional Gaussian distributions and four-dimensional neural voxels. A decomposition-basse neural voxel encoding algorithm, based on HexPlane (Hexagonal Plane), is introduced to effectively construct Gaussian features from the four-dimensional neural voxels. This approach then applies a lightweight MLP to predict the Gaussian deformation new time steps.
MSTH (Wang et al., 2024a): A method for efficiently reconstructing dynamic 3D scenes in both time and space. It decouple the representation of dynamic radiance fields into time-invariant 3D hash encoding and time-varying 4D hash encoding. By using uncertainty-guit guided masks as weights, the algorithm avoids the large number of hash collisions caused by the additional time dimension.
M5D-GS (Hu et al., 2025b): This method addresses 3D dynamic scene reconstruction for objects undergoing large deformations. builds on a 3D-GS framework to achieve high-fidelity, real-time rendering of dynamic objects. The framework explicitly decouples object motion and deformation estimation and employs a coarse-to fine matching strategy to initialize large motions.
To evaluate the quality of videos generated by CloDS in the DVC forward process, need to compare CloDS with video prediction models. The descriptions of the related baselines are as follows:
MAU (Chang et al., 2021): MAU attempts to simulate temporal information in videos by expand- ing the temporal receptive field and aggregating it through an attention mechanism. The proposed module is used in the bottleneck part of the autoencoder, where spatial and temporal information is combined across different MAU layers.
4

---

Published as a conference paper at ICLR 2026
TAU (Tan et al., 2023): TAU captures time evolution by decomposing temporal attention into intraframe static attention and inter-frame dynamic attention. is argued that mean square error loss mainly focuses intraframe differences, which motivates the introduction of differential divergence regularization to address inter-frame variations. By keeping the spatial encoder and decoder simple with 2D convolutional neural networks, intentionally implement the proposed TAU modules and surprisingly find that the resulting model achieves performance competitive with recurrent-based models.
MMVP (Zhong et al., 2023): MMVP is an end-to-end trainable two-stream pipeline. It uses mo- tion matrices which helps the motion prediction become more focused, and efficiently reduces the information loss in appearance to represent appearance-agnostt motion patterns.
Sim VP (Gao al., 2022) SimVP simpler yet more effective video prediction model. is entirely CNN-based and trained end-to-end using MSE loss, without introducing any additional techniques or complex strategies.
D.4 EXPERIMENT PLATFORM
We conduct comprehensive experiments utilizing the NVIDIA H800 80GB PCIe paired with the Intel(R) Re) Platinum 8380 CPU @ 2.30GHz.
D.5 TRAIN DETAILS
For MGN, used the settings in the original paper (Pfaff et al., 2020). When training the Gaussian components with multiple trajectories, the first time step of all training trajectories used train the Gaussian components. The experiments in this paper are conducted complete pass through the first time step of all trajectories is considered an epoch. In each epoch, we loop 100 times for each time step. During the training of the Dynamics Learning GNN, we train the model for for 1000 epochs. The quantitative experimental results in the paper are the averages over five trials. Our code available at https: //gi thub. com/ wwwnot- zzy //co/....
H800 GPUs. One
D.6 INFERENCE TIME CLODS
Our method is built upon neural simulator and mesh-based Gaussian splatting render: The neural physical simulator accelerates traditional physics solvers. And Gaussian splatting, which renders scenes through Gaussian volume projection, is significantly faster than conventional renderer. The rendering speed (FPS) on single H800 is reported in Figure S.1. As shown, our pipeline is substan- tially more efficient than traditional approaches.
E ROBUSTNESS TO INITIAL MESH ERRORS
Since the method in this paper requires us- ing the mesh from the first time step of the trajectories for deploying the Gaussian components to obtain the cloth's Gaus- impor-
Inference (FPS)
50
40
sian component representation, tant assumption of CloDS is that an esti- mate of the initial cloth mesh is available.
However, estimating the initial mesh may
introduce errors due to various factors, so
we need to demonstrate the robustness of CloDS to the noise of initial mesh. To as- CDG, we add different noise to the initial mesh and then use the SMGS rendered from the noisy ini- tial mesh to train the C Finally, we test on the same test data as in Table 2. The results presented in Table S.I. To more intuitively examine the model's inference behavior under noise perturbations, we visualize the corresponding results in Figure S.2. It be observed that although emror in the initial mesh reduce ClolD'ss
Saunel Simulattor
Saussiann Ronder
Cless (ours)
Physte-bsse [Aresin)
Trediionll
Praditional
(Blendw)
sess CloDS's robustness
Figure S.1: Inference Time CloDS (Neural Simula- tor Gaussian Render) and tradition pipline (Physic- based Simulator Traditional Render).
5

---

Published as a conference paper at ICLR 2026
t=2
t=12
t=22
t=32
t=42
t=300
Reference
CloDS (Ours)
Gaussian Noise
Translation Noise
Scaling
Noise
Figure S.2:
Visualization of CDR under different noise conditions, corresponding Table S.1. Videos are available Part 11 URL..
ability to learn cloth dynamics, CloDS still enables MGN to learn reliable cloth dynamics even in the presence of mesh errors.
The results show that after training with multiple trajectories, the video prediction model performs significantly worse on new cloth poses compared to CloDS. This is due to the strong deformations of the cloth, which exhibit chaotic behavior: small changes in the initial state lead to drastically different subsequent states. Additionally, it is observed that after training multiple trajectory videos, the video prediction model's extrapolation performance worse than when trained on single trajectory. while CloDS achieves better performance. This is because cloth dynamics are complex, and video prediction models struggle to effectively learn the dynamies of cloth. In contrast, CloDS maps 2D images to the 3D physical representation of the cloth, enabling it to better learn cloth dynamics from observation.
F RENDERED TEXTURES
To demonstrate the effectiveness and generalization ability of the CloDS, we rendered the cloth using different textures. All texture images used in this study are shown in Figure S.3. We first used the texture Figure e S.3a to render multi-view videos, which produced the experimental results in Figure 5, Figure 6, and Table 2, showing that CloDS can learn cloth dynamics from multi-view videos in an unsupervised manner. We then used the texture in Figure S.3b to render a cylindrical cloth, demonstrating that method generalizes to different cloth shapes (shown in Figure 7a). Finally, we used the texture in Figure S.3c to demonstrate that our method can adapt to various textures (shown in Figure 7b).

---

Published as a conference paper at ICLR 2026
ICLR International Conference Leamming Representations
ICLR International Conference Leaminn Representations
Figure S.3: (a): Texture used in Figure 6. (b): Texture used in Figure 7a. (c): Texture used in Figure 7b.
Table S.2: Performance on cloth dynamic video prediction. "Extrapolation" refers to testing the last 100 frames of the 10 training trajectories, while "Generalization' refers to testing the first 100 frames of 2 unseen trajectories.
Extrapolation
Generalization SSIM
Method
PSNR (dB)1
SSIM
LPIPS
RMSEL
PSNR (dB)f
LPIPS
RMSEL
MAU
17.1273
0.9531
0.03743
0.13431
12.6789
0,9291
0.04817
0.16531
TAU
17,5812
0.9563
0.03472
0.13386
13.0322
0.9391
0.04532
0.16087
MMVP
17.5723
0.9523
0.03389
0.13549
12.9535
0,9380
0.04486
0.15791
Sim
18.9722
0.9557
0.03246
0.12704
14.2391
0.9438
0.04593
0. 0414028
CloDS (ours)
27.7331
0,9835
0.00832
0.03987
25.2380
0.9798
0.01178
0.05901
G MORE RELATED WORK
Video prediction model. Video prediction models learn object dynamics from videos to predict future states from initial frames (Gao et al., 2022; Tang al., 2023; Zhong al., 2023). These mod- els are typically tested on datasets such as TaxiBJ (Zhang et al., 2017) and Moving MNIST (Srivas- tava et al., 2015) which focus on simple motion with minimal occlusion/deformation In contrast, CDG demands 3D-aware modeling of highly deformable objects. In CloDS, we use dynamics learning model to learn the cloth dynamics in 3D space and employ Gaussian Splatting Rendering as mapping between 3D space and 2D pixel space. This allows the dynamic prediction model to learn 3D spatial information from the pixel space.
H MORE RESULT
H.1 VIDRO PREDICTION MODEL TRAINED MULTIPLE TRAIECTORIES.
Currently, video prediction models can learn the motion patterns of objects from videos. How- ever, these models typically focus on predicting rigid bodies, with less attention given to highly deformable objects like cloth. The results in the table 2 indicate that video prediction models are less effective than CloDS at learning the cloth dynamics by training on specific initial states. To further compare the generalization abilities of CloDS and video prediction models, we train video prediction models the first 300 time steps of 10 trajectories and then test its interpolation per- formance on 2 unseen trajectories. Additionally, to assess the impact of increased training data on model performance, test the extrapolation performance the last 100 time steps of the 10 training trajectories. The experimental results are shown in Table S.2.
H.2 ERROR ACCUMULATION IN MESH EXTRACTION.
When mapping the 2D pixel space to the 3D space, SMGS is used to recursively extract the cloth mesh for all time steps, allowing the Dynamic Learning GNN to learn the cloth dynamics. Since Mc-1 depends on Mr, errors can accumulate during the recursive extraction process. To explore the accumulation of errors, we calculated the error between the extracted mesh and the ground truth as it evolves over time steps, as shown in Figure S.4a, with the mean and variance presented in Figure S.4b. We can observe that as time progresses. the accumulated error does not increase dramatically but rather stabilizes. This indicates the stability and effectiveness of using SMGS for mesh extraction.
7

---

Published as a conference paper at ICLR 2026
0.36
b
0.20
0.34
0.15
0.32
0.5
0.10
0.30
0.08
0.0
0.2
0.4
0.02
0.00
250 300
0.00
6-57 PC
PGT PGTURMEARE
0.0
50
100
150
250
250
300
350
400
step
Type
Time step
P-GT
RE
A-GT-Unviewed
E-GT
CloDS
CloDS (mio Curgel
Figure S.4: (a): Error accumulation, mean and variance. "P-GT" is the error between the predicted and ground truth meshes. "P-E" is the error between the predicted and extracted meshes. "E-GT" is the error between the extracted and ground truth meshes. These are the experimental results on the viewed trajectories (1-50). "P-GT-Un Viewed" is the error between the predicted and the extracted meshes on unviewed trajectories. (b): Error accumulation when with and without Ledge-
t=2
t=12
t=22
t=32
t=42
t=300
Ground Truth
Extracted Mesh (CloDS)
Predicted Mesh (CloDS)
Extracted Mesh (wlo fute)
Figure S.5: The mesh visualizations corresponding to the results in Figure S.4. "Extracted mesh (CloDS)" and "Predicted mesh (CloDS)" correspond to the results presented in Figure S.4a, while "Extracted mesh (w/o Ledge)" corresponds to the results shown in Figure S.4b. Videos are available in Part at URL.
H.3 ERROR ACCUMULATION IN DYNAMICAL SYSTEM PREDICTION.
When predicting the dynamics of the system, we use the rollout method to recursively predict the cloth state based on the initial state. Since the model does not learn the stationary state of the cloth, this simulation can lead to error accumulation, causing prediction errors to increase dramatically over time. Figure S.4a shows the error between the predicted mesh the ground truth and the ex- tracted mesh over time. For clearer visualization, we present the ground-truth mesh, the extracted mesh, and the predicted mesh Figure S.5. We observe that even with full rollout, the error does not increase drastically, which confirms the stability of the cloth dynamics learned by CloDS.
Additionally, Figure S.4a shows that the mean error between the prediction and the ground truth, as well the extracted mesh, are quite similar, indicating that the cloth dynamics learned by CloDS are close to the real cloth dynamics. However, the variance of the error between the prediction and the ground truth is larger than that between the prediction and the extracted mesh. This is because CloDS primarily uses the extracted mesh to enable the GNN to learn the cloth dynamics, leading to more stable emor between the prediction and the extracted mesh.
8

---

Published as a conference paper at ICLR 2026
t=20
t=40
t=60
t=80
t=100
t=400
Reference (Render)
151
ALR
CLRR
Reference (Mesh)
CloDS (Ours)
Figure S.6: Visualization results under lighting conditions. "Reference (Image)" shows the rendered images with lighting. "Reference (Mesh)" shows the corresponding meshes. "CloDS (ours)" shows the meshes predicted by CloDS. Videos available in Part at URL.
H.4 NECESSITY OF MESH REPRESENTATION
In traditional simulators, particle-based representation is effective and well-suited for cloth simu- lation. However, neural network (NN) methods, if gradient descent is used randomly update particle positions without mesh-based constraints on their relative positions, NN will fail to con- verge. This is why this paper introduces geometry loss based on mesh (Eq. 8). Without edge loss, the method degrades to particle-base approach that ignores mesh topology. It can be seen that from the resulting error shown Figure S.4b that the absence mesh topology constraints leads to accumulation of node errors. To more clearly illustrate the necessity of the mesh representation, we visualize in Figure S.5 the mesh extracted without the edge loss (shown as "Extracted Mesh (w/o )e)ge)"). Moreover, by inspecting the point-cloud baseed LIP results in Figure 9a, observe that the cloth shape collapses rapidly during inference. This further corroborates the necessity of mesh-based representations for addressing the CDG problem.
H.5 CLODS PERFORMANCE UNDER LIGHTING CONDITIONS.
To demonstrate that our method
Table S.3: Performance of CloDS in CDG under lighting and without lighting Conditions.
can also learn cloth dynamics under complex environmental conditions, we introduced lighting effects dur- ing rendering generate multi-view video data with light and shadow variations. We then used CloDS to learn cloth dynamics under complex lighting conditions. The visual and quantitative results are shown in Figure S.6 and Table S.3. The generated videos are available in Part at URL. The results indicate that even with lighting CloDS still learns cloth dynamics effectively. However, compared with the performance under no-light conditions, lighting reduces CloDS performance. This is mainly because the same region may ex- hibit different colors due to shadows and illumination, which disrupts the temporal consistency of the images. This inconsistency causes inaccurate mesh node estimation and reduces the accuracy of subsequent training. Although Table S.3 shows a performance drop under lighting compared to no-light conditions, CloDS still learns cloth dynamics effectively.
Interpolation
Extrapolation
Method
AVG
de
AAN
du
light
0.1582-0.044 0.056
0.1573-0.049
0.1541
who light
0.0313
0.1305
0.1312-0.037
0.1336:0.048
Moreover, CloDS significantly outperforms existing geometry- approaches. As shown in our response to Figure 9, current geometry-ave aware baselines already struggle to capture complex cloth geometry even under simple lighting conditions, and we sincerely thank you for this constructive suggestion baseline construction.

---

Published as a conference paper at ICLR 2026
Table S.4: Performance of CloDS on the CDR task under different camera number.
Performance of CloDS on CDR task Table S.5:
under different Gaussian component density.
Interpolation JAVG $55000
Extrapolation
Interpolation JAGG $5000
Extrapolation
Method
Method
de-300
JAYG $5000
dc-300
d4-300
$5300
dr-400
0.1362
0.1371
0.1401
0.1396
0.1348
0.1389
0.1367
0.1387
n
5
d: 0.082
0.077
4: 0.081
4: 0,069
0.075
# 0.063
0.082
4: 0.066
0.1347
0.1353
0.1334
0.1413
0.1352
0.1368
0.1321
0.1423
11 20
10
+
+:
0.066
# 0.064
0.049
+ 0.058
0.1313
0.1305
0.1312
0.1336
0.1313
0.1305
0.1312
0.1336
30
20
+ 0.016
= 0.039
+: 0.037
0.0.88
+ 0.016
= 0.039
+ 0.037
+ 0.018
H.6 THE IMPACT OF THE NUMBER OF CAMERAS.
For each trajectory, we set up 30 identical camera positions following NeuroFluid (Guan et al., 2022). We further explored the impact of number of cameras on Novel View Synthesis in the Dy- namic Scenes task. As shown in Table S.4, more views improve performance by capturing finer details, despite gains from additional cameras are modest.
H.7 THE IMPACT GAUSSIAN COMPONENT DENSITY
A higher density of Gaussian components allows for a more detailed representation of the 2D spatial features in the image. To assess the impact of Gaussian component density on model performance, we evaluate CloDS with varying densities and test its performance on the CDG task using the viewed trajectories. The experimental results are presented Table S.5. It can be observed that increasing the Gaussian component density leads to slight improvement in model performance. However, training with lower density still enables the model to effectively learn the cloth dynamics.
H.8 CLODS vs. SORA
As representative of current generative models, Sora (Liu et al., 2024) can produce visually realistic videos by observing videos. However, its physical understanding is severely limited (Motamed et al., 2025). To evaluate Sora's performance on the CDG task, we provide the initial frame of a cloth and the prompt "A flag fluttering in the wind" to generate a cloth motion video. The generated videos are available in Part at URL., is evident that the results lack adherence to physical principles.
H.9 EXTRACTED INITIAL MESH FROM THE INITIAL FRAME.
We use 2D gaussian splatting Huang et al. (2024) to extracte the Initial mesh M, from the first frames I..N. In general, the process consists of reconstructing a static field, rendering multi-view depth maps from this field, and finally applying TSDF fusion followed by Marching Cubes obtain the initial triangular mesh. Concretely, we first use the first-frame images I together with their camera intrinsics and extrinsies to obtain Gaussian representation of the cloth via 2D Gaussian Splatting (2DGS). Unlike 3D Gaussian Splatting, 2DGS represents the surface using 2D elliptical disks embedded in 3D space. Each disk is parameterized by 3D center, two tangent-plan rections, and the corresponding anisotropic scales. During forward rendering. 2DGS extends the CUDA renderer of 3DGS to additionally output per-pixel depth and surface normals, which are re- quired for extracting the initial mesh M1. Next, we perform forward rendering pass for all training views, producing the depth and normal maps D1:NN, For each view i, we back-project every pixel into 3D space using the predicted depth Dj and the known camera parameters, generating a multi- view point cloud. We then define a voxel grid at a fixed resolution and construct a Truncated Signed Distance Function (TSDF) within this volume. All depth maps are fused into the TSDF using the built-in TSDF integration module of Open3D Zhou et al. (2018), resulting in consistent volumetric reconstruction. Finally, applying Marching Cubes to the fused TSDF yields the initial mesh M,. See Algorithm S.3 for algorithmic details.
di-
10

---

Published as a conference paper at ICLR 2026
Algorithm S.3 Initial mesh extraction from first- first-frame NeRF-style data using 2D Gaussian Splatting
Input: Multi-view RGB images 1::N of the first frame; camera intrinsics and extrinsics {K;, Ti) voxel size u: truncation threshold r; number training iterations T. Output Initial triangle mesh M1.
1: (Optional) SEM calibration point cloud.
2: if sparse calibration point cloud P is not provided then {/--))
3: Run SIM on N1:N } to obtain camera poses and sparse points
4: P (p
5: else
6: Use the given and camera poses.
7: end if
8: Initialize 2D Gaussian splats.
9: for each point ps E P do
Set Gaussian center Cą f- Ph
11: Initialize orthonormal tangential basis (1h c2 (x) with th 1h 1g
12: Initialize scales s s t- Sjnit
13: Initialize opacity ag Qimi and SH appearance coefficients a₁
14: end for
15: Let 0 collect all Gaussian parameters {ck, th th sht st oz, ax).
16: Train 2DGS on the first-frame views.
17: for = to T do
18:
for = to No do
D- for each training view
19:
Render color image f from 0 using the 2DGS renderer Simultancously render geometric buffers: depth samples (zij} along each ray,
median-depth map D
surface- normal map N
21:
end for
Compute color reconstruction loss (0, I1:N (1:N)
22:
Compute depth distortion loss La(O, {2g})
24:
Compute normal consistency loss Ln (0, {Di, Ni))
25:
L La + ala + 3Ln.
26: Update < VoL using Adam
27: end for
28: Denote the optimized parameters as d*.
29: Render per-view depth maps from 2DGS.
30: for i= = 1 to N do
31: Using 0*, render median-depth map Dj for view
32: end for
33: TSDF fusion.
34: Initialize a TSDF volume V with voxel size v and truncation
for i = N do
36:
for each pixel in Dj do
Back-project (ar, Dj (x)) via (K, T;) a 3D point q
38: Update TSDF values and weights in V for voxels within distance 7 of
39:
end for
40:
end for
41: Mesh extraction.
42: Apply Marching Cubes to the zero-level set of V to obtain watertight mesh Mj
43: return Mj
11


Reviews:
[{"review_id": "TsGVY9xq9Q", "reviewer": "Reviewer_bT4z", "summary": "The authors in this paper presents a method for cloth dynamics grounding. Cloth dynamics are learned from visual observations under unknown conditions without physical supervision.", "questions": "refer to the weakness section", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The use of  Spatial Mapping Gaussian Splatting to establish a mapping between the 2D pixel space and 3D space is interesting. SMGS handles large deformations and severe self-occlusion by using both relative and absolute positions of the Gaussian components. This design ensures an accurate mapping between the 2D and 3D spaces during rendering.", "weaknesses": "- The visual results are shown under wind force, it would have been interesting to see cloth dynamics under various type snd source of forces e.g. objects colliding with cloth. How to model them inside the current framework.\n- A detailed analysis on cloth-cloth collision, cloth-object collision is missing.\n- Do add following relevant references under neural garment simulator GarSim: Particle Based Neural Garment Simulator WACV 2023 and GenSim: Unsupervised Generic Garment Simulator CVPR 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3SvuLwfL3m", "reviewer": "Reviewer_kW74", "summary": "This paper learns cloth dynamics from video. The approach first reconstructs a mesh from images via Gaussian splatting, then applies a mesh-based neural simulator to model dynamics in the mesh domain. Experiments show performance comparable to or better than selected baselines.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "* The method learns cloth dynamics directly from video.\n* The method achieves performance comparable to approaches trained on ground-truth mesh data.", "weaknesses": "1. Clarity and consistency. The writing is unnecessarily complex. If I understand correctly, a simple and clear description would be: learn dynamics directly from videos by first performing video-to-geometry grounding, then training a dynamics model on the grounded meshes. Also, there appears to be a typo/inconsistency: Equations (7) and (9) for geometry should take the same input parameters; please verify and correct.\n2. Related work coverage (missing citations). Given the focus on data-driven, mesh-based cloth simulation, the Related Work should include additional existing work (e.g., [1,2,3]). In particular, [3] also learns cloth dynamics from multi-view videos and addresses more complex scenarios (richer appearance, human body interactions). Please discuss [3] in more detail and, if feasible, compare against the pipeline in [3].\n3. Dataset scope and realism. The current dataset appears limited (a single cloth and ~120 videos of trajectories). Compared to [3], this setting may be simplistic. Please either expand the number/diversity of training and test videos or report results on established real-world datasets (e.g., those used in [3]) to demonstrate robustness and generality.\n4. Experimental protocol and reporting. Since results are reported over 20 trajectories, include mean $\\pm$ std across trajectories to reflect variability. Additionally, please report inference time (e.g., FPS/latency on a specified GPU) to quantify simulator efficiency.\n\n\n[1]. Santesteban, et al. Self-Supervised Collision Handling via Generative 3D Garment Models for Virtual Try-On. CVPR 2021\n[2]. Shao, et al. Towards Multi-Layered 3D Garments Animation. ICCV 2023.\n[3]. Rong, et al. Gaussian Garments: Reconstructing Simulation-Ready Clothing with Photorealistic Appearance from Multi-View Video. 3DV 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NWSbSixHQD", "reviewer": "Reviewer_FVhY", "summary": "This paper introduces Cloth Dynamics Splatting (CloDS), an unsupervised, visual-only framework for Cloth Dynamics Grounding, which aims to learn cloth dynamics from multi-view videos under unknown conditions without direct physical supervision. The core strength lies in its ability to bridge the gap between 2D visual observations and 3D physical representations for highly deformable materials, a significant challenge in the field.", "questions": "See weaknesses", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper clearly defines a new and challenging problem, Cloth Dynamics Grounding, which focuses on unsupervised learning of cloth dynamics solely from visual data.\n\n- The introduction of Spatial Mapping Gaussian Splatting, a mesh-based Gaussian splatting module, provides a differentiable mapping between 2D pixel space and 3D geometry. The proposed dual-position opacity modulation in SMGS is a clever solution to address severe self-occlusions and large non-linear deformations inherent to cloth dynamics. For the mesh-based gaussian splatting, there are relative work that should be cited.\n\n[1]SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering\n\n[2]real-time large-scale deformation of gaussian splatting\n\n[3]VR-GS: A Physical Dynamics-Aware Interactive Gaussian Splatting System in Virtual Reality\n\n[4]Recent Advances in 3D Gaussian Splatting\n\n- CloDS achieves performance close to fully mesh-supervised methods on the CDG task, demonstrating effective unsupervised dynamics learning.", "weaknesses": "- The method assumes an initial mesh state ($M_1$) is available to build the initial Gaussian component representation. Although robustness to initial mesh errors is analyzed, I still suggest that some visual results should be prepared and presented to incorporate the results reported in FigureS.2.\n\n- Performance degrades under complex lighting conditions due to temporal inconsistency caused by shadows and illumination, suggesting the current approach is sensitive to visual changes beyond pure geometry and dynamics.\n\n- How is the initial mesh $M_1$ \"extracted from the initial frame $I_1^{1:N}$ via 2D Gaussian Splatting\"? Is this step fully unsupervised, or does it rely on any pre-trained model, shape priors, or a fixed template mesh? A brief explanation of how $M_1$ is obtained would clarify the unsupervised visual-only premise.\n\n- The comparison to video prediction models is strong, but since a key challenge is 3D-aware modeling, a comparison to other geometry-aware, unsupervised methods (e.g., scene flow-based approaches or other particle-based visual grounding methods like NeuroFluid, adapted for cloth) would further solidify the value of the DVC and SMGS approach for this task.\n\n- DeepFashion3D is also an impressive cloth dataset, some evaluations on the dataset are more encouraged.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)