# PhyDeformer: High-Quality Non-Rigid Garment Registration with Physics-Awareness

Boyang Yu<fnref n="1" />, Frederic Cordier<fnref n="2" />, Hyewon Seo<fnref n="1" />†

## Abstract

Accurately registering 3D garment meshes to real-world image data is a fundamental yet challenging task in computer vision and graphics, with applications in virtual try-on systems, digital fashion, performance capture, and virtual content creation. This problem involves recovering detailed, non-rigid garment geometry from partial, noisy, and often ambiguous visual cues extracted from 2D or reconstructed 3D data. A key challenge lies in aligning garment templates with target shapes while preserving realistic fabric behavior and accommodating variations in body shape, garment fit, and pose. We present PhyDeformer, a new deformation method for high-quality garment mesh registration. It operates in two phases: In the first phase, a garment grading is performed to achieve a coarse 3D alignment between the mesh template and the target mesh, accounting for proportional scaling and fit (e.g. length, size). In the second phase, the graded mesh is refined to capture fine-grained geometric details of the 3D target through a localized optimization process, leveraging a Jacobian-based deformation framework. Both quantitative and qualitative evaluations on synthetic and real garment data demonstrate the effectiveness and robustness of our method in achieving accurate and visually plausible registrations. The code and base meshes generated and evaluated in this paper are available at https://github.com/MLMS-CG/PhyDeformer.

## 1. Introduction

Real 3D garment data is becoming increasingly prevalent in virtual clothing, enhancing realism in many applications such as gaming, film, fashion, and virtual try-on. The integration of such real-world garment into virtual clothing is often challenging due to the large variations and high-frequency details present in realistic garments geometries. Aligning virtual template garments to 3D image of real clothing can accelerate the recovery and parametric adaptation of garment shape and fit, enabling more efficient synthesis, editing, and simulation in downstream digital fashion pipelines.

In this paper, we propose a deformation method tailored for high-quality garment mesh registration. Our two-stage mesh deformation approach, named PhyDeformer, effectively performs non-rigid registration to capture a wide variety of garment shapes, handling both large modifications (e.g., proportional resizing) and intricate details, such as folds and wrinkles. In the first stage, garment grading is performed to achieve a coarse 3D alignment between the template and the target mesh, accounting for proportional scaling and fit. In the second stage, the graded garment mesh undergoes refinement using Jacobian-based deformation, guided by both the reconstruction loss and the physics-based constraints. As a result, we obtain a resulting mesh that fits the target well while maintaining physical plausibility.

## 2. Related work

### 2.1. Garment registration

Existing approaches typically estimate vertex-wise displacement maps to register a single-layer mesh [SYMB21, MYR*20] or multiple garment layers [PMPHB17, TBTPM20, CPY*21] to 3D scans of clothed humans acquired using body scanners. [SYMB21, MYR*20] employ an energy-minimization framework with objectives that combine data fidelity and regularization terms. While effective in capturing body shapes under tight-fitting garments, their single-surface representation of both body and clothing imposes inherent limitations. In particular, this approach struggles to accurately model loose or flowing garments (e.g., skirts, dresses) that deviate significantly from the underlying body shape. [CPY*21] attempts to address this limitation by employing a multi-stage alignment scheme that progressively optimizes the vertex displacements. However, the consecutive intermediate optimizations introduce complexity, which may hinder reproducibility. The Parser-Net framework of Tiwari et al. [TBTPM20] offers a learning-based alternative to a similar problem: They register a unified body-clothing template mesh to the input scan, then map a single mesh registration to multiple garment-specific representation of clothing via a learned regressor. However, fine geometric details such as wrinkles and folds are not explicitly modeled and may be lost during the vertex-wise regression.

Lahner et al [LCT18] propose a two-stage approach for realistic

<footnote n="1">¹ICube laboratory, University of Strasbourg, France</footnote>

<footnote n="2">²LMIA, University of Haut Alsace, Strasbourg, France</footnote>

<footnote>† Corresponding author:seo@unistra.fr</footnote>
cloth animation. First, they learn a low-dimensional deformation subspace from 4D scan sequences, enabling the retargeting of a template garment to new body poses and shapes for reconstructing realistic global cloth deformations. Then, a conditional generative adversarial network (cGAN) enhances these results by predicting high-frequency, pose-dependent wrinkle details from coarse normal maps, ensuring temporal coherence throughout motion sequences. While the predicted wrinkles appear visually realistic, the high-frequency details are encoded as normal maps rather than actual geometric displacements of the mesh.

Another line of work fits garments by optimizing their 2D sewing patterns through differentiable simulation [YCS24, LCL*24]. The key idea is to jointly optimize the pattern geometry and simulation parameters so that the simulated garment closely matches the target garment shape. While this produces physically simulated garment results, the approach is computationally expensive as each optimization step requires running a full quasi-static simulation.

## 2.2. Garment reconstruction

Learning-based generative models can also be used for 3D garment reconstruction to match a target garment shape. These methods typically predict geometric deformations such as vertex-wise displacements [PLPM20, DLLG*23] or implicit field updates [JZH*20, CPA*21, LGF24] to align a template mesh or learned garment representation with the target geometry. Predictions are obtained from pretrained networks and can be further refined by optimizing latent variables via backpropagation. TailorNet [PLPM20] predicts garment deformation as a function of pose, body shape, and style, enabling registration of a template garment to a target body with high-frequency wrinkle detail through low- and high-frequency displacement components. ISP [LGF24] represents garments as 2D sewing pattern panels parameterized as signed distance functions, which are mapped to 3D to register multi-layered garments onto a target body via a differentiable 2D-to-3D mapping. SMPLicit [CPA*21] adopts a unified implicit representation conditioned on SMPL pose and shape, allowing for registration, reconstruction, and editing of a wide variety of garment types directly from scans or images. DrapeNet [DLLG*23] learns a latent space for garments of arbitrary topology and couples it with a self-supervised draping network that predicts vertex-wise displacements to fit garments onto target body shapes while avoiding interpenetrations, providing a fully differentiable pipeline for both garment generation and registration from 3D scans or images. Nevertheless, the predictions are influenced by dataset biases and may fail to achieve the high fitting precision required for accurate garment alignment.

## 3. Method

Figure 1 illustrates an overview of our method. PhyDeformer registers a given template mesh geometry to the target garment mesh through a two-stage process. In the first stage, the linear grading module followed by draping captures the overall geometry including size and proportions. In the second stage, a refinement is achieved by optimizing a displacement mapping $\phi: R^3 \to R^3$ over the vertices through Jacobians guided by a set of losses.

<figure><img src="image_1.png" /><figcaption>**Figure 1:** Given an input 3D base mesh and a target garment, we first perform linear grading on the base mesh to achieve an initial alignment. It is further refined by optimizing per-triangle Jacobians.</figcaption></figure>

### 3.1. Coarse grading

Given a template garment mesh $X_{base}$ and its 2D sewing pattern $U_{base}$, the initial draping shape $X_{init}$ is simulated on the underlying body B. Note that the difference between the resulting mesh $X_{init}$ and the target T may be substantial at this stage. To bridge the gap, a coarse geometric deformation is performed to capture the overall geometric shape of the target garment, such as length and proportion, by using a set of key measurements on 3D open contours. A 3D open contour is composed of edges connected to only one adjacent triangle, which often carry design features, representing elements such as necklines, hem contours, cuffs, etc. These open contours are extracted from both the draped and target meshes, each paired with its counterpart. As in [YCS24], longitudinal distances between corresponding contours are measured along the body skeleton, together with the circumference differences, to adjust the 2D sewing pattern. The updated 2D pattern is then remeshed and re-draped to produce $X_{LG}$, a simulated shape that reflects the changes of 2D pattern in 3D geometry. Readers may refer to [YCS24] for more technical details.

### 3.2. Non-rigid geometry refinement

We further refine the simulated garment $X_{LG}$ (referred to as the source mesh hereafter) obtained from the previous stage through mesh deformation. A straightforward approach to deformation involves directly optimizing the coordinates of the source mesh vertices. However, with this method, each vertex's gradient influences only its own displacement, which can result in the excessive exposure of high-frequency details and lead to undesirable artifacts. Aiming to preserve the structure and topology of the source mesh while achieving the deformation process in a single step, we opt for deforming the vertices X of the garment mesh indirectly using a set of per-triangle Jacobians (face gradients), inspired by [GAG*23, AGK*22]. In this setting, the objective is to find the optimal deformation map that captures the target shape. Starting with
the variables $\mathbf{J}_i \in \mathbb{R}^{3x3}$ initialized with per-triangle Jacobians of the source mesh, an optimization process updates the Jacobian field using multiple loss terms, detailed below.

As the optimization is applied to $\mathbf{J}_i$ in a per-triangle manner, a Poisson equation is solved to find new vertex positions well connected $\phi: \mathbb{R}^3 \to \mathbb{R}^3$, such that the Jacobians $\nabla_i(\phi)$ of this deformation mapping for each triangle is closest to $\mathbf{J}_i$ in the least square sense.

$$ \phi^* = \min_{\Phi} \sum_{i=1}^{|F|} A_i \|\nabla_i(\phi) - \mathbf{J}_i\|^2, \quad (1) $$

where $A_i$ is the face area, $\nabla_i$ is defined as the face gradient operator, $\nabla_i(\phi)$ denotes the Jacobian of $\phi$ at triangle $f_i \in F$. Compared to vertex-wise optimization, the propagated gradients $\frac{\partial \mathcal{L}}{\partial f_i}$ influence a larger surface mesh area, leading to a globally-coherent deformation $\phi$ which is indirectly optimized by $\mathbf{J}_i$.

**Losses.** At each iteration, the deformed garment geometry $X \leftarrow \phi^*(X)$ is compared with the target $T = \{t\}$, and the Adam algorithm is used to minimize a loss function that drives the deformation of the garment mesh. We employ a set of loss terms to ensure the refined garment mesh conforms to the target while maintaining physical realism. Several of these loss terms, specifically those arising from membrane strain energy and bending energy ($\mathcal{L}_s, \mathcal{L}_b$) are inspired by prior work, notably SNUG [SOC22].

We use a reconstruction loss to evaluate the similarity between $X$ and $T$. This loss comprises Chamfer distances $\mathcal{L}_{CF}$ calculated between both the surfaces and the open contours,

$$ \mathcal{L}_{rec} = \mathcal{L}_{CF}(X, T) + \mathcal{L}_{CF}(X_{open}, T_{open}). \quad (2) $$

The cosine distance of normals $\mathcal{L}_n$ between $X$ and $T$ is also measured, which is written as:

$$ \mathcal{L}_n = \frac{1}{|X|} \sum_x^{|X|} (1 - \langle \mathbf{n}_x, \mathbf{n}_{\tilde{t}} \rangle) + \frac{1}{|T|} \sum_t^{|T|} (1 - \langle \mathbf{n}_t, \mathbf{n}_{\tilde{r}} \rangle), \quad (3) $$

where $\mathbf{n}_x$ and $\mathbf{n}_{\tilde{t}}$ are the unit normal vectors at point $x \in X$ and $\tilde{t} = \arg\min_{t \in T} (\|x - t\|)$ respectively, and vice versa for $\mathbf{n}_t$ and $\mathbf{n}_{\tilde{r}}$.

Strain loss is employed to ensure that the shapes of the triangles in the deformed garment resist stretching. This loss term is based on the Saint Venant Kirchhoff (StVK) elastic material model, which is formulated as:

$$ \mathcal{L}_s = \sum_i^{|F|} \left( \frac{\lambda}{2} \cdot \mathrm{tr}(\mathbf{G}_i)^2 + \mu \cdot \mathrm{tr}(\mathbf{G}_i^2) \right) A_i, \quad (4) $$

where $\lambda$ and $\mu$ represent the Lamé coefficients, which are respectively set to 16.3 and to 13.5 in our experiments. $A_i$ is the area of $i$-th triangle, and $\mathbf{G}_i$ the green strain tensor. This regularization term constrains the deformation, ensuring that the shapes of the triangles do not deviate excessively from the original, undeformed geometry.

Bending loss is employed to penalize changes in discrete curvature as a function of the dihedral angle between edge-adjacent triangles, which is formulated as follows:

$$ \mathcal{L}_b = \sum_j^{|E|} \frac{\kappa}{2} \cdot \alpha_j^2, \quad (5) $$

where $\kappa$ represents the bending stiffness set to $4e-5$, $E$ is the edges, and $\alpha_j$ the radian angle between two adjacent triangles. This loss term effectively constrains the deformation of the garment to prevent excessive bending.

Finally, collision loss is employed to prevent the interpenetration of garment vertices with the body mesh (if available). This is achieved by penalizing the negative distance between garment nodes and their closest point on the body surface with a cubic energy term:

$$ \mathcal{L}_c = \sum_n^{|X|} \max(\epsilon - \mathrm{sdf}(x), 0)^3, \quad (6) $$

where $\mathrm{sdf}(\cdot)$ represents the distance between the query point and the body surface, and $\epsilon$ the chosen minimal distance threshold between the body and the garment. This loss term constrains the deformation of the garment to prevent the intersection with the body mesh, leading to a more physically correct outcome.

All of these loss terms are combined to form the final objective to guide the deformation of the source mesh:

$$ \mathcal{L} = \mathcal{L}_{rec} + \lambda_n \mathcal{L}_n + \lambda_s \mathcal{L}_s + \lambda_b \mathcal{L}_b + \lambda_c \mathcal{L}_c. \quad (7) $$

# 4. Results and Experiments

## 4.1. Implementation details

We conducted the experiments using a NVIDIA 3090 GPU, 24Gb RAM, and an Intel i7-5220R CPU. We run the optimization for 1500 iterations until the convergence is reached, which takes approximately 5 minutes. The stretching regularization term is added after 500 iterations. The learning rate was set to 0.002 in the Adam optimizer. The loss weights were set as: $\lambda_n = 0.01$, $\lambda_s = 1$, $\lambda_b = 0.1$, and $\lambda_c = 0.01$ in our experiments. We use template base models selected from the Berkeley Garment Library [NSO12].

## 4.2. Evaluation on synthetic garments

To demonstrate the effectiveness of our garment registration approach, we leveraged the geometry of 3D garments from two synthetic 3D datasets as targets. LAVA lab dataset [KL21] contains a wide range of high-quality garment meshes simulated over the average female body at T pose; Sewfactory [LXL*23] extends to a more comprehensive dataset containing diverse garment styles and human shapes in various poses. We selected several representative 3D garment geometries from the datasets, which serve as the targets. For the base meshes, we utilize a set of garment templates that cover basic clothing categories (e.g., shirts, skirts, dresses, etc.). These base meshes, presented in a canonical pose, are publicly available with the accompanying code to support future research in 3D garment modeling.

We qualitatively evaluated the results with state-of-the-art works: Tailornet [PLPM20], SMPLicit [CPA*21], Drapenet and ISP [DLLG*23, LGF24], and IGPM [YCS24]. Note that the parameter space representing the garment geometry differs from each other: the $\gamma$ garment style parameter for TailorNet [PLPM20], the latent vector $z = [z_{cut}, z_{style}]$ describing garment cut and style
<figure><img src="image_2.png" /><figcaption>**Figure 2:** Comparison of 3D garment reconstruction of our method with others [PLPM20, CPA*21, DLLG*23, LGF24]. Our method reproduces faithful garment shapes, even accounting for intricate geometry details like wrinkles on large sleeves. Best viewed on screen zoomed-in.</figcaption></figure>

for SMPLicit [CPA*21], the latent codes z's encoding the garment characteristics in ISP [LGF24], and the coordinates of effective control points and material parameters in IGPM. Drapenet [DLLG*23] uses unsigned distance functions (UDFs) and requires extra computations for meshing, and it is sensitive to the initialization of latent code for optimization. Both ISP [LGF24] and Drapenet can produce certain geometric details in the registration to garments, but when it comes to challenging posed garment samples such as S01 and S05, the performance decreases and leads to visible defects. IGPM [YCS24] exploits the inverse cloth simulation to achieve coarse-to-fine alignment, but it fails to capture the exact wrinkle patterns and the inverse simulation is relatively expensive to execute. In contrast, PhyDeformer reconstructs accurate 3D geometry, for both loose and tight garments of different subjects with less computational cost.

For quantitative evaluation of the geometric similarity, two metrics are used: Chamfer distance (L2 norm, scaled by $e^3$) to the ground truth mesh vertices, and the normal similarity to measure the orientation consistency of the surface. As shown in Table 1 and Table 2, our method outperforms others in the 3D reconstruction of

posed garments, which is confirmed by qualitative results, shown in Figure 2 and Figure 3.

<figure><img src="image_3.png" /><figcaption>**Figure 3:** Qualitative comparison of 3D garment reconstruction using our method and other approaches [YCS24, DLLG*23, LGF24].</figcaption></figure>

### 4.3. Evaluation on 3D scan data

To showcase the capability of our refinement stage in handling fine-grained, wrinkle-level details from real scans, as well as its ease of integration with other coarse fitting techniques, we performed a qualitative evaluation using the GarmCap dataset [LZZ*23]. We adopted their rigged smooth template and coarse fitting, refining the alignment using our second step. As shown in Figure 4, our method outputs reasonable, high-quality reconstruction of the 3D scanned garments.

### 5. Ablation study

We conduct ablation studies to evaluate how the components affect the overall fitting quality. Some results are shown in Table 3 for the garment S01, the most representative data in our experiments.

**Without linear grading.** In Figure 5 (b) we illustrate that our method's performance decreases when the linear grading stage is disabled. The linear grading provides better initialization, thereby contributing to improved overall performance.

**Without Jacobians.** In Figure 6, we show that replacing Jacobians with vertex displacements in the optimization decreases the resulting surface quality. Representing the deformation through vertex displacements exposes the per-vertex high-frequency mode of the deformation, thus leading to localized noisy gradients that deteriorate the triangulation of the meshes(holes) as illustrated in Figure 6.
<table><caption>**Table 1:** Quantitative comparison results in 3D garment reconstruction on LAVA lab dataset. The best results are in boldface.</caption><thead><tr><th rowspan="2">Garments</th><th rowspan="2">SMPLicit</th><th colspan="4">Chamfer distance / Normal similarity</th></tr><tr><th>TailorNet</th><th>Drapenet</th><th>ISP</th><th>IGPM</th><th>Ours</th></tr></thead><tbody><tr><td>T-shirt</td><td>1.4 / -</td><td>0.331 / 0.081</td><td>0.689 / 0.129</td><td>0.297 / 0.094</td><td>0.112 / 0.049</td><td><strong>0.101 / 0.017</strong></td></tr><tr><td>Dress</td><td>3.2 / -</td><td>1.305 / 0.161</td><td>0.619 / 0.135</td><td>0.189 / 0.131</td><td>0.110 / 0.075</td><td><strong>0.096 / 0.029</strong></td></tr><tr><td>Shorts</td><td>1.3 / -</td><td>1.036 / 0.050</td><td>0.131 / 0.048</td><td>0.202 / 0.095</td><td>0.126 / 0.043</td><td><strong>0.088 / 0.019</strong></td></tr><tr><td>Pants</td><td>2.9 / -</td><td>2.587 / 0.104</td><td>0.485 / 0.085</td><td>0.185 / 0.077</td><td>0.142 / 0.049</td><td><strong>0.084 / 0.019</strong></td></tr><tr><td>Skirt</td><td>6.5 / -</td><td>1.30 / 0.063</td><td>- / -</td><td>0.435 / 0.093</td><td>0.106 / 0.014</td><td><strong>0.086 / 0.007</strong></td></tr></tbody></table>

<table><caption>**Table 2:** Quantitative evaluation in 3D garment reconstruction on Sewfactory dataset.</caption><thead><tr><th rowspan="2">Garments</th><th colspan="4">Chamfer distance / Normal similarity</th></tr><tr><th>Drapenet</th><th>ISP</th><th>IGPM</th><th>Ours</th></tr></thead><tbody><tr><td>S01</td><td>0.346 / 0.235</td><td>0.431 / 0.232</td><td>0.191 / 0.203</td><td><strong>0.159 / 0.058</strong></td></tr><tr><td>S02</td><td>17.031 / 0.165</td><td>17.61 / 0.214</td><td>0.083 / 0.089</td><td><strong>0.047 / 0.021</strong></td></tr><tr><td>S03</td><td>0.798 / 0.156</td><td>0.182 / 0.164</td><td>0.205 / 0.124</td><td><strong>0.076 / 0.061</strong></td></tr><tr><td>S04</td><td>- / -</td><td>3.024 / 0.340</td><td>0.398 / 0.276</td><td><strong>0.228 / 0.27</strong></td></tr><tr><td>S05</td><td>1.831 / 0.268</td><td>0.87 / 0.186</td><td>0.090 / 0.07</td><td><strong>0.138 / 0.028</strong></td></tr></tbody></table>

<figure><img src="image_4.png" /><figcaption>**Figure 4:** Qualitative results on the GarmCap dataset. From left to right: posed templates, coarse fittings, refined fittings by PhyDeformer, and the target shape. Best viewed when zoomed in.</figcaption></figure>

(b). Clipping the gradients can alleviate this but still causes sharp "burrs" on the deformed mesh shown in Figure 6 (c).

**Without loss terms.** Figure 7 shows that our refinement stage with all losses can lead to large and smooth deformations for the final fitting. To assess the effectiveness of each loss term, we conducted an ablation study by omitting individual losses — specifically, the contour loss, normal loss, and bending loss.

<figure><img src="image_5.png" /><figcaption>**Figure 5:** Results obtained without linear grading: (a) posed mesh without linear grading, (b) deformed mesh, (c) target mesh.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>**Figure 6:** Results of optimizing vertex displacements instead of Jacobians: (a) source mesh, (b) deformed mesh with naive vertex-displacements optimization, (c) deformed mesh with clipped gradients to avoid holes caused by "NaNs", (d) target mesh.</figcaption></figure>

As shown in Table 3, the removal of the open contour loss term did not affect much the global geometric shape, though it led to an increase in Chamfer distance. The consequence is more perceptible in Figure 7 (b), where a visible failure around the collar region can be observed. Eliminating the normal restriction significantly compromised the resulting deformed mesh, as demonstrated in Figure 7 (c). Similarly, omitting the bending loss introduced evident bumpy artifacts, shown in Figure 7 (d). Quantitative results in Table 3 confirm that our two-phase refining strategy enhances the geometric accuracy of the output.

<table><caption>**Table 3:** Quantitative results of the ablation study on garment S01.</caption><thead><tr><th>Method</th><th>Chamfer distance</th><th>Normal similarity</th></tr></thead><tbody><tr><td>w/o linear grading</td><td>0.188</td><td>0.106</td></tr><tr><td>w/o contour loss</td><td>0.088</td><td>0.056</td></tr><tr><td>w/o normal consistency</td><td>0.124</td><td>0.173</td></tr><tr><td>w/o bending loss</td><td>0.158</td><td>0.052</td></tr><tr><td>Ours</td><td><strong>0.047</strong></td><td><strong>0.021</strong></td></tr></tbody></table>

## 6. Robustness to noisy targets

We assess the method’s robustness to noise in the target shape by introducing Gaussian noise to the vertex coordinates of the target
<figure><img src="image_7.png" /><figcaption>**Figure 7:** Results of our ablation study on loss terms: (a) source mesh, (b) without contour loss, (c) without normal loss, (d) without bending loss, (e) with all losses, (f) target mesh.</figcaption></figure>

mesh, with standard deviation ranging from 0.5 to 1 cm. When evaluating PhyDeformer with these modified targets (see Figure 8), we observe a decline in performance as noise levels increase. This indicates that PhyDeformer can effectively manage minimal noise levels, typical of high-accuracy 3D scanning systems.

<figure><img src="image_8.png" /><figcaption>**Figure 8:** The alignment results on target meshes with synthetic noise of two different levels.</figcaption></figure>

## 6.1. Use case: Registration for inverse garment simulation

PhyDeformer offers significantly better efficiency than the fully physics-based simulation-embedded method IGPM [YCS24], whereas the latter offers simulation-ready assets for more flexible reusability. PhyDeformer can provide a registered target mesh with consistent topology. After the linear grading stage, the output is registered to the raw target mesh, the resulting deformed mesh is utilized as a new target for the differentiable simulation optimization, replacing the Chamfer distance loss with the per-vertex loss (MSE, mean square error). We refer to this scheme as *Hybrid* in contrast to the baseline IGPM. In Figure 9, we showcase how the hybrid strategy outperforms the vanilla IGPM in terms of the speed of convergence. Since MSE is more expressive, it also leads to more accurate reconstruction.

<figure><img src="image_9.png" /><figcaption>**Figure 9:** We evaluate the Chamfer Distance error for hybrid and IGPM [YCS24]. On the right, we also illustrate the qualitative results for comparison: (a) Target; (b) New target by PhyDeformer; (c) IGBM with Hybrid scheme, and (d) IGPM.</figcaption></figure>

## 7. Conclusion and limitation

In this work, we introduce PhyDeformer, which is lightweight yet capable of producing high-quality registered meshes with detailed wrinkles. We optimize per-face Jacobians to predict smooth mesh deformations, preserving the intrinsic topology of the source shape. Physics-driven loss terms are employed to constrain Jacobian-based deformation, ensuring approximate adherence to physically faithful garment shapes. Consequently, our method generates high-quality geometric outputs by reproducing both low-frequency shape changes and high-frequency details. Aligned garment meshes enable a wide range of applications, such as texture transfer, deformation capture, and physical parameter estimation for draping on human avatars.

In our current setting, we assume that the garment data consist of well-segmented scans or synthetic inputs. We also aim to explore the potential of integrating PhyDeformer with a 2D pattern estimation method. This integration could enhance efficiency by enabling case-specific Jacobian optimization, thereby avoiding the computational burden of large-scale optimization in differentiable simulation frameworks or the need to train deep learning models on extensive datasets.

## Acknowledgments

This work has been funded by the binational project "Synthetic Data Generation and Sim-to-Real Adaptive Learning for Real-World Human Daily Activity Recognition of Human-Care Robots (21YS2900)" granted by the ETRI, South Korea.

## References

[AGK*22] AIGERMAN N., GUPTA K., KIM V. G., CHAUDHURI S., SAITO J., GROUEIX T.: Neural jacobian fields: Learning intrinsic mappings of arbitrary meshes. *SIGGRAPH* (2022). 2

[CPA*21] CORONA E., PUMAROLA A., ALENYÀ G., PONS-MOLL G., MORENO-NOGUER F.: Smplicit: Topology-aware generative model for clothed people. In *CVPR* (2021). 2, 3, 4

[CPY*21] CHEN X., PANG A., YANG W., WANG P., XU L., YU J.: Tightcap: 3d human shape capture with clothing tightness field. *ACM Trans. Graph.* 41, 1 (nov 2021). doi:10.1145/3478518.1
[DLLG*23] DE LUIGI L., LI R., GUILLARD B., SALZMANN M., FUA P.: Drapenet: Garment generation and self-supervised draping. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition* (2023), pp. 1451–1460. 2, 3, 4

[GAG*23] GAO W., AIGERMAN N., GROUEIX T., KIM V., HANOCKA R.: Textdeformer: Geometry manipulation using text guidance. In *ACM SIGGRAPH 2023 Conference Proceedings* (2023), pp. 1–11. 2

[JZH*20] JIANG B., ZHANG J., HONG Y., LUO J., LIU L., BAO H.: Bcnet: Learning body and cloth shape from a single image. In *Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XX 16* (2020), Springer, pp. 18–35. 2

[KL21] KOROSTELEVA M., LEE S.-H.: Generating datasets of 3d garments with sewing patterns. In *Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks* (2021), Vanschoren J., Yeung S., (Eds.), vol. 1. 3

[LCL*24] LI Y., CHEN H.-Y., LARIONOV E., SARAFIANOS N., MATUSIK W., STUYCK T.: DiffAvatar: Simulation-ready garment optimization with differentiable simulation. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)* (June 2024). arXiv:2311.12194. 2

[LCT18] LAHNER Z., CREMERS D., TUNG T.: Deepwrinkles: Accurate and realistic clothing modeling. In *Proceedings of the European conference on computer vision (ECCV)* (2018), pp. 667–684. 1

[LGF24] LI R., GUILLARD B., FUA P.: Isp: Multi-layered garment draping with implicit sewing patterns. *Advances in Neural Information Processing Systems 36* (2024). 2, 3, 4

[LXL*23] LIU L., XU X., LIN Z., LIANG J., YAN S.: Towards garment sewing pattern reconstruction from a single image. *ACM Transactions on Graphics* (TOG) 42, 6 (2023), 1–15. 3

[LZZ*23] LIN S., ZHOU B., ZHENG Z., ZHANG H., LIU Y.: Leveraging intrinsic properties for non-rigid garment alignment. In *IEEE/CVF International Conference on Computer Vision (ICCV)* (2023). 4

[MYR*20] MA Q., YANG J., RANJAN A., PUJADES S., PONS-MOLL G., TANG S., BLACK M. J.: Learning to dress 3d people in generative clothing. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition* (2020), pp. 6469–6478. 1

[NSO12] NARAIN R., SAMII A., O'BRIEN J. F.: Adaptive anisotropic remeshing for cloth simulation. *ACM Transactions on Graphics* (TOG) 31, 6 (2012), 1–10. 3

[PLPM20] PATEL C., LIAO Z., PONS-MOLL G.: Tailornet: Predicting clothing in 3d as a function of human pose, shape and garment style. In *IEEE Conference on Computer Vision and Pattern Recognition (CVPR)* (jun 2020), IEEE. 2, 3, 4

[PMPHB17] PONS-MOLL G., PUJADES S., HU S., BLACK M. J.: Clothcap: Seamless 4d clothing capture and retargeting. *ACM Transactions on Graphics* (ToG) 36, 4 (2017), 1–15. 1

[SOC22] SANTESTEBAN I., OTADUY M. A., CASAS D.: Snug: Self-supervised neural dynamic garments. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition* (2022), pp. 8140–8150. 3

[SYMB21] SAITO S., YANG J., MA Q., BLACK M. J.: SCANimate: Weakly supervised learning of skinned clothed avatar networks. In *Proceedings IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)* (June 2021). 1

[TBTPM20] TIWARI G., BHATNAGAR B. L., TUNG T., PONS-MOLL G.: Sizer: A dataset and model for parsing 3d clothing and learning size sensitive 3d clothing. In *Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16* (2020), Springer, pp. 1–18. 1

[YCS24] YU B., CORDIER F., SEO H.: Inverse garment and pattern modeling with a differentiable simulator. In *Computer Graphics Forum* (2024), Wiley Online Library, p. e15249. 2, 3, 4, 6