Structure-Preserving 3D Garment Modeling with Neural Sewing Machines
Authors: Xipeng Chen
Guangrun Wang
Dizhong Zhu
Xiaodan Liang
Philip Torr
Liang Lin
Venue: NeurIPS 2022 Accept (2022)

Abstract: 3D Garment modeling is a critical and challenging topic in the area of computer vision and graphics, with increasing attention focused on garment representation learning, garment reconstruction, and controllable garment manipulation, whereas existing methods were constrained to model garments under specific categories or with relatively simple topologies. In this paper, we propose a novel Neural Sewing Machine (NSM), a learning-based framework for structure-preserving 3D garment modeling, which is capable of learning representations for garments with diverse shapes and topologies and is successfully applied to 3D garment reconstruction and controllable manipulation. To model generic garments, we first obtain sewing pattern embedding via a unified sewing pattern encoding module, as the sewing pattern can accurately describe the intrinsic structure and the topology of the 3D garment. Then we use a 3D garment decoder to decode the sewing pattern embedding into a 3D garment using the UV-position maps with masks. To preserve the intrinsic structure of the predicted 3D garment, we introduce an inner-panel structure-preserving loss, an inter-panel structure-preserving loss, and a surface-normal loss in the learning process of our framework. We evaluate NSM on the public 3D garment dataset with sewing patterns with diverse garment shapes and categories. Extensive experiments demonstrate that the proposed NSM is capable of representing 3D garments under diverse garment shapes and topologies, realistically reconstructing 3D garments from 2D images with the preserved structure, and accurately manipulating the 3D garment categories, shapes, and topologies, outperforming the state-of-the-art methods by a clear margin.

Structure-Preserving 3D Garment Modeling with Neural Sewing Machines
Xipeng Chen', Guangrun Wang", Dizhong Zhu?, Xiaodan Liang', Philip H. Torr?, Liang Lin
Sun Yat-sen University
2 University of Oxford
chenxp37@mai 1t. edu. wanggrun@gmal com,
Amos ZhuShotmail com, 1iangxc9mmai. sysu. edu. cn, pellipp.oorrenn ox. ac. uk, linliang@ieee.co org
Abstract
3D Garment modeling is a critical and challenging topic in the area of computer vision and graphics, with increasing attention focused on garment representation learning, garment reconstruction, and controllable garment manipulation, whereas existing methods were constrained to model garments under specific categories with relatively simple topologies. In this paper, we propose a novel Neural Sewing Machine (NSM), (carning-basee framework for structure- structure-preservvnne preserving 3D garment modeling, which is capable of learning representations for garments with diverse shapes and topologies and is successfully applied to 3D garment reconstruction and controllable manipulation. To model generic garments, obtain sewing pattern embedding via a unified sewing pattern encoding module, as the sewing pattern accurately describe the intrinsic structure and the topology the 3D garment. Then we use a 3D garment decoder to decode the sewing pattern embedding into 3D garment using the UV-position maps with masks. To preserve the intrinsic structure of the predicted 3D garment, we introduce an inner-panel structure-preserve preserving loss, an inter-panel prnettre-prreeevvin loss, and a surface-normal loss in the learning process of our framework. We evaluate NSM on the public 3D garment dataset with sewing patterns with diverse garment shapes and categories. Extensive experiments demonstrate that the proposed NSM is capable representing 3D garments under diverse garment shapes and topologies, realistically reconstructing 3D garments from 2D images with the preserved structure, and accurately manipulating the 3D garment categories, shapes, and topologies, outperforming the state-of of-the-ar methods by a clear margin.
first
Introduction
long-standing vision in computer vision and graphics generic 3D garment modeling, which discover patterns in the clothing data, provide better understanding of the clothing data, and learn reconstructable and manipulable representations for the clothing data. In the past decades, the dominant paradigm in 3D garment modeling resolves to physics-based simulators, which were computationally expensive and time-consuming
335 Recently, deep learning has been introduced and has made great strides (24] 49 allowing wide range of applications, e.g., 3D virtual try-on
3
[88. garment design 44 and virtual digital humans I
Despite achieving remarkable progress, prior works only model specific categories clothes with relatively simple topologies. Specifically, template-bus basedd methods [17 49 defined templates with fixed topology model garments for each category, which have difficulty
12
use pre-
*Conresponding author is Liang Lin.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Jacket hood
Jacket
Jumpsutt
T-shirt
Pants
Dress
Skirt (style
Skirt Gitye Bi
Skirt (styte
Figure I: Illustration of our NSM for modeling garments with diverse shapes and topologies. The first and third rows denote the predicted UV-position maps with masks. The second and fourth rows denote the predicted 3D garment shape.
in representing real garments with diverse styles and categories. SMPL-based methods (21 use displacement on am SMPL model to represent different kinds of garments, assuming that the garments share the same topology with human bodies, but this assumption doesn' 't hold for generic garments, e.g., a skirt doesn't 't have legs (see Fig. 3. Recently, UV-parrameteriattion baaiiioo-baedddd methods convert a garment to UV position map representation by storing the mesh shape and topology on a learned 2D map representation, but they use either pre-defined UV maps or rely on human body mesh parameterization. Without generic garment modeling, existing methods have shown inferior capability in 3D garment reconstruction and controllable garment manipulation.
[1] [8 30 45
46 177
In this paper, we propose neural sewing machine (NSM), a learning- based approach for modeling generic garments with diverse shapes and topologies. Mimicking physi- cal sewing machine that is utilized for modeling realistic clothes, define NSM model 3D garments using sewing patterns that are widely-used terms physical gar- ment sewing and production. A sewing pattern describes what atomic panels that a garment is composed of and together (see Fig. 3
PixelZMesh
AnchorUDF
NSM
Ground Truth
how these atomic panels are Such sewing patterns can not only truly reflect the garment model in the physical world but also uniquely describe the generic garments with diverse categories and shapes. With this strong prior, our NSM approximates the garment surface by mapping the 2D atomic panels to the surface of the 3D shape, enforcing each warped panel to be as isometric as possible to the corresponding 3D garment (see Fig. D as opposed to previous methods that implicitly predict a 3D garment from a latent code without modeling explicit local structure [38 47
Figure Topologies 3D garments ob- tained from different methods (Best view via zoom-in).
30 39]. Our NSM
guarantees that the garment surface locally resembles the Euclidean plane and the mapping between the 3D garment and the 2D panels is isometric when only limited stretches exist on the garment
surface (see comparisons in Fig. 2
While the idea of an NSM is elegant, designing it faces multiple challenges. First, unifying the embeddings for generic garments with diverse categories, shapes, and topologies is challenging. To achieve this goal, we statistically analyze the panels of different garment categories in the dataset and construct basic panel groups, based which we use PCA build parametrie unified garment encoder. Second, mapping sewing pattern embeddings to 3D shapes is hard, which requires warped panel to be isometric to the 3D garment. To solve this problem, we decouple 3D garments into sewing- sewing-pattern- pattern-base based UV-position maps with masks and use an inversed PCA and CNN to build generic garment decoder. Third, existing deep models tend filter out high-frequency signals and ignore modeling the seams between panels, preventing them from perceiving the 3D structure. To avoid this, we propose an inner-panel loss, an inter-panel loss, and surface-normal loss to learn the pineeeerepreserveed representations for the 3D garments.

---

garment
Sewing pattern
Panall top front
Panel top back
Panel dress front
Panel dress back
Figure 3: Definition of sewing pattem. A 3D skirt is composed of four panels, and each arrowed dashed line indicates the two edges from different panels are stitched together.
Thanks to our NSM, we can model generic garments with reconstructable and manipulable represen- tations, allowing rich potential applications. For example, our NSM can arbitrarily edit the category and shape of garments as if it were physical sewing machine. This interpretable manipulation is quite beneficial for garment design. As another example, our NSM can reconstruct 3D garments from 2D images with high reconstruction accuracy and high-quality 3D structure preservation.
In summary, this paper makes five non-trivial contributions.
We propose an NSM, novel 3D garment modeling method capable of modeling generic garments. Our NSM takes the sewing pattern as a solid prior, which can truly reflect the fidelity and diversity of clothes in the real world, so it can model the real garment structures and learn general garment representations.
A unified sewing pattern encoder for generic garments with diverse categories, shapes, and topologies is proposed in our NSM, where we statistically construct basic panel groups for sewing patterns based on which we use PCA for encoding.
A generic garment decoder for mapping sewing pattern embeddings 3D shapes is proposed in our NSM, where we decouple the 3D garment shapes into sewing-pattent based UV- position maps with masks and use an inversed-PCA and CNN for decoding.
Our NSM addresses the inability of deep models to achieve fidelity of 3D garment structure by proposing novel crructur-preservy panel loss.
losses, including an inner-panel loss and an inter-
Extensive experiments show that NSM achieves state-of- the-art results in multiple 3D garment modeling tasks on the large 3D garment dataset with sewing patterns, including representing 3D garments under diverse garment shapes and topologies, realistically re- constructing 3D garments from 2D images with the preserved structure, and accurately manipulating the 3D garment categories, shapes, and topologies.
2 Related Work
Physics-based simulators. Physics-based methods dating back to the 1980s required enormous computational effort due to the large number of triangular faces that need to be processed (31]. Later, to reduce the computational complexity, some works proposed efficient methods, such as using but searching for optimal hyperparameters for these simulators still required a lot of time. Later, to reduce the hyperparameter search time, some works proposed to learn, estimate, or speculate hyperparameters [2 37 29,, but the controlled environment and simulator were indispensable conditions. Nevertheless, the complexity of physics- based simulators is high, with lot of working time required.
low-resolution simulators and small models 14 36 25
Template-based models. An intuitive way model 3D clothes is use templates. Pioneeringly, Chen et al. [7] defined various garment part templates as a database and searched the required parts from the database to sew them into clothes. However, this searching and-matching impractical since the real world is dependent on infinite garment part templates. To solve this problem, parametric templates were proposed [17 49 13 [3 34 [3] in which templates of the same type but different sizes can be described by values. Although this significantly reduces the template number, designing parametric templates still inefficient. Hence, some works proposed using human bodies as templates
and added displacement to SMPL [20] (a typical human body model) [21] 6 32,
33 2 However,
these methods assume that garments and humans share a topology, which doesn't 't hold for generic
garments, e.g., skirt doesn't t have legs. To make the templates more descriptive of real-workes
clothes, recently, sewing patterns have been considered promising templates [00
28 16
However, the use of sewing patterns is still similar to ordinary parametric templates, so still faces
3

---

DPPD PD
PCA bases
Mask maps
Sewing pattern
Embedding
PCA
Unified Sewing
UV-position maps
Pattern Encoding
Network D
Structure preserving
learning
Sewing patterns
PCA coefficients
322
4
B
A
B'
Lyee
lly
y2ll
Linn
A
d''s'l
basic panel groups
(a)
(b)
PCA
X2
Line = (x (x2))
Enar
cos (Tt n2))
(c)
(d)
Figure 4: This is the Framework of our NSM. The sewing pattern is first encoded into an embedding by the unified sewing pattern encoding module. Then, the UV-position maps with masks are predicted from the
embedding by neural network D and the inverse PCA transform PCA We train NSM by optimizing four
losses to ensure 3D garment structure preservation: (a) reconstruction loss; (b) inner-panel structure-preservin loss; (c) inter- mner-pancl structure- pratturr-rreervingg loss; (d) surface-normar loss.
the dilemma of not being able to model generic clothes. Even state-of-the-ard methods still rely on physical simulators for clothing generation [16], failing to learn reconstructable and manipulable representations. Unlike all previous cemplate-bacet methods, our NSM model garments with complicated topologies; Novelly, our NSM uses a sewing machine to solve the generic garment modeling problem that is intractable with existing template-based methods.
Learned UV parameterization. UV parameterization is the process of mapping a 3D surface to a 2D plane. Usually, this mapping is bijective and means that we can store the 3D vertices continuously
2D UV plane. This allows 2D convolutional neural networks operate on 3D shape, which has
been introduced into neural garment modeling called UV-position maps [28 30
41
45
46 17 [8].
To formulate the 3D garment deformation, some works 45
introduce the UV-position maps for enhancing and producing realistic and detailed garment surface deformation from coarse mesh parameterization. Other works 28 30 use UV-position maps for encoding 3D garment with different shapes and topologies and utilize the auto-encoder structure to learn garment representation or generation. However, existing methods use either fixed pre-defined UV maps or rely on human body mesh parameterization. In contrast, our NSM makes flexible use of UV-position maps with the help of sewing patterns. We observe that the sewing pattern one solution the UV mapping process with low distortion garment is simulated by sewing the panels garment surface. Hence, we simultaneously model the 2D panel shapes and garment 3D shapes on UV-position maps, allowing NSM represent garments with diverse shapes and topologies.
46
Neural Sewing Machines
In this work, we study the problem of modeling 3D garments with diverse shapes and topologies.
Previous methods (30 24 32 21 were constrained model garments under specific categories or with relative simple topologies, and failed to learn reconstructable and manipulable representations. To this end, we propose Neural Sewing Machine (NSM), learning framework for structure- preservng 3D garment modeling capable of learning representations for garments with diverse shapes and topologies, as shown in Fig. 4 As sewing patterns accurately describe the intrinsic structure and the topology of the 3D garment, we first use unified garment encoding module to obtain an embedding from the sewing pattern. Then, we use 3D garment decoding module to decode sewing pattern embeddings to 3D garment shape using the representation of UV-position maps with masks [0). To allow 3D structure- prepturviireeeiiigggg. learning, we introduce inner-panel loss, an inter-panel loss, and nurface-normal loss. In the following, will demonstrate the components of NSM in detail.

---

3.1 Unified Sewing Pattern Encoding
We first introduce the definition of the sewing pattern, then we demonstrate how to encode the sewing patterns from different garment categories into a shared latent space.
Previously, [S] defined sewing pattern by set of 2D panels along with the stitch information about how the panels are stitched together. For example, in Fig. [3 sewing pattern of skirt had 4 panels named top front, top back, dress front and dress back. Each panel corresponded to part of the 3D garment, and the correspondences were featured by the colors. The stitch information of how panels were stitched together on the 3D garment shape was featured by the arrowed dashed lines between panel edges.
To obtain unified sew pattern embeddings for generic garments with different categories and shapes, we extend the definition in [15] to a hierarchical graph (see Fig. 5). Formally, a garment is defined by sewing patter,, and a sewing pattern is defined by a graph G (V, E), where the graph node V represents the set of all seamed edges of all panels in a garment; the graph adjacencies E represents the stitch information of how panels were stitched together. Then, the previous definition of a sewing pattern can be rewritten by G = (P(V), E), where P(V) is a panel containing several seamed edges. However, using (V, E) or (P, E) describe garment could be too specific and will overfit garment, which is unsuitable for describing generic garments and for getting generic sewing pattern embeddings. To obtain generic sewing pattern embeddings, we introduce basic panel groups. As shown in Fig. 5 a basic panel group B(V) or B is a combination of several panels, and thus a garment (i.e., a sewing pattern) can be represented as a combination of the basic panel groups BV). For example, the skirt in Fig. contains two basic panel groups; the three garments in the left concern of Fig. (each column is a garment) contain three, two, two basic panel groups, respectively. Finally, a sewing pattern is rewritten by G = (B(V), Note that the basic panel groups are statistically collected from all garment categories in the sewing patterns dataset [5).
With the definition of sewing patterns in terms of basic panel groups, we are able to compute unified sewing pattern embeddings for a garment. First, we discrete each seamed edge into m points, with each point repre- sented by 2D coordinate, and for a given basic panel group B, having seamed edges, we have a raw repre- x2, Second, assume that there are
B
sentation B; E [ total of N data containing this basic panel group B, in the whole dataset; then, we compute the mean shape B, E Rlxmx2 for these N samples. Third, we compute a PCA subspace for these N samples after subtracting the mean shape B, and we obtain the PCA coefficients "f" E R (1 < S N) for each sample and PCA components A; E Rlxxxxxx, where h is the num- ber of PCA components. Fourth, an embedding 7 of a sewing pattern is obtained by concatenating all the PCA coefficients % of the basic panel groups (leaving blank if a garment sewing pattern does not contain some basic panel groups). Fig presents a detailed illustration of these processes.
Figure 5:
Definition of sewing pattern in terms of basic panel groups
3.2 3D Garment Decoding
Given the embedding for a sewing pattern sample, we decode the 3D garment shape that is draped on the T-pose human body. To represent 3D garment under diverse categories and encrypt the topologies and shape details of the garments, UV-position maps with mask maps (28 30) are introduced to represent the 3D garment shape, which store the 3D coordinates of the garment at its UV coordinates and the masks indicate the 2D panels shapes of the 3D garment, as shown in Fig. 4 Note that previous methods only use a single map to store 3D garment by registering the 3D garment onto human body, which has limitation for representing loose garment garment with complex topology. In contrast, we introduce multiple maps with each map individually representing one panel of the garment, which is capable of representing garment with arbitrary topology and shape.
Specifically, we use a CNNd network D to predict the UV-position maps for panels [= from the garment representation 7, where Y E IR MxW x3 is the UV-position map for the panel and T is the panel number. The network D paraterized by θD uses the Decoder architecture in 27]
{T')T
D(y; 0D).
5

---

Kllire Kler
Kllinn Mileer Rilaar
May
Wleer
Mihien
Rhies Mlare Ranee
Figure 6: Illustration of the losses in structure preserving learning. Cean is benefit for learning reasonable inner panel deformation. Ciua helps to preserve the inter- aiterppanel topology structure. Loor helps preserve local structure on the garment.
The mask maps introduced represent the detailed 2D panel shape of the garment. Instead of predicting the mask maps by a neural network, which tends to miss the small parts of the masks. We use the inverse PCA transform PCA-1 decode the mask maps (M') Jt=1 from the garment representation 7,, where M E (0, 10.1) M indicate the 2D panel shape:
(M PCA (Y)
The 3D garment mesh can be readout from the UV-position maps {Y')7, and the mask maps [M")== 3D space. The garment readout process provided in the Appendix.
3.3 Structure Preserving Learning
It is not trivial to learn the structured deformable 3D garment with diverse shapes and topologies from the embedding that is encoded from the 2D sewing patterns. To preserve the intrinsic structure of the predicted 3D garment, except from the 3D reconstruction loss, we introduce an inner-pane structure- presurrvngg loss, an inter- inter-panel structure- structure-preserving preserving loss, and a surface-norma loss the learning process of our NSM. In the following, we introduce the training losses in our NSM.
3D Reconstruction Loss: We optimize the predicted UV-position maps {Y*)²= using the 3D reconstruction loss free as following, where ÿ is the ground truth UV-position map and M' the ground truth mask map for the panel. The loss Lree constrains the 3D coordinates of the prediction and the ground truth should be the same at the corresponding UV-coordinates:
Lrer Σ
Elrr
M pe
Inner-panel Structure Preserving Loss: In practice, the neural networks tend to learn the low frequency signal (24] and garment usually has folds on its surface. The 3D Reconstruction loss solely cannot guarantee the network D) to learn an isometric mapping between the UV-coordinates and the 3D coordinates, as shown Fig. [ The garment surface flat as the panels are deformed with wrong scale. Hence, we introduce the inner mener structure preserving loss Linn to learn an isometric mapping, which helps to preserve the intrinsic structure within each deformed panel, where is constant that indicates the scaling between the 3D coordinate and the UV coordinate. The loss Livn constrains that the length of a curve on a panel should be consistent no matter how the panel deformed in 3D space:
H-IW-1
Linn ΣΣΣ
||xts Y. 11, + ||x2s Y1
(4)
The above losses only model the inner-panel geometry and topology. As garment consists different panels that are stitched together following the stitch information, we use the inter-panel structure preserving loss Lint encode this inter-panel topology knowledge. As shwon Fig. 6 when the loss List is removed, there exists a gap between two edges that should be stitched together. Specifically, given two stitched edges e and e the panels
Inter-panel Structure Preserving Loss:

---

pl and P'2 using the stitch information E, illustrated in Figure 5 we use bilinear interpolation to obtain their 3D coordinates on their UV-position maps Y and Y where of m= and
ei Im=x denote the UV coordinates of m points on the two edges. The loss Lint demonstrates that two stitched edges are point-wise stitched together:
Lint
Σ #1
a(e rh) (( y ||,
(5)
To further approximate the deformation on the 3D garment surface, we introduce the surface norm loss Loor as following. As shown in Fig. [ the loss Loor helps to preserve local structure on the garment surface. The normal map N is first calculated from the predicted UV-position map Y by finding the normal vector that is perpendicular to the tangential plane of each 3D point from the UV-position map Y'. Then we calculate the cosin loss for each point on the 3D garment surface as following with ground truth normal map Ñ' and the mask map Mr
Surface Normal Loss:
OY du
0Y du
0Y dv
N
x
/(1l
||||
II).
(6)
du
T W
Enor -E
cos(NN * Mi Mig)
The total training loss & in the learning process of NSM is given as following:
L = Oreecrec + ainnfinn + Cint + Unorrron
(8)
4 Experiments
Dataset. Here we use the largest 3D garment dataset with sewing patterns introduced in 15 It covers a variety of garment designs, including variations of t-shirts, jackets, dresses, skirts, jumpsuits and pants. Compared with the previous 3D garment datasets [32 4] this dataset contains more diverse 3D garment shapes and topologies. We use about 22400 samples from 12 base categories in the experiments; each sample contains sewing pattern, 3D garment mesh draped on T-pose SMPL (0 and a rendered image of the 3D garment. We use the first 90% part of the data for each base type as the training set and the remaining as the test set.
Evaluation Metrics. To evaluate the quality of 3D garment reconstruction, following 47 38 we calculate the Chamfer distance (Chamfer) and the average point-to-surface Euclidean distance (P2S) between the predicted and ground truth 3D point clouds. Furthermore, evaluate the quality of the topology structure of the 3D garment, we introduce the mean geodesic length error (MGLE) to measure the topology structure difference between two 3D meshes. Given two points 2; and æy on the ground truth 3D mesh surface, we find two points nf and xy on the predicted 3D mesh surface that are closet points x; and æ; respectively. The geodesic length error is calculated as follows:
K-1
K
MGLE K(K-1)
Σ
Σ 1=411
|9( 2,, -9(06, 2/)]1;
where y is the geodesic length between two points on the mesh computed using the algorithm in [9]. For each garment, as the calculation of geodesic length is relatively slow, we sample K = 20 points and calculate MGLE between every two points. The sensitivity analysis of K is provided in the Appendix.
Implementation Details. The weights {Aree: Oinnn, Giint, anom } in the training loss L are set to {1, 10-3, 3, 10 1001000-00 2)-2). The number of PCA components h is set to 12, which guarantees the sewing pattern shape is reconstructed with small errors. The basic panel group number are set to 10 and the total panel number is set to 33. The scaling between the 3D coordinate and the UV coordinate is set to 1.5. The width W and height H of the UV-position maps and the mask maps are set as 128. Our model is trained with a GTX 2080 GPU with the learning rate as -e and the batch size as 8 for 40 epochs.
7

---

4.1 Comparison of Single-view Reconstruction
To validate the generalization ability of our framework, conduct experiments single-view 3D shape reconstruction. We first train NSM on the training set with paired sewing patterns and the 3D garment annotations. Then we train encoder for mapping the input image to the sewing pattern embedding y using a CNN-based architecture [27] as the backbone of the encoder. More
details about training process are provided in Appendix. 38 AnchorUDF 47 and BCNet [13
The results are shown in Table [ We compare our NSM with three state-of- the-art single-view reconstruction meth- ods, i.e., Pixel2Mesh Pixel2Mesh rep- resents 3D objects as deformable 3D meshes, and AnchorUDF uses Unsigned Distance Function (UDF) to represent 3D garments, and BCNet uses template mesh to represent garments for each cate- gory. Our NSM achieves a 2.08cm error for chamfer, a 1. 1.90cm error for P2S, and a 3. 3.73cm error for MGLE, achieving better performance than all methods for Chamfer and P2S, achieving better performance than Pixel2Mesh and AnchorUDF for MGLE. Note that BCNet uses template meshes with predefined topology and our method achieves compatible performance with BCNet for MGLE.
Table Chamfer, P2S errors and MGLE (em) for different single-view reconstruction methods on the 3D garment dataset.
Methods
Chamfer
PZS 4
MGLE
Pixel2Mesh [38]
5.23
3.84
10.81
AnchorUDF [47
3.31
4.17
4.61
BCNet [3
4.69
4.33
3.42
NSM
2.08
1.90
3.73
Visually, as the three competitors implicitly model 3D garment shapes from compressed features or use registered template mesh, they may have limitations for recovering detailed garment structures, as shown in Fig. 7 In contrast, our NSM explicitly models the garment structure by introducing the sewing patterns, which helps to preserve the detailed garment structures. Besides, these three methods either use topology templates or recover the topology via post-process, which brings errors in modeling complicatedly-sttin -tttutcuued garments (e.g.. wrinkled clothes and fine structures), as shown in Fig. 7 Differently, our NSM can recover detailed garment topology structures thanks to the learnable UV-position maps with masks.
4.2 Reconstruction from Sewing Pattern
To validate the garment representation ability of our NSM, we evaluate NSM for reconstructing 3D garments from sewing patterns. We train NSM the training set with sewing patterns and 3D garment annotations. In the testing, given a sewing pattern, we first obtain the embedding by PCA transform using the PCA components pre-computed at the training phase. Then, the UV-position maps with masks (= 3D garments) are predicted from the embedding using the 3D garment decoder.
We report the experimental results in Table 2 the Chamfer distance error is 1.65cm, the P2S error is 1.46cm, and the MGLE error is 3.54cm. Fig. [ presents the qual- itative visualization, showing that framework can effectively rep- resent 3D garments from different categories with diverse shapes and topologies. More comparisons are provided in the Appendix.
Table 2: Chamfer, P2S errors and MGLE (cm) obtained by our framework for 3D garment reconstruction from sewing pattern on the 3D garment dataset.
NSM
Lshin
jacket
dress
skirt
jumpsuit
punts
All
Chamler &
148
2.17
1.77
1.61
0.06
1.53
1.65
P2S 4
1.77
208
1.27
1.19
0.73
2.13
1.46
MGLE
3.13
4.13
4.26
3,46
2.32
2.65
3.54
4.3 Controllable Garment Editing
We demonstrate our framework is capable of controllable garment editing. Given source 3D garment and the predicted sewing pattern (obtained using a PointNet (26) trained on paired 3D garments and sewing pattern embeddings), we can accurately edit the 2D panel shapes or classes, from which our NSM recovers the corresponding edited 3D garment. Previous methods [2 30 model and edit 3D garments via the implicit neural representation, having difficulty with controllable garment editing. In contrast, sewing pattern embedding is more interpretable, and the structure- preserving 3D garment modeling guarantees that the editing on 2D panels can be accurately reflected 3D garments.
As shown in Fig. NSM supports freely editing on the 3D garment shape and topology with preserved intrinsic structure. And NSM can edit garments with significant shape variations or transfer
8

---

PixelZMesh
AnchorUDF
BCNet
NSM
Ground truth
Figure 7: Visual comparison of the learned topology and geometry of different single-view reconstruction methods on the 3D garment dataset.
Source
Source
Target seuinn pattom
balteng
Kaning Saess
Predicted sowing patterns
Source gament
Target 50 germent
Source 30 gernest
Terget sowing pattem
tassop SOTee
Edttig pinol
Prodicted
Editing
Predicted snwing
ponei chopa
Editing pasel shape
Figure 8: Illustration of controllable garment editing. Given source 3D garment and the predicted sewing pattern, we obtain the edited target garment by editing the panel shape or panel class the sewing pattern.
garments from one category to another by editing on the 2D panels, even for some categories that are shown in the training
4.4 Model Analysis
Ablation Study. We conduct ablation studies to provide insights into the effectiveness of each component in our framework, as shown in Table Only using the 3D reconstruction loss free for training, the framework achieves an MGLE eтror of 4.38cm. When the ooner-panel structure-pret prreeervigpseriinggggg. loss Linn is added, the MGLE error increases from 4.38cm 4.69. This might because inner- panel structure-presevvee
Table 3: Quantitative ablation study.
Evec
Linn
Lent
Enor
Chamfer
P2S
MGLB
1.81
1.69
4.38
1.80
1.76
4.69
1.80
1.68
3.62
1.65
1.46
3,54
loss Linn affects the inter-panel topology structure. When the inter-panel
structure- prreturrrppeerviiggg... loss Line is added, the MGLE error drops from 4.69cm to 3.62. When the surface normal loss Cnor is added, changes in Chamfer and P2S eгrоr are not obviously in our ablation study. But as shown in Figure 6 when the losses used for srructure-preservvee garments with correct structures. This indicates that chamfer and P2S errors cannot truly reflect the quality of the topology structure of the 3D object. On the contrary, the MGLE error reflect the topology structure of the 3D object since evaluates the geodesic length error on the 3D object surface.
full model achieves the MGLE error 3.54cm. Note that the
learning removed, the model can not output 3D
Interpolation Results. We also show the results of our NSM for interpolating in the sewing pattern embedding space, shown in Fig. Given the source embedding Y and target embedding "t.

---

0.25
I 0.5
0.75
0
0.25
0.5 0.75
Figure 9: Visualization of the interpolation in the sewing pattern embedding space of NSM. The first and third rows denote the predicted UV-position maps with masks. The second and forth rows denote the predicted 3D garment shapes.
the interpolated embedding is given as % ar, + (1-aa) We predict the UV-positinn maps with masks (= 3D garments) from the interpolated embedding nr As we can see, the changes 2D panels and 3D garment shapes consistent.
Conclusion
We present Neural Sewing Machine (NSM) for structure-preserviing 3D garment modeling. The sewing pattern embedding obtained by the unified sewing pattern encoder. Then the UV-position maps with masks are predicted from the embedding. To preserve the intrinsic structure of the 3D garment, introduce inner-panel structure loss, inter-pancl structure loss, and surface-normi loss. Experiments show that NSM is capable of representing garments with diverse shapes and topologies, reconstructing 3D garments from single images, and editing 3D garments by controlling the panel shape and class.
Acknowledgement. This work was supported in part by National Key R&D Program of China under National Natural Science Foundation of China (NSFC) under Grant
Grant 2e.22000AAA110 No.U19A2073 and No.61976233, Guangdong Province Basic and Applied Basic Research (Regional Joint Fund-Key Key) Grant No. No.2019B1511 1 Guangdong Outstanding Youth Fund (Grant No. 2021B1515020061) the UKRI grant Turing AI Fellowship EP/W002981/1 and the EPSRC/MURI grant EP/N019474/L. We also thank the Royal Academy Engineering.
Limitations and broader impact
The method in this paper aims to build 3D garment models as realistic as possible, which has significant positive effects the real world, such facilitating garment design, virtual try-on, and virtual human design in the metaverse. Currently, we do not model the garment deformation with different human pose. This is a challenge problem for loose garments with diverse shape variations. Our solution is to use "virtual garment bones" 23 or diffused skinning [19]. Another limitation of our work is that our NSM encoder requires the panels to be relatively regular. While some garments may have irregular panels, like the several garments found in South Asia, East Asia and Africa. We will expand this in our future work. We believe that this work does not have any negative impact on society.
References
[1] Salman Ahmed, Lukman Irshad, Onan Demirel, and Irem Y Tumer. comparison between virtual reality and digital human modeling for proactive ergonomic design. International conference on human--computer interaction, pages 3 21. Springer, 2019.
[2] Thiemo Alldieck, Marcus Magnor, Bharat Lal Bhatnagar, Christian Theobalt, and Gerard Pons-Mo Moll. Learning to reconstruct people in clothing from single rgb camera. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1175-1 1186, 2019.
10

---

[3] Aric Bartle, Alla Sheffer. Vladimir G Kim. Danny M Kaufman, Nicholas Vining, and Floraine Berthouzoz. Physics-driven pattern adjustment for direct 3d garment editing. ACM Trans. Gruph., 35(4):50-1, 2016.
[4] Hugo Bertiche, Meysam Madadi, and Sergio Escalera. Cloth3d: clothed 3d humans. In European Conference on Computer Vision, pages 344-359. Springer, 2020.
[5] Bharat Lal Bhatnagar, Cristian Sminchisescu, Christian Theobalt, and Gerard Pons-Moll. Combining implicit function learning and parametric models for human reconstruction. In European Conference on Computer Vision, pages 311 1329.. Springer, 2020.
[6] Bharat Lal Bhatnagar, Garvita Tiwari, Christian Theobalt, and Gerard Pons-Moll. Multi-garmen net: Learning dress 3d people from images. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5420-5430, 2019.
[7] Xiaowu Chen, Bin Zhou, Fei-Xiang Lu, Lin Wang, Lang Bi, and Ping Tan, Garment modeling with a depth camera. ACM Trans. Graph., 34(6):203-1, 2015.
[8] Enric Corona, Albert Pumarola, Guillem Alenya, Gerard Pons-Moll, and Francesc Moreno- Noguer. Sm- plicit: Topology-aware generative model for clothed people. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11875 1855-11855 11885, 2021.
[9] Keenan Crane, Clarisse Weischedel, and Max Wardetzky. The heat method for distance computation. Communicattons of the ACM, 60(1 (0(19))0--99 2017.
[10] Philippe Decaudin, Dan Julius, Jamie Wither, Laurence Boissieux, Alla Sheffer, and Marie-Paule Cani. Virtual garments: A fully geometric approach for clothing design. In Computer Graphics Forum, volume 25, pages 625 634. Wiley Online Library, 2006.
[11] Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C. Russell, and Mathieu Aubry. A papier-máché approach to learning 3d surface generation. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 216-224, 2018.
[12] Fangzhou Hong. Liang Pan, Zhongang Cai, and Ziwei Liu. Garment4d: Garment reconstruction from point cloud sequences. Advances in Neural Information Processing Systems, 34:27940-27951, 2021.
[13] Boyi Jiang. Juyong Zhang, Yang Hong. Jinhao Luo, Ligang Liu, and Hujun Bao. Benet: Learning body and cloth shape from a single image. In European Conference on Computer Vision, pages 18-35. Springer, 2020.
[14] Ladislav Kavan, Dan Gerszewski, Adam W Bargteil, and Peter-Pike Sloan. Physics-inspired upsampling for cloth simulation in games. In ACM SIGGRAPH 20/ papers, pages 1-10. 2011.
[15] Maria Korosteleva and Sung-Hes Hee Lee. Generating datasets of 3d garments with sewing patterns. arXiv preprint 109.0069905533. 2021.
[16] Maria Korosteleva and Sung-Hee Lee. Neuraltailor: Reconstructing sewing pattern structures from 3d point clouds of garments. arXiv preprint arXiv:2201. 2022.
[17] Zorah Lahner, Daniel Cremers, and Tony Tung. Deepwrinkles: Accurate and realistic clothing modeling. In Proceedings of the European Conference on Computer Vision (ECCV), pages 667-684, 2018.
[18] Verica Lazova, Eldar Insafutdinov, and Gerard Pons-Moll. 360-degree textures of people in clothing from a single image. In 2019 International Conference on 3D Vision (3DV). pages 643-653. IEEE, 2019.
[19] Siyou Lin, Hongwen Zhang, Zerong Zheng, Ruizhi Shao, and Yebin Liu. Learning implicit templates for point-based clothed human modeling. arXiv preprint arXiv:2207.06040 2022.
[20] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael Black. Smpl: A skinned multi-person linear model. ACM Irunsactions graphics (TOG), 34(6): 1-1,, 2015.
[21] Qianli Ma, Jinlong Yang, Anurag Ranjan, Sergi Pujades, Gerard Pons-Moll, Siyu Tang, and Michael J Black, Learning to dress 3d people generative clothing. In Proceedings of the IÉEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4478, 2020.
[22] Eder Miguel, Derek Bradley, Bernhard Thomaszewski, Bernd Bickel, Wojeiech Matusik, Miguel A Otaduy, and Steve Marschner. Data-driven estimation of cloth simulation models. Computer Graphics Forum, volume 31, pages 519-528. Wilcy Online Library, 2012.
[23] Xiaoyu Pan, Jiaming Mai, Xinwei Jiang, Dongxue Tang, Jingxiang Li, Tianjia Shao, Kun Zhou, Xiaogang Jin, and Dinesh Manocha. Predicting loose-fitting garment deformations using bone-driven motion networks. In Special Interest Group on Computer Graphics and Interactive Techniques Conference Proceedings, pages 1-10, 2022.
[24] Chaitanya Patel, Zhouyingcheng Liao, and Gerard Pons-Moll. Tailoret: Predicting clothing in 3d as function of human pose, shape and garment style. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7365-7375, 2020.
[25] Xavier Provot et al. Deformation constraints in a mass-spring model to describe rigid cloth behaviour. In Graphics interface, pages 147-147. Canadian Information Processing Society, 1995.
11

---

[26] Charles R Qi. Hao Su. Kaichun Mo. and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660. 2017.
[27] Olaf Ronneberger, Philipp Fischer, and Thomas Brox, U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer asssisee intervention, pages 234 241. Springer, 2015.
[28] Shen, Junbang Liang, and Ming C Lin. Gan-based garment generation using sewing pattern images. In European Conference on Computer Vision, pages 225-247. Springer, 2020.
[29] Carsten Stoll, Juergen Gall, Edilson De Aguiar, Schastian Thrun, and Christian Theobalt. Video-based reconstruction of animatable human characters. ACM Transactions on Graphics (TOG), 29(6):1-10, 2010.
[30] Zhaoqi Su, Tao Yu, Yangang Wang, and Yebin Liu. Deepeloth: Neural garment representation for shape and style editing. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[31] Demetri Terzopoulos, John Platt, Alan Barr, and Kurt Fleischer. Elastically deformable models. In Proceedings of the 14th annual conference on Computer graphics and interactive techniques, pages 205-214, 1987.
[32] Garvita Tiwari, Bharat Lal Bhatnagar, Tony Tung, and Gerard Pons-Moll. Sizer: A dataset and model for parsing 3d clothing and learning size sensitive 3d clothing. In European Conference on Computer Vision, pages 1-18. Springer. 2020.
[33] Lokender Tiwari and Brojeshwar Bhowmick. Deepdraper: Fast and accurate 3d garment draping over a 3d human body. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1416-1426, 2021.
[34] Raquel Vidaurre, Igor Santesteban, Elena Garces, and Dan Casas. Fully convolutional graph neural networks for parametric virtual try-on. In Computer Graphics Forum, volume 39, pages 145-156. Wiley Online Library, 2020.
[35] Pascal Volino and Nadia Magnenat- Magnenat-Thalmamn. Accurate garment prototyping and simulation. Computer- Aided Design and Applications, 265):645-654, 2005.
[36] Huamin Wang, Florian Hecht, Ravi Ramamoorthi, and James F O' Brien.. Example-based wrinkle synthesis for clothing animation. In ACM SIGGRAPH 2010 papers, pages 1-8. 2010.
[37] Huamin Wang. James F O'Brien, and Ravi Ramamoorthi. Data-driven elastic models for eloth: modeling and measurement. ACM transactions on graphics (TOG), 30(4):1-12, 2011.
[38] Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In Proceedings the European conference on computer vision (ECCV), pages 52-67, 2018.
[39] Tuanfeng Y Wang, Duygu Ceylan, Jovan Popovic, and Niloy J Mitra. Learning shared shape space for multimodal garment design. arXiv preprint arXiv: 11335, 2018.
[40] Donglai Xiang. Fabian Prada, Chenglei Wu, and Jessica Hodgins. Monoclothcap: Towards temporally coherent clothing capture from monocular rgb video. In 2020 International Conference on 3D Vision (3DV), pages 322-332. IEFF, 2020.
[41] Yi Xu, Shanglin Yang, Wei Sun, Li Tan, Kefeng Li, and Hui Zhou. 3d virtual garment modeling from rgh images. In 2019 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pages 37-45. IEEE, 2019.
[42] Shan Yang. Tanya Ambert, Zherong Pan, Ke Wang, Licheng Yu, Tamara Berg, and Ming C Lin. Detailed garment recovery from a single-view image. arXiv preprint (ok.:v6600..225500 2016.
[43] Shan Yang. Zherong Pan, Tanya Amert, Ke Wang, Licheng Yu, Tamara Berg, and Ming C Lin. Physics- inspired garment recovery from a single-view image. ACM Transactions on Graphics (TOG), 175:111-14, 2018.
[44] Dongliang Zhang, Yongjin Liu, Jin Wang, and Jituo Li. An integrated method of 3d garment design. The journal of the Textile Institute, 109(12):159--160) 2018.
[45] Meng Zhang, Tuanfeng Wang, Duygu Ceylan, and Niloy J Mitra. Deep detail enhancement for any garment. In Computer Graphics Forum, volume 40, pages 399-411. Wiley Online Library, 2021.
[46] Meng Zhang, Tuanfeng Y Wang, Duygu Ceylan, and Niloy J Mitra. Dynamic neural garments. ACM Transactions on Graphics (TOG), 40(6): 15, 2021.
[47] Fang Zhao, Wenhao Wang. Shengcai Liao, and Ling Shao. Learning anchored unsigned distance functions with gradient direction alignment for single-view garment reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12674 12683, 2021.
12

---

amount spent on [48] Fuwei Zhao, Zhenyu Xie. Michael Kampffimeyer, Haoye Dong, Songfang Han, Tianxiang Zheng, Tao Zhang, and Xiaodan Liang. M3d-vton: A monocular-to-3d virtual try-on network. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13239-13249, 2021.
[49] Heming Zhu, Yu Cao, Hang Jin, Weikai Chen, Dong Du, Zhangye Wang. Shuguang Cui, and Xiaoguang Han. Deep fashion3d: A dataset and benchmark for 3d garment reconstruction from single images. In European Conference on Computer Vision, pages 512-530. Springer, 2020.
Checklist
1. For all authors...
(a) Do the main claims made the abstract and introduction accurately reflect the paper's 's contribu- tions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) you state full set of assumptions of theoretical results? [N/A]
(h) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in supplemental material as URL)? [Yes] supplemental.
NSM
(b) Did you specify the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e.g.. with respect to the random seed after running experiments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Our experiments are conducted on Ubuntu System, Intel(R) Core(TM) i5 8408 CPU(2.00HH2). GTX 2080.
4. If you are using existing assets (e.g.. code, data, models) or curating/releasing new assets...
(a) your work existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No] The data used in work are publicly released.
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 'r us-
ing/curating? [Yes] The data used in our work are publicly released.
(c) Did you discuss whether the data you are using/curating contains personally identifiable in- formation or offensive content? [Cs]] The data used in work does not contain personally identifiable information offensive content.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable?
[N/A]
(b) you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants participant compensation? [N/A]
13


Reviews:
[{"review_id": "z2YxgGZpZT8", "reviewer": "Reviewer_HS7n", "summary": "This paper looks into the problem of 3D garment modeling and reconstruction which is a challenging problem within the wider area of 3D generative modeling and reconstruction. The problem is especially challenging because it is unclear as to what constitutes a good representation for 3D clothes that captures the wide variety of garments we see around us. This paper proposes to use sewing patterns as a representation of 3D garments and proposes a learning based framework - Neural Sewing Machine - capable of generating garments of varied topology. The authors discuss challenges and propose solutions to both encoding and decoding using this representation. They additionally propose three kinds of losses – inner-panel structure preserving loss, inter-panel structure preserving loss and surface normal loss. The authors claim that their method outperforms the prior state of the art. \n", "questions": "1. L59 : “Our NSM guarantees …” – I don’t understand how this statement is true? It’s not that the 2D UV map is an isometric parameterization of any kind. Maybe I am missing something?\n\n2. The authors make comments about physics-based simulators in several places and believe that their method is superior to this line of work. I have absolutely no clue how using physically based simulators for interacting with the clothes is a “contemporary line of work” to the proposed method, as the two are completely different items? What’s the context or argument here? Also, the authors claim that their method creates garments which can be manipulated but nowhere do they qualify or provide any evidence for this one. \n\n3. In the dataset, the authors say L245 - “ We use the first 90% part of the data for each base type … and remaining as the test set”. Where is the validation set? If the authors fine tuned their model on the test set then this basically throws all of experimental evaluation into jeopardy. \n\n4. I also didn’t understand the section on controllable garment editing. It looks interesting but I could not parse through the procedure involved in getting that. \n\nI firmly believe that the idea has a lot of potential, but I am not convinced with the experimental evaluation provided here. Unless the authors can address the concerns (primarily with comparisons to relevant methods) I don't think there is any merit to accepting paper. This is simply because the main hypothesis of this paper is that using sewing patterns as a representation is inherently superior to other kinds of representation. And in the absence of state of the art results on relevant comparisons, I don't see how else the authors would validate this hypothesis.  ", "limitations": "I am surprised that the authors didn't talk about obvious limitations of their work — both technical and in terms of its social impact. \n\nDue to the choice of their representation, I doubt that their method generalizes to other clothing styles and the authors neither experiment with the generalization nor provide any comments on it. This to me seems like a severe limitation of the proposed work. However, I don't think this is any reason to reject the paper or the idea as this is not the main hypothesis/claim of the paper. \n\nThe dataset used in this paper contains clothing styles which are not very inclusive. In particular, there are several garments found in South Asia, East Asia and Africa which would be hard to be captured by the PCA basis used in the paper — which primarily contains popular garments worn in the western countries. I believe that the lack of inclusivity of garments found in other (and often underrepresented) regions of the world fundamentally limits this work and moreover prevents even a discussion about the needs of such underrepresented regions and cultures as it relates to automatic 3D garment representation and generation. ", "rating": 6, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\nIn my evaluation, the proposal of this paper is very good. Indeed, representation of 3D garments is a challenge and this paper provides an interesting sewing pattern based representation. The authors also do a great job of highlighting various design choices involved in both encoding and decoding this representation and thus building their proposed – Neural Sewing Machine. In particular, I believe the following are the set of strengths of this paper: \n\n1. Unified Sewing Pattern Encoding: I like this encoding scheme as it allows for representing clothes of diverse categories, shapes and topologies. The fact that the authors chose PCA to come up with the encoding is again very interesting – something well suited for this problem setting as there is only so much variety in clothing. \n\n2. 3D Garment Decoding: The authors provide a principled approach to decode the encoding by learning a 2D UV map and a mask map which together represent the various panels. \n\n3. Structure Preserving losses: The proposed losses are intuitive and crucial for getting good results. The ablation that the authors provided validates their importance. \n\n4. Mean Geodesic Length Error: I particularly liked this metric to evaluate the quality of topology and the authors show that their method gives good results on this metric. \n\nBefore I mention the weaknesses of the paper, I want to emphasize that the authors have presented an interesting method to model and generate 3D garments and I personally feel that the idea has a lot of potential. However, I think that authors have left out several important experiments and their analysis does not adequately back the claims they are making. Moreover, I don’t find their choice of baselines very convincing. I also find the paper is hard to read in several places with figures missing captions/ not being clear enough. \n\n1. I think the authors should do a better job at describing the related work. They mention them all in a single line L112, L115 but that’s pretty much what the reader gets. At the present moment, it is hard to properly contextualize the contribution of this paper with respect to existing literature. \n\n2. The authors show comparison with general purpose and ‘freeform’ baselines such as pixel2mesh and anchorUDF which is good to know but by no means is a fair comparison in my opinion. Such weak baselines tell little about the efficacy of the proposed method. The authors dedicate half a page in the related work on template-based models and garment structure modeling – which clearly shows that there are plenty of closely related recent papers (with open source code) that specifically target the garment generation problem. However, the authors don’t provide any comparisons for these. Therefore I am not convinced that this paper is really state of the art (not that it matters). \n\n3. I am also curious to see an ablation for the UV position maps as an intermediary representation. Particularly, how well does it compare against those approaches that used an alternative image based representation. \n\n4. I think the paper would benefit if authors spend some time in making the figures more explanatory. For example, in Fig 6 there are no captions as to which one is the prediction and which is ground truth. Moreover, I qualitatively cannot make sense what are the artifacts manifested by each loss term ablation. Perhaps an inset focusing on the artifact will help. \n\n5. Novelty on surface normal loss – I think authors should not claim novelty on this one as it is pretty much well known in the area of 3D reconstruction. \n\n6. While the authors introduce a new metric – Mean Geodesic Length Error (MGLE) to measure the quality of topology, I believe that the authors should have provided more analysis for the behavior of this metric such as the effect of the number of samples K. This is crucial because this metric seems to be important to show the superiority of the proposed method and I suspect that for different values of the sample K, the results will be very different. I hope to see results with a much larger value of K (on the order of a few 1000s atleast). \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "s32zVx52JP7", "reviewer": "Reviewer_1Ro2", "summary": "This work focuses on modeling 3D garments with a novel neural sewing machine (NSM).  The NSM is a learning framework that can effectively encode garment representations, which could facilitate 3D garment construction and manipulation.  Firstly, it utilizes a unified sewing pattern encoding module to embed sewing patterns.  Then, the embeddings are decoded into a 3D garment by using the UV-position maps with masks.  A few loss terms are introduced to preserve the inner-panel structures of the 3D garments.  Extensive experiments validate its superiorities over previous SoTA methods.", "questions": "Generally, I think it is a nice paper, and maybe the authors can provide more real-captured images as I mentioned in the ``Weakness''.", "limitations": "The manuscript has a section to discuss the limitations and broader impact.  But the method limitations have not been properly discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Using sewing patterns for 3D garment modeling is a good idea and also a nice contribution, which supports both 3D garment reconstruction and controllable editing.\n2. Extensive experiments show that the proposed method outperforms previous SoTA methods quantitatively and qualitatively.\n\nWeakness:\n1. This manuscript claims that previous methods failed to learn reconstructable and manipulable representations with complicated topology.  Maybe a few more examples with complicated or straight shapes can be provided.  \n\n2. Also, all the experiments are conducted on synthetic data, and it would be nice if real-captured or even in-the-wild images can be used for evaluation.\n\n3. It seems like this work can only reconstruct the garments with the canonical pose, and many local details (e.g. wrinkle) are missing even if they can be observed in the input image.  And the possible penetrations between the reconstructed 3D garments and the 3D human body are not considered during reconstruction and manipulation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Ku8DI5Frk6", "reviewer": "Reviewer_RjUE", "summary": "This paper introduces a network model to reconstruct 3D garments based on sewing pattern input. The sewing patterns are first gone through PCA for coefficients to compose the embedding. Later, masks and uv position maps are generated using the coefficients. The pixels from the masked maps are used to reconstruct the 3D mesh. The paper introduces new loss functions to regularize the model behavior using deformation and sewing edge alignment rules, as well as normal consistentcy. Results show that the proposed method works better in a single-view garment reconstruction task and is capable of controllable editing and interpolation.", "questions": " See above.", "limitations": " See above.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n - The paper is the first one that I know of that addresses the importance of modeling garments in the context of sewing patterns, which is the key to develop a physically correct as well as easy-to-use garment generation models.\n\nWeakness:\n - The method does not model tightness of the cloth. What it really does is merely a *garment template mesh* generation model for a given sewing pattern shape. While it is not a critical component, the authors should discuss it either in future work or in the limitation section.\n - It is still not clear how the position maps compose the final 3D mesh. Although there are several sentences in the caption of the figure in the appendix that mentions it, it is not in sufficient detail. How is the mesh triangulated around the sewing pattern boundary where the edges are defined in lines but the inner vertices are defined as grid points? What does the algorithm do when the 3D coordinates of the sewing edges from two patterns are different? Is there any post-process to smooth out the neighboring vertex locations caused by the difference? If not, would that generate unnatural artifacts? There are a lot of critical details on this topic but the authors did not discuss them in detail.\n - The fabric material is not modeled. This is not a necessary component but should be mentioned in the future work section.\n - It is not clear how the basic panel groups are defined and it is not shown that what has been included would be sufficient to support *generic* garments. There seems to be only 9 basic panel groups. How are they chosen? Which factors are considered? Which garments can be supported by the 9 groups and which cannot? For example, can the model support T-shirts with pockets or dresses with lace/ribbon decorations using the 9 groups? It is a critical component of the paper but it is not discussed in depth.\n - It seems in Fig. 8 that the garment editing for the panel class is manual. It doesn't actually help too much because the user will have to specify the new pattern shape themselves which could be tedious.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DmpNUG2NkR", "reviewer": "Reviewer_WXEe", "summary": "This paper proposes the Neural Sewing Machine (NSM) for structure-preserving 3D garment modeling. Previous methods mostly model garments in a class-specific way, which is not flexible and easily extendable. In this work, each garment is modeled as several basic panels and their stitching relations. The basic panels are encoded by PCA from a large-scale dataset. The stitching relations as well as the 3D structure of each panel are encoded in the UV-position maps. For each image input, NSM predict PCA coefficients for each basic panel that corresponds to the observed garment. Based on the predicted PCA coefficient, a CNN-based network is trained to predict the UV-position maps. Four training targets are introduced to facilitate the supervised training. The reconstruction results are the best compared with previous baselines. Utilizing the structure-preserving modeling, controllable garment editing and interpolation are also shown to demonstrate the versatility of this kind of garment modeling.", "questions": "1. Section 4.2, what is the input for the reconstruction from sewing pattern?\n2. Why inner-panel structure preserving loss learns an isometric mapping between UV and 3D coordinates?\n3. For the controllable garment editing, can we draw the panel shape by hand?", "limitations": "The limitation is not discussed. Please discuss more on the limitation. Please see the above 'Weakness' section for my comments on the proposed garment modeling.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n1. Considering the sewing pattern into the garment modeling process and unifying modeling for basic garment classes are important contributions towards practical real-world garment modeling.\n2. Both qualitative and quantitative results surpasses previous SOTA methods.\n3. Ablation study proves the effectiveness of the proposed training targets.\n4. The application of controllable garment editing is interesting.\n\n## Weaknesses\n1. The experiments are all done on a synthetic dataset, which might not be enough to fully prove its practicality when facing real-world data. For example, given a real photo of a clothed human, can NSM still give correct reconstructions on the clothes?\n2. It seems like the sewing pattern/ topology of several basic panels are solely represented by the predicted UV-position maps. It might lead to two problems. a) For two corresponding edges, the predicted UV-position are not guaranteed to be perfectly aligned, which would give broken geometry when stitching panels. b) For problem a, a solution would be to find correspondences and force them to align spatially. However, finding such correspondences is not trivial under the current modeling. The edge points in the predicted UV-position map are not necessarily spatially ordinal.\n3. It seems like NSM is only capable of reconstruct clothes in the canonical space. Can NSM takes photos of a posed clothed human as input? Can NSM output posed clothes as observed in the input image? A simple synthetic dataset for this would be Cloth3D.\n\n## Typo\nTable 3, headers, four L_{rec}.\n\n## Missing References\n[1] Xiang, D., Prada, F., Wu, C., & Hodgins, J. (2020, November). Monoclothcap: Towards temporally coherent clothing capture from monocular rgb video. In 2020 International Conference on 3D Vision (3DV) (pp. 322-332). IEEE.\n\n[2] Hong, F., Pan, L., Cai, Z., & Liu, Z. (2021). Garment4D: Garment reconstruction from point cloud sequences. Advances in Neural Information Processing Systems, 34, 27940-27951.\n\n[3] Bhatnagar, B. L., Sminchisescu, C., Theobalt, C., & Pons-Moll, G. (2020, August). Combining implicit function learning and parametric models for 3d human reconstruction. In European Conference on Computer Vision (pp. 311-329). Springer, Cham.\n\n[4] Alldieck, T., Magnor, M., Bhatnagar, B. L., Theobalt, C., & Pons-Moll, G. (2019). Learning to reconstruct people in clothing from a single RGB camera. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 1175-1186).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept