SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ren Li Benoit Guillard Pascal Fua
Computer Vision Lab, EPFL
ren.li@epfl.ch benoit.guillard@epfl.ch pascal.fua@epfl.ch
Many approaches to draping individual garments on human body models are realistic, fast, and yield outputs that are differentiable with respect to the body shape on which they are draped. However, they are either unable to handle multi-layered clothing, which is prevalent in everyday dress, or restricted to bodies in T-pose. In this paper, we introduce a parametric garment representation model that addresses these limitations. As in models used by clothing designers, each garment consists of individual 2D panels. Their 2D shape is defined by a Signed Distance Function and 3D shape by a 2D to 3D mapping. The 2D parameterization enables easy detection of potential collisions and the 3D parameterization handles complex shapes effectively. We show that this combination is faster and yields higher quality reconstructions than purely implicit surface representations, and makes the recovery of layered garments from images possible thanks to its differentiability. Furthermore, it supports rapid editing of garment shapes and texture by modifying individual 2D panels.
Draping virtual garments on body models has many applications in fashion design, movie-making, virtual try-on, virtual and augmented reality, among others. Traditionally, garments are represented by 3D meshes and the draping relies on physics-based simulations (PBS) [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12] to produce realistic interactions between clothes and body. Unfortunately PBS is often computationally expensive and rarely differentiable, which limits the scope of downstream applications. Hence, many recent techniques use neural networks to speed up the draping and to make it differentiable. The garments can be represented by 3D mesh templates [13, 14, 15, 16, 17, 18, 19, 20], point clouds [21, 22], UV maps [23, 24, 25, 26, 27], or implicit surfaces [28, 29, 30]. Draping can then be achieved by Linear Blend Skinning (LBS) from the shape and pose parameters of a body model, such as SMPL [31].
Even though all these methods can realistically drape individual garments over human bodies, none can handle multiple clothing layers, despite being prevalent in everyday dress. To address overlapping clothing layers such as those in Fig. 1 while preserving expressivity, we introduce an Implicit Sewing Pattern (ISP), a new representation inspired by the way fashion designers represent clothes. As shown in Fig. 2, a sewing pattern is made of several 2D panels implicitly represented by signed distance functions (SDFs) that model their 2D extent and are conditioned on a latent vector  $\mathbf{z}$ , along with information about how to stitch them into a complete garment. To each panel is associated a 2D to 3D mapping representing its 3D shape, also conditioned on  $\mathbf{z}$ . The 2D panels make it easy to detect collisions between surfaces and to prevent interpenetrations. In other words, the garment is made of panels whose 2D extent is learned instead of having to be carefully designed by a human and whose
Figure 1: Multi-layered garment draping. Top: Draping multiple layers of garments over one body (left) and modifying the body's shape (right). Bottom: Draping the same set of 5 garments over bodies with varying poses and shapes.
3D shape is expressed in a unified  $uv$  space in which a loss designed to prevent interpenetrations can easily be written.
This combination enables us to model layered garments such as those of Fig. 1 while preserving end-to-end differentiability. This lets us drape them realistically over bodies in arbitrary poses, to recover them from images, and to edit them easily. Doing all this jointly is something that has not yet been demonstrated in the Computer Vision or Computer Graphics literature. Furthermore, most data driven draping methods rely on synthetic data generated with PBS for supervision purposes. In contrast, ISPs rely on physics-based self-supervision of [18, 13]. As a result, at inference time, our approach can handle arbitrary body poses, while only requiring garments draped over bodies in a canonical pose at training time. Our code is available at https://github.com/liren2515/ISP.
Garment Draping. Garment draping approaches can be classified as physics-based or data-driven. Physics-based ones [3, 32, 33, 34, 35, 12] produce high-quality results but are computationally demanding, while data-driven approaches are faster, sometimes at the cost of realism. Most data-driven methods are template-based [13, 14, 15, 17, 19, 18, 20, 16, 18, 13], with a triangulated mesh modeling a specific garment and a draping function trained specifically for it. As this is impractical for large garment collections, some recent works [22, 21, 36] use 3D point clouds to represent garments instead. Unfortunately, this either prevents differentiable changes in garment topology or loses the point connections with physical meanings. The approaches of [24, 25] replace the clouds by UV maps that encode the diverse geometry of garments and predict positional draping maps. These UV maps are registered to the body mesh, which restricts their interpretability and flexibility for garment representation and manipulation. The resulting garments follow the underlying body topology, and the ones that should not, such as skirts, must be post-processed to remove artifacts. Yet other algorithms [29, 30, 37] rely on learning 3D displacement fields or hierarchical graphs for garment deformation, which makes them applicable to generic garments.
While many of these data-driven methods deliver good results, they are typically designed to handle a single garment or a top and a bottom garment with only limited overlap. An exception is the approach of [38] that augments SDFs with covariant fields to untangle multi-layered garments. However, the untangling is limited to garments in T-pose. In contrast, our proposed method can perform multi-layered garment draping for bodies in complex poses and of diverse shapes.
Garments as Sets of Panels. Sewing patterns are widely used in garment design and manufacturing [39, 40, 41, 42, 43]. Typically, flat sewing patterns are assembled and then draped using PBS. To automate pattern design, recent works introduce a parametric pattern space. For example, the methods of [44, 45] introduce a sparse set of parameters, such as sleeve length or chest circumference, while [27] relies on principal component analysis (PCA) to encode the shape of individual panels. It requires hierarchical graphs on groups of panels to handle multiple garment styles. By contrast, our approach relies on the expressivity of 2D Sign Distance Functions (SDFs) to represent the panels.
To promote the use of sewing patterns in conjunction with deep learning, a fully automatic dataset generation tool is proposed in [46]. It randomly samples parameters to produce sewing patterns and uses PBS to drape them on a T-posed human body model to produce training pairs of sewing patterns and 3D garment meshes.
Garments as Implicit Surfaces. SDFs have become very popular to represent 3D surfaces. However, they are primarily intended for watertight surfaces. A standard way to use them to represent open surfaces such as garments is to represent them as thin volumes surrounding the actual surface [28, 29], which can be represented by SDFs but with an inherent accuracy loss. To address this issue, in [30], the SDFs are replaced by unsigned distance functions (UDFs) while relying on the differentiable approach of [47] to meshing, in case an actual mesh is required for downstream tasks. However, this requires extra computations for meshing and makes training more difficult because surfaces lie at a singularity of the implicit field. This can result in unwanted artifacts that our continuous UV parameterization over 2D panels eliminates.
In clothing software used industrially [39, 40, 41], garments are made from sets of 2D panels cut from pieces of cloth which are then stitched together. Inspired by this real-world practice, we introduce the Implicit Sewing Patterns (ISP) model depicted by Fig. 2. It consists of 2D panels whose shape is defined by the zero crossings of a function that takes as input a 2D location  $\mathbf{x} = (x_u, x_v)$  and a latent vector  $\mathbf{z}$ , which is specific to each garment. A second function that also takes  $\mathbf{x}$  and  $\mathbf{z}$  as arguments maps the flat 2D panel to the potentially complex 3D garment surface within the panel, while enforcing continuity across panels. Finally, we train draping networks to properly drape multiple garments on human bodies of arbitrary shapes and poses, while avoiding interpenetrations of successive clothing layers.
3.1 Modeling Individual Garments
We model garments as sets of 2D panels stitched together in a pre-specified way. Each panel is turned into a 3D surface using a uv parameterization and the networks that implement this parameterization are trained to produce properly stitched 3D surfaces. To create the required training databases, we use the PBS approach of [46]. Because it also relies on 2D panels, using its output to train our networks has proved straightforward, with only minor modifications required.
Flat 2D Panels. We take a pattern  $P$  to be a subset of  $\Omega = [-1,1]^2$  whose boundary are the zero crossings of an implicit function. More specifically, we define
$$
\mathcal {I} _ {\Theta}: \Omega \times \mathbb {R} ^ {| \mathbf {z} |} \rightarrow \mathbb {R} \times \mathbb {N}, (\mathbf {x}, \mathbf {z}) \mapsto (s, c), \tag {1}
$$
where  $\mathcal{I}_{\Theta}$  is implemented by a fully connected network. It takes as input a local 2D location  $\mathbf{x}$  and a global latent vector  $\mathbf{z}$  and returns two values. The first,  $s$ , should approximate the signed distance to the panel boundary. The second,  $c$ , is a label associated to boundary points and is used when assembling the garment from several panels. Then, boundaries with the same labels should be stitched together. Note that the  $c$  values are only relevant at the panel boundaries, that is, when  $s = 0$ . However, training the network to produce such values only there would be difficult because this would involve a very sparse supervision. So, instead, we train our network to predict the label
Figure 2: Implicit Sewing Pattern (ISP). (a) The 3D mesh surface for a shirt with the front surface in gray and the back one in blue. (b) The front and back 2D panels of the ISP. The numbers denote the labels of seamed edges  $E_{c}$ , and indicate how to stitch them when  $c > 0$ . (c) We use an implicit neural representation for the signed distance and for the edge labels, denoted here by the different colors. (d) Interpolation in the latent space allows topology changes, here from a sleeveless shirt to a long-sleeve open jacket. The top rows show the front and back panels, the bottom row the reconstructed meshes. (e) To parameterize a two-panel garment, we train the implicit network  $\mathcal{I}_{\Theta}$  to predict the signed distance field  $(s_f / s_b)$  and the edge label field  $(c_f / c_b)$  that represent the two panels. They are mapped to 3D surfaces by the network  $\mathcal{A}_{\Phi}$ .
$c$  of the closest boundary for all points, as shown in Fig. 2(c), which yields a much better behaved minimization problem at training time.
In this paper, we model every garment  $G$  using two panels  $P^f$  and  $P^b$ , one for the front and one for the back.  $E^f$  and  $E^b$  are labels assigned to boundary points. Label 0 denotes unstitched boundary points like collar, waist, or sleeve ends. Labels other than zero denote points in the boundary of one panel that should be stitched to a point with the same label in the other.
In practice, given the database of garments and corresponding 2D panels we generated using publicly available software designed for this purpose [46], we use an auto-decoding approach to jointly train  $\mathcal{I}_{\Theta}$  and to associate a latent vector  $\mathbf{z}$  to each training garment. To this end, we minimize the loss function obtained by summing over all training panels
$$
\mathcal {L} _ {\mathcal {I}} = \sum_ {\mathbf {x} \in \Omega} | s (\mathbf {x}, \mathbf {z}) - s ^ {g t} (\mathbf {x}) | + \lambda_ {C E} \mathrm {C E} (c (\mathbf {x}, \mathbf {z}), c ^ {g t} (\mathbf {x})) + \lambda_ {r e g} \| \mathbf {z} \| _ {2} ^ {2}, \tag {2}
$$
with respect to a separate latent code  $\mathbf{z}$  per garment and the network weights  $\Theta$  that are used for all garments. CE is the cross-entropy loss,  $s^{gt}$  and  $c^{gt}$  are the ground-truth signed distance value and the label of the closest sequenced edge of  $\mathbf{x}$ , and  $\lambda_{CE}$  and  $\lambda_{reg}$  are scalars balancing the influence of the different terms. We handle each garment having a front and a back panel  $P^f$  and  $P^b$  by two separate network  $\mathcal{I}_{\Theta_f}$  and  $\mathcal{I}_{\Theta_b}$  with shared latent codes so that it can produce two separate sets of  $(s,c)$  values at each  $(x_u,x_v)$  location, one for the front and one for the back.
From 2D panels to 3D Surfaces. The sewing patterns described above are flat panels whose role is to provide stitching instructions. To turn them into 3D surfaces that can be draped on a human body, we learn a mapping from the 2D panels to the 3D garment draped on the neutral SMPL body. To this end, we introduce a second function, similar to the one used in AtlasNet [48]
$$
\mathcal {A} _ {\Phi}: \Omega \times \mathbb {R} ^ {| \mathbf {z} |} \rightarrow \mathbb {R} ^ {3}, (\mathbf {x}, \mathbf {z}) \mapsto \mathbf {X}. \tag {3}
$$
It is also implemented by a fully connected network and takes as input a local 2D location  $\mathbf{x}$  and a global latent vector  $\mathbf{z}$ . It returns a 3D position  $\mathbf{X}$  for every 2D location  $\mathbf{x}$  in the pattern. A key difference with AtlasNet is that we only evaluate  $\mathcal{A}_{\Phi}$  for points  $\mathbf{x}$  within the panels, that is points for which the signed distance returned by  $\mathcal{I}_{\Theta}$  of Eq. (1) is not positive. Hence, there is no need to deform or stretch  $uv$  patterns. This is in contrast to the square patches of AtlasNet that had to be, which simplifies the training.
Given the latent codes  $\mathbf{z}$  learned for each garment in the training database, as described above, we train a separate set of weights  $\Phi_f$  and  $\Phi_b$  for the front and back of the garments. To this end, we minimize a sum over all front and back training panels of
$$
\mathcal {L} _ {\mathcal {A}} = \mathcal {L} _ {C H D} + \lambda_ {n} \mathcal {L} _ {\text {n o r m a l}} + \lambda_ {c} \mathcal {L} _ {\text {c o n s i s t}}, \tag {4}
$$
$$
\mathcal {L} _ {\text {c o n s i s t}} = \sum_ {c > 0} \sum_ {\mathbf {x} \in E _ {c}} \left\| \mathcal {A} _ {\Phi_ {f}} (\mathbf {x}, \mathbf {z}) - \mathcal {A} _ {\Phi_ {b}} (\mathbf {x}, \mathbf {z}) \right\| _ {2} ^ {2}. \tag {5}
$$
where  $\mathcal{L}_{CHD}$ ,  $\mathcal{L}_{normal}$ , and  $\mathcal{L}_{consist}$  are the Chamfer distance, a normal consistency loss, and a loss term whose minimization ensures that the points on the edges of the front and back panels sewn together— $E_{c}$  for  $c > 0$ —are aligned when folding the panels in 3D,  $\lambda_{n}$  and  $\lambda_{c}$  are scalar weights. This assumes that the front and back panels have their seamed edges aligned in 2D, i.e., for  $c > 0$  and  $\mathbf{x} \in \Omega$ , if  $\mathbf{x}$  has label  $c$  on the front panel, then it should also have label  $c$  in the back panel, and vice-versa. Our experiments show that  $\mathcal{L}_{consist}$  reduces the gap between the front and back panels.
Meshing and Preserving Differentiability.  $\mathcal{A}_{\Phi}$  continuously deforms 2D patches that are implicitly defined by  $\mathcal{I}_{\Theta}$ . To obtain triangulated meshes from these, we lift a regular triangular 2D mesh defined on  $\Omega$  by querying  $\mathcal{A}_{\Phi}$  on each of its vertex, as in [48]. More specifically, we first create a square 2D mesh  $\mathbf{T} = \{V_{\Omega}, F_{\Omega}\}$  for  $\Omega$ , where vertices  $V_{\Omega}$  are grid points evenly sampled along orthogonal axis of  $\Omega$  and faces  $F_{\Omega}$  are created with Delaunay triangulation. Given the latent code  $\mathbf{z}$  of a specific garment, for each vertex  $v \in V_{\Omega}$ , we can obtain its signed distance value  $s$  and edge label  $c$  with  $(s, c) = \mathcal{I}_{\Theta}(v, \mathbf{z})$ . We construct the 2D panel mesh  $\mathbf{T}_P = \{V_P, F_P\}$  by discarding vertices of  $\mathbf{T}$  whose SDF is positive. To get cleaner panel borders, we also keep vertices  $v \in V_{\Omega}$  with  $s(v, \mathbf{z}) > 0$  that belong to the mesh edges that cross the 0 iso-level, and adjust their positions to  $v - s(v, \mathbf{z}) \nabla s(v, \mathbf{z})$  to project them to the zero level set [49]. The front and back panel 2D meshes are then lifted to 3D by querying  $\mathcal{A}_{\Phi}$  on each vertex. During post-processing, their edges are sewn together to produce the final 3D garment mesh  $\mathbf{T}_G(\mathbf{z})$ . More details can be found in the Supplementary Material.
$\mathcal{I}_{\Theta}$  performs as an indicator function to keep vertices with  $s \leq 0$ , which breaks automatic differentiability. To restore it, we rely on gradients derived in [50, 51] for implicit surface extraction. More formally, assume  $v$  is a vertex of the extracted mesh  $\mathbf{T}_G$  and  $\mathbf{x}$  is the point on the UV space that satisfies  $v = \mathcal{A}_{\Phi}(\mathbf{x}, \mathbf{z})$ , then, as proved in the Supplementary Material,
$$
\frac {\partial v}{\partial \mathbf {z}} = \frac {\partial \mathcal {A} _ {\Phi}}{\partial \mathbf {z}} (\mathbf {x}, \mathbf {z}) - \frac {\partial \mathcal {A} _ {\Phi}}{\partial \mathbf {x}} \nabla s (\mathbf {x}, \mathbf {z}) \frac {\partial s}{\partial \mathbf {z}} (\mathbf {x}, \mathbf {z}). \tag {6}
$$
Hence, ISP can be used to solve inverse problems using gradient descent, such as fitting garments to partial observations.
3.2 Garment Draping
We have defined a mapping from a latent vector  $\mathbf{z}$  to a garment draped over the neutral SMPL model [31], which we will refer to as a rest pose. We now need to deform the garments to conform to bodies of different shapes and in different poses. To this end, we first train a network that can deform a single garment. We then train a second one that handles the interactions between multiple garment layers and prevents interpenetrations as depicted in Fig. 3.
Single Layer Draping. In SMPL, body templates are deformed using LBS. As in [19, 29, 30], we first invoke an extended skinning procedure with the diffused body model formulation to an initial rough estimate of the garment shape. More specifically, given the parameter vectors  $\mathbf{B}$  and  $\Theta$  that control the body shape and pose respectively, each vertex  $v$  of a garment mesh  $\mathbf{T}_G$  is transformed by
$$
v _ {i n i t} = W \left(v _ {\left(\mathbf {B}, \boldsymbol {\Theta}\right)}, \mathbf {B}, \boldsymbol {\Theta}, w (v) \mathcal {W}\right), \tag {7}
$$
$$
v _ {(\mathbf {B}, \Theta)} = v + w (v) B _ {s} (\mathbf {B}) + w (v) B _ {p} (\Theta),
$$
where  $W(\cdot)$  is the SMPL skinning function with skinning weights  $\mathcal{W} \in \mathbb{R}^{N_B \times 24}$ , with  $N_B$  being the number of vertices of the SMPL body mesh, and  $B_s(\mathbf{B}) \in \mathbb{R}^{N_B \times 3}$  and  $B_p(\Theta) \in \mathbb{R}^{N_B \times 3}$  are the
shape and pose displacements returned by SMPL.  $w(v) \in \mathbb{R}^{N_B}$  are diffused weights returned by a neural network, which generalizes the SMPL skimming to any point in 3D space. The training of  $w(v)$  is achieved by smoothly diffusing the surface values, as in [29].
This yields an initial deformation, such that the garment roughly fits the underlying body. A displacement network  $\mathcal{D}_s$  is then used to refine it.  $\mathcal{D}_s$  is conditioned on the body shape and pose parameters  $\mathbf{B},\Theta$  and on the garment specific latent code  $\mathbf{z}$ . It returns a displacement map  $D_{s} = \mathcal{D}_{s}(\mathbf{B},\Theta ,\mathbf{z})$  in UV space. Hence, the vertex  $v_{init}$  with UV coordinates  $(x_{u},x_{v})$  in  $\Omega$  is displaced accordingly and becomes  $\tilde{v} = v_{init} + D_s[x_u,x_v]$ , where  $[\cdot ,\cdot ]$  denotes standard array addressing.  $\mathcal{D}_s$  is implemented as a multi-layer perceptron (MLP) that outputs a vector of size  $6N^2$ , where  $N$  is the resolution of UV mesh  $\mathbf{T}$ . The output is reshaped to two  $N\times N\times 3$  arrays to produce the front and back displacement maps.
To learn the deformation for various garments without collecting any ground-truth data and train  $\mathcal{D}_s$  in a self-supervised fashion, we minimize the physics-based loss from [18]
$$
\mathcal {L} _ {p h y} = \mathcal {L} _ {\text {s t r a i n}} + \mathcal {L} _ {\text {b e n d}} + \mathcal {L} _ {\text {g r a v i t y}} + \mathcal {L} _ {\text {B G c o l}}, \tag {8}
$$
where  $\mathcal{L}_{strain}$  is the membrane strain energy caused by the deformation,  $\mathcal{L}_{bend}$  the bending energy raised from the folding of adjacent faces,  $\mathcal{L}_{gravity}$  the gravitational potential energy, and  $\mathcal{L}_{BGcol}$  the penalty for body-garment collisions.
Multi-Layer Draping. When draping independently multiple garments worn by the same person using the network  $\mathcal{D}_s$  introduced above, the draped garments can intersect, which is physically impossible and must be prevented. We now show that our ISP model makes that straightforward first in the case of two overlapping garments, and then in the case of arbitrarily many.
Consider an outer garment  $\mathbf{T}_G^o$  and an inner garment  $\mathbf{T}_G^i$  with rest state vertices  $V_{o}$  and  $V_{i}$ . We first drape them independently on a target SMPL body with  $\mathcal{D}_s$ , as described for single layer draping. This yields the deformed outer and underlying garments with vertices  $\tilde{V}_{o}$  and  $\tilde{V}_{i}$ , respectively. We then rely on a second network  $\mathcal{D}_m$  to predict corrective displacements, conditioned on the outer garment geometry and repulsive virtual forces produced by the intersections with the inner garment.
In ISP, garments are represented by mapping 2D panels to 3D surfaces. Hence their geometry can be stored on regular 2D grids on which a convolutional network can act. In practice, we first encode the rest state of the outer garment  $\mathbf{T}_G^o$  into a 2D position map  $M_r$ . The grid  $M_r$  records the 3D location of the vertex  $v_o \in V_o$  at its  $(x_u, x_v)$  coordinate as  $M_r[x_u, x_v] = v_o$  within the panel boundaries,  $M_r[x_u, x_v] = (0, 0, 0)$  elsewhere. Concatenating both front and back panels yields a  $N \times N \times 6$  array, that is, a 2D array of spatial dimension  $N$  with 6 channels. After draping, the same process is applied to encode the geometry of  $\mathbf{T}_G^o$  into position map  $M_d$ , using vertices  $\tilde{V}_o$  instead of  $V_o$  this time. Finally, for each vertex  $\tilde{v}_o \in \tilde{V}_o$ , we take the repulsive force acting on it to be
$$
f \left(\tilde {v} _ {o}\right) = \max  \left(0, \left(\tilde {v} _ {i} - \tilde {v} _ {o}\right) \cdot \mathbf {n} _ {i}\right) \mathbf {n} _ {i}, \tag {9}
$$