ULNeF: Untangled Layered Neural Fields for Mix-and-Match Virtual Try-On
Authors: Igor Santesteban
Miguel A. Otaduy
Nils Thuerey
Dan Casas
Venue: NeurIPS 2022 Accept (2022)
Keywords: Neural fields
deformable objects
contact handling
cloth
virtual try-on

Abstract: Recent advances in neural models have shown great results for virtual try-on (VTO) problems, where a 3D representation of a garment is deformed to fit a target body shape. However, current solutions are limited to a single garment layer, and cannot address the combinatorial complexity of mixing different garments. Motivated by this limitation, we investigate the use of neural fields for mix-and-match VTO, and identify and solve a fundamental challenge that existing neural-field methods cannot address: the interaction between layered neural fields. To this end, we propose a neural model that untangles layered neural fields to represent collision-free garment surfaces. The key ingredient is a neural untangling projection operator that works directly on the layered neural fields, not on explicit surface representations. Algorithms to resolve object-object interaction are inherently limited by the use of explicit geometric representations, and we show how methods that work directly on neural implicit representations could bring a change of paradigm and open the door to radically different approaches.

ULNeF: Untangled Layered Neural Fields for Mix-and-Match Virtual Try-On
Igor Santesteban
Miguel A. Otaduy
Universidad Rey Juan Carlos
Universidad Rey Juan Carlos
Madrid, Spain
Madrid, Spain mig miguel. otaduy@urjc.es
igor wwwtsstteaanuurrccccoomm
Nils Thuerey
Dan Casas
Technical University of Munich Germany
Universidad Rey Juan Carlos
Madrid, Spain
nils. www.ttheeeeuum..e. de
dan. casasuurjc.es
Abstract
Recent advances in neural models have shown great results for virtual try-on (VTO) problems, where 3D representation of garment is deformed to fit target body shape. However, current solutions are limited to a single garment layer, and cannot address the combinatorial complexity of mixing different garments. Motivated by this limitation, we investigate the use of neural fields for mix-and-match VTO, and identify and solve fundamental challenge that existing neural-field methods cannot address: the interaction between layered neural fields. To this end, we propose neural model that untangles layered neural fields represent collision-free garment surfaces. The key ingredient is a neural untangling projection operator that works directly on the layered neural fields, not on explicit surface representations. Algorithms to resolve object-object interaction are inherently limited by the use of explicit geometric representations, and we show how methods that work directly neural implicit representations could bring change of paradigm and open the door to radically different approaches.
Introduction
In virtual try-on (VTO) applications, a computer model of 3D garment displayed together with an avatar of person, to communicate how the garment deforms based on the shape and pose of the person. Neural processing methods have shown great success to solve the problem of VTO [38, 57, 50, 23,5, 84], by leveraging low- wow-dimensionalit parameterizations of body shape and pose [36, 29]. These methods train a 3D deformation model of one garment (or a predefined outfit), and provide accurate approximation of physics simulation at runtime, while requiring just a small fraction of the computational cost of physical simulations.
However, state-of-the-ar VTO is limited to wearing a single garment or a predefined outfit, but in real life we combine many clothes to create different outfits. Unfortunately, existing garment-specific or outfit-specific neural processing solutions cannot address the combinatorial complexity of mix- and-match VTO. In fact, the problem of mix-and-mate VTO poses novel challenges to machine learning algorithms, as it drives the attention toward object interaction problems where each object (e.g., each garment) is geometrically complex, and the space object-object interactions cannot be exhaustively trained.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Object-object interaction has been solved traditionally using explicit geometric representations [43, 2]. Recent advances neural models of implicit representations have enabled radically new solutions to many problems, with the ability to efficiently encode parametric models [55, 15, 1, 10, 48, 27, 72]. However, for object-ohject interaction, the applicability of neural implicit models is still limited to solving proximity queries against explicit representations, and interaction is actually solved on these explicit representations [83].
Motivated by the challenges of mix-and-match VTO, we introduce a novel approach to resolve multi-object interaction problems, which works directly on implicit representations of the objects. We represent multiple possibly colliding objects (e.g., multiple garments) using layered variant of neural fields [75], and we design an algorithm that untangles these layered neural fields to represent collision-fre free objects. In Section 3 we describe how layered neural fields can be parameterized by deformation codes represent multiple deformable surfaces, and we introduce a neural untangling projection operator that works directly on the layered neural fields, not on surface representations. The result of compositing the neural projection with the layered neural fields is untangled layered neural fields (ULNeFs).
In Section 4 show how ULNeF's can be used to efficiently solve mix-and-match VTO for multiple garment layers. As a preprocess, each garment layer is represented using a parametric neural field that is trained for this garment in isolation. At runtime, given code that represents body shape, we optimize for the deformation of the garment layers that is free of collisions. using ULNeFs as the fundamental tool to resolve garment collisions. We demonstrate challenging mix--and-mate VTO examples with multiple layers of clothing resolved interactively.
In summary, contribution is threefold:
1. A neural field formulation, based on covariant fields, capable of encoding open surfaces with holes, as well as inside-outside information (Section 3.1).
2. A neural projection operator that directly projects entangled surfaces, encoded as neural fields, to untangled configurations, coined as ULNeF (Section 3.2).
3. A downstream task using ULNeFs to enable interactive and accurate ankk-mattmhhh VTO (Section 4).
ULNeF's could see applicability in other object-object interaction problems beyond mix-and-match VTO. Algorithms to resolve object-object interaction are inherently limited by the use of explicit geometric representations. We show how methods that work directly implicit representations could bring a change of paradigm and open the door to radically different approaches. Moreover, novel neural-field representations are hard to leverage in situations that involve object interaction and/or collisions, but extend their applicability.
methods rooted on implicit representations could further
2 Related Work
Neural Fields. Over the last few years, neural fields [75] have emerged as an alternative to the well- established polygonal meshes to represent shapes. Neural fields build on the -asso well-established- idea that shapes can be represented as a level set of implicit functions, but propose that such function compact,
can be learned with neural network [11, 39, 40]. These neural representations continuous, and are easily differentiable, which opens the door to a large variety of applications in many fields including Computer Vision [54, 85], Computer Graphics [62, 42, 18, 78], and Robotics [61, 65, 47].
Many works leverage the neural fields capabilities to encode human- features for variety of tasks. This has enabled, for example, impressive advances in reconstructing clothed humans directly from RGB [54, 25, 80], RGB-D [35, 64, 82, 17], and point cloud [12] input. Alternatively, some methods use neural fields to ease the fitting of a parametric human model [36, 29] to sparse inputs [71, 6, 7], which yields to detailed 3D reconstructions that can be articulated by the underlying skeleton.
Closer to ours are the works that use neural fields for modeling 3D deformable bodies [15, 45, 1, 41, 30, 31] and clothed humans [10, 48, 27, 55, 72]. This is contrast to previous works that tackle these 3D modeling tasks with explicit mesh-based models for humans [9, 56, 37, 52], which

---

typically requires accurate surface registrations, and limits the surface details by the mesh topology. A common strategy is learn dynamic neural fields in a canonical space, reproducing pose-dependent deformations observed in detailed scans [55, 10] or partial depth maps [48, 72, 17]. The learned field then articulated using forward skinning techniques. Despite the realism of the output deformations, learned fields encode a single surface for clothing and body. In contrast, our formulation defines how to mix and untangle different fields to allow the editing of clothing styles.
Neural fields have also been used to encode both appearance and volumetric information of a scene, representation known as Neural Radiance Fields (NeRF) [42]. Follow up works showed that NeRFs can be used also to encode articulated objects [76, 46, 77] and dynamic scenes [53, 19, 49]. Specific for humans, A-NeRF [63] transforms NeRF features using an skeleton, and demonstrates that novel motions and viewpoints can be synthesized. NeuralBody [51] appends learnable features to the vertices of a surface body model, enabling free-viewpoint rendering of animatable humans. Similarly, Kwon ef al. [32] enrich parametric surface human aggregating spatio-temporal density and color information using transformers. Orthogonal to these works, we do not encode appearance or volume density, but propose novel formulation to allow the untangled combination of garments encoded using layered fields, which enriches existing representations for humans.
3D Virtual Try-On. This area of research aims at estimating how a digital representation of a 3D
garment deforms as a function of the underlying body shape. Physics-based cloth simulation [3,43] 43] has been the natural choice to model this problem, however, given the high computational cost of each simulation time step, recent methods have explore learning- based alternatives [73, 57, 50, 5, 23]. These works aim at learning function that dircetly outputs deformed mesh of garment given an input feature that describes the target body shape. To this end, the common solution is to use per- vertex supervised strategies that leverage large datasets of simulated [22, 57, 50, 4] or reconstructed 3D garments [37, 52]. Notably, recent methods have proposed self-supervised strategies based on physics- losses [5, 58].
All 3D VTO methods described above use an explicit model to encode the garment mesh. Closest to ours is SMPLicit [13], recent work that encodes parametric garments as unsigned distance fields learned from data. This allows for controllable garment style, but since an unsigned field used it complicates the extraction of isosurface, and does not allow for inside/outside queries to model collisions.
Garment-bode collision handling is particularly challenging in learning- based VTO. Since at inference time most methods directly predict garment vertex positions (i.e., there is not explicit collision check), residual regression errors can potentially lead to garment vertices that penetrate the body. To alleviate this, most methods [5, 50, 57] include term in their loss function to penalize collisions, but generalization of this term beyond the training dataset is difficult. In fact, an expensive postprocessing step to fix collisions is common in most state-of-the-a
learning-based methods, but some speed-up methods exist [66]. To circumvent the need for any fix, Santesteban ef al. [59] propose to learn generative subspace that encode collision-fre garment configurations, but cannot handle more than one layer. In contrast, our ULNeF approach proposes a completely new formulation based on neural fields that can deal with multiple garments.
Image-Based Based Virtual Try-On. Virtual try-on has also been approached using image-based methods. The aim is to generate compelling 2D images of dressed people, without dealing with any 3D model. Early works based pose-dependent interpolation between different images [26] have been outperform with modern techniques based on convolutional neural networks (CNNs) [24, 69]. Subsequent works further improve the quality of the synthesized images [33, 81, 79, 20], solve artifacts by reducing the reliance on 2D segmentation [28, 2 21], support mix-and- mnthh VTO [44, 34, 14], and synthesize images for arbitrary poses [16, (0).
3 Untangled Layered Neural Fields
The core of our work is a method that takes as input neural fields, which implicitly represent N possibly colliding surfaces, and outputs N projected fields the ULNeeFs- which encode collision- free implicit surfaces and minimize the displacement with respect to the input. By using implicit representations, the surfaces are defined as zero-sets of scalar functions. Then, the untangling
3

---

fi
Gi
Untangled
f
92(x,8)
Layered
Position and shape code x.p
(2))) K200
O hb)
f'
Neural
Fields
9% ().
fN
f20)) (x)
Neural untangling Section
Implicit surface model
Section 3.1
Figure 1: Overview UL.eF. We propose a formulation to encode parametric open surfaces using a novel neural representation 9% (z,, (3) that encapsulates two fields: signed distance field f(x) that represents the garment surface and provides a notion of inside-outside; and a covariant field h(r) ) that models the volume near the openings that other garments can pass through without producing tangled configurations. Having a set of surfaces with potentially entangled configurations, and leveraging their corresponding fields f (z) and h2 * (z), we propose neural projection operator that directly deforms the input fields ft (x) such that their zero-sets do not collide.
operation reduces to modifying the scalar functions, which in practice shifts the zero-sets. Figure depicts summary of our main building block.
In this section, we present the untangling operation as an optimization formulated on scalar field values, and we show how this optimization can be efficiently learned with neural model. Beforehand, we first diseuss specifies of the implicit representation of garments, and how we further parameterize the garments as a function of additional settings, in our case body shape.
3.1 Implicit Surface Model
Surfaces can represented implicitly as the zero-set of their distance field. Formally, given a distance field f(z), R$, the surface is the set X suffers from two challenging aspects when applied cloth untangling. First, untangling requires inside-outside information to resolve queries. Second, garments open surfaces with holes that allow inner layers to pass through, introducing great complexity to the process of collision detection. To the best of our knowledge, develop first neural model of garments that addresses these challenges with an implicit representation.
f(z) 0}. However, this implicit representation
To this end, we represent the garment using two fields: signed distance field f(x) that represents garment surface and provides a notion of inside-outside and a covariant field l((x) that models the volume near the openings that other garments can pass through without producing tangled configurations. We construct the signed distance field by calculating the euclidean distance to the surface and computing the sign sign((z p) n) where is the closest surface point and is the normal vector at point The covariant field [8] is computed via a Hermite Radial Basis Function (HRBF) [74] fit by constraining the normals of the seams of cutout regions. Using these fields, can detect a point & is in a tangled configuration if f (x) < 0 and h(x) < 0. Hence, points with h(x)) 0 are considered to be unproblematic, they lie in the volume extending the surface holes.
surface with holes, e,g., a garment surface g, can be represented implicitly using this pair of fields, g(xx) = ((x)). We use neural model with parameters O to represent these two fields. Moreover, using a neural model allows us to further parameterize the surface based on an additional code B. which yields a model g(z, B, Onelds). In our case, parameterize garment surfaces function of body shape [36], but the implementation could be extended to include other codes such as body pose [57, 50, 23] garment design parameters [68, 60], in previous neural VTO models.
We train the neural surface model in a supervised way, with loss terms for errors in the fields and their gradients with respect to ground-trutt data. Additionally, we encode points x using Fourier Features [67], but omit the text to simplify the notation. Formally:
L, + Ln + (Ror + Lan
acids = arg min
ox 85

---

Σ
(2) BB, nnels) - (2, []]
(2)
B
Σ
(1( B, Onews) her BB]
B
Lor 8r
Σ >
||
a1 dhr
(z,, BB Onelás)
OfGI (æ, [B)1
(4)
tee
B
0h (zz, B3 Onela)) Oh ||
abGE On
(ar, B)]
(5)
B
In the supplementary document we provide additional details about the architecture of the neural network, training hyperparameters, and our strategy to sample B and x.
3.2 Neural Untangling
Let us take input possibly colliding implicit surfaces {X;} } defined by pairs of signed-distanee and covariance fields fü (2), h (x), respectively. Note that the surfaces can be further parameterized by a code B discussed above. However, we drop this parameterization in this section, as it does not affect the untangling operation. The subindex i denotes the order in which the surfaces should be layered, with surface + 1 above, i.e., outside, surface i.
We perform untangling by outputting N implicit surfaces {X,} defined by signed distance fields f1(x). We seek surfaces that are as close as possible the input surfaces, but remain collision free.
Thanks to the implicit surface representation. untangling can be formulated as a local operation on the field values at positions J2 f&) E RN h = (hi, he ........ ...))..)))))))) E R., with components f = (z), h = h (x). and it outputs a vector of field values f = f2....f E RN. We denote the local untangling operation as f = Off h²). Applying the local untangling operation positions we obtain the untangled layered field representation f(xr), as shown schematically Fig. 1.
R$. Formally, untangling takes input two vectors of field values f"
The definition of the local untangling operation borrows from the method by Buffet et al. [8]. We define this operation as the following optimization:
f = fff h²) = arg min || 111
+
ΣΣ ft.
(6)
oo
ft < and h3 0 and fj > and hī <
Hfi,, h7, J3.
o
otherwise
In a nutshell, this optimization returns the closest collision- free field values. The collision loss H() penalizes the total loss when point is outside the top surface j and inside the bottom surface i. Buffet et al. [8] designed an algorithm of complexity O(N3) to solve the local untangling operation.
Instead, we propose neural model with parameters Ountangl that learns the untangling operation. Then, (Qumtang)) can be regarded as a projection operator that projects colliding field values to the closest collision-free values, the ULNeFs f. Importantly, note that this neural projection operator is trained only once for any arbitrary combination of surfaces, as it operates on the field values, not on the actual surfaces. Hence, once trained, this model naturally generalizes to unseen garments at train time.
We train the neural projection operator by randomly sampling values of f E RN from from (.22, 1.5) and h² E R. from 2d(-1.0, 1.0). For each pair of f* and h" we compute ground -trthhh values of the untangled surfaces for using the method by Buffet et al. [8].
Ountangl arg min
Σ 110(F, h², Ountangl) forll,
Please check the supplementary document for details about training data, architecture and parameters.
4 Mix-and-Match VTO
Using the ULNeeFs presented in the previous section, we now describe how we solve the problem of mix-and-match VTO. The input to the VTO problem consists of neural parametric models of garments
5

---

PREPROCESS (Per garment)
Explicit Garment Model
Implicit Garment Model
Garment
Cloth simulation
canonical space
Implit
Optimization
min
Ieplii
B
Target body
Original surfaces
Untangled surfaces
Final result
Figure 2: Pipeline of our method for mix-and- mix-and-match VTO. We first preprocess a dataset of garments by simulating each of them in variety of human shapes. Then, we transform garments into canonical space, and learn shape-dependent explicit and implicit models. At runtime, we infer explicit and implicit shape-dependent garment deformations, use UL.NeF to untangle the implicit representations, and optimize the explicit surfaces to fit into the resulting untangled fields.
trained for each garment in isolation, together with value of the parametric code to be evaluated (in our case, body shape (3). We describe an optimization problem that takes the per-ge ger-gammett models and finds untangled collision-free garments with minimal deformation. The central ingredient of this optimization is the fast evaluation of ULNeFs, as we search for the optimum. Figure depicts our mix-and-match pipeline.
To the problem, similar to other VTO methods [59, 50, 5], all garment deformations represented in canonical body space, free of shape or pose deformations. We start by describing the body model and the canonical-space garment deformation, and then present the optimization of untangled garments.
4.1 Explicit Garment Model
Our explicit garment model builds top of the parametric SMPL body model [36] and uses explicit geometry (i.e., vertices and triangles) to represent garment deformations. SMPL defines a template surface Tp, local shape- and pose-dependent deformations with respect to this template, and provides a skinning transformation to world space. We have limited our implementation to shape- dependent transformations; therefore, omit pose and skinning transformations in the following, In SMPL, a point on the body surface M3 for shape is defined as:
Mg (z:
(z. (), TB,
where B, represents a deformation modeled using shape-dependent blend shapes.
To represent neural garment deformations, we follow a similar formulation and, specifically, we use the canonical garment model introduced by Santesteban al. [59]. This canonical model diffuses surface body properties (in our case, the shape-dependent blend-shape deformations B.) beyond body surface, any point in R3, This diffusion strategy allows retrieve accurate per-point shape-dependent deformations. Hence, following the same formulation as for the body surface in (8), point the garment surface Mg is obtained by transforming the garment in canonical space X:
Ma(#, B)
(x. B). E xE((B,
where B2 B) is the diffused shape blendshape that outputs per-point 3D deformations function of the point E R3 and shape parameter B.. Notice that X(B) is the deformed garment (i.e., it encodes shape-dependent wrinkles), but does not include the body shape deformations because is encoded in our canonical space. Hence, Br,r, B) needed add body shape deformations. This visualized in Figure 2, bottom row, after the optimization step the untangled surfaces shown in

---

the canonical space, and then (z, B) is added to bring the deformations into the full space. See supplementary document for more details.
The garment model capable producing accurate and fast deformations of a single garment, but combining the output of multiple models results in deeply tangled surfaces. In the following section, we describe our approach to solve this issue by leveraging ULNeFs.
4.2 Optimization of Untangled Garments
To obtain untangled garment surfaces (B) for specific shape code B, reconstruct the zero-sets of ULNeFs. Note that ULNeFs define implicit surfaces (X:(B)), and here we search for explicit mesh-based discretizations. Moreover, since ULNeess defined by per-garment neural parametric fields, we use the method presented in Section 3. to train an implicit equivalent of each explicit garment model.
To reconstruct the zero-sets of ULNeFs, first we initialize possibly colliding garments (B)) using the per-garment explicit models. We have observed that just projecting mesh vertices to the zero-sets could yield large triangle distortions. Therefore, when searching for the untangled garment surfaces, add penalty term to minimize triangle distortion. Formally, we obtain each untangled garment surface by solving the following optimization:
(B) arg min
Eprojection + Estrain
(10)
projection
Σ
[i(2,
(11)
zEX,(B)
Extrain
Σ ((F(T)TF(TT) (EX:(B)
D)l22
(12)
In the Eprojection term, we evaluate the untangled field f; for all vertices in the garment mesh. Note that this requires first evaluating per-garment fields, followed by the neural projection, as shown in Figure 1. In the Estrain term, we evaluate the squared Frobenius norm of Green strain for all triangles i' the garment mesh, with F the deformation gradient.
We solve the optimization (10) using L-BFGS, We have observed that initialization with the per- garment meshes (X;(B) is key for fast convergence the optimization. Note also that the gradient computation requires the gradient of the ULNeFs, which is easily obtained thanks to the automatic differentiation capabilities of machine learning frameworks.
Evaluation
5.1 Quantitative Evaluation
In Table we present an ablation study of the different terms and encodings used to train the implicit representation for open surfaces described in Section 3.1. For each ablation, show the error of the two fields used our representation. Results demonstrate that both the encoding of input points with Fourier Features [67] and the supervision of the gradients contribute training the model.
Table 2 evaluates the runtime performance of our approach. Specifically, we compare the evaluation time of the untangling operator of Buffet et al., [8] (i.e., solving Equation 6) vs. forward pass of our learned projection operator. demonstrates that for complex outfits with thousands of vertices (the outfits shown in Figure range from 15k to 30k vertices), our approach runs up to two order of magnitude faster. Similarly, our formulation to evaluate the covariant fields f and is also significantly faster.
5.2 Qualitative Evaluation
In Figure we present a qualitative ablation of study of the different terms used to learn our implicit garment model described in in Section 3. We show that using Fourier Features [67] to encode points, as well as supervising the gradient loss is required obtain a accurate neural fields to encode detailed garments.
7

---

Ours
W/ Fourier feats. [67]
W/o gradient supervision
Error (T-shirt)
1.1mm
0.8mm
2.0mm 1.6mm
1.0mm
5.3mm 6.6mm
0.9mm
Error (Dress)
1.3mm
2.0mm
2.0mm
1.9mm
Table
Ablation study of the different terms of our implicit surface model described in Section 3.1.
Untangling operator
Field evaluation
No vertices
Buffet er al. [8]
Ours
Buffet et al. [8]
Ours
0.04 ms
0.24 ms
0.08 ms
0.36 ms
5000
81.6 ms
0.68 ms
2.08 ms
0.77 ms
15000
238.5 ms
1.74 ms
6.02 ms
2.00 ms
30000
508.0 ms
3.35 ms
12.1 ms
3,92 ms
Table 2:
Comparison of runtime performance of the main components of ULNeF. We use the authors' implementation to compare the performance of the untangling operator, and an efficient GPU reimplementation to compare the fields. This comparison was conducted in a regular desktop PC equipped with an AMD Ryzen 2700 CPU, an Nvidia GTX 1080 GPU, and 32GB of RAM.
In Figure 3, and in the supplementary video, we show qualitative results of mix-and-match VTO. For each example, show the entangled result that state-of-the-art VTO methods [59, 59, 50] produce when predicting the deformations for multiple garments, without any postprocess.
Conclusions
Motivated by the shortcoming of state-of-the-art data-driven methods for virtual try-on (VTO) (i.e., the inability to deal with multiple garments), we have identified fundamental limitations in emerging surface representations based on neural fields [75]. These limitations effectively prevent current neural field methods from modelling common scenarios such as contact and interaction between objects.
To address this, we have presented Untangled Layered Neural Fields (ULNeF), a novel neural approach project entangled implicit surfaces to an untangled configuration. The zero-set of the projected fields is collision-free and minimizes the difference with respect to the input surfaces (i.e., maintain the fine-scale details). Importantly, ULNNeFs generalize to neural fields unseen at train time. Additionally. they are highly efficient to evaluate since they only require a forward pass (i.e., a projection, and no iterative optimization) to find the untangled fields.
We have demonstrated the applicability of ULNeF's mix-and -nt-mhhh-ttch VTO, where we untangle the implicit representations of parametric garments, and optimize the garment explicit surfaces driven by the untangled fields. This enables for first time interactive, accurate, and collision ffree combinations of 3D garments, while being able manipulate the underlying body shape.
Discussion. In Section 4 we showcased a downstream task with ULNeFs consisting mix-and- match VTO. We assume that a requirement for such task is to untangle the input garments while keeping the original mesh topology, and show that this can be achieved by fitting the input meshes to the untangled fields (Section 4.2). However, we want to stress that if fixed mesh topology is not a requirement, marching cubes could be used directly after projecting the neural fields with ULNeF.
All all, we believe UL.NeF makes an important step towards modeling interactions neural fields. Future research could explore the use of ULNeeF in other scenarios that need to account for contact (e.g.. hand-object interaction) and are currently limited by explicit surface models.
Limitations.
The proposed approach for VTO using ULNeFs has only been validated with garments in T-pose. The root of this limitation is the difficulty in extending the for-

---

Figure 3: Given a set garments (left insets), existing VTO methods [57] infer their fit into a target body shape but produce a heavily entangled results (left). In contrast, ULNeF untangles the garments by directly projecting their neural fields into an collision-free configuration. Since ULNeF allows to specify the desired order, different outfits be created (center and right).

---

(a) Ground-truth
(b) Ours
(c) W/o Fourier feats. [67]
(d) W/ gradient loss
Figure 4: Qualitative ablation study of our implicit garment model described in Section 3.1. For this
particular figure, we use Marching Cubes to extract the surface.
mulation based on covariant fields to more complex poses, but we believe that represent- ing garments in an unposed canonical space could be helpful to circumvent this issue. Similarly, garments with highly curved boundaries (e.g.. neck or sleeves) can be problematic because the computed covariant field might code wrong surface semantics. No- tice that our formulation based variant fields used to approximate the signed distance values around the open areas (i.e., semantic information about the position of given point with respect to the surface). There- fore, since the estimation of such co- variant fields rely on the normals of the vertices, areas with high curvature (i.e., non- non-smooth normals) can lead to wrong fields. potential solution to this issue could be investigating an alternative to our covariant fields that leads to a smoother representation.
Input
Output
Figure 5: Limitations. ULNeF struggles with input configu-
rations (left) with vertices located too far from the untangling area or in the border of the garment. Untangled results (right) exhibit residual collisions in such specific areas.
Additionally, although achieve significant speed up compared to previous works [8], our overall runtime is in the order of 200ms per frame. While this is good enough for interactive mix-and-mate VTO applications in static pose, falls short of producing real-time animations of untangled outfits. Hence, further improvements towards reducing the computational cost remain open avenues for future works.
The work was funded in part by the European Research Council (ERC Consol- idator Grant no. 772738 TouchDesign) and Spanish Ministry of Science (RT12211--09699) 9-90694--9110 Virnnig)
Acknowledgments.
References
[1] Thiemo Alldieck, Hongyi Xu, and Cristian Sminchisescu. imGHUM: Implicit Generative Models of 3D Human Shape and Articulated Pose. In Proc. of IEEE International Conference on Computer Vision (ICCV), 2021.
[2] Sheldon Andrews and Kenny Erleben. Contact and friction simulation for computer graphics. In ACM SIGGRAPH 2021 Courses, SIGGRAPH '21, 2021. doi: 100
3464571.
10

---

[3] David Baraff and Andrew Witkin. Large Steps in Cloth Simulation. In Proc. of Conference on Computer Graphies and Interactive Techniques (SIGGRAPH), page 43-54, 1998.
[4] Hugo Bertiche, Meysam Madadi, and Sergio Escalera. CLOTH3D: Clothed 3D Humans. In Proc. of European Conference on Computer Vision (ECCV), 2020.
[5] Hugo Bertiche, Meysam Madadi, and Sergio Escalera. PBNS: Physically Based Neural Simu- lation for Unsupervised Garment Pose Space Deformation. ACM Transactions on Graphics (Proc. SIGGRAPH Asia), 40(6), dee 2021. doi 10. 1ew.111114448850 1145/347851333400477. 3480479.
[6] Bharat Lal Bhatnagar, Cristian Sminchisescu, Christian Theobalt, and Gerard Pons-Moll. Combining Implicit Function Learning and Parametric Models for 3D Human Reconstruction. In Proc. of European Conference on Computer Vision (ECCV), 2020.
[7] Bharat Lal Bhatnagar, Cristian Sminchisescu, Christian Theobalt, and Gerard Pons-Moll. LoopReg: sulf-super Learning of Implicit Surface Correspondences, Pose and Shape for 3D Human Mesh Registration. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[8] Thomas Buffet, Damien Rohmer, Loic Barthe, Laurence Boissieux, and Marie-Paule Cani. Implicit Untangling: Robust Solution for Modeling Layered Clothing. ACM Transactions on Graphics (Proc. SIGGRAPH Asia), 38(4), 2019. doi: 10. 1145/3306346 3323010.
[9] Dan Casas and Miguel Otaduy. Learning Nonlinear Soft-Tissue Dynamics for Interactive Avatars. Proc. of the ACM on Computer Graphics and Interactive Techniques, I(1), 2018. doi: 10. 1145/3201887
[10] Xu Chen, Yufeng Zheng, Michael J Black, Otmar Hilliges, and Andreas Geiger. SNARE: Differentiable Forward Skinning for Animating Rin-R Neural Implicit Shapes. In Proc. of IEEE International Conference on Computer Vision (ICCV), 2021.
[11] Zhiqin Chen and Hao Zhang. Learning Implicit Fields for Generative Shape Modeling. In Proc. of Computer Vision and Pattern Recognition (CVPR), 2019.
[12] Julian Chibane, Thiemo Alldieck, and Gerard Moll. Implicit functions feature space for 3d shape reconstruction and completion. In Proc. of Computer Vision and Pattern Recognition (CVPR), 2020.
[13] Enric Corona, Albert Pumarola, Guillem Alenyå, Gerard Pons-Moll, and Francesc Moreno- Noguer. SMPLicit: Topology- Generative Model for Clothed People. In Proc. of Computer Vision and Pattern Recognition (CVPR), 2021.
[14] Aiyu Cui, Daniel McKee, and Svetlana Lazebnik. Dressing order: generation for pose transfer, virtual try-on and outfit editing. In Proc. of IEEE International Conference on Computer Vision (ICCV), pages 14638-14647, October 2021.
Recurrent person image
[15] Boyang Deng, John P Lewis, Timothy Jeruzalski, Gerard Pons-Moll, Geoffrey Hinton, Moham- mad Norouzi, and Andrea Tagliasacchi. NASA: Neural Articulated Ahape Approximation. In Proc. of European Conference on Computer Vision (ECCV), 2020.
[16] Haoye Dong, Xiaodan Liang, Xiaohui Shen, Bochao Wang, Hanjiang Lai, Jia Zhu, Zhiting Hu, and Jian Yin. Towards multi-pose guided virtual try-on network. In Proc. of IEEE International Conference on Computer Vision (ICCV), pages 9025-9034, 2019. doi : 10. 00/./////..... ICCV.2019. 00912.
[17] Zijian Dong, Chen Guo, Jie Song, Xu Chen, Andreas Geiger, and Otmar Hilliges. PINA: Learning Personalized Implicit Neural Avatar from Single RGB-D Video Sequence. In Proc. of Computer Vision and Pattern Recognition (CVPR), 2022.
[18] Yilun Du, Yinan Zhang, Hong-Xing Yu, Joshua B. Tenenbaum, and Jiajun Wu. Neural radiance flow for 4d view synthesis and video processing. In Proc. of IEEE International Conference on Computer Vision (ICCV), 2021.

---

[19] Guy Gafni, Justus Thies, Michael Zollhöfer, and Matthias Nießner. Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction. In Proc. of Computer Vision and Pattern Recognition (CVPR), 2021.
[20] Chongjian Ge, Yibing Song, Yuying Ge, Han Yang, Wei Liu, and Ping Luo. Disentangled cycle consistency for highly-realistic virtual try-on. In Proc. of Computer Vision and Puttern Recognition (CVPR). 2021.
[21] Yuying Ge, Yibing Song, Ruimao Zhang, Chongjian Ge. Wei Liu, and Ping Luo. Parser-free virtual try-on via distilling appearance flows. Proc. of Computer Vision and Pattern Recognition (CVPR), 2021.
[22] Peng Guan, Loretta Reiss, David A Hirshberg, Alexander Weiss, and Michael J Black. DRAPE: DRessing Any PErson. ACM Transactions on Graphics (Proc. SIGGRAPH), 31(4), 2012. doi: 1145/2185520 2185531.
[23] Erhan Gundogdu, Victor Constantin, Amrollah Seifoddini, Minh Dang, Mathieu Salzmann, and Pascal Fua. GarNet: two-stream network for fast and accurate 3D cloth draping. In Proc. of IEEE International Conference on Computer Vision (ICCV), 2019. doi :00:10.00 101111099/C00 /IC/// 2019. 00883.
[24] Xintong Han, Zuxuan Wu, Zhe Wu, Ruichi Ya, and Larry S Davis. VITON: An Image-base Virtual Try-on Network. In Proc. of Computer Vision and Pattern Recognition (CVPR), 2018.
[25] Tong He, John Collomosse, Hailin Jin, and Stefano Soatto. Geo-pifu: Geometry and pixel aligned implicit functions for single-view human reconstruction. In Advances in Neural Infor- mation Processing Systems (NeurIPS), pages 9276-9287, 2020.
[26] Anna Hilsmann, Philipp Fechteler, and Peter Eisert. Pose space image based rendering. Com- puter Graphics Forum, 32, 2013. doi hattps://don org/ 10. 1111/cgf. 12046.
[27] Zeng Huang, Yuanlu Xu, Christoph Lassner, Hao Li, and Tony Tung. ARCH: Animatable Reconstruction of Clothed Humans. In Proc. IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 3093-3102, 2020.
[28] Thibaut Issenhuth, Jérémie Mary, and Clément Calauzènes. Do not mask what you do not need mask: a parser free virtual try-on. Proc. of European Conference on Computer Vision (ECCV), 2020.
[29] Hanbyul Joo, Tomas Simon, and Yaser Sheikh. Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies. In Proc. of Computer Vision and Pattern Recognition (CVPR), 2018.
[30] Korrawe Karunratanakul, Adrian SpurT, Zicong Fan, Otmar Hilliges, and Siyu Tang. A skeleton- driven neural occupancy representation for articulated hands. In International Conference on 3D Vision (3DV), 2021.
[31] Korrawe Karunratanakul, Jinlong Yang. Yan Zhang. Michael J Black, Krikamol Muandet, and Siyu Tang. Grasping field: Learning implicit representations for human grasps. In International Conference on 3D Vision (3DV), pages 333-344, 2020.
[32] Youngjoong Kwon, Dahun Kim, Duygu Ceylan, and Henry Fuchs. Neural human performer: Learning generalizable radiance fields for human performance rendering. Advances in Neural Information Processing Systems (NeurIPS), 2021.
[33] Hyug Jae Lee, Rokkyu Lee, Minseok Kang, Myounghoon Cho, and Gunhan Park. La-viton: A network for looking-attractive virtual try-on. In Proc. of IEEE International Conference on Computer Vision (ICCV) Workshops, Oct 2019.
[34] Kedan Li, Min Jin Chong, Jingen Liu, and David Forsyth. Toward accurate and realistic outfits visualization with attention to details. Proc. of Computer Vision and Puttern Recognition (CVPR), 2021.
12

---

[35] Zhe Li, Tao Yu, Chuanyu Pan, Zerong Zheng, and Yebin Liu. Robust dd self self-portraits in seconds. In Proc. of Computer Vision and Pattern Recognition (CVPR), pages 1344-1353, 2020.
[36] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. SMPL: A skinned multi-person linear model. ACM Transactions on Graphics (Proc. SIGGRAPH Asia), 1-66) 2015. doi 10. 0011/111112991 2818013.
[37] Qianli Ma, Jinlong Yang, Anurag Ranjan, Sergi Pujades, Gerard Pons-Moll, Siyu Tang, and Michael J. Black. Learning to Dress 3D People in Generative Clothing. In Proc. of Computer Vision and Puttern Recognition (CVPR), 2020.
[38] Meysam Madadi, Hugo Bertiche, Wafa Bouzouita, Isabelle Guyon, and Sergio Escalera. Learn- ing Cloth Dynamics: 3D+Texture Garment Reconstruction Benchmark. In Hugo Jair Escalante and Katja Hofmann, editors, Proc. of the NeurlPS 2020 Competition and Demonstration Track, volume 133 of Proceedings of Machine Learning Research, pages 57-76. PMLR, 06-12 Dec 2021.
[39] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy Networks: Learning 3D Reconstruction in Function Space. In Proc. of Computer Vision and Pattern Recognition (CVPR), 2019.
[40] Mateusz Michalkiewicz, Jhony K Pontes, Dominic Jack, Mahsa Baktashmotlagh, and Anders Eriksson. Deep level sets: Implicit surface representations for 3d shape inference. In Proc. of IEEE International Conference on Computer Vision (ICCV), 2021.
[41] Marko Mihajlovic, Yan Zhang, Michael Black, and Siyu Tang. LEAP: Learning Articulated Occupancy of People. In Proc. of Computer Vision and Pattern Recognition (CVPR), 2021.
[42] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In Proc. of European Conference on Computer Vision (ECCV), 2020.
[43] Rahul Narain, Armin Samii, and James F O'brien. Adaptive Anisotropic Remeshing for Cloth Simulation. ACM Transactions on Graphics (Proc. SIGGRAPH Asia), 31(6):1-10, 2012. doi: 110.11114222661 2366171.
[44] Assaf Neuberger, Eran Borenstein, Bar Hilleli, Eduard Oks, and Sharon Alpert. Image based virtual try-on network from unpaired data. In Proc. of Computer Vision and Pattern Recognition (CVPR), pages 5 5183-5192, 2020. doi:10. L009///P42200000.. 2020. 2020.00523.
[45] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Occupancy Flow: 4D Reconstruction by Learning Particle Dynamics. In Proc. of IEEE International Conference on Computer Vision (ICCV), 2019.
[46] Atsuhiro Noguchi, Xiao Sun, Stephen Lin, and Tatsuya Harada. Neural articulated radiance field. In Proc. of IEEE International Conference on Computer Vision (ICCV), 2021.
[47] Joseph Ortiz, Alexander Clegg. Jing Dong, Edgar Sucar, David Novotny, Michael Zollhoefer, and Mustafa Mukadam. isdf: Real-time neural signed distance fields for robot perception. In Robotics: Science and Systems (RSS), 2022.
[48] Pablo Palafox, Aljaz Bozic, Justus Thies, Matthias NieBner, and Angela Dai. Neural parametric models for 3d deformable shapes. In Proc. of IEEE International Conference on Computer Vision (ICCV), 2021.
[49] Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin-Brualla Nerfies: Deformable neural radiance fields. Proc. of IEEE International Conference on Computer Vision (ICCV), 2021.
[50] Chaitanya Patel, Zhouyingcheng Liao, and Gerard Pons-Moll. The Virtual Tailor: Predicting Clothing in 3D as Function of Human Pose, Shape and Garment Style. Proc. of Computer Vision and Pattern Recognition (CVPR), 2020.
13

---

[51] Sida Peng. Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou, Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In Proc. of Computer Vision and Puttern Recognition ( (CVPR), pages 9054-9066 2021.
[52] Gerard Pons-Moll, Sergi Pujades, Sonny Hu, and Michael J. Black. ClothCap: Seamless 4D clothing capture and retargeting. ACM Transactions on Graphics (Proc. SIGGRAPH), 36(4), 2017. doi: 10. http//3372999..073777
[53] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-NeRF: Neural Radiance Fields for Dynamic Scenes. In Proc. of Computer Vision and Pattern Recogni- lion (CVPR), 2021.
[54] Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo. PIFuHD: Multi-Level Pixel- Aligned Implicit Function for High-Resolution 3D Human Digitization. In Proc. of Computer Vision and Pattern Recognition (CVPR), 2020.
[55] Shunsuke Saito, Jinlong Yang. Qianli Ma, and Michael J. Black. SCANimate: Weakly super- vised learning of skinned clothed avatar networks. In Proc. of Computer Vision and Puttern Recognition (CVPR), 2021.
[56] Igor Santesteban, Elena Garces, Miguel A. Otaduy, and Dan Casas. SoftSMPL: Data-driven Modeling of Nonlinear Soft-tissue Dynamics for Parametrie Humans. Computer Graphics Forum (Proc. Eurographics), 39(2), 2020. doi 10. 1111/cgf. 13912.
[57] Igor Santesteban, Miguel A. Otaduy, and Dan Casas. Learning -Based Animation of Clothing for Virtual Try-On. Computer Graphics Forum (Proc. Eurographics), 38(2), 2019. doi:
10. 10.1111/cgff 13643.
[58] Igor Santesteban, Miguel A Otaduy, and Dan Casas. SNUG: Self-Supervised Neural Dynamic Garments. Proc. of Computer Vision and Pattern Recognition (CVPR), 2022.
[59] Igor Santesteban, Nils Thuerey, Miguel A Otaduy, and Dan Casas. Self-Supervised Collision Handling via Generative 3D Garment Models for Virtual Try-On. Proc. of Computer Vision and Pattern Recognition (CVPR), 2021.
[60] Yu Shen, Junbang Liang, and Ming C. Lin. GAN-based Garment Generation Using Sewing Pattern Images. In Proc. of European Conference on Computer Vision (ECCV), 2020.
[61] Anthony Simeonov, Yilun Du, Andrea Tagliasacchi, Joshua B. Tenenbaum, Alberto Rodriguez, Pulkit Agrawal, and Vincent Sitzmann. Neural Descriptor Fields: SEuia-Equivariant Object Representations for Manipulation, 2021. arXiv: 21112.2211..... 05124.
[62] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3d-structure-awe aware neural scene representations. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[63] Shih- Shih-Yang Su, Frank Yu. Michael Zollhöfer, and Helge Rhodin. A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[64] Zhuo Su, Lan Xu, Zerong Zheng, Tao Yu, Yebin Liu, and Lu Fang. RobustFusion: Hu- man Volumetric Capture with Data-Driven Visual Cues Using a RGBD Camera. In Proc. of European Conference on Computer Vision (ECCV), page 246-264, 2020. doi 10. 978-3-030-58548- 8_1.. 15.
[65] Edgar Sucar, Shikun Liu, Joseph Ortiz, and Andrew J. Davison. iMAP: Implicit Mapping and Positioning in Reme- In Proc. of IEEE International Conference Computer Vision (ICCV), pages 6229-6238, 2021.
[66] Qingyang Tan, Zherong Pan, and Dinesh Manocha. Lcollision: Fast generation of collision-free human poses using learned non- con-eenttattion constraints. Proceedings of the AAAI Conference on Artificial Intelligence, 35(5):3913-392) 2021. doi : 10.. 1609/ awao//aaai../. v3515 16510.
14

---

[67] Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich- Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[68] Raquel Vidaurre, Igor Santesteban, Elena Garces, and Dan Casas. Fully Convolutional Graph Neural Networks for Parametric Virtual Try-On. Computer Graphics Forum (Proc. SCA), 39(8), 2020. doi: 10. 1111/cgf..... 14109.
[69] Bochao Wang, Huabin Zheng, Xiaodan Liang, Yimin Chen, Liang Lin, and Meng Yang. Toward Staracteristic-presevvy image-based virtual try-on network. In Proc. of European Conference on Computer Vision (ECCV), pages 589-604, 2018.
[70] Jiahang Wang, Tong Sha, Wei Zhang, Zhoujun Li, and Tao Mei. Down to the last detail: Virtual try-on with fine-grained details. In Proc. of ACM International Conference on Multimedia, page 466-474, 2020. doi: 10. 0145///394111.3111391
[71] Shaofei Wang, Andreas Geiger, and Siyu Tang. Locally Aware Piecewise Transformation Fields for 3D Human Mesh Registration. In Proc. of Computer Vision and Pattern Recognition (CVPR), 2021.
[72] Shaofei Wang. Marko Mihajlovic, Qianli Ma, Andreas Geiger, and Siyu Tang. MetaAvatar: Learning Animatable Clothed Human Models from Few Depth Images. In Advances Neural Information Processing Systems (NeurIPS), 2021.
[73] Tuanfeng Y Wang, Duygu Ceylan, Jovan Popović, and Niloy J Mitra. Learning Shared Shape Space for Multimodal Garment Design. ACM Transactions on Graphics (Proc. SIGGRAPH Asia), 37(6), 2018, doi: 100...11553222220 3275074.
[74] Holger Wendland. Scattered data approximation. volume 17. Cambridge university press, 2004.
[75] Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural Fields in Visual Computing and Beyond. Computer Graphics Forum, 2022. doi: 10. 11f.//gf...f.... 14505.
[76] Hongyi Xu, Thiemo Alldieck, and Cristian Sminchisescu. H-nerf: Neural radiance fields for rendering and temporal reconstruction of humans in motion. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[77] Gengshan Yang, Minh Vo, Neverova Natalia, Deva Ramanan, Vedaldi Andrea, and Joo Hanbyul. BANMo: Building Animatable 3D Neural Models from Many Casual Videos. In Proc. of Computer Vision and Pattern Recognition (CVPR), 2022.
[78] Guandao Yang, Serge Belongie, Bharath Hariharan, and Vladlen Koltun. Geometry Processing with Neural Fields. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[79] Han Yang. Ruimao Zhang, Xiaobao Guo, Wei Liu, Wangmeng Zuo, and Ping Luo. Towards reato-re virtual try-on by adaptively generating-preserving image content. In Proc. of Computer Vision and Pattern Recognition (CVPR), 2020.
[80] Ze Yang, Shenlong Wang, Siva Manivasagam, Zeng Huang, Wei-Chiu Ma, Xinchen Yan, Ersin Yumer, and Raquel Urtasun. S3: Neural Shape, Skeleton, and Skinning Fields for 3D Human Modeling. In Proc. of Computer Vision and Pattern Recognition (CVPR), 2021.
[81] Ruiyun Yu, Xiaoqi Wang, and Xiaohui Xie. Vinfp: An image-based virtual try-on network with body and clothing feature preservation. In Proc. of IEEE International Conference on Computer Vision (ICCV), 2019.
[82] Tao Yu, Zerong Zheng, Kaiwen Guo, Pengpeng Liu, Qionghai Dai, and Yebin Liu. Function4D: Real-time Human Volumetric Capture from Very Sparse Consumer RGBD Sensors. In Proc. of Computer Vision and Pattern Recognition (CVPR), 2021.
[83] Ryan& S. Zesch, Bethany R. Witemeyer, Ziyan Xiong, David I. W. Levin, and Shinjiro Sueda. Neural Collision Detection for Deformable Objects, 2022. arXiv: 2202, 02309.
15

---

[84] Meng Zhang, Tuanfeng Y. Wang, Duygu Ceylan, and Niloy J. Mitra. Dynamic Neural Garments.
ACM Transactions on Graphics (Proc. SIGGRAPH Asia), 40(6), 2021.
[85] Zerong Zheng, Tao Yu, Yebin Liu, and Qionghai Dai. PaMIR: Parametric Model-Conditioned Implicit Representation for Image-based Human Reconstruction. IEEE Transactions on Pattern Analysis and Machine Intelegence, 2021.
16


Reviews:
[{"review_id": "aUzi11DlHyL", "reviewer": "Reviewer_Czdg", "summary": "The paper proposed a framework for  virtual try-on with multiple garments. The idea is first encode the the open surface implicitly with signed distance field and covariant field, and to combine multiple garment with collision free condition, the paper further proposed a formulation to use neural network to predict the untangle layered implicit surfaces, comparing with previous work (Buffet et al. [8]) which uses optimization process to achieve collision free. The final algorithm is formalizing the VTO as an optimization problem (Eq. 10), which needs to achieve both collision free and minimize distortions of the triangles. Qualitatively, the results achieves better performance comparing with previous works on VTO. ", "questions": "See Weakness section above", "limitations": "See Weakness section above", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n1. The paper is well written and easy to follow. \n\n2. The final performance on VTO looks better than baselines, qualitatively speaking. \n\n\nWeakness\n\n1. Novelty. As summarized above (and also in the paper), there're two new things proposed in the paper: a) A neural field formulation to encode open surfaces, b)  utilizing a network to predict the untangled distance field, instead of optimization procedure. However, are these two novel? Why using them is correct choice? Regarding the encoding of open surfaces, there're already many published works [a, b], why the proposed method is better than these work? The paper didn't provide an explanation on this, nor comparisons. Regarding using a network to predict untangled distance field, this is a new approach to me, however, Buffet et al. [8] can solve the fundamental limitation mentioned in the abstract \"the interaction between layered neural fields.\", to me, the benefit is only using a neural network is much faster than [8], are there other advantages?\n\n2. Limitation. There are a bunch of limitation of proposed methods:\n\n- The experiment is only designed for canonical pose, extending to arbitrary poses is not clear (It's not as simple as mentioned in the Line 160-161), different poses will make the garment fitting more complicated,  and ULNef module might also have generalization issue.\n- The ULNef is limited to a fixed number of garment, the input is N dimension vector and output is also N dimension vector, how does it generalize to different number of garments?\n- The model would require to fit each module for each garment, how could this scale up to real world application, where we can have thousands of different clothes. \n- Generalization of ULNef. It's hard to say whether ULNef can generalize to different garments, e.g. trained one one set of garment, but tested on a different set of garments (there's also no analysis of this)\n\n3. Expeirment. The experiment has many problems:\n\n- The biggest issue for experiment is the model is only trained and tested on 5 garments (See Supplement Sec 1), this is very limited in evaluating the performance to check the whether the result can generalize in a larger scale. \n- It would be better to ablate each module proposed in the framework. I think there are missing at least two baselines for comparisons: i) To show the effect of proposed idea of encoding open surfaces, it would be better to compare at least one of [a], [b]. ii) To show the effect of ULNef, one baseline is not using a neural network to predict $\\textbf{f}$, but use [8] to get $\\textbf{f}$\n\n \nIn summary, this paper has many problems in both the method and experiment side, and I think it's not ready for publication at current status, so I vote for a reject. \n\n[a] NEURAL UNSIGNED DISTANCE FIELDS FOR IMPLICIT FUNCTION LEARNING\nJulian Chibane, Aymen Mir, Gerard Pons-Moll\nNeurIPS 2020\n\n[b] 3PSDF: Three-Pole Signed Distance Function for Learning Surfaces with Arbitrary Topologies\nWeikai Chen, Cheng Lin, Weiyang Li, Bo Yang\nCVPR 2022\n\n[POST REBUTTAL]\nAfter reading the rebuttal and other reviews, I'm convinced of the results provided in the paper, but I still do not think the novelty is great enough for neurips, so I only raised my score to boardline accept. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "U-Fo418W9eY", "reviewer": "Reviewer_LMYH", "summary": "Based on recent success on neural cloth simulation for a single garment, the authors extend previous works to solve multi-layer cloth prediction while avoiding collisions. The author leverages the untangling power of the implicit surface model from Buffet et al. and neural cloth representation from Santesteban et al., designing a new neural untangling layer directly predicting collision-resolved multi-layer cloth pieces. The authors have shown a cool demo in the video that they can directly predict multi-layer cloth on the human bodies while tuning the human height and width.", "questions": "1. Sec 4.2 and Sec. 3.2 both are inspired by numerical optimization, why not also trained a network to replace the optimization in  Sec. 4.2?\n\n2. In Eq. 9 the X(beta) is already dependent on shape parameters, why also need a diffusion step? Directly train a network to replace Mg(x, beta) maybe better?", "limitations": "The work is only evaluated on simplified canonical space.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pro:\n\n1. To extend a neural network-based virtual try-on system to a multi-layer cloth system is very novel and ambitious. The method utilizes a neural implicit representation of cloth and uses an optimization-inspired method to untangle different cloth pieces. The authors have shown cool visualization results that they can handle up to ~5 layers of cloth pieces stacking on a human body. In the ablation study, they also include results that using Fourier features and gradient loss can generate a better implicit representation of the garment model.\n\n2. Another exciting part of this paper is the authors have built a comprehensive pipeline for the mix-and-match VTO task. From neural explicit/implicit representation to untangling, to find the final zero-set surface using optimization. It's not easy to fuse several challenging components to a final runnable pipeline.\n\nCon:\n\n1. Sec 4.2 about optimization of untangled garments is not well-written and very unclear. In line 242, they mentioned that \"evaluating per-garment fields, followed by the neural projection (untangling I guess?)\", not sure if it is only queried once to find the zero-level surface or done repeatedly during the optimization. Also, it is unclear if it is solving the {X_i} with all the pieces together or only finding layer by layer. I suggest the authors add an algorithm section to detailly include how they combine L-BFGS and the untangling operator.\n\n2. Current results are limited by the evaluation, the authors simplified the problem to T-pose (with only height, and width), without pose parameters.\n\nSome related work:\nCollision avoidance based on neural network and optimization:\n LCollision: Fast Generation of Collision-Free Human Poses using Learned Non-Penetration Constraints\nQ Tan, Z Pan, D Manocha\nAAAI Conference on Artificial Intelligence", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Ds-hHppxTd", "reviewer": "Reviewer_U9t8", "summary": "This paper presents ULNef, which leverages layered neural fields to create collision-free clothing surfaces for mix-and-match virtual try-on. ", "questions": "- I encourage the authors to add visual results of ablations and failures cases.\n\n- Comparing with pure cloth simulation methods would be better provided.", "limitations": "The authors have discussed several limitations of the proposed method. I am interested in looking at some visual failure cases and more in-depth analysis.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strength*\n\n- This paper aims to resolve an interesting problem that how to fit multi-layer garments to a target body shape while avoiding colliding among garment layers. The authors propose a novel model that untangles the layered neural fields with neural implicit representations (i.e., a signed distance field and a covariant field). This potentially introduces a new paradigm for solving related problems.\n\n- The visual results clearly verify the effectiveness of the proposed method.\n\n- The way of obtaining SDF from open surfaces is interesting and has proven to be effective.\n\n- I like the pipeline of solving layered VTO and the results are promising.\n\n*Weaknesses*\n\n- Only a quantitative ablation study is given in Table 1. A visual comparison should be given for understanding how the Fourier features and gradient supervision contribute to reducing the garment collision qualitatively.\n\n- The authors briefly discussed the limitations without showing visual failure examples. In my opinion, illustrating some failure cases (for example, how does the method work on a posed body, or extreme body shapes say very large or small $\\beta$? Will the results look bad?) can help to guide future research directions.\n\n- Can the mix-and-match VTO be directly achieved by using cloth simulation? Is it possible to compare the proposed method with its simulation counterparts?\n\n- Minor: Figure 1 could be more informative. For now, it is hard to make sense out of this simple diagram.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8mQZjCAb1N1", "reviewer": "Reviewer_q8FC", "summary": "This work presents a novel approach to mix-and-match 3D virtual try-on.  The proposed model is able to untangle layered neural fields (how garments are represented) by introducing a projection operator that works directly on neural implicit representations. The composition of the layered neural field with the neural projection is what the authors refer to as a ULNeF.  At runtime, the model is able to take as input a representation of the body shape and optimize for a layering of the garments by using ULNeFs to resolve collisions.", "questions": "Are there any failure cases with the ULNeF model? For example, are there any types of clothing with certain features that would prove problematic for this approach?\n\nWhat would happen if the ordering of the garments did not make sense? For example, if one were to layer the pants on top of the dress?\n\n[159-161] Have the authors performed any experiments with parameterizing additional codes, other than body shape?\n", "limitations": "The authors have acknowledged that the proposed method has only been demonstrated on a single pose (T-pose).  The work is based on simulated models, and does not involve real humans or garments, so potential negative societal impact is lessened..", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Mix-and-match is a very challenging problem in VTO, and the authors demonstrate results on up to 7 layers of garments, which is quite impressive.\n\n* ULNeFs could be applicable to other object-object interaction problems, as the authors point out.\n\n* The neural model introduced that learns the untangling operation results in significant efficiency gains (“up to two orders of magnitude faster” [258-259]).\n\n* The authors provide sufficient details about implementation, such that this work would be reproducible by others.\n\nWeaknesses:\n* No comprehensive quantitative evaluation is performed to measure the quality of the mix-and-match results, besides the ablation study of terms and encodings used for training the implicit surface model.  Only a handful of qualitative examples are shown, though the video in the supplemental material shows a compelling interactive demo.\n\n* The authors acknowledge that they have only validated their work on a single pose, using simulated models and clothing.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept