Title: Greatness in Simplicity: Unified Self-Cycle Consistency for Parser-Free Virtual Try-On
Venue: NeurIPS 2023
Accepted: yes
Keywords: parser-free virtual try-on

self-cycle consistency

human analysis and understanding

fashion synthesis

Markov Random Field
Abstract: Image-based virtual try-on tasks remain challenging, primarily due to inherent complexities associated with non-rigid garment deformation modeling and strong feature entanglement of clothing within human body. Recent groundbreaking formulations, such as in-painting, cycle consistency, and knowledge distillation, have facilitated self-supervised generation of try-on images. However, these paradigms necessitate the disentanglement of garment features within human body features through auxiliary tasks, such as leveraging 'teacher knowledge' and dual generators. The potential presence of irresponsible prior knowledge in the auxiliary task can serve as a significant bottleneck for the main generator (e.g., 'student model') in the downstream task. Moreover, existing garment deformation methods lack the ability to perceive the correlation between the garment and the human body in the real world, leading to unrealistic alignment effects. To tackle these limitations, we present a new parser-free virtual try-on network based on unified self-cycle consistency (USC-PFN), which enables robust translation between different garments using just a single generator, faithfully replicating non-rigid geometric deformation of garments in real-life scenarios. Specifically, we first propose a self-cycle consistency architecture with a circular mode. It utilizes real unpaired garment-person images exclusively as input for training, effectively eliminating the impact of irresponsible prior knowledge at the model input end. Additionally, we formulate a Markov Random Field to simulate a more natural and realistic garment deformation. Furthermore, USC-PFN can leverage a general generator for self-supervised cycle training. Experiments demonstrate that our method achieves state-of-the-art performance on a popular virtual try-on benchmark.

Greatness in Simplicity: Unified Self-Cycle Consistency for Parser-Free Virtual Try-On

Abstract

Image-based virtual try-on tasks remain challenging, primarily due to inherent complexities associated with non-rigid garment deformation modeling and strong feature entanglement of clothing within human body. Recent groundbreaking formulations, such as in-painting, cycle consistency, and knowledge distillation, have facilitated self-supervised generation of try-on images. However, these paradigms necessitate the disentanglement of garment features within human body features through auxiliary tasks, such as leveraging 'teacher knowledge' and dual generators. The potential presence of irresponsible prior knowledge in the auxiliary task can serve as a significant bottleneck for the main generator ( e.g. , 'student model') in the downstream task. Moreover, existing garment deformation methods lack the ability to perceive the correlation between the garment and the human body in the real world, leading to unrealistic alignment effects. To tackle these limitations, we present a new parser-free virtual try-on network based on unified self-cycle consistency ( USC-PFN ), which enables robust translation between different garments using just a single generator, faithfully replicating non-rigid geometric deformation of garments in real-life scenarios. Specifically, we first propose a self-cycle consistency architecture with a circular mode. It utilizes real unpaired garment-person images exclusively as input for training, effectively eliminating the impact of irresponsible prior knowledge at the model input end. Additionally, we formulate a Markov Random Field to simulate a more natural and realistic garment deformation. Furthermore, USC-PFN can leverage a general generator for self-supervised cycle training. Experiments demonstrate that our method achieves state-of-the-art performance on a popular virtual try-on benchmark.

1 Introduction

To provide consumers with an online try-on experience similar to physical stores, researchers have focused on image-based virtual try-on (VTON), allowing individuals to virtually try on garments available in online shops. The mainstream image-based approach of VTON involves the replacement of a person's clothing area with an in-shop garment image. However, all current datasets are paired, meaning one person corresponds to only one piece of clothing they are wearing. To capture underlying correlations between unpaired images from random combinations in the dataset, three architectures have been used: in-painting methods [8, 3, 9, 10, 4, 11-14], knowledge distillation methods [11, 6, 7], and cycle consistency methods[15, 2] (see Figure 1).

Real

Person

Garment

Masked

Person

Fake

Person

(c) Knowledge Distillation (Two-way)

Real

Person

Fake

Person

Garment 1

Garment 2

Real

Person

Fake

Person

Garment 1

Garment 2

+R

+R

+R

Pseudo

Person

Garment

+R

Fake

Person

T

S

G

T

S

G2

G

Masked Person

Real Person

pre-trained

(b) In-Painting (One-way)

(d) Self-Cycle Consistency (Ours)

A

The in-painting-based method, inspired by [16], addresses variations in garment styles by masking the upper body of the person image and training the model to learn the reproduction of masked regions. This paradigm uses a person representation ( e.g. , semantic map) as input to provide masked body information; however, slight errors in the person representation can result in occluded and unrealistic outcomes. Moreover, this paradigm faces challenges when dealing with cases where the same person tries on different garments due to their reliance on one-to-one paired garment-person images during training (see Figure 2). Therefore, the knowledge distillation-based structure [11] is proposed, which pretrains an in-painting-based teacher model to provide prior try-on result images for training the student model. This paradigm introduced a new pipeline that enables self-supervised training using only unpaired images as input. However, the potential presence of irresponsible teacher knowledge can still reproduce some issues of in-painting methods. Additionally, a method is proposed in [1, 2] that employs two CNNs to achieve the cycle training. In this approach, the output of CNN 1 as input is fed to CNN 2 and then minimizes the difference between the output of CNN 2 and the input of CNN 1 to achieve cycle consistency. However, the complex dual structure poses challenges in network convergence, and reliance on person representation throughout the entire network hinders the generation of high-quality virtual try-on results.

In this task, Thin Plate Splines (TPS) [17, 18, 8, 3, 10, 4, 11, 2, 12] is commonly used to deform garments. However, TPS focuses only on pixel transformations around the control point, often causing excessive distortion. Another non-rigid deformation technique, called Appearance Flow (AF) [19, 9, 6, 7, 14], predicts a pixel-level displacement field of garments for pixel-by-pixel transformations. However, it lacks spatial prior knowledge, resulting in an inaccurate perception of the spatial relationship between the garment and the human body. Recently, the Moving Least Squares (MLS) [20, 13] has shown promise in handling deformation for simple poses. However, there are difficulties in handling complex poses. These issues can lead to misalignment, hindering accurate simulation of the natural non-rigid deformation that occurs during the interaction between the garment and the body (see Figure 2).

Therefore, we propose a virtual parser-free try-on network based on self-cycle consistency to address the aforementioned limitations. First, we explore the relationship between garments and the human body. We use an auxiliary network to disentangle irrelevant features such as texture, color, and shape in the latent feature space of the human body. This approach preserves valuable features like depth and lighting while removing entanglements with unrelated features. Consequently, the network can authentically simulate the interaction between different garments and the human body in real-life scenarios, without being affected by the garment it wears. Furthermore, it has been confirmed that the person representation appearing at the input end of the network has potential negative impacts on performance [11, 6, 7]. Moreover, we argue that there is no inherent translation between multiple

domains in virtual try-on tasks. Therefore, we introduce a shared-weight circular architecture where the parser is shifted from the input end to the output end, specifically for loss computation. This approach allows us to bypass the parser during the inference stage. The main contributions of our paper can be summarized as follows:

We propose USC-PFN, a novel parser-free virtual try-on network that generates highly realistic try-on results using only unpaired data as input and a single CNN as the generator, offering a new perspective for the virtual try-on task.

Wefirst leverage the Markov Random Field to estimate a deformation field to establish precise dense pixel-level correspondences between the in-shop garment and the person. This is a new approach to address the misalignment caused by a lack of spatial perception, and simulate the non-rigid deformation and interaction of the garment in real-life scenarios.

We propose a concise and effective self-cycle consistency paradigm for the parser-free virtual try-on task, and extensive experiments on the popular benchmark show that our framework outperforms state-of-the-art methods both qualitatively and quantitatively.

2 Related Work

Parser-Free Virtual Try-On The formulations mentioned earlier (see Figure 1) can be subdivided based on whether the parser is used in image-to-image translation [21, 16]. It has been demonstrated that the parser can improve try-on results but may introduce unreliable semantic errors [11, 6, 7]. Initially, [8, 3] used pose heatmap and body mask as person representation to provide body's geometry information. Later, [10] introduced a parser to estimate an "after-try-on" semantic map instead of the above person representation. To avoid unreliable semantic information, [11, 6, 7] trained a "student model" without the parser using pretrained teacher knowledge. However, the potential presence of irresponsible teacher knowledge can still affect the "student model," making it challenging to train robust architectures without

DCTON

FS-VTON

RT-VTON

Ours

Garment

TPS

AF

MLS

MRF

Cycle

Consistency

Knowledge

Distillation

In-Painting

Self-Cycle

Consistency

Formulation

Deformation

the parser in the input end. To address this challenge, we present a parser-free pipeline called self-cycle consistency. Our goal aims to achieve high-quality virtual try-on results using a general network, minimum input, and simple but efficient architecture.

Non-rigid Deformation In virtual try-on tasks, common non-rigid image deformation methods include Thin Plate Spline (TPS) [17, 18], Moving Least Squares (MLS) [20], and Appearance Flow (AF) [19]. As shown in Figure 2, TPS is the most widely used due to its ability to provide high-quality deformation while preserving the garment's image structure [8, 3, 10, 4, 11, 2, 12]. However, TPS is limited by its local mode, which only significantly affects pixels near control points. MLS is another method introduced recently, but it is not suitable for large-scale deformation [13]. AF is the most flexible method, deforming the garment by estimating pixel displacement vectors between the source and target shapes [9, 6, 7, 14]. However, none of these methods can fully mimic the natural interaction between the garment and the human body due to a lack of spatial perception. Our proposed method, which is based on Markov Random Field system with unpaired images as input, can obtain realistic spatial deformation.

Self-Cycle Consistency CycleGAN [1] utilized a self-supervised learning framework for cyclically learning the translation between unpaired images from different domains. [2] extended this with a deconstruction method for different garment styles to address the failure of cycle consistency in virtual try-on tasks. However, the complex dual-model structure and reliance on the parser hinder the generation of realistic results. To break through these limitations, following StarGAN [22], we propose a single-model self-cyclic learning translation paradigm between different garment styles.

Garment 2

Supervision

Deformed

Garment 2

Real

Person 1

Fake

Person 2

Pseudo

Person 2

Deformed

Garment 1

Fake

Person 1

G

φ

Fake / Real

Ds

G

θ

Non-rigid Garment

Deformer

G

A

Auxiliary Deformer

f

Deformation Field

G

φ

Self-cyclic Image

Generator

Discriminator

Pre-trained

Supervision

EPS

f

Deformed

Garment

GT

f

Deformed

Garment 2

Real

Person 1

Garment 1

Real

Person

: Supervision

BI

BI

BI

: Bilinear

Interpolation

G

θ

G

A

G

θ

G

φ

WS

Weight Sharing

WS

3 Methodology

Notation. In this work, we use ( g , p ) to represent paired garment-person images where the person p is wearing garment g . Moreover, we randomly select the garment image g ′ to form unpaired garment-person images ( g ′ , p ) with p . The try-on result of ( g , p ) and ( g ′ , p ) are denoted as ˆ p and ˆ p ′ , respectively. In general, " ′ " represents the target image, and " ˆ " denotes the transformation result. In addition, " ↦→ " represents the mapping relationship of the entire architecture, while " ⇒ " represents the mapping relationship of the sub-modules.

Overview. The objective of the virtual try-on task is to naturally dress an arbitrary in-shop garment image g ′ ∈ R 3 × H × W on a reference person image p ∈ R 3 × H × W to obtain a try-on result ˆ p ′ ∈ R 3 × H × W . As illustrated in Figure 3, we first propose a self-cycle consistency framework, USC-PFN, a new virtual try-on paradigm that does not require the human parser and other person representation at the model input end for generating highly realistic try-on images. The framework is broken up into two sub-networks: (i) a non-rigid garment deformer G θ , and (ii) a self-cyclic image generator G φ . They are mainly used to construct a cyclic mapping V t : ( p , g ′ ) ↦→ ˆ p ′ , (ˆ p ′ , g ) ↦→ ˆ p , where p ≈ ˆ p , i.e. , ˆ p is the same to the real image p .

3.1 Formulation

Garments Deformation. In ( p , g ′ ) ↦→ ˆ p ′ , the initially flat garment g ′ must be deformed to match the posture of the person p before synthesizing the target try-on result ˆ p ′ , denotes as ( p , g ′ ) θ ⇒ ˆ g ′ , ( p , ˆ g ′ ) φ ⇒ ˆ p ′ . Our key idea involves using G θ to model the non-rigid deformation based on Markov Random Field (MRF) [23], to seek an optimal spatial transformation solution ˆ f ∗ ∈ R 2 × H × W ( i.e. optimal deformation field ˆ f ∗ = G ∗ θ ( p , g ′ ) ) that minimizes the dissimilarity between the warped garment ˆ g ′ and the garment ˆ p ′ g worn on the target result ˆ p ′ , where ˆ g ′ is obtained by using bilinear interpolation ( δ ): ˆ g ′ = ϱ ( ˆ f , g ′ ) . This can be formalized as minimizing the MRF energy function:

where λ r and λ p are trade-off hyper-parameters. V + represents the prior pseudo-label from the pre-trained garment deformer and ˜ f A is the deformation field of ˆ p ′ g .

where E reg is regularization term of ˆ f , which composed of spatial position weights w ϕ of pixel points and a smoothing term R sm . R sm is used to ensure that each element in ˆ f and its neighborhood have similar values. D [ · , · ] indicates quantification of similarity or difference. During garment deformation, we find that the deformation range of the edge region is often larger than the central region. Therefore, we use the two-dimensional Gaussian distribution [24] with the centroid and variance of pixel coordinates in the cloth region, as w ϕ to maintain the flexibility of the edge region.

In (ˆ p ′ , g ) ↦→ ˆ p , this process is similar to the one above, i.e. , (ˆ p ′ , g ) θ ⇒ ˆ g , (ˆ p ′ , ˆ g ) φ ⇒ ˆ p , but the difference is that we remove E dat from Eq.(1) to ensure more accurate results.

Synthesis of Try-On Results. Recall that the goal of the virtual try-on task is to synthesize the target try-on result after warping the garment as if it was shot from realistic scenes. Therefore, reconstructing the real image p is an essential task to form self-supervised training, which can be formalized as minimizing the following objective:

where V is training set. ˆ p ′ denotes p wearing the garment g ′ . This differentiable generation appears to have completeness, however, the absence of ˆ p ′ in V forces the process to be compensated by extra person representation R (semantic maps, pose heatmaps, etc.) as input. This means that results are heavily constrained by R , and any errors that occur can directly affect the final try-on effect. To tackle this problem, [6, 7] utilized a pre-trained and parser-based "teacher network" T to generate ˆ p ′ :

Yet they cannot handle that irresponsible teacher knowledge brings to the learning process of their "student network" S . [2] introduced the cycle-consistency structure that employed and trained cyclically two CNNs ( e.g. , N 1 and N 2 ), where let N 1 provide ˆ p ′ for N 2 :

where the structural difference between ˆ p and p is minimized to ensure cycle-consistency. However, decomposing the body and then synthesizing it back together based on semantics leads to occlusion and artifacts due to some parsing errors. Relying on prior knowledge weakens the learning of the garment of adaptive spatial structure on the human body. Additionally, the dual-model structure is complex, resulting in longer training times and unstable convergence. Inspired by [22], we reformulate the synthesis process as an unpaired garment style translation problem. Specifically, we provide ˆ p ′ for ourselves ( G φ ) by ourselves (weight-sharing structure):

Inference. During inference, our goal is to synthesize the desired try-on result ˆ p just by feeding arbitrary garment-person pairs ( p , g ′ ) you need into G ∗ θ and G ∗ φ :

3.2 Network Architecture

Non-rigid Garment Deformer (NGD) G θ . The deformer G θ can adopt arbitrary encoder-decoder networks. As shown in Eq. (1), to fit Markov Random Field, we first pre-trained G θ by feeding paired images ( p , g ) to learn a real deformation field ˜ f , i.e. ˜ f = G θ ( p , g ) . At this point, pre-trained G θ contains the full latent alignment information, including shape, color, perspective changes, texture variations, shadows and lighting, and depth of field, between the paired images ( p , g ) . However, features such as color, shape, and texture do not need to be learned, because the input garment varies greatly during inference, but g and p used for training are paired. Therefore, G θ is necessarily invalid when unpaired images are fed. To disentangle the aforementioned feature entanglement, we employ an auxiliary deformer G A , which takes the garment g , prior Densepose descriptor p d ∈ R 1 × H × W [25], and the pose heatmap p h ∈ R 18 × H × W as input to learn the same goal as pre-trained G θ , i.e. , ˜ f A = G A ( p d , p h , g ) . Because p d and p h are not correlated with g in terms of color, texture, and shape, G A does not face the challenges mentioned above and can handle any type of garment effortlessly. However, unfortunately, p d and p h also lack perspective changes, shadows, lighting,

and depth of field. As a result, the deformation results obtained are often not realistic. Therefore, we use ˜ f to supervise the training of G A , ˜ f A ≈ ˜ f , supplementing its spatial perception abilities. After obtaining the optimal G ∗ A , we continue to train the deformer G θ by taking unpaired ( p , g ′ ) as input and f A as supervision, to disentangle the feature entanglement of G θ :

Self-cyclic Image Generator (SIG) G φ . The generator G φ can adopt arbitrary encoder-decoder networks. Our goal is to optimize G φ by taking unpaired images ( p , ˆ g ′ ) . The training process is divided into two steps: 1) generating the try-on result ˆ p ′ by taking unpaired images ( p , ˆ g ′ ) as input, i.e. ˆ p ′ = G φ ( p , ˆ g ′ ) . Immediately after, 2) taking ˆ p ′ and warped garment ˆ g as input of G φ to generate try-on result ˆ p , i.e. ˆ p = G φ (ˆ p ′ , ˆ g ) . Different from the current methods that rely on an additional network to provide prior pseudo-label ˆ p ′ , training G φ directly with self-cycle consistency is not work due to the absence of ground truth of ˆ p ′ , G φ is more inclined to learn the mapping: p ↦→ ˆ p when we jointly perform the two steps. Therefore, we introduce additional supervision on the upper body of the generated ˆ p ′ to take over the role of the label in [22]. In simple terms, the functionality of the person representation R has shifted from the input side to the supervision side. In addition, the self-cycle consistency can be ensured by enforcing ˆ p ≈ p to establish self-supervision in step 2.

3.3 Self-Cycle Consistency Training Strategy

Training of NGD. As shown in Eq.(1), Eq.(2), and Eq.(8), to achieve the MRF energy minimization, we design multiple loss functions. E dat is implemented with the pixel-level L 1 loss to encourage the similarity between ˆ f and ˜ f A . E reg is composed of the weights w ϕ and the smoothing term R sm (implemented as second-order smooth constraint L sec [6]). E pix is implemented with L 1 loss and VGG perceptual loss L per [26] as the evaluation of structural similarity. Finally, we define the overall loss function of NGD as:

where λ r and λ d are hyperparameters that balances the relative contributions of sub-loss terms.

Training of SIG. To self-supervised cyclic optimization of SIG, we propose a self-cycle consistency loss consisting of L 1 loss and perceptual loss L per , respectively, to maintain consistency between the try-on result ˆ p and the real person p :

To make ˆ p ′ as close as possible to the real distribution of person, we employ a discriminator D s to introduce adversarial loss:

Additionally, the adversarial loss L G adv of generator G φ is introduced simultaneously. In addition to the aforementioned losses, we propose a skin reconstruction loss L sr to forcefully supervise the skin reconstruction of ˆ p ′ in the arm and neck regions. To do this, we pre-trained a model SR, which can be fed into deformed garment ˆ g ′ and parts of the skin s p to reconstruct the envisaged arm and neck regions of ˆ p ′ :

where ˆ p ′ s denotes skin regions of ˆ p ′ . Furthermore, to generate the garment part of ˆ p ′ and preserve the invariant body content ( e.g. head, trouser), we introduce the garment reconstruction loss L gr and content preservation loss L cp :

where ˆ p ′ g denotes garment part of ˆ p ′ , ˆ p ′ c and p c denote the same content of ˆ p ′ and p , respectively. 2 Finally, we define the overall loss function of SIG as:

To prevent the unreliable result ˆ p ′ generated in step 1 to wrongly guide step 2 , we introduce an adjustable gated mechanism late in the training to prevent irresponsible backpropagation:

where α is the adjustable gated factor that determines whether backpropagation is enabled.

CP-VTON+

ACGPN

DCTON

PF-AFN

SDAFN

FS-VTON

RT-VTON

Ours

Garment

Person

CVPRW

CVPR

CVPR

CVPR

ECCV

CVPR

CVPR

4 Experiments

Dataset. We conduct experiments using the VITON dataset [8], which consists of 16,253 image groups. Each group includes a front-view female image p , an in-shop garment image g with its mask p M , a reference semantic map p p , and a pose heatmap p h . The image size for each is 256 pixels × 192 pixels. The training set contains 14,221 groups, and the test set contains 2,032 groups. All evaluations and visualizations are performed using the test set. Note that the in-shop garment and the garment worn on the person are identical in the training set, while they differ in the test set.

Implementation Details. The USC-PFN is implemented in PyTorch and trained on a single Nvidia Tesla V100 GPU running Ubuntu 16.04. During training, a batch size of 16 is used for 100 epochs, and the Adam optimizer [29] is employed with parameters β 1 = 0 . 5 and β 2 = 0 . 999 , and the initial learning rate is set to 1 e -4 with linear decay after 50 epochs. The USC-PFN consists of NGD and SIG, both have the same structure as Res-UNet [30], and the discriminator is from pix2pixHD [16]. In the loss functions, the λ r = 20 and λ p = 0 . 25 in the L ngd . The λ scyc = 1 , λ G adv = 0 . 1 , λ sr = 50 , λ gr = 1 , and λ cp = 10 in the L t sig .

Baselines. We perform comparative evaluations of our model's performance, which includes the deformer and generator, against publicly available state-of-the-art methods. To this end, we leverage ten popular methods, including VITON[8], CP-VTON[3], Cloth-flow[9], CP-VTON+[27], ACGPN[4], DCTON[2], PF-AFN[6], ZFlow[31], SDAFN[28], and RT-VTON[13], as baseline methods for quantitative evaluation. In particular, we select seven most cutting-edge methods [27, 4, 2, 6, 28, 13, 7] for qualitative evaluation.

Evaluation Metrics. Quantitative evaluations are carried out in both paired and unpaired settings to compare our approach with baseline methods. For the paired setting, we use Structure Similarity (SSIM) [32] to evaluate the visual quality and diversity of the generated images. In the unpaired setting, as there is no ground truth available, FID is directly employed to evaluate the distributional similarity between the generated and real images. Note that we do not use Inception Score (IS) [33]

-: official code or data are not provided. IP: in-painting; CC: cycle consistency; KD: knowledge distillation; SC: self-cycle consistency.

in the evaluation [34, 35], as it is only effective in datasets similar to ImageNet [36]. Specifically, a lower score of FID indicates a higher quality of the result.

4.1 Experimental Evaluation

Qualitative Results. In Figure 4, we perform visual comparison with seven methods, including SOTA in-painting methods [27, 4, 13, 28], the SOTA cycle consistency method [2], and SOTA knowledge distillation methods [6, 7]. In particular, we specifically select four striped garments to visually showcase the deformation effects of each method. It indicates that all methods can achieve approximate garment alignment and tryon synthesis. However, in terms of garment alignment, the TPS-based methods

(columns 3, 4, and 5) exhibit excessive local fabric deformation and misaligned spatial correspondence. The AF-based methods (columns 6, 7, and 8) demonstrate overly flexible and unconstrained deformation effects, which are particularly noticeable in the first row. The MRF-based method (column 9) shows semi-rigid deformation effects in highly non-rigid body poses, such as misalignment in the arm region. In terms of try-on synthesis, the in-painting methods (columns 3, 4, 7, and 9) exhibit mismatches between the given garments and the synthesized try-on results due to parsing errors, particularly noticeable in the arms region. The cycle consistency method (column 5), being modular in nature, shows artifacts along the module boundaries. The knowledge distillation methods (columns 6 and 8) occasionally suffer from similar issues as the in-painting methods, both in terms of clothing deformation and try-on synthesis, due to irresponsible teacher knowledge. In contrast, our proposed method (last column) can achieve the most realistic results benefits from two key factors. Firstly, it leverages MRF's perception of human spatial structure to faithfully simulate the process of cloth deformation after try-on, capturing realistic fabric draping effects. Secondly, it utilizes the self-cycle consistency paradigm to accurately reconstruct the appearance of the garment worn on the body, all without the need for additional parsers or complex designs.

Quantitative Results. We evaluate the quantitative results of different methods under the same configuration for a fair comparison. We use SSIM for measuring structural similarity in the paired setting and FID for measuring distributional similarity in the unpaired setting. Table 1 presents our quantitative results, including the SSIM and FID scores of baseline methods [8, 3, 9, 27, 4, 2, 6, 31, 28, 13] and the proposed USC-PFN, on the VITON dataset. In terms of SSIM, USC-PFN

Person

Garment

Full

Flow

Flow

Flow

Flow

Person

Garment

Full

Flow

Flow

Flow

Flow

w/o

w/o

Edat

0

MRF

0

'ST

AL

evTs

LeVTN

SLAO

w/

w/oEdai

w/oMRP

w/oLsr

surpasses the SOTA in-painting method SDAFN [28] by 0.03, the SOTA knowledge distillation method PF-AFN[6] by 0.02, and the SOTA cycle consistency method DCTON [2] by 0.08. In terms of FID, it surpasses the SOTA in-painting method RT-VTON [13] by 1.19 and the SOTA cycle consistency method DCTON by 4.35. Although PF-AFN obtains a lower FID score, some of their results appear to be worse than ours 3 . Additionally, PF-AFN requires complex training with the assistance of the teacher model. The result demonstrates the superiority of our self-cycle consistency approach while showcasing the significant advantage of generating high-quality try-on images.

Computational Complexity. In addition, to demonstrate the superior performance of our network, we calculate the computational costs of ACGPN[4], DCTON[2], PF-AFN[6], and our method as shown in Table 2. On the same Tesla V100 GPU, we have an appropriate number of parameters and the fewest FLOPs (floating point operations), yet our inference speed is nearly four times faster than ACPGN and about twice as fast as DCTON. This demonstrates the real-time capability of our method and its ability to achieve high-quality visual performance with fewer computational resources. Note that our architecture can employ any encoder-decoder network, allowing for flexible selection of the network based on computational resources.

Ablation Study. In the ablation study, we evaluate the effectiveness of NGD and SIG in USC-PFN. Table 3 and Figure 5 provide both quantitative and qualitative results. Firstly, we demonstrate that the auxiliary deformer G A successfully disentangles the color, texture, and shape correlations in the pre-trained G θ . The experiments further validate the necessity of the data term E dat in MRF, as it plays a crucial role in ensuring the accurate deformation of the garment. Moreover, we conduct an experiment by removing the entire MRF module to emphasize its essential in USC-PFN. Finally, the efficacy of L sr in SIG is validated, emphasizing its critical role in supervising skin generation.

Limitations and Discussion. There are still limitations to USC-PFN that need to be addressed in future work. Firstly, due to the complex non-rigid deformation of garment images, it is challenging to achieve good convergence during end-to-end training; that is, integrating the garment deformer G θ into the image generator G φ remains a difficult task. Secondly, accurately aligning garments with human poses is challenging due to the complexity and variability of human

poses. Thus, finding a more optimal closed-form solution is necessary to solve the garment deformation task. Finally, we still require extensive supervision during the training, and it is still an urgent issue to find a solution to train a high-quality network with less supervision.

5 Conclusion

In this paper, we present USC-PFN, a parser-free virtual try-on network that utilizes a self-cycle consistency pipeline, which only uses one model to cyclically learn trying on different styles of

garments. To get a more natural and realistic garment alignment, USC-PFN first incorporates the Markov Random Field as a non-rigid deformation method, by enhancing the deformer's perception of human spatial structure, thereby mimicking the natural interaction between the garment and the person. In addition, unlike existing paradigms, our method does not require input information from parsers, teacher knowledge, or complex person representations for the generator. Instead, it solely relies on the garment and human images as input to train and infer try-on results. The result indicates that USC-PFN outperforms the state-of-the-art methods significantly in terms of subjective and objective evaluations, demonstrating its superior performance. In the future, we plan to extend this architecture to different tasks of image-to-image translation.

Acknowledgments and Disclosure of Funding

This work was in part supported by the National Key Research and Development Program of China (Grant No. 2022ZD0160604) and NSFC (Grant No. 62176194), and the Key Research and Development Program of Hubei Province (Grant No. 2023BAB083), the Project of Sanya Yazhou Bay Science and Technology City (Grant No. SCKJ-JYRC-2022-76, SKJC-2022-PTDX-031), and the Project of Sanya Science and Education Innovation Park of Wuhan University of Technology (Grant No. 2021KF0031).

References

Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision , pages 2223-2232, 2017.

Chongjian Ge, Yibing Song, Yuying Ge, Han Yang, Wei Liu, and Ping Luo. Disentangled cycle consistency for highly-realistic virtual try-on. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16928-16937, 2021.

Bochao Wang, Huabin Zheng, Xiaodan Liang, Yimin Chen, Liang Lin, and Meng Yang. Toward characteristic-preserving image-based virtual try-on network. In Proceedings of the European conference on computer vision (ECCV) , pages 589-604, 2018.

Han Yang, Ruimao Zhang, Xiaobao Guo, Wei Liu, Wangmeng Zuo, and Ping Luo. Towards photorealistic virtual try-on by adaptively generating-preserving image content. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 7850-7859, 2020.

Ning Wang, Jing Zhang, Lefei Zhang, and Dacheng Tao. Gt-must: Gated try-on by learning the mannequinspecific transformation. In Proceedings of the 30th ACM International Conference on Multimedia , pages 2182-2190, 2022.

Yuying Ge, Yibing Song, Ruimao Zhang, Chongjian Ge, Wei Liu, and Ping Luo. Parser-free virtual try-on via distilling appearance flows. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8485-8493, 2021.

Sen He, Yi-Zhe Song, and Tao Xiang. Style-based global appearance flow for virtual try-on. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3470-3479, 2022.

Xintong Han, Zuxuan Wu, Zhe Wu, Ruichi Yu, and Larry S Davis. Viton: An image-based virtual try-on network. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7543-7552, 2018.

Xintong Han, Xiaojun Hu, Weilin Huang, and Matthew R Scott. Clothflow: A flow-based model for clothed person generation. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10471-10480, 2019.

Ruiyun Yu, Xiaoqi Wang, and Xiaohui Xie. Vtnfp: An image-based virtual try-on network with body and clothing feature preservation. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10511-10520, 2019.

Thibaut Issenhuth, Jérémie Mary, and Clément Calauzenes. Do not mask what you do not need to mask: a parser-free virtual try-on. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XX 16 , pages 619-635. Springer, 2020.

Benjamin Fele, Ajda Lampe, Peter Peer, and Vitomir Struc. C-vton: Context-driven image-based virtual try-on network. In Proceedings of the IEEE/CVF winter conference on applications of computer vision , pages 3144-3153, 2022.

Han Yang, Xinrui Yu, and Ziwei Liu. Full-range virtual try-on with recurrent tri-level transform. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3460-3469, 2022.

Chenghu Du, Feng Yu, Minghua Jiang, Ailing Hua, Xiong Wei, Tao Peng, and Xinrong Hu. Vton-scfa: A virtual try-on network based on the semantic constraints and flow alignment. IEEE Transactions on Multimedia , 25:777-791, 2023.

Nikolay Jetchev and Urs Bergmann. The conditional analogy gan: Swapping fashion articles on people images. In Proceedings of the IEEE international conference on computer vision workshops , pages 2287-2292, 2017.

Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Highresolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 8798-8807, 2018.

Fred L. Bookstein. Principal warps: Thin-plate splines and the decomposition of deformations. IEEE Transactions on pattern analysis and machine intelligence , 11(6):567-585, 1989.

Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. Advances in neural information processing systems , 28, 2015.

Tinghui Zhou, Shubham Tulsiani, Weilun Sun, Jitendra Malik, and Alexei A Efros. View synthesis by appearance flow. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14 , pages 286-301. Springer, 2016.

Scott Schaefer, Travis McPhail, and Joe Warren. Image deformation using moving least squares. In ACM SIGGRAPH 2006 Papers , pages 533-540. 2006.

Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1125-1134, 2017.

Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo. Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 8789-8797, 2018.

Alexander Shekhovtsov, Ivan Kovtun, and Václav Hlaváˇ c. Efficient mrf deformation model for non-rigid image matching. Computer Vision and Image Understanding , 112(1):91-99, 2008.

Carl Friedrich Gauss. Theoria motus corporum coelestium in sectionibus conicis solem ambientium , volume 7. FA Perthes, 1877.

RA Guler, Natalia Neverova, and IK DensePose. Dense human pose estimation in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA , pages 18-23, 2018.

Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and superresolution. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14 , pages 694-711. Springer, 2016.

Matiur Rahman Minar, Thai Thanh Tuan, Heejune Ahn, Paul Rosin, and Yu-Kun Lai. Cp-vton+: Clothing shape and texture preserving image-based virtual try-on. In CVPR Workshops , volume 3, pages 10-14, 2020.

Shuai Bai, Huiling Zhou, Zhikang Li, Chang Zhou, and Hongxia Yang. Single stage virtual try-on via deformable attention flows. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XV , pages 409-425. Springer, 2022.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 2015.

Foivos I Diakogiannis, François Waldner, Peter Caccetta, and Chen Wu. Resunet-a: A deep learning framework for semantic segmentation of remotely sensed data. ISPRS Journal of Photogrammetry and Remote Sensing , 162:94-114, 2020.

Ayush Chopra, Rishabh Jain, Mayur Hemani, and Balaji Krishnamurthy. Zflow: Gated appearance flow-based virtual try-on with 3d priors. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5433-5442, 2021.

Kalpana Seshadrinathan and Alan C Bovik. Unifying analysis of full reference image quality assessment. In 2008 15th IEEE International Conference on Image Processing , pages 1200-1203. IEEE, 2008.

Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems , 29, 2016.

Shane Barratt and Rishi Sharma. A note on the inception score. arXiv preprint arXiv:1801.01973 , 2018.

Mihaela Rosca, Balaji Lakshminarayanan, David Warde-Farley, and Shakir Mohamed. Variational approaches for auto-encoding generative adversarial networks. arXiv preprint arXiv:1706.04987 , 2017.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a virtual try-on method. At inference time, no parser information (e.g., pose, segmentation map) is needed; given a person's image and a garment, the method is able to output a try-on image. To learn this model with a dataset of paired person and garment images, a deformation network is learned to warp the garment to match the person. Meanwhile, a generator is learned to combine the warped new garment and the person to create the try-on image. The generator is trained with a self-cycle-consistency objective, and the intuition is simple: a try-on image should be within the distribution of real images (adversarial loss), and trying on the person's original cloth on an already tried-on image should look like the original image (cycle-consistency loss). The method has multiple other components in it, making it hard to add this to the summary. The proposed method is compared with prior work and shows similar performance with the state of the art.\n\nSTRENGTHS:\n1. From the qualitative comparisons, using ground truth deformation field as supervision instead of other alternatives (appearance flow, TPS) seems to help reduce artifacts. Especially in the case where the clothes have stripes.\n\n2. The work proposes a good idea to use an auxiliary deformer, that takes in only pose information (without RGB) of the person as input to predict the garment deformation. It makes sense that it will help generalizing perdiction when the garment is not paired with the person (the training set is all paired up). Since at inference time, only RGB input are given, this auxiliary deformer is used to generate pseudo ground truths for unpaired garment and person images. In short, auxiliary deformer \"teaches\" the final deformer to generalize better, which is a good idea.\n\nWEAKNESSES:\n1. In the related work section, it is mentioned that using parser information can introduce unreliable errors and artifacts. And even with teacher-student training, this issue can still lead to robustness issues. Meanwhile, I do not understand why the proposed method alleviates the issue. The auxiliary deformation network, which is essential to the method (see Fig. 5), is trained with parser information (dense pose). It will be great to clarify this.\n\n2. Writing is hard to understand. It prevents readers from understanding what exactly the algorithm is. See the points below:\n\n3. In line 189, the Lsec loss is refered to a citation without mentioning what exactly it is. Is it precisely the regularization term in Eq. 2?\n\n4. The skin region is mentioned in the method section (s_p, p _s). How do you locate, define, or detect these regions? Same question holds for the content preservation loss: how do you locate the invariant content region (head, trouser, etc.)?\n\n5. Is there any reason that the cycle-consistency loss does not need to be separated into skin, content and garment reconstruction loss? It seems compatible, and I don't understand why the loss function design cannot be applied symmetrically.\n\n6. In figure 5, I do not understand the visualization in the left column of SIG w/o MRF. Having an explanation of the visualization is nice. Also, although I understand other visualziations are representing the deformation field, it is still good to mention it in the caption.\n\nQUESTIONS:\n1. The paper mentioned that prior work uses two CNNs to train models cyclically, while the proposed method uses one CNN to train a self-cycle. However, is there a reason why training one model directly with self-cycle consistency will not work? Also, I am guessing that the skin, content, and garment reconstruction loss is one of the important components to make training converge to a reasonable solution. It will be great to confirm this.\n\n2. In my opinion, the draft needs a major update, especially in the method section. If the revision explains the design choices more clearer, I am happy to change my ratings.\n\nLIMITATIONS:\n1. The paper mentions that it is challenging to achieve good convergence during end-to-end training. Does this mean that a two-stage training (pretraining NGD then training SIG) is required for good results? Or does this mean in general the current method does not attain good convergence?\n\n2. In line 290, what does it mean by closed-form solution here?\n\n-----------------\nRaised score to a borderline accept.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new parser-free virtual try-on network (USC-PFN) to use only unpaired images as input to generate realistic try-on results. \nTo address the core warping problem in virtual try-on, it models the deformation field estiamtion by using the Markov Random Field. To train the try-on generator by using unpaired data, it proposes a self-cycle consistency pipeline.\nExtensive  comparisons with the state-of-the-art methods on VITON benchmark demonstrate its superiority and the ablation study also shows the effectiveness of different modules in the proposed method.\n\nSTRENGTHS:\n- This paper explores the parser-free virtual try-on, which is challenging and with great significance for the image-based virtual try-on.\n\n- For the first time, this paper introduces the Random Markov Field into the non-rigid garment deformer, which is quite different from the deformation module in previous methods.\n\n- It proposed a novel self-cycle consistency pipeline for the training of the try-on generator by using unpaired images.\n\n- The authors conduct extensive comparisons with the existing state-of-the-art methods to illustrate the superiority of the proposed methods. The ablation study also show the effectiveness of the different modules in the proposed method.\n\nWEAKNESSES:\n- One core technical contribution in this paper is that it model the garment warping by using the Markov Random Field. However, the authors do not provide an clear explanation about it. What is the difference between it and the widely used appearance flow? why can it outperform the TPS-based or appearance flow-based methods?\n\n- The authos only conduct experiments on the VITON benchmark, in which the image resolution is quite low (256 x 192). However, most of the advanced methods focus on the higher resolution virtual try-on (e.g., 512 x 384, 1024 x 768), which are more closed to the real world \ntry-on scanerio.\n\n- The writing is not straightforward amd several descriptions are a bit obscure. For example, (1) Although  the garment deformer and the auxiliary deformer receive different inputs, in Figure 3, they seems to take the same inputs (i.e., person image and the garment image). (2) In line131, it is confused how to obtain the deformation field $\\tilde{f}$.\n\n- Some decipt in the main paper is a bit overclaimed. In line 300, the authors claim that USC-PFN can solely rely on the garment and human image for training. However, it still required some human condition like human parsing, densepoe during training NGD and SIG.\n\nQUESTIONS:\n- For the non-rigid deformation module, is the pre-trained deformer necessary for the training of the final deformer? In my opinon, it is unnecessary to use supervision provided by the pre-trained deformer  (which is trained by using paired images) for the training of auxiliary deformer, since the ground truth deformation field is not necessary for the training of deformer (velidated in the previous works like PF-AFN[1], FS-VTON[2]). Once obtaining the auxiliary deformer, we could train the final deformer by using the unpaired images.\n\n- When training the Self-cyclic Image Generator (SIG), several local region supervisions (i.e., skin loss, garment loss, preserved content loss) are employed to facilitate the self-supervised cyclic training, which should resort to the human parsing to obtain the specific local region. Thus, the parsing error might still affact the training procedure. However, some knowledge distillation method like PF-AFN[1] does not face with parsing error issue since they only use the global supervision when the the parser-free student network. My question is how does USC-PFN alleviate the influence derived the parsing error during training.\n\n- Since the SIG can be trained with unpaired images, is it possible for USC-PFN to leverage large amout of unpaired images from Internet during the trianing of SIG? Will such the increasing in training data facilitate the performance of USC-PFN?\n\n[1] Ge et al. \"Parser-Free Virtual Try-on via Distilling Appearance Flows\", CVPR 2021.\n\n[2] He et al. \"Style-Based Global Appearance Flow for Virtual Try-On\", CVPR 2022.\n\nLIMITATIONS:\nSome limitations of this paper have been disscussed in the main paper. Other limitation I concern is the image resolution, since it is quite improtant for the real world try-on scanerio.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the challenges in generating high-quality virtual try-on images, specifically focusing on non-rigid garment deformation and unpaired garment-person images. Existing methods rely on disentangling garment domains with the aid of \"teacher knowledge\" or dual generators, which can limit the quality of try-on results. Additionally, current garment deformation techniques struggle to mimic natural interaction between garments and the human body, resulting in unrealistic alignment effects. To overcome these limitations, the authors propose a Unified Self-Cycle Consistency for Parser-Free virtual try-on Network (USC-PFN). USC-PFN utilizes a single generator and incorporates Markov Random Field for more realistic garment deformation.\n\nSTRENGTHS:\n- The paper is easy to understand.\n- The experiment results seems okay, but not excellent compared to the baselines.\n- The proposed method requires less computational costs compared to the baselines such as ACGPN and DCTON during inference.\n\nWEAKNESSES:\n- The primary limitation of this paper lies in the lack of significant performance improvement compared to the existing virtual try-on baselines. Particularly, when examining the performance comparison in Table 1, it raises doubts about the meaningful enhancement in terms of FID, which is even higher than that of PF-AFN. Furthermore, the disparity in frames per second (FPS) between PF-AFN and the proposed approach is not significant.\n- The proposed self-cyclic image generator is inspired by StarGAN; however, it is unclear what specific advantages it offers. It appears to train a generator with clothing as a condition instead of employing other models that utilize cycle consistency loss. Nevertheless, it is uncertain whether this approach is novel and impactful enough to be presented at a top-tier conference..\n- The newly proposed non-rigid garment deformer based on Markov Random Field lacks sufficient examples to demonstrate its superior warping capabilities compared to existing warping methods. Various virtual try-on methods have also proposed different approaches to enhance the performance of warping. However, it remains unclear how the proposed method establishes its superiority in this regard qualitatively and quantitatively.\n- Most recent virtual try-on models have demonstrated their performance on high-resolution datasets such as VITON-HD [1] or Dresscode [2]. Additionally, parser-free virtual try-on models like DC-VTON have also shown their performance on images of around 512x384 resolution. However, this paper is lacking references to related works and only presents evidence of its performance on the low-resolution VITON dataset, which is somewhat disappointing.\n\n[1] VITON-HD: High-Resolution Virtual Try-On via Misalignment-Aware Normalization, CVPR 2021\n[2] Dress Code: High-Resolution Multi-Category Virtual Try-On, ECCV 2022\n\nQUESTIONS:\nDoes the paper include a comparative analysis between the proposed warping method and other existing warping methods? I contend that it is imperative to conduct a direct comparison, encompassing both qualitative and quantitative evaluations, with well-established warping techniques such as TPS transformation and Appearance Flow.\n\nLIMITATIONS:\nMentioned above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presented a system for image based virtual try-on. The main contribution consists (1) a parser free virtual try-on network trained with unpaired data; (2) a MRF based deformation estimation network; (3) a cycle consistency based training method. The paper performed experiments on the VITON Zalando dataset. The paper is evaluated with several prior work as baselines, including VITON[8], CP-VTON[3], Cloth-flow[9] etc. The main evaluation metrics are SSIM for pair data and FID for data with no ground truth. In terms of quantitative performance, the proposed method achieves better or comparable results than previous art. In terms of qualitative results, the generated images are visually comparable, and some times slightly better than previous work.\n\nSTRENGTHS:\n+ The paper is addressing an interesting problem.\n\n+ The main idea is interesting and seems novel to me. Although cycle consistency are not new ideas and it has been used in previous work on virtual try on, the overall combination of ideas seems still new in this specific domain.\n\n+ Extensive experiment shows state-of-the-art quantitative results. The qualitative results have less deformation artifact than previous work.\n\nWEAKNESSES:\nI'm not an expert on virtual try-on so I will mainly rely on the other reviewers opinion for a more informative feedback about the weakness of the paper. On feedback I have is that the results are often overly smooth on the garment area. The paper showed a lot of results of clothing with stripe pattern and the stripes are often very smooth, and not reflecting the pose and shape of human body very well.This is perhaps due to the way that the deformation is learned. It would be great to discuss the potential cause of this over-smoothness and also perform ablations studies.\n\nQUESTIONS:\nPlease address the aforementioned questions.\n\nLIMITATIONS:\nThe limitation is discussed in section 4 and seems sufficient to me.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a virtual garment try-on method. A cycle-consistency loss allows for self-supervision, i.e. the method does not require paired data (of the same person wearing different garments) as supervision. Unlike a previous method [2] that also uses cycle consistency, the same network weights are used for both the forward and backward step of the cycle. Additionally, a new garment deformation model based on Markov Random Fields is used instead of previous methods that use one of thin plate splines, appearance flow, or moving least squares. The authors show improved results over almost all previous methods, and similar results to one previous method [6].\n\nSTRENGTHS:\n- Cycle consistency using shared weights makes sense. It might seem like a small change, but does seem to require support from newly introduced losses.\n- The new deformation model is a contribution, although it is unfortunately not evaluated separately from the full try-on model.\n- The evaluation is reasonable and results show a significant advantage over the previous cycle-consistent method, and a significant advantage over most other methods (except one).\n\nWEAKNESSES:\n- The exact differences to [2] that give the proposed method an advantage over [2] are unclear. The authors allude to  advantages over [2] in several passages, but never fully describe what the differences to [2] are exactly (apart from re-using the same weights for both directions in the cycle) or where the advantages come from. For example:\n\t- In Section 2, the authors mention that [2] prioritizes paired garment-person images which can lead to difficulties with unpaired data. However, the proposed method also uses paired (garment, person) images, so what exactly is the difference? This needs to be clarified.\n\t- in Section 3.1, the authors mention a 'deconstruction of the human body' that leads to some problems in [2]. It is unclear what the authors refer to here. Do the authors refer to the Densepose descriptor used in [2]? If so, the authors would first have to mention that [2] needs to use the Densepose descriptor, so the reader has enough context to understand this statement. Just looking at the Eq 5 and 6, the only difference between the proposed method and [2] seems to be that the same weights are re-used for both directions. If there are any other differences, these need to be described explicitly near Eq 5 before referring to them. This needs to be clarified.\n\t- Also in Section 3.1, the authors mention that previous methods (including [2]) need an 'a-priori label'. Again, the authors need to describe what this refers to, otherwise the differences to [2] are hard to understand.\n- The existing method [6] seems to have a similar performance. A discussion of other advantages over [6] would be good.\n\t- FID is better for [6] while SSIM is better for the proposed method.\n\t- PF-AFN look a bit more natural to me in all examples of Figure 4 except the first row, and maybe some details in the fourth row. It looks more natural to me because it seems to better follow the shape of the body, the proposed method looks flatter, as if it does not fully follow the body shape.\n- The evaluation could be improved with additional ablations:\n\t- Using SIG with the same deformation method that [2] uses, to have a more direct comparison of using cycle consistency with shared vs. non-shared weights in the forward/backward steps.\n\t- Comparing the performance of NGD to previous deformation methods separately from the full try-on pipeline.\n- The exposition is often confusing and should be improved.\n\t- The introduction emphasizes that the method only uses unpaired data, but it seems paired (garment A, person wearing A) data was still used. The introduction should clarify which kinds of pairs were not used as training. I assume the author refer to (person wearing A, person wearing B) pairs, but this needs to be clarified in the introduction.\n\t- The description of the deformation fields in Eq. 1 and the paragraph before it is a bit confusing. $\\hat{f}$ is described as the optimal deformation field, but in the equation it is used as the variable that is being optimized (I would have expected  $\\hat{f}$ to be the result of the minimization in Eq. 1, rather than the variable being minimized over).\n\t- At the end of Section 3, 'infinitely close' is probably not the right phrasing, since i) its unclear what 'infinitely close' means (why not say 'the same'?) and ii) the goal is to reconstruct $p$ with $\\hat{p}$, but in practice, they will not be the same.\n\t- $R_{sm}$ is not defined in Eq. 2. The type of smoothing term should at least be described shortly.\n\t- A bit more information should be given in the weights $w_\\phi$ in Eq. 2. How is the Gaussian constructed? How are its mean and variance computed? I assume something like the centroid and variance of pixel coordinates in the cloth region. A brief mention would be good.\n\t- In Eq. 2 and 3 it would be good to denote which variables the minimization is over (i.e. $\\text{argmi}n_{\\hat{f}}$ or $\\text{argmi}n_{\\phi}$.\n\t- In Eq. 3, it should probably be \"$\\dots\\text{with} \\dots$\" instead of \"$\\dots s.t. \\dots$\"\n\t- Above Eq. 7 the authors mention that additional supervision is introduced for the upper body, but do not mention how it is used, what it consists of, etc. If this is described later on, it would be better to not mention this yet, to avoid confusion, or to re-structure the sections so the reader does not need to know information that will only be provided later on to understand the current text.\n\t- In Eq. 7 the notation is confusing. I did not follow why every variable has an additional ' in the notation. Why not use the same input notation as before: $(g', p)$, and output is $\\hat{g}'$?\n\t- In Section 3.2, some design choices should be discussed:\n\t\t- Why is $\\mathcal{G}_\\mathcal{A}$ not used directly instead of $\\mathcal{G}_\\theta$? (I assume because the authors want to avoid using Densepose in the cycle training, but it should be discussed why, and ideally an ablation should be given.)\n\t\t- Why does $\\mathcal{G}_\\theta$ need to be pre-trained instead of directly starting training with $\\mathcal{G}_\\mathcal{A}$ and then finetuning?\n\t- Eq. 8 is unclear, it would be clearer to explicitly describe the loss used to train $\\mathcal{G}_\\theta$ in the second phase, probably something like $\\min_\\theta \\mathcal{G}_\\theta(g', p) - \\mathcal{G}_\\mathcal{A}(g', p_d, p_h)$\n\t- In Section 3.3, the definition of $L_{sec}$ should be given, or at least a short description of what this loss does.\n\t- In Eq. 9, $\\mathcal{L}_{per}$ is not defined, it is only defined later on in the next subsection.\n\t- Above Eq. 15, step 1 and step 2 are not defined. These probably refer to the forward and backward steps in the cycle, but they should be defined more explicitly.\n\t- The loss in Eq. 15 needs a bit more motivation. why is backpropagation disabled if $\\hat{p}'$ is more similar to $p$ than $\\hat{p}$? Could this not happen quite frequently at the start of the training? Some discussion is needed.\n\t- A bit more information is needed how SSIM is computed. Is it computed by using the garment as input that the input person is already wearing, and is a held-out test set of paired data used?\n\t- In the ablation study when removing the MRF module, what deformation method is used instead?\n\nDetails:\n- Near line 80: self-consistency consistency -> self-cycle consistency\n- In Section 3.2, the paragraph titles should give the acronyms of the two modules in paranthesis (NGS, SIG), since the acronyms are used later on in the text.\n\nQUESTIONS:\n- Please clarify the setup that was used to compute SSIM.\n- Please clarify all differences that [2] that could cause the improved performance shown in the results.\n- Please describe all advantages of the proposed method over [6], and what you consider to be the most important advantages.\n\nLIMITATIONS:\nSome limitations have been discussed. Impacts from potentially biased datasets could be mentioned.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}