Greatness in Simplicity: Unified Self-Cycle Consistency for Parser-Free Virtual Try-On
Authors: Chenghu Du
junyin Wang
Shuqing Liu
Shengwu Xiong
Venue: NeurIPS 2023 poster (2023)
Keywords: parser-free virtual try-on
self-cycle consistency
human analysis and understanding
fashion synthesis
Markov Random Field
TL;DR: This paper proposes a new self-cycle consistency parser-free virtual try-on pipeline (USC-PFN) that enables the robust translation between different garment styles using only a general generator, while achieving SOTA performance.

Abstract: Image-based virtual try-on tasks remain challenging, primarily due to inherent complexities associated with non-rigid garment deformation modeling and strong feature entanglement of clothing within human body. Recent groundbreaking formulations, such as in-painting, cycle consistency, and knowledge distillation, have facilitated self-supervised generation of try-on images. However, these paradigms necessitate the disentanglement of garment features within human body features through auxiliary tasks, such as leveraging 'teacher knowledge' and dual generators. The potential presence of irresponsible prior knowledge in the auxiliary task can serve as a significant bottleneck for the main generator (e.g., 'student model') in the downstream task. Moreover, existing garment deformation methods lack the ability to perceive the correlation between the garment and the human body in the real world, leading to unrealistic alignment effects. To tackle these limitations, we present a new parser-free virtual try-on network based on unified self-cycle consistency (USC-PFN), which enables robust translation between different garments using just a single generator, faithfully replicating non-rigid geometric deformation of garments in real-life scenarios. Specifically, we first propose a self-cycle consistency architecture with a circular mode. It utilizes real unpaired garment-person images exclusively as input for training, effectively eliminating the impact of irresponsible prior knowledge at the model input end. Additionally, we formulate a Markov Random Field to simulate a more natural and realistic garment deformation. Furthermore, USC-PFN can leverage a general generator for self-supervised cycle training. Experiments demonstrate that our method achieves state-of-the-art performance on a popular virtual try-on benchmark.

Greatness in Simplicity: Unified Self-Cycle Consistency for Parser-Free Virtual Try-On
Chenghu Du
Junyin Wang' Shuqin Liu'
Shengwu Xiong 12,.35,+ 1.2.3..
Wuhan University of Technology. Shannhaii AI Laboratory
SSnnya Science and Education Innovation Park, Wuhan University of Technology
WWuhan Textile University. 'Qiongtai Normal University {duch, wjy199708, xiongss)@whur Swtpt//ttuttuuuuueee edu. en, Liushugingutuu@foxaaa com chenghu. .oiiuiggiti github. 1o/USC-PFN/
Abstract
Image-based virtual try-on tasks remain challenging, primarily due to inherent complexities associated with non-rigid garment deformation modeling and strong feature entanglement of clothing within human body. Recent groundbreaking for- mulations, such as in-painting, cycle consistency, and knowledge distillation, have facilitated self-supervised generation of try-on images. However, these paradigms necessitate the disentanglement of garment features within human body features through auxiliary tasks, such as leveraging 'teacher knowledge' and dual gener- ators. The potential presence irresponsible prior knowledge in the auxiliary task can serve as significant bottleneck for the main generator (e.g., 'student model') in the downstream task. Moreover, existing garment deformation methods lack the ability to perceive the correlation between the garment and the human body in the real world, leading to unrealistic alignment effects. To tackle these limitations, present
parser-free virtual try-on network based unified self-cycle consistency (USC-PFN), which enables robust translation between differ- ent garments using just single generator, faithfully replicating non-rigid geometric deformation of garments in real-life life scenarios. Specifically, we first propose a self-cycle consistency architecture with circular mode. It utilizes real unpaired garment-person images exclusively as input for training, effectively eliminating the impact of irresponsible prior knowledge the model input end. Additionally, we formulate Markov Random Field to simulate a more natural and realistic garment deformation. Furthermore, USC-PFN can leverage general generator for self-supervised cycle training. Experiments demonstrate that our method achieves state-of-the-ann performance popular virtual try-on benchmark.
Introduction
To provide consumers with an online try-on experience similar to physical stores, researchers have focused on image-based virtual try-on (VTON), allowing individuals to virtually try on garments available in online shops. The mainstream image-based approach of VTON involves the replacement of person's is clothing area with an in-shop garment image. However, all current datasets are paired, meaning person corresponds only one piece of clothing they wearing. To capture underlying correlations between unpaired images from random combinations in the dataset, three architectures have been used: in-painting methods 8
[4 knowledge distillation methods [T] 6 7].
and cycle consistency methods [5 2
(see Figure
"Shengwu Xiong is corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

+R
+R
Real Person
Garment
Garment
Pscudo
Fake
Person
Person
Person
Cycle Consistency (Two-way)
(c) Knowledge Distillation (Two-way)
Person
Real
Masked
Person
Garment
Person
Fale
Person
Real
Garment
Prmon
Garment 2
(b) In-Painting (One-way)
Syll-Cycle Consistency (Ours)
Figure 1: Comparison of virtual try-on pipelines. "+R" denotes that additional person representation (e.g. semantic map) may required. (a) cycle consistency approaches 2 require two generators implement bidirectional translation between various garment styles. (b) in-painting approaches
3 are implemented by reproducing the masked person's 's garment. (c) knowledge distillation
approaches 66. 7 pre-trained "teacher knowledge" optimize student model. (d) USC-PFN utilizes self-cycle consistency without the need for additional person representation, enabling single generator perform bidirectional translation between various garment styles.
The in- ppaintnng-bssed method, inspired by [6]. addresses variations in garment styles by masking the upper body of the person image and training the model learn the reproduction of masked regions. This paradigm uses person representation (e.g., semantic map) as input to provide masked body information; however, slight errors the person representation can result in occluded and unrealistic outcomes. Moreover, this paradigm faces challenges when dealing with cases where the same person tries different garments due to their reliance one-to-one paired garment-person images during training (see Figure Therefore, the knowledge distillation basod structure [ is proposed, which pretrains an in aa--aantiin--aae based teacher model to provide prior try-on result images for training the student model. This paradigm introduced new pipeline that enables self celf-supervised training using only unpaired images input. However, the potential presence of irresponsible teacher knowledge can still reproduce some issues of in-painting methods. Additionally, method is proposed in 2 that employs two CNNs to achieve the cycle training. this approach, the output of CNN, as input is fed to CNN2 and then minimizes the difference between the output of CNN, and the input of CNN, to achieve cycle consistency. However, the complex dual structure poses challenges in network convergence, and reliance on person representation throughout the entire network hinders generation of high-quality virtual try-on results.
12 is commonly used to deform garments. However, TPS focuses only on pixel transformations around the control point, often causing excessive distortion. Another non-rigid deformation technique, called Appearance Flow predicts pixel-level displacement field of garments for pixel-by-pixt transformations. However, it lacks spatial prior knowledge, resulting in an inaccurate perception of the spatial relationship between the garment and the human body. Recently, the Moving Least Squares (MLS)
In this task, Thin Plate Splines (TPS) 11 18
3
(AF)
20 [3) has shown promise in handling deformation for simple poses. However, there are difficulties handling complex poses. These issues can lead to misalignment, hindering accurate simulation of the natural non-rigid deformation that occurs during the interaction between the garment and the body (see Figure 2
Therefore, we propose virtual parser-free try-on network based on self-cycle consistency to address the aforementioned limitations. First, we explore the relationship between garments and the human body. We use an auxiliary network to disentangle irrelevant features such as texture, color, and shape in the latent feature space of the human body. This approach preserves valuable features like depth and lighting while removing entanglements with unrelated features. Consequently, the network can authentically simulate the interaction between different garments and the human body in real-life scenarios, without being affected by the garment it wears. Furthermore, it has been confirmed that the person representation appearing the input end of the network has potential negative impacts on performance [1] [7]. Morcover, we argue that there is no inherent translation between multiple

---

domains in virtual try-on tasks. Therefore, we introduce shared-weight circular architecture where the parser shifted from the input end the output end. specifically for loss computation. This approach allows us to bypass the parser during the inference stage. The main contributions of our paper can be summarized as follows:
We propose USC-PFN, a novel parser-free virtual try-on network that generates highly realistic try-on results using only unpaired data as input and single CNN the generator, offering new perspective for the virtual try-on task.
We first leverage the Markov Random Field to estimate a deformation field to establish precise dense pixel-level correspondences between the in-shop garment and the person. This is new approach to address the misalignment caused by lack of spatial perception, and simulate the non-rigid deformation and interaction of the garment in real-life scenarios.
We propose a concise and effective self-cycle consistency paradigm for the parser-free virtual try-on task, and extensive experiments on the popular benchmark show that our framework outperforms state-of-the-ar methods both qualitatively and quantitatively.
2
Related Work
Parser-Free Virtual Try-On The formula-
tions mentioned earlier (see Figure 1 be subdivided based on whether the parser is used
Formalation
in image-to-image translation 21 [6]. It has been demonstrated that the parser can improve try-on results but may introduce unreliable se- mantic errors [1I 6, Initially, [8] 3 used pose heatmap and body mask as person repre- sentation to provide body's geometry informa- tion. Later, I0] introduced a parser to estimate an "after-try-on' semantic map instead of the above person representation. To avoid unreli-
kyytr
Aroweelgr Bailivan
In Pardtet
Deformation
TBSS
MES
MRI
Garment
DCTON
FS-VTON
RT-VTON
Ours
able semantic information, [1] 6 7 trained "student model" without the parser using pre- trained teacher knowledge. However, the poten- tial presence of irresponsible teacher knowledge can still affect the "student model," making it challenging to train robust architectures without the parser in the input end. To address this challenge, we present parser-free pipeline called self-cycle consistency. Our goal aims to achieve high-quality virtual try-on results using a general network, minimum input, and simple but efficient architecture.
Figure 2: Comparisons of virtual try-on (DCTON (2, PF-AFN [6], FS-VTON (7), RT-VTON [3] and ours) and different deformation methods (TPS, AF, MLS, and our proposed MRF).
Non-rigid Deformation In virtual try-on tasks, common non-rigid image deformation methods include Thin Plate Spline (TPS) [17 18) Moving Least Squares (MI.S [20]. and Appearance Flow (AF) 19). As shown in Figure TPS is the most widely used due to its ability to provide high-quality
deformation while preserving the garment's image structure [8] 3 10
[2]. However, TPS is
limited by its local mode, which only significantly affects pixels near control points. MLS is another method introduced recently, but it is not suitable for large-scale deformation [3]. AF is the most flexible method, deforming the garment by estimating pixel displacement vectors between the source and target shapes 6 14], However, none of these methods can fully mimic the natural interaction between the garment and the human body due to lack of spatial perception. Our proposed method, which is based on Markov Random Field system with unpaired images as input, can obtain realistic spatial deformation.
Self-Cycle Consistency CycleGAN ① utilized a self superviedd learning framework for cyclically learning the translation between unpaired images from different domains. [2] extended this with
a deconstruction method for different garment styles to address the failure of cycle consistency in
virtual try-on tasks. However, the complex dual-model structure and reliance on the parser hinder the generation of realistic results. To break through these limitations, following StarGAN 22), we propose single-model self-cyclic learning translation paradigm between different garment styles.
3

---

Pre-trained
- Supervision
BI Interpolation
Ga
f
GT
Deformed
Real Person
Fake Person
Supervision
Garment &
Real Person
Bl
Deformed
Carment
R
Supervision
Discriminator
G
Garment&
95
D
Real Person P
Deformed Parmentg
Go
Fake
Det.
Pseudo
Deformed
Garment g
Person p'
Person p'
Garment
Fake Real
Non-rigid Garment
Self-eyelic Image
Deformer
Auxiliary Deformer
Deformation Field
Generator
Weight Sharing
Figure 3: Overview of our self-cycle consistency network: paired garment-person images (g. p) and an arbitrary garment g' serve as the training data. Left: G9 is pre-trained using paired images to obtain the ground-truth deformation field f for the auxiliary deformer GA to optimize fA. Then, Go is further trained by using fĄ of 9A with unpaired images. Right: 9, takes unpaired (g', , p) to generate fake person p'. Then, the real person p reconstructed by re-feeding p' and g into GG, ensuring self-cycle consistency ~ p. DD is the discriminator that learns to distinguish real and fake images. EPS represents enhanced pixel-level supervision and Det. represents detaching tensor p' from the computation graph.
3 Methodology
Notation. In this work, we use (g. p) to represent paired garment pernnnt-eeroonno images where the person p wearing garment g. Moreover, we randomly select the garment image g' to form unpaired garment-person images (g', p) with p. The try-on result of (g. p) and (g', p) are denoted as p and p', respectively. In general, */" represents the target image, and denotes the transformation result. In addition, "r>" represents the mapping relationship of the entire architecture, while "=" represents the mapping relationship of the sub-modules.
The objective the virtual try-on task is to naturally dress an arbitrary in-shop garment image g' Hx μ3xH* As illustrated in Figure 3 we first propose a self-cycle consistency framework. USC-PFN, new virtual try-on paradigm that does not require the human parser and other person representation at the model input end for generating highly realistic try-on images. The framework is broken up into two nub-networks:
Overview.
reference person image p x to obtain try-on result p'
non-rigid garment deformer Go. and (ii) a self-cyclic image generator 9e- They are mainly used to construct cyclic mapping V. g') + p', (p'.g) + P. where = p. i.e., the same the real image p.
3.1 Formulation
Garments Deformation. In (p.g') p', the initially flat garment g' must be deformed match the posture of the person p before synthesizing the target try-on result p', denotes as (p.g') - g', (p.g') - p'. Our key idea involves using Go to model the non-rigid deformation based on Markov Random Field (MRF) (23]. to seek an optimal spatial transformation solution f" x (i.e. optimal deformation field f G (p, g')) that minimizes the dissimilarity between the warped garment g' and the garment P's worn on the target result p', where g' is obtained by using bilinear interpolation (6): g' e(f.g').
This can be formalized as minimizing the MRF energy function:
minimize
Σ
Edat (Ea, + Erer (f) + X Epio (D's 8
(I)
where X, and A9 are trade-off eeper-parameters 4 represents the prior pseudo-label from the pre-traing tre-trained garment deformer and fA is the deformation field of P'g
East = argming j f
f/|.
Ereg wo Nom (f), Epic = argming D [ 8']
(2)

---

where Ereg is regularization term of f, which composed of spatial position weights wo of pixel points and a smoothing term Rym- Ram is used ensure that each element in f and its neighborhood have similar values. D [--] indicates quantification of similarity or difference. During garment deformation, we find that the deformation range of the edge region is often larger than the central region. Therefore, we use the two-dimensional Gaussian distribution [24] with the centroid and variance of pixel coordinates in the cloth region, as w to maintain the flexibility of the edge region.
In (j', g) + p. this process is similar to the one above, i.e., (p', g) - 8: (D', g) - p. but the difference is that we remove Edat from Eq. to ensure more accurate results.
Synthesis of Try-On Results. Recall that the goal of the virtual try-on task is to synthesize the target try-on result after warping the garment as if it was shot from realistic scenes. Therefore, reconstructing the real image p an essential task to form self-supervised training, which can be formalized as minimizing the following objective:
minimize
Σ <RDDEVV
argmin, D Dp. with (p'.g) $ P.
where V is training set. p' denotes wearing the garment g'. This differentiable generation appears have completeness, however, the absence of p' in V forces the process to be compensated by extra person representation R (semantic maps, pose heatmaps, etc.) as input. This means that results are heavily constrained by R, and any errors that occur can directly affect the final try-on effect. To tackle this problem, 6 7 utilized a praited and parser-based "teacher network" T to generate p':
p' = T (R. P = S (p'.g).
(4)
Yet they cannot handle that irresponsible teacher knowledge brings to the learning process of their "student network" S. [2] introduced the sycle-consistency structure that employed and trained cyclically two CNNs N1 and N2), where Wi provide p' No::N:
p' = N (R, W (R.P'.g).
(5)
where the structural difference between P and p is minimized to ensure cycle-consistency. However, decomposing the body and then synthesizing it back together based on semantics leads to occlusion and artifacts due to some parsing errors. Relying on prior knowledge weakens the learning of the garment of adaptive spatial structure on the human body. Additionally, the dual-model structure is complex, resulting in longer training times and unstable convergence. Inspired by [22]. we reformulate the synthesis process an unpaired garment style translation problem. Specifically, provide p' for ourselves (Gp) by ourselves wweght--shring structure):
p = G. (p' g), with p' G (p.g').
(6)
Inference. During inference, our goal is to synthesize the desired try-on result P just by feeding arbitrary garment-person pairs (p. g') you need into 9 and 9p:
P 9 (p.g), with g' e (9% (p. (p.g')g)).
3.2 Network Architecture
Non-rigid Garment Deformer (NGD) Go. The deformer Go can adopt arbitrary encoder-decoder networks. As shown in Eq. ( to fit Markov Random Field, we first pre-trained Go by feeding paired images (p, g) to learn real deformation field f, i.e. f = Go (p, g). At this point, pre-trained Go contains the full latent alignment information, including shape, color, perspective changes. texture variations, shadows and lighting, and depth of field, between the paired images (p.g). However, features such as color, shape, and texture do not need to be learned, because the input garment varies greatly during inference, but g and p used for training are paired. Therefore, Go is necessarily invalid when unpaired images are fed. To disentangle the aforementioned feature entanglement, we employ an auxiliary deformer GA, which takes the garment g. prior Densepose descriptor Pa E R.. Hx (25). and the pose heatmap Ph E R18x R18xHXW HXW as input to learn the same goal as pre-trained Go. i.e., f = G. (pa, (P,,Ph,.5 g). Because Pa and Ph are not correlated with y in terms of color, texture, and shape, GA does not face the challenges mentioned above and can handle any type of garment effortlessly. However, unfortunately, Pa and Ph also lack perspective changes, shadows, lighting,
5

---

and depth of field. As result, the deformation results obtained are often not realistic. Therefore, we use to supervise the training of GA. fA ~ f, supplementing its spatial perception abilities. After obtaining the optimal GΛ. we continue to train the deformer Go by taking unpaired (p. ga) as input and fĄ as supervision, to disentangle the feature entanglement of Go:
f ~ f, with f = Go gp. f - G (Pa. Ph. g').
(8)
Self-cyelic Image Generator (SIG) Gg- The generator GG can adopt arbitrary encoder-decoder networks. Our goal is to optimize 9.p by taking unpaired images (p, g'). The training process is divided into two steps: 1) generating the try-on result p' by taking unpaired images (p. g') as input, i.e. p' = Go (p. g Immediately after. 2) taking p' and warped garment as input of G to generate try-on result p. i.e. p = GΦ (p',g). network to provide prior pseudo-label p'. training Gep directly with self-cycle consistency is not work due to the absence of ground truth of p', 9 is more inclined to learn the mapping: p + when we jointly perform the two steps. Therefore, we introduce additional supervision on the upper body of the generated p' to take over the role of the label in [22]. In simple terms, the functionality of the person representation R has shifted from the input side to the supervision side. In addition, the self-cycle consistency can be ensured by enforcing p R p to establish relf-superviiion in step 2.
g). Different from the current methods that rely on an additional
3.3 Self-Cyclle Consistency Training Strategy
Training of NGD. As shown in Eq. [1. Eq. ② and (8. to achieve the MRF energy minimization, we design multiple loss functions. Edat implemented with the pixel-level Ci loss to encourage the similarity between and fA. Ereg is composed of the weights we and the smoothing term Rsm (implemented second-order smooth constraint Lsec (6). Eepir is implemented with L1 loss and VGG perceptual loss per (26 as the evaluation of structural similarity. Finally, we define the overall loss function of NGD as:
Eaga = Ldat + Xy + Ap (Li + pper);
where X, and Aa are hyperparameters that balances the relative contributions of sub-loss terms.
Training of SIG. To self- sulf-supervised cyclic optimization of SIG, we propose self-cycle consistency loss consisting of L loss and perceptual loss Lper, respectively, to maintain consistency between the try-on result p and the real person
Lsaye
|p- pll + ||p P per
(10)
To make p' close possible to the real distribution of person, employ discriminator D, to introduce adversarial loss:
D = Ep [log (D, (p)) + Egg, logg (1 Dz (9 (8))]
(11)
Additionally, the adversarial loss LG of generator Go is introduced simultaneously. In addition to the aforementioned losses, we propose a skin reconstruction loss Ler to forcefully supervise the skin reconstruction of p' in the arm and neck regions. To do this, we pre-trained model SR, which can be fed into deformed garment g' and parts of the skin Sp to reconstruct the envisaged arm and neck regions of p':
Lar = |RR (g' sp) P'll
(12)
where P9 denotes skin regions of p'. Furthermore, to generate the garment part of p' and preserve the invariant body content (e.g. head, trouser), we introduce the garment reconstruction loss Cgr and content preservation loss Cep:
Egr = 11B - g''.,,
Lop = Il - -Peli
(13)
where P'g denotes garment part of p', P'. and pe denote the same content of p' and p. respectively Finally, we define the overall loss function of SIG as:
C..g Ascyefsccye + X CC + Aur + Agr + Aep
(14)
To prevent the unreliable result p' generated in step wrongly guide step 2, we introduce an adjustable gated mechanism late in the training to prevent irresponsible hackpropagation:
1, |p -plp'1l, |lp -
Leig alh'sig with
0, otherwise,
(15)
where a the adjustable gated factor that determines whether backpropagation is enabled.
We provide the means to obtain Sp. P'. Pg. and pe in the supplementary material.

---

Person
Garment
CP-VTON-
ACGPN
DCTON
PF-AFN
SDAFN
ES-VTON
RT-VTON
Ours
Figure 4: Qualitative results of different methods (CP-VTON+ 27 6F SDAFN 28, RT-VTON FS-VTON and ours) in the unpaired setting.
ACGPN DCTON PF-
4
Experiments
Dataset. We conduct experiments using the VITON dataset 8 which consists of 16,253 image groups. Each group includes front-view female image p. in-shop garment image g with its mask PM. a reference semantic map Pp. and a pose heatmap Ph- The image size for each is 256 pixels 192 pixels. The training set contains 14,221 groups, and the test set contains 2,032 groups. All evaluations and visualizations are performed using the test set. Note that the in-shop garment and the garment worn on the person are identical in the training set, while they differ in the
Implementation Details. The USC-PFN is implemented in PyTorch and trained on a single Nvidia Tesla V100 GPU running Ubuntu 16.04. During training, a batch size of 16 is used for 100 epochs, and the Adam optimizer (29] is employed with parameters B, 0.5 and 3g 0.999, and the initial learning rate is set to le with linear decay after epochs. The USC-PFN consists NGD and SIG, both have the same structure Res-UNet [30], and the discriminator is from pix2pixHD [16]. In the loss functions, the A, = 20 and Ap = 0.25 the Engd- The Aecye 1, G 0.1, Agr = 50, Agr and dep 10 in the 2.g.'
Baselines. We perform comparative evaluations of our model's performance, which includes the deformer and generator, against publicly available state-of-the-art methods. To this end, leverage ten popular methods, including VITON[8) CP-VTON(3, Cloth-flow[9] CP-VTON+22 [27). ACGPN 4 DCTON(2). PF-AFN ZFlow 31], SDAFN 28). and [3), as baseline methods for quantitative evaluation. In particular, we select seven most cutting-edge methods for qualitative evaluation.
27
2 28 13 7
Evaluation Metries.
Quantitative evaluations are carried out in both paired and unpaired settings to compare our approach with baseline methods. For the paired setting, we use Structure Similarity (SSIM) 32 to evaluate the visual quality and diversity of the generated images. In the unpaired setting, as there no ground truth available, FID directly employed to evaluate the distributional similarity between the generated and real images. Note that do not use Inception Score (IS) (33

---

Table 1: Quantitative results of different methods on VITON, 'Def.' represents different warping methods. 'Syn.' represents different synthesizing paradigms. 'Parser' indicates whether the parser used in the model during inference. The best result are in bold and the second best result are in blue.
Methods VITON
Publication
Def.
Syn.
Parser
SSIM t
FID 4
8
CVPR 2018
TPS
IP
Y
0.74
55.71
CP- CP-TTON
ECCV 2018
TPS
IP
Y
0.72
24.45
Cloth-flow
ICCV 2019
AF
IP
Y
0.84
14.43
CP-VTON+
27
CVPRW 2020
TPS
IP
Y
0.75
21.04
ACGPN
4
CVPR 2020
TPS
IP
Y
0.84
16.64
DCTON
CVPR 2021
TPS
CC
Y
0.83
14.82
PF-AFN 6
CVPR 2021
AF
KD
N
0.89
10.09
ZFlow 31
ICCV 2021
AF
IP
Y
0.88
15.17
RT-VTON
13
CVPR 2022
MLS
IP
Y
11.66
SDAFN 28
ECCV 2022
AF
IP
N
0.88
12.05
USC-PEN (Ours)
This Work
MRF
SC
N
0.91
10.47
officiaal code data are not provided. IP: puinting; CC: cycle comsistency; KD: knowledge distillation SC: selle consistency.
the evaluation
14 35 as it only effective in datasets similar to ImageNet 36
Specifically,
lower score of FID indicates higher quality of the result.
4.1 Experimental Evaluation
Qualitative Results. In Figure we perform visual comparison with seven methods, including SOTA in-painting methods 27
Table 2: Analysis of time cost and computational com- plexity between ACGPN DCTON2], PF-AFN and ours.
the SOTA ey-
cle consistency method (2]. and SOTA knowledge distillation methods [6] 77
Methods
Time
#Params
FLOPs
FPS
In particular, we specifically select four striped garments to visually showcase the deformation effects of each method. indicates that all methods can achieve approximate garment alignment and try- on synthesis. However, in terms of gar- ment alignment, the TPS-based methods (columns 3, 4, and 5) exhibit excessive local fabric deformation and misaligned spatial correspon- dence. The AF-based methods (columns 7, and 8) demonstrate overly flexible and unconstrained deformation effects, which are particularly noticeable in the first row. The MRF-based method (col- umn 9) shows semi-rigid deformation effects in highly non-rigid body poses, such misalignment in the arm region. In terms of try-on synthesis, the in-painting methods (columns 3, 4, 7, and 9) exhibit mismatches between the given garments and the synthesized try-on results due parsing errors, particularly noticeable in the arms region. The cycle consistency method (column 5), being modular in nature, shows artifacts along the module boundaries. The knowledge distillation methods (columns 6 and 8) occasionally suffer from similar issues as the in- in-painting methods, both in terms of clothing deformation and try-on synthesis, due to irresponsible teacher knowledge. In contrast, our proposed method (last column) can achieve the most realistic results benefits from two key factors. Firstly, leverages MRF's perception of human spatial structure to faithfully simulate the process of cloth deformation after try-on, capturing realistic fabric draping effects. Secondly, it utilizes the self-cycle consistency paradigm to accurately reconstruct the appearance of the garment worn on body, all without the need for additional parsers complex designs.
ACGPN DCTON
w~0h
139M
206G
10
2
4444
153M
194G
19
PF-AFN 6
99M
69G
34
Ours
~32h
140M
46G
39
Quantitative Results. We evaluate the quantitative results of different methods under the same configuration for a fair comparison. We use SSIM for measuring structural similarity the paired setting and FID for measuring distributional similarity in the unpaired setting. Table T presents our quantitative results, including the SSIM and FID scores of baseline methods {8] 27 2
6. 3.31 28 [3] and the proposed USC-PFN, the VITON dataset. In terms of SSIM, USC-PFN
8

---

syo SUC
MRR
avlo
Persin
Garment
Flow
Flow
Flow
Flow
Full
w/jo
Vile MRE
or/o Cur
Persen
Garment
Flaw
Vlow
Flow
Flaw
Full
Figure 5; Ablation studies on the effect NGD and SIG in the unpaired setting.
surpasses the SOTA in-painting method SDAFN [28] by 0.03, the SOTA knowledge distillation method PF-AFN by 0.02, and the SOTA cycle consistency method DCTON [2] by 0.08. In terms of FID, it surpasses the SOTA in-painting method RT-VTON [3] by 1.19 and the SOTA cycle consistency method DCTON by 4.35, Although PF-AFN obtains a lower FID score, some of their results appear to be worse than ours f Additionally, PF-AFN requires complex training with the assistance of the teacher model. The result demonstrates the superiority of our self-cycle consistency approach while showcasing the significant advantage of generating high-quality try-on images.
addition, to demonstrate the superior performance our network, calculate the computational costs of ACGPN DCTON PF-AFN and method as shown in Table 2 On the same Tesla V100 GPU, we have an appropriate number of parameters and the fewest FLOPs (floating point operations), yet our inference speed is nearly four times faster than ACPGN and about twice as fast as DCTON. This demonstrates the real-time capability of our method and its ability to achieve high-quality visual performance with fewer computational resources. Note that our architecture can employ any encoder-decoder network, allowing for flexible selection of the network based computational resources.
Computational Complexity.
Ablation Study. In the ablation study, we evaluate the effectiveness of NGD and SIG in USC-PFN, Table Figure providd both quantitative and qualitative results. Firstly, we demonstrate that the auxiliary deformer GA successfully disentangles the color, texture, and shape correlations in the pre-trained Go. The experiments further validate the necessity of the data term Edat in MRF, as plays a crucial role in ensuring the accurate deformation of the garment. Moreover, we conduct an experiment by removing the entire MRF module emphasize its essential USC-PFN, Finally, the efficacy of fer in SIG is validated, emphasizing its critical role in supervising skin generation.
Limitations and Discussion. There are still limitations to USC-PFN that need be addressed in future work. Firstly, due to the complex non-rigid de- formation of garment images, it is chal- lenging to achieve good convergence dur- ing end-to-end training; that is, integrat- ing the garment deformer Go into the im- age generator G.p remains difficult task. Secondly, accurately aligning garments with human poses is challenging due to the complexity and variability of human
Table 3: Ablation studies of NGD and SIG in the unpaired setting. Lower score of FID indicates higher quality of results.
NGD
FID 4
SIG
FID +
w/o GA w/o Edat
42.23
w/o MRF
14.28
30.53
w/o Ler
15.44
Full NGD
25.44
Full SIG
10.47
poses. Thus, finding a more optimal closed-form solution necessary to solve the garment deforma- tion task. Finally, we still require extensive supervision during the training, and it is still urgent issue to find a solution to train a high-quality network with less supervision.
Conclusion
In this paper, we present USC-PFN, a parser-free virtual try-on network that utilizes a self-cycle consistency pipeline, which only uses one model to cyclically learn trying on different styles of
See visual comparisons in the supplementary material.

---

garments. To get more natural and realistic garment alignment, USC-PFN first incorporates the Markov Random Field as a non-rigid deformation method, by enhancing the deformer's 's perception of human spatial structure, thereby mimicking the natural interaction between the garment and the person. In addition, unlike existing paradigms, our method does not require input information from parsers, teacher knowledge, or complex person representations for the generator. Instead, it solely relies on the garment and human images as input to train and infer try-on results. The result indicates that USC-PEN outperforms the state-of-the-art methods significantly in terms of subjective and objective evaluations, demonstrating its superior performance. In the future, we plan to extend this architecture to different tasks of image-to- to-immae translation.
Acknowledgments and Disclosure of Funding
This work was in part supported by the National Key Research and Development Program of China (Grant No. 2022ZD0160604) and NSFC (Grant No. 62176194), and the Key Research and Development Program of Hubei Province (Grant No. 2023BAB083), the Project of Sanya Yazhou Bay Science and Technology City (Grant No. SCKI-JYRC-2002-766 SKJC-2022-PTDX PTDX--31 and the Project of Sanya Science and Education Innovation Park of Wuhan University of Technology (Grant No. 202 IFF03I...
References
[1] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference computer vision. pages 2223-2232, 2017.
[2] Chongjian Ge, Yibing Song, Yuying Ge, Han Yang. Wei Liu, and Ping Luo. Disentangled cycle consistency highly- realisti virtual try-on. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2021.
[3] Bochao Wang, Huabin Zheng, Xiaodan Liang, Yimin Chen, Liang Lin, and Meng Yang. Toward characteristic-pressene image-based virtual try-on network. In Proceedings of the European conference computer vision (ECCV), pages 589 209, 2018
[4] Han Yang, Ruimao Zhang, Xiaobao Guo, Wei Liu, Wangmeng Zuo, and Ping Luo. Towards photo- realistic virtual try-on by adaptively generating preserving image content. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition. pages 7850-7859. 2020.
generating-peservingg
[5] Ning Wang, Jing Zhang, Lefei Zhang, and Dacheng Tao. Gt-must: Gated try-on by learning the mannequin- specific transformation. In Proceedings of the 30th ACM International Conference on Multimedia, pages 2182-2190, 2022.
[6] Yuying Ge, Yibing Song, Ruiman Zhang, Chongjian Ge, Wei Liu, and Ping Luo. Parser-free virtual try-on via distilling appearance flows. In Proceedings of the IEFF//VV conference computer vision and pattern recognition, pages 8485-8493, 2021.
[7] Sen He, Yi-Zhe Song, and Tao Xiang, Style-based global appearance flow for virtual try-on. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3470-3479, 2022.
[8] Xintong Han, Zuxuan Wu, Zhe Wu, Ruichi Yu, and Larry Davis. Viton: An image-based virtual try-on network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7543-7552, 2018
[9] Xintong Han, Xiaojun Hu, Weilin Huang, and Matthew R Scott. Clothflow; A dow-based model for clothed person generation. In Proceedings of the IEEE/CVF international conference computer vision, pages 10471-10480, 2019.
[10] Ruiyun Yu, Xiaoqi Wang, and Xiaohui Xie. Vinfp: An image-based virtual try-on network with body and clothing feature preservation. In Proceedings of the IEEE/CVE international conference on computer vision, pages 1051 2019.
[11] Thibaut Issenhuth, Jérémie Mary, and Clément Calauzenes. Do not mask what you do not need to mask: a parser- free virtual try-on. In Computer Vision Vision-ECCV 2020: 16th European Conference, Glasgow, UK August 23-28, 2020, Proceedings, Part XX 16, pages 619 635. Springer, 2020.
10

---

[12] Benjamin Fele. Ajda Lampe, Peter Peer, and Vitomir Struc. C-vton: Context-driven image-based virtual try-on network. In Proceedings of the IEEE/CVE winter conference applications computer vision, pages 3144-3153, 2022.
[13] Han Yang, Xinrui Yu, and Ziwei Liu. Full-range virtual try-on with recurrent tri-level transform. In Proceedings of the IERE/CVF Conference on Computer Vision and Pattern Recognition, pages 3460-3469, 2022.
[14] Chenghu Du, Feng Yu, Minghua Jiang, Ailing Hua, Xiong Wei, Tao Peng, and Xinrong Hu. Vton-sefa: A virtual on network based on the semantic constraints and flow alignment. IEEE Transactions on Multimedia, 25:777-791, 2023.
[15] Nikolay Jetchev and Urs Bergmann. The conditional analogy gan: Swapping fashion articles people images. In Proceedings of the IEEE international conference computer vision workshops, pages 2287-2292, 2017.
[16] Ting- Wang. Ming- Liu, Jun- Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE conference on computer vision and pattern recogmition, pages 8798-8807, 2018.
[17] Fred L. Bookstein. Principal warps: Thin-plate splines and the decomposition of deformations. IEEE Transactions on pattern unalysis and machine intelligence, 116)):677-585, 1989.
[18] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. Advances in neural information processing systems, 28, 20155
[19] Tinghui Zhou, Shubham Tulsiani, Weilun Sun, Jitendra Malik, and Alexei A Efros. View synthesis by appearance flow. In Computer Vision-ECCV 2016; 14th European Conference, Amsterdam, The Netherlands, October 11-14, 14, 2016, Proceedings, Part 14, pages 286 301. Springer, 2016.
[20] Scott Schaefer, Travis McPhail, and Joe Warren. Image deformation using moving least squares. ACM SIGGRAPH 2006 Papers, pages 533-540. 2006.
[21] Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125 1134, 2017.
[22] Yunjey Choi, Minje Choi, Munyoung Kim, Jung- Joog-Woo Ha, Sunghun Kim, and Jacgul Choo. Stargam: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8789-8797, 2018
[23] Alexander Shekhovtsov, Ivan Kovtun, and Václav Hlaváč. Efficient mrf deformation model for non non-rigid image matching. Computer Vision and Image Understanding, 1 12(1))91199, 2008
[24] Carl Friedrich Gauss. Theoria motus corporum coelestium sectionibus conicis solem ambientium. volume 7. FA Perthes, 1877.
[25] RA Guler, Natalia Neverova, and IK DensePose. Dense human pose estimation the wild. Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, pages 18,23, 2018.
[26] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super- resolution. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I1 14, pages 694-711 Springer, 2016.
[27] Matiur Rahman Minar, Thai Thanh Tuan, Heejune Ahn, Paul Rosin, and Yu-Kun Lai. Cp-vton+: Clothing shape and texture preserving image-based virtual try-on. In CVPR Workshops, volume 3, pages 10-14, 2020.
[28] Shuai Bai, Huiling Zhou, Zhikang Li, Chang Zhou, and Hongxia Yang. Single stage virtual try-on via deformable attention flows. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 27, 2022, Proceedings. Part XV, pages 409 425, Springer, 2022.
[29] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 2015.
[30] Foivos I Diakogiannis, François Waldner, Peter Caccetta, and Chen Wu. Resunet-a: A deep learning framework for semantic segmentation of remotely sensed data. ISPRS Journal of Photogrammetry and Remote Sensing, 162.94 -14, 2020.

---

[31] Ayush Chopra, Rishabh Jain, Mayur Hemani, and Balaji Krishnamurthy. Zîlow: Gated appearance flow-based virtual try-on with 3d priors. Proceedings of the IEEE/CVF International Conference Computer Vision, pages 5433-5442, 2021.
[32] Kalpana Seshadrinathan and Alan C Bovik. Unifying analysis of full reference image quality assessment. In 2008 15th IEEE International Conference on Image Processing, pages 1200-1203, IEEE, 2008.
[33] Tim Salimans, Ian Goodfellow, Wajeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances neural information processing systems, 29, 2016.
[34] Shane Barratt and Rishi Sharma. A note the inception score. arXiv preprint arXiv: 100.018001.0193.. 2018.
[35] Mihaela Rosca, Balaji Lakshminarayanan, David Warde-Farley, and Shakir Mohamed. Variational ap-
proaches for auto-encoding generative adversarial networks. arXiv preprint arXiv: 2017.
[36] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. leee, 2009.
12


Reviews:
[{"review_id": "HxxFB6Sgmf", "reviewer": "Reviewer_nwZk", "summary": "The authors present a virtual garment try-on method. A cycle-consistency loss allows for self-supervision, i.e. the method does not require paired data (of the same person wearing different garments) as supervision. Unlike a previous method [2] that also uses cycle consistency, the same network weights are used for both the forward and backward step of the cycle. Additionally, a new garment deformation model based on Markov Random Fields is used instead of previous methods that use one of thin plate splines, appearance flow, or moving least squares. The authors show improved results over almost all previous methods, and similar results to one previous method [6].", "questions": "- Please clarify the setup that was used to compute SSIM.\n- Please clarify all differences that [2] that could cause the improved performance shown in the results.\n- Please describe all advantages of the proposed method over [6], and what you consider to be the most important advantages.", "limitations": "Some limitations have been discussed. Impacts from potentially biased datasets could be mentioned.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Cycle consistency using shared weights makes sense. It might seem like a small change, but does seem to require support from newly introduced losses.\n- The new deformation model is a contribution, although it is unfortunately not evaluated separately from the full try-on model.\n- The evaluation is reasonable and results show a significant advantage over the previous cycle-consistent method, and a significant advantage over most other methods (except one).", "weaknesses": "- The exact differences to [2] that give the proposed method an advantage over [2] are unclear. The authors allude to  advantages over [2] in several passages, but never fully describe what the differences to [2] are exactly (apart from re-using the same weights for both directions in the cycle) or where the advantages come from. For example:\n\t- In Section 2, the authors mention that [2] prioritizes paired garment-person images which can lead to difficulties with unpaired data. However, the proposed method also uses paired (garment, person) images, so what exactly is the difference? This needs to be clarified.\n\t- in Section 3.1, the authors mention a 'deconstruction of the human body' that leads to some problems in [2]. It is unclear what the authors refer to here. Do the authors refer to the Densepose descriptor used in [2]? If so, the authors would first have to mention that [2] needs to use the Densepose descriptor, so the reader has enough context to understand this statement. Just looking at the Eq 5 and 6, the only difference between the proposed method and [2] seems to be that the same weights are re-used for both directions. If there are any other differences, these need to be described explicitly near Eq 5 before referring to them. This needs to be clarified.\n\t- Also in Section 3.1, the authors mention that previous methods (including [2]) need an 'a-priori label'. Again, the authors need to describe what this refers to, otherwise the differences to [2] are hard to understand.\n- The existing method [6] seems to have a similar performance. A discussion of other advantages over [6] would be good.\n\t- FID is better for [6] while SSIM is better for the proposed method.\n\t- PF-AFN look a bit more natural to me in all examples of Figure 4 except the first row, and maybe some details in the fourth row. It looks more natural to me because it seems to better follow the shape of the body, the proposed method looks flatter, as if it does not fully follow the body shape.\n- The evaluation could be improved with additional ablations:\n\t- Using SIG with the same deformation method that [2] uses, to have a more direct comparison of using cycle consistency with shared vs. non-shared weights in the forward/backward steps.\n\t- Comparing the performance of NGD to previous deformation methods separately from the full try-on pipeline.\n- The exposition is often confusing and should be improved.\n\t- The introduction emphasizes that the method only uses unpaired data, but it seems paired (garment A, person wearing A) data was still used. The introduction should clarify which kinds of pairs were not used as training. I assume the author refer to (person wearing A, person wearing B) pairs, but this needs to be clarified in the introduction.\n\t- The description of the deformation fields in Eq. 1 and the paragraph before it is a bit confusing. $\\hat{f}$ is described as the optimal deformation field, but in the equation it is used as the variable that is being optimized (I would have expected  $\\hat{f}$ to be the result of the minimization in Eq. 1, rather than the variable being minimized over).\n\t- At the end of Section 3, 'infinitely close' is probably not the right phrasing, since i) its unclear what 'infinitely close' means (why not say 'the same'?) and ii) the goal is to reconstruct $p$ with $\\hat{p}$, but in practice, they will not be the same.\n\t- $R_{sm}$ is not defined in Eq. 2. The type of smoothing term should at least be described shortly.\n\t- A bit more information should be given in the weights $w_\\phi$ in Eq. 2. How is the Gaussian constructed? How are its mean and variance computed? I assume something like the centroid and variance of pixel coordinates in the cloth region. A brief mention would be good.\n\t- In Eq. 2 and 3 it would be good to denote which variables the minimization is over (i.e. $\\text{argmi}n_{\\hat{f}}$ or $\\text{argmi}n_{\\phi}$.\n\t- In Eq. 3, it should probably be \"$\\dots\\text{with} \\dots$\" instead of \"$\\dots s.t. \\dots$\"\n\t- Above Eq. 7 the authors mention that additional supervision is introduced for the upper body, but do not mention how it is used, what it consists of, etc. If this is described later on, it would be better to not mention this yet, to avoid confusion, or to re-structure the sections so the reader does not need to know information that will only be provided later on to understand the current text.\n\t- In Eq. 7 the notation is confusing. I did not follow why every variable has an additional ' in the notation. Why not use the same input notation as before: $(g', p)$, and output is $\\hat{g}'$?\n\t- In Section 3.2, some design choices should be discussed:\n\t\t- Why is $\\mathcal{G}_\\mathcal{A}$ not used directly instead of $\\mathcal{G}_\\theta$? (I assume because the authors want to avoid using Densepose in the cycle training, but it should be discussed why, and ideally an ablation should be given.)\n\t\t- Why does $\\mathcal{G}_\\theta$ need to be pre-trained instead of directly starting training with $\\mathcal{G}_\\mathcal{A}$ and then finetuning?\n\t- Eq. 8 is unclear, it would be clearer to explicitly describe the loss used to train $\\mathcal{G}_\\theta$ in the second phase, probably something like $\\min_\\theta \\mathcal{G}_\\theta(g', p) - \\mathcal{G}_\\mathcal{A}(g', p_d, p_h)$\n\t- In Section 3.3, the definition of $L_{sec}$ should be given, or at least a short description of what this loss does.\n\t- In Eq. 9, $\\mathcal{L}_{per}$ is not defined, it is only defined later on in the next subsection.\n\t- Above Eq. 15, step 1 and step 2 are not defined. These probably refer to the forward and backward steps in the cycle, but they should be defined more explicitly.\n\t- The loss in Eq. 15 needs a bit more motivation. why is backpropagation disabled if $\\hat{p}'$ is more similar to $p$ than $\\hat{p}$? Could this not happen quite frequently at the start of the training? Some discussion is needed.\n\t- A bit more information is needed how SSIM is computed. Is it computed by using the garment as input that the input person is already wearing, and is a held-out test set of paired data used?\n\t- In the ablation study when removing the MRF module, what deformation method is used instead?\n\nDetails:\n- Near line 80: self-consistency consistency -> self-cycle consistency\n- In Section 3.2, the paragraph titles should give the acronyms of the two modules in paranthesis (NGS, SIG), since the acronyms are used later on in the text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Thanks for your diligent efforts and valuable suggestions. We have provided both quantitative and qualitative results of our garment deformer on VITON and a high-definition VITON-HD (512×384) datasets (see https://github.com/anony-conf/results-USC-PFN).\n---\n### Weaknesses:\n- **W-A1:** The differences with [2] are: 1) [2] uses dual-network but only one network for inference, leading to convergence challenges and increased computational overhead. Our method achieves efficient convergence with a single shared-weight network, requiring fewer parameters and FLOPs. 2) [2] relies on human parsing as input during both training and inference, making the results sensitive to incorrect parsing. Our results only takes garment and person images as input, thus avoiding this issue. 3) [2] segments clothing and skin, which may cause artifacts at boundaries after their synthesis. 4) [2] introduces a multi-encoder network for feature extraction, adding complexity. Our method can utilize any generator as our training network, achieving results surpassing [2]. 5) [2] uses a globally deformable STN for clothing deformation. We introduce an MRF-based deformator achieving SOTA performance. 6) Our method has more advanced performance than [6]. In summary, our approach offers a novel solution for virtual try-on tasks.\n---\n- **A1-1:** Sorry for your confusion. In Sec. 2, the \"self-cycle training\" is our own architecture, not [2]. [2] itself refers to its approach as 'disentangled cycle-consistency.'\" Moreover, both our method and [2] utilize paired images.\n---\n- **A1-2:** In Sec. 3.1, 'deconstruction of the human body' refers to [2] introduced a parsing  network to generate skin and clothing separately. Incorrect parsing can lead to erroneous try-on results. Our method only take clothing and person images as inputs, thus avoiding the issues.\n---\n- **A1-3:** The term 'a-priori label' refers to the use of the Densepose or human parsing as input. Our network completely eliminates any such a-priori labels from the input.\n---\n- **W-A2-1:** Thanks for your comments. FID is not universally suitable for [6], as there are some erroneous images in datasets, which might lead to slight disadvantages in FID. We utilized a publicly available augmented VITON dataset for the purpose of generalization validation. The table below demonstrates that our FID significantly outperforms PF-AFN.\n\n|Methods|SSIM ↑|FID ↓|\n|:--:|:--:|:--:|\n|ACGPN|0.81|20.75|\n|Cloth-flow|0.86|13.05|\n|PF-AFN|0.87|12.19|\n|**Ours**|**0.90**|**10.48**|\n---\n- **W-A2-2:** In Fig. 4, PF-AFN exhibits excessive deformation of the lines on the abdomen and arms, which is highly unnatural and indicative of clothing deformation failure. Our results adhere to the body shape, especially in the arm regions. Some instances appearing flattened are due to our suppression of excessive folding during training, as an abundance of folds could potentially obscure clothing details.\n---\n- **W-A3-1:** We replaced NGD with the STN from [2] to generate try-on results. The qualitative and quantitative results are shown in Table below and Fig. 8.1 (https://github.com/anony-conf/results-USC-PFN).\n\n|Methods|SSIM ↑|FID ↓|KID ↓|\n|:--:|:--:|:--:|:--:|\n|DCTON [2]|0.83|16.32|0.915|\n|**Ours**|**0.89**|**10.29**|**0.229**|\n---\n- **W-A3-2:** We have conducted qualitative and quantitative experiments for NGD on both VITON (Table 1) and the high-definition VITON-HD datasets (Table 2). See Table below and Fig. 8.1.\n\n|Methods|Pub.|Warping|FID-P ↓|KID-P ↓|FID-UP ↓|KID-UP ↓|\n|:--:|:--:|:--:|:--:|:--:|:--:|:--:|\nACGPN|CVPR 2020|TPS|42.10|2.009|41.48|2.048|\nDCTON|CVPR 2021|TPS|42.80|2.170|42.19|2.126|\nPFAFN|CVPR 2021|AF|22.81|0.785|23.90|0.860|\nSGAFN|CVPR 2022|AF|20.07|0.552|20.38|0.481|\n**Ours** (NGD)|/|**MRF**|**18.70**|**0.390**|**19.50**|**0.355**|\n---\n|Methods|Pub.|Warping|FID-P ↓|KID-P ↓|FID-UP ↓|KID-UP ↓|\n|:--:|:--:|:--:|:--:|:--:|:--:|:--:|\n|VITON-HD|CVPR 2021| TPS|32.968|1.407|32.93|1.353|\n|HR-VITON|ECCV 2022|AF|25.499|0.926|24.826|0.759|\n|**Ours** (NGD)|/|**MRF**|**19.060**|**0.418**|**22.861**|**0.504**|\n---\n- **W-A4:** Thanks for your professional and meticulous comments, which are crucial for enhancing the quality of our paper. We have made major revisions based on these suggestions. However, due to the limitations of the rebuttal space, we regret that we are unable to write down the answer to each of your concerns raised below individually. Once again, we sincerely appreciate your diligent efforts.\n---\n### Questions:\n- **Q-A1:** The setup that was used to compute SSIM has already been explained on the open-source webpage (https://github.com/anony-conf/USC-PFN/).\n---\n- **Q-A2:** All differences between our work and [2] have been summarized in the aforementioned **W-A1**.\n---\n- **Q-A3:** 1) [6] relies on prior knowledge to guide the student model, which may involve irresponsible teacher knowledge. Our approach does not employ a similar structure; instead, it self-guides for convergence. 2) [6] is unable to mitigate the impact of errors in the teacher knowledge, whereas we have avoided this aspect. 3) [6] designs a complex structured deformer, while we have the flexibility to adopt any generator.\n- The most significant advantage is that [6] requires meticulously designed and unique complex clothing deformer, whereas we can employ any network to serve the architecture, resulting in more realistic deformation effects. In try-on synthesis stage, [6] relies on a parser-based prior model, while we rely solely on our own approach. Hence, our method offers a completely novel approach and architecture for the virtual try-on task.\n---\n### Limitations:\n- **A:** We conducted experiments on VITON-HD and augmented VITON datasets to demonstrate the generalization and robustness of our method, unaffected by potential biased dataset influences.\n---\n> We hope that our responses have provided you with new insights into our work. We look forward to you giving us a chance by increasing your approval of our paper."}, {"review_id": "KP7NIXh9SA", "reviewer": "Reviewer_ugBm", "summary": "The paper presented a system for image based virtual try-on. The main contribution consists (1) a parser free virtual try-on network trained with unpaired data; (2) a MRF based deformation estimation network; (3) a cycle consistency based training method. The paper performed experiments on the VITON Zalando dataset. The paper is evaluated with several prior work as baselines, including VITON[8], CP-VTON[3], Cloth-flow[9] etc. The main evaluation metrics are SSIM for pair data and FID for data with no ground truth. In terms of quantitative performance, the proposed method achieves better or comparable results than previous art. In terms of qualitative results, the generated images are visually comparable, and some times slightly better than previous work.", "questions": "Please address the aforementioned questions.", "limitations": "The limitation is discussed in section 4 and seems sufficient to me.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The paper is addressing an interesting problem.\n\n+ The main idea is interesting and seems novel to me. Although cycle consistency are not new ideas and it has been used in previous work on virtual try on, the overall combination of ideas seems still new in this specific domain.\n\n+ Extensive experiment shows state-of-the-art quantitative results. The qualitative results have less deformation artifact than previous work.", "weaknesses": "I'm not an expert on virtual try-on so I will mainly rely on the other reviewers opinion for a more informative feedback about the weakness of the paper. On feedback I have is that the results are often overly smooth on the garment area. The paper showed a lot of results of clothing with stripe pattern and the stripes are often very smooth, and not reflecting the pose and shape of human body very well.This is perhaps due to the way that the deformation is learned. It would be great to discuss the potential cause of this over-smoothness and also perform ablations studies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Thank you for your diligent efforts and valuable suggestions in the peer review process.\n---\n### Strengths:\n**S-A:** Thanks for your comments. There is a significant difference between our main idea and the disentangled cycle consistency approach [A]. The differences are summarized as follows:\n\n1) [A] uses dual-network during training, but only uses one network for inference, leading to convergence challenges and increased computational overhead. In contrast, our method achieves efficient convergence with a single shared-weight network, requiring fewer parameters and FLOPs.\n\n2) [A] relies on human parsing as input during both training and inference, making the results sensitive to incorrect parsing. Our approach only takes garment and person images as input, thus avoiding above issue.\n\n3) [A] employs disentangled cycle consistency to segment clothing and skin, which may cause artifacts at boundaries after their synthesis. Our method directly uses the whole person image as input, thus avoiding this issue.\n\n4) [A] introduces a multi-encoder network for feature extraction, adding complexity. Our approach can utilize any generator as our training network, achieving results surpassing [A].\n\n5) [A] uses a globally deformable STN network for clothing deformation. We introduce an MRF-based clothing deformator achieving SOTA performance.\n\n6) Our method outperforms [A] in terms of realism, algorithmic complexity, model size, FPS, and generalization.\n- In summary, our approach possesses significant advantages and impact to offer a novel solution for virtual try-on tasks.\n\n### Weaknesses:\n- **W-A1:** Thanks for your comments. In fact, this task is based on deep learning for image generation, thus sharing commonality with other image generation tasks. In comparison to previous virtual try-on literature, our paper provides comprehensive qualitative and quantitative experiments, comparing against state-of-the-art methods [2,3,4,6,7,8,9,13,27,28,31], to demonstrate the effectiveness of the proposed framework. The novelty of our approach is outlined in the introduction and related work sections. Moreover, we have adequately addressed and supplemented relevant experiments raised by other reviewers. Therefore, we sincerely appreciate your impartial and responsible evaluation. We have conducted experiments on selected state-of-the-art literature to showcase the comprehensiveness of our experiments, and the results are as follows.\n\n|Methods|Publication|Number of Baselines|Qualitative / Quantitative Results on Clothing Deformation|High-Resolution Dataset VTON-HD|\n|:--:|:--:|:--|:--:|:--:|\n|[2]|CVPR 2021|5 ( [ 3, 4, 8, 15, 27] )         |×/×|√|\n|[6]|CVPR 2021|5 ( [ 3, 4, 9, 11, 27] )         |×/×|√|\n|[28]|ECCV 2022|6 ( [ 3, 4, 6, 9, 11, 31] )     |×/×|×|\n|[7]|CVPR 2022|8  ( [2, 3, 4, 6, 8, 9, 27, 31] )|×/×|×|\n|[13]|CVPR 2022|3  ( [27, 4, 2] )               |√/×|×|\n|Ours|This Work|10                              |Added/Added|Added|\n- As can be observed, the experiments conducted for our method are thorough and comprehensive.\n---\n- **W-A2:** Sorry for your confusion. Firstly, the selection of a substantial number of striped garments in the experiments was done to highlight the detailed and efficient performance of the newly proposed clothing deformation algorithm, MRF, in clothing deformation. This is because choosing clothing with large solid color blocks and no patterns would not showcase the significant pixel displacement caused by excessive deformation, whereas striped patterns allow for the clear visualization of the direction of lines when subjected to extensive deformation.\n\n- Regarding the challenge of dealing with overly smooth clothing regions, this is indeed a rather tricky aspect of virtual try-on tasks. Through preliminary experimental validation, we observed that **the perceptual loss in the overall loss function directly affects the smoothness of the clothing.** When the smoothness is excessive, there are fewer wrinkles in the clothing; conversely, there are more wrinkles when it's less smooth.\n\n- To validate this hypothesis, we conducted ablation experiments regarding the perceptual loss and $L_1$ loss. We controlled the hyperparameters of both to adjust their significance in the overall loss. The results of the ablation experiments are presented in Table 1 and Figure 8.2 (https://github.com/anony-conf/results-USC-PFN).\n---\n\n|$\\lambda_1$|$\\lambda_p$|FID ↓|\n|:--:|:--:|:--:|\n|0|0|10.66|\n|1|0|17.45|\n|0|1|10.76|\n|1|1|**10.57**|\n|1|10|11.98|\n|10|1|11.46|\n\n---\n- It can be observed that the network converges well only when the perceptual loss is present. When $\\lambda_1=1$ and $\\lambda_p=1$, meaning both are not artificially controlled, the convergence is optimal. Furthermore, from the fourth column of Figure 8.2, it can be seen that the wrinkles on the clothes are most realistic when both losses are utilized, while the second column without the perceptual loss exhibits an excessively smooth appearance. Thus, we have preliminary evidence that overly smooth clothing is a result of the perceptual loss. We can infer that the introduction of the adversarial loss would lead to more realistic wrinkles in the clothing.\n\n- However, overall, the appearance of wrinkles results in localized darkened pixels. If not controlled properly, these darkened regions might appear in unexpected areas of clothing, thereby obscuring the original details of the clothing. As a result, current virtual try-on methods primarily focus on the authenticity of clothing deformation first, and then address finer details like wrinkles.\n---\n> We sincerely thanks for your dedicated efforts. We hope that our responses have provided you with new insights into our work. We look forward to you giving us a chance by increasing your approval of our paper. If you have more questions or encounter broken links, please comment, and we'll assist you quickly."}, {"review_id": "c6KhaXC369", "reviewer": "Reviewer_XB1J", "summary": "The paper addresses the challenges in generating high-quality virtual try-on images, specifically focusing on non-rigid garment deformation and unpaired garment-person images. Existing methods rely on disentangling garment domains with the aid of \"teacher knowledge\" or dual generators, which can limit the quality of try-on results. Additionally, current garment deformation techniques struggle to mimic natural interaction between garments and the human body, resulting in unrealistic alignment effects. To overcome these limitations, the authors propose a Unified Self-Cycle Consistency for Parser-Free virtual try-on Network (USC-PFN). USC-PFN utilizes a single generator and incorporates Markov Random Field for more realistic garment deformation.", "questions": "Does the paper include a comparative analysis between the proposed warping method and other existing warping methods? I contend that it is imperative to conduct a direct comparison, encompassing both qualitative and quantitative evaluations, with well-established warping techniques such as TPS transformation and Appearance Flow.", "limitations": "Mentioned above.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "- The paper is easy to understand.\n- The experiment results seems okay, but not excellent compared to the baselines.\n- The proposed method requires less computational costs compared to the baselines such as ACGPN and DCTON during inference.", "weaknesses": "- The primary limitation of this paper lies in the lack of significant performance improvement compared to the existing virtual try-on baselines. Particularly, when examining the performance comparison in Table 1, it raises doubts about the meaningful enhancement in terms of FID, which is even higher than that of PF-AFN. Furthermore, the disparity in frames per second (FPS) between PF-AFN and the proposed approach is not significant.\n- The proposed self-cyclic image generator is inspired by StarGAN; however, it is unclear what specific advantages it offers. It appears to train a generator with clothing as a condition instead of employing other models that utilize cycle consistency loss. Nevertheless, it is uncertain whether this approach is novel and impactful enough to be presented at a top-tier conference..\n- The newly proposed non-rigid garment deformer based on Markov Random Field lacks sufficient examples to demonstrate its superior warping capabilities compared to existing warping methods. Various virtual try-on methods have also proposed different approaches to enhance the performance of warping. However, it remains unclear how the proposed method establishes its superiority in this regard qualitatively and quantitatively.\n- Most recent virtual try-on models have demonstrated their performance on high-resolution datasets such as VITON-HD [1] or Dresscode [2]. Additionally, parser-free virtual try-on models like DC-VTON have also shown their performance on images of around 512x384 resolution. However, this paper is lacking references to related works and only presents evidence of its performance on the low-resolution VITON dataset, which is somewhat disappointing.\n\n[1] VITON-HD: High-Resolution Virtual Try-On via Misalignment-Aware Normalization, CVPR 2021\n[2] Dress Code: High-Resolution Multi-Category Virtual Try-On, ECCV 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Thank you for your diligent efforts and valuable suggestions.\n---\n### Weaknesses:\n- **W-A1:** Thanks for your comment. Our method significantly outperforms both RT-VTON (CVPR 2022) and SDAFN (CVPR 2022), which are more advanced than PFAFN. Moreover, our method outperforms PFAFN in KID and SSIM. We compare the performance improvements of the three methods over PFAFN; see Table 1.\n\n|Table 1|SSIM↑|FID↓|KID↓|SSIM*|FID*|KID*|\n|:--:|:--:|:--:|:--:|:--:|:--:|:--:|\n|PFAFN|0.89|10.06|0.264|/|/|/|\n|RT-VTON|-|11.66|-|-|- 14.2%| - |\n|SDAFN|0.88|12.05|-|- 1.1%|- 18.0%| - |\n|**Ours**|**0.91**|10.47|**0.249**|+ 2.2%|- 4.1%| + 5.7%|\n- In Table 1, our method shows the most significant improvement. In addition, we introduced a publicly available augmented VITON test set (see Table 2) and the high-definition VITON-HD dataset for a comprehensive and fair evaluation.\n\n|Table 2|SSIM ↑|FID ↓|\n|:--:|:--:|:--:|\n|ACGPN|0.81|20.75|\n|Cloth-flow|0.86|13.05|\n|PF-AFN|0.87|12.19|\n|**Ours**|**0.90**|**10.48**|\n- In Table 2, the SSIM of our method reaches 0.90, an increase of 3.4% compared to PFAFN. The FID is 10.48, which is a reduction of 14% compared to PFAFN. Next, we counted the performance of baseline methods on VITON-HD (512$\\times$ 384) (see Table 3).\n\n|Table 3|Pub.|Parsing as input|SSIM ↑|FID ↓|KID ↓|\n|:--:|:--:|:--:|:--:|:--:|:--:|\n|CP-VTON|ECCV 2018|Y|0.791|30.25|4.012|\n|ACGPN|CVPR 2020|Y|0.858|14.43|0.587|\n|DCTON|CVPR 2021|Y|0.810|15.55|/|\n|VITON-HD|CVPR 2021|Y|0.843|11.64|0.300|\n|HR-VITON|ECCV 2022|Y|0.878|9.90|0.188|\n|**Ours**|/|**N**|**0.899**|**9.10**|**0.159**|\n- In Table 3, our approach has also surpassed the SOTA method, HR-VITON, this demonstrates the significant improvements in performance, highlighting the significance of our method. **The qualitative results on the VITON-HD are in Fig. 4 (https://github.com/anony-conf/results-USC-PFN).**\n- Furthermore, regarding FPS, we mentioned in line 275 of the paper that the parameters, FLOPs, and FPS of our network can vary with the used network. We intentionally reduced the *ngf* parameter in NGD (see Table 4), both parameters and FLOPs are significantly reduced.\n\n|Table 4|#Params|FLOPs|FPS|FID-Clothing ↓|SSIM-TryOn ↑|\n|:--:|:--:|:--:|:--:|:--:|:--:|\n|ACGPN|139M|206G|10|42.10|0.84|\n|DCTON|153M|194G|19|42.80|0.83|\n|PF-AFN|99M|69G|34|22.81|0.89|\n|Ours (All ngf=64)|140M|46G|39|18.70|0.91|\n|Ours (NGD ngf=32)|**87M**|**29G**|**40**|**18.85**|**0.91**|\n- We also calculated the complexity on VITON-HD (see Table 5).\n|Table 5|Weight|#Params|FLOPs|FPS|\n|:--:|:--:|:--:|:--:|:--:|\n|VITON-HD|588M|154M|1689.7G|3.76|\n|HR-VITON|586M|148M|1555.4G|4.09|\n|**Ours**|**334M**|**87M**|**467.9G**|**22.19**|\n---\n- **W-A2:** Our method offers a novel viewpoint for virtual try-on tasks. In comparison to previously proposed solutions like disentangled cycle consistency [A] and knowledge distillation [B], its advantages are summarized as:\n1) [A] uses dual-network during training, but only uses one network for inference, leading to convergence challenges and increased computational overhead. In contrast, our method achieves efficient convergence with a single shared-weight network, requiring fewer parameters and FLOPs.\n2) [A] relies on human parsing as input during both training and inference, making the results sensitive to incorrect parsing. In [B], teacher network training and inference depend on the parser, where erroneous human parsing in teacher knowledge can mislead the student network. Our approach only takes garment and person images as input, thus avoiding above issues.\n3) [A] employs disentangled cycle consistency to segment clothing and skin, which may cause artifacts at boundaries after their synthesis. Our method directly uses person images as input, avoiding this issue.\n4) [A] introduces a multi-encoder network for feature extraction, adding complexity. [B] employs a complex clothing deformator for clothing deformation. Our approach can utilize any generator as our training network, achieving results surpassing [A] and [B].\n5) [A] uses a globally deformable STN network for clothing deformation. [B] employs less controllable appearance flows for clothing deformation. We introduce an MRF-based clothing deformator achieving SOTA performance.\n6) Our method outperforms the methods based on [A] and [B] in terms of realism, algorithmic complexity, model size, FPS, and generalization.\n- In summary, our approach possesses significant advantages and impact to offer a novel solution for virtual try-on tasks.\n---\n- **W-A3:** We have added qualitative and quantitative results on VITON (see Table 6) and VITON-HD (see Table 7) datasets. Qualitative results are shown in Fig. 3 and 4 (https://github.com/anony-conf/results-USC-PFN).\n\n|Table 6|Pub.|Warping|FID-P ↓|KID-P ↓|FID-UP ↓|KID-UP ↓|\n|:--:|:--:|:--:|:--:|:--:|:--:|:--:|\nCP-VTON|ECCV 2018|TPS|43.95|2.233|42.13|2.112|\nACGPN|CVPR 2020|TPS|42.10|2.009|41.48|2.048|\nDCTON|CVPR 2021|TPS|42.80|2.170|42.19|2.126|\nPFAFN|CVPR 2021|AF|22.81|0.785|23.90|0.860|\nSGAFN|CVPR 2022|AF|20.07|0.552|20.38|0.481|\n**Ours**|/|MRF|**18.70**|**0.390**|**19.50**|**0.355**|\n---\n|Table 7|Pub.|Warping|FID-P ↓|KID-P ↓|FID-UP ↓|KID-UP ↓|\n|:--:|:--:|:--:|:--:|:--:|:--:|:--:|\n|VITON-HD|CVPR 2021| TPS|32.968|1.407|32.93|1.353|\n|HR-VITON|ECCV 2022|AF|25.499|0.926|24.826|0.759|\n|**Ours**|/|MRF|**19.060**|**0.418**|**22.861**|**0.504**|\n---\n- **W-A4:** Thanks for your suggestion. We have added qualitative and quantitative experiments on the high-resolution dataset VITON-HD with a resolution of $512\\times 384$, as shown in Table 3, and Fig. 3 and 4 (https://github.com/anony-conf/results-USC-PFN).\n---\n### Questions:\n- **Q-A1:**  We have added direct comparison experiments with well-established warping techniques such as TPS and Appearance Flow, please refer to **W-A3**.\n---\n> Thanks for your dedicated efforts. We hope that our responses have provided you with new insights into our work. We look forward to you giving us a chance by increasing your approval of our paper."}, {"review_id": "xEP1zsDki4", "reviewer": "Reviewer_RUsU", "summary": "This paper proposes a new parser-free virtual try-on network (USC-PFN) to use only unpaired images as input to generate realistic try-on results. \nTo address the core warping problem in virtual try-on, it models the deformation field estiamtion by using the Markov Random Field. To train the try-on generator by using unpaired data, it proposes a self-cycle consistency pipeline.\nExtensive  comparisons with the state-of-the-art methods on VITON benchmark demonstrate its superiority and the ablation study also shows the effectiveness of different modules in the proposed method.", "questions": "- For the non-rigid deformation module, is the pre-trained deformer necessary for the training of the final deformer? In my opinon, it is unnecessary to use supervision provided by the pre-trained deformer  (which is trained by using paired images) for the training of auxiliary deformer, since the ground truth deformation field is not necessary for the training of deformer (velidated in the previous works like PF-AFN[1], FS-VTON[2]). Once obtaining the auxiliary deformer, we could train the final deformer by using the unpaired images.\n\n- When training the Self-cyclic Image Generator (SIG), several local region supervisions (i.e., skin loss, garment loss, preserved content loss) are employed to facilitate the self-supervised cyclic training, which should resort to the human parsing to obtain the specific local region. Thus, the parsing error might still affact the training procedure. However, some knowledge distillation method like PF-AFN[1] does not face with parsing error issue since they only use the global supervision when the the parser-free student network. My question is how does USC-PFN alleviate the influence derived the parsing error during training.\n\n- Since the SIG can be trained with unpaired images, is it possible for USC-PFN to leverage large amout of unpaired images from Internet during the trianing of SIG? Will such the increasing in training data facilitate the performance of USC-PFN?\n\n[1] Ge et al. \"Parser-Free Virtual Try-on via Distilling Appearance Flows\", CVPR 2021.\n\n[2] He et al. \"Style-Based Global Appearance Flow for Virtual Try-On\", CVPR 2022.", "limitations": "Some limitations of this paper have been disscussed in the main paper. Other limitation I concern is the image resolution, since it is quite improtant for the real world try-on scanerio. ", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This paper explores the parser-free virtual try-on, which is challenging and with great significance for the image-based virtual try-on.\n\n- For the first time, this paper introduces the Random Markov Field into the non-rigid garment deformer, which is quite different from the deformation module in previous methods.\n\n- It proposed a novel self-cycle consistency pipeline for the training of the try-on generator by using unpaired images.\n\n- The authors conduct extensive comparisons with the existing state-of-the-art methods to illustrate the superiority of the proposed methods. The ablation study also show the effectiveness of the different modules in the proposed method. ", "weaknesses": "- One core technical contribution in this paper is that it model the garment warping by using the Markov Random Field. However, the authors do not provide an clear explanation about it. What is the difference between it and the widely used appearance flow? why can it outperform the TPS-based or appearance flow-based methods?\n\n- The authos only conduct experiments on the VITON benchmark, in which the image resolution is quite low (256 x 192). However, most of the advanced methods focus on the higher resolution virtual try-on (e.g., 512 x 384, 1024 x 768), which are more closed to the real world \ntry-on scanerio.\n\n- The writing is not straightforward amd several descriptions are a bit obscure. For example, (1) Although  the garment deformer and the auxiliary deformer receive different inputs, in Figure 3, they seems to take the same inputs (i.e., person image and the garment image). (2) In line131, it is confused how to obtain the deformation field $\\tilde{f}$.\n\n- Some decipt in the main paper is a bit overclaimed. In line 300, the authors claim that USC-PFN can solely rely on the garment and human image for training. However, it still required some human condition like human parsing, densepoe during training NGD and SIG.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Thank you for your diligent efforts and valuable suggestions.\n---\n### Weaknesses:\n- **A1:** Thanks for your comments. Datasets consist of paired garment-person images $(g,p)$, where $p$ wearing $g$. Existing methods use the person representation $I_p$ and $g$ to find dense spatial correspondences between $g$ and $p$. They adjust garment shape to minimize shape differences between deformed garment $\\hat g$ and the Ground Truth (GT, garment on $p$). Due to the lack of GT flow field, appearance flow (AF) -based methods indirectly predict the flow field $F$ by the supervision of warped garment's shape ($\\hat g$) [19], i.e., calculate the loss between $\\hat g$ and GT. Two issues arise: 1) $I_p$ lacks $p$'s depth and spatial info, leading to rigid garment deformation and unrealistic fitting. 2) Optimizing $F$ indirectly by optimizing $\\hat g$'s shape with pixel similarity disregards structural and depth correlations between $\\hat g$ and $p$, causing excessive deformation. Similarly, TPS-based methods predict control points to calculate the same flow field. So, the drawbacks of both methods are the same. And TPS is more rigid.\n- Our method tackled the problems. 1) We use $p$ instead of $I_p$ as input to input depth and spatial info. But $p$ and $g$ are paired, to solve this issue, 2) we introduce Markov Random Field (Eq. 1) for clothing deformation, which supervises the estimated deformation field $f$ by GT field $\\bar f$. However, $\\bar f$ does not exist in the dataset. Therefore, we introduce the auxiliary deformer $G_A$ to learn depth and spatial info from $p$. $G_A$, pretrained with densepose descriptor $p_d$, is used to remove irrelevant priors (color, texture, and shape features of $p$'s garment) in extracted features of $p$, ensuring only depth and spatial info remains. Thus, GT field $\\bar f$ can be generated by $G_A$.\n- In summary, our approach directly eliminates relevant info from the feature of $p$ containing rich spatial and depth info, and employs MRF principles to directly supervise $f$, enhancing the model's spatial awareness. We provide both qualitative and quantitative results on low-resolution and high-resolution datasets in **A2**, to demonstrate the effectiveness of our deformer.\n---\n- **A2:** Thanks for your suggestion. To demonstrate the effectiveness of our method, we have added qualitative and quantitative results of both NGD and SIG on VITON-HD dataset in https://github.com/anony-conf/results-USC-PFN , Sec. 1 to 5.\n---\n- **A3:** (1) We sincerely apologize for the ambiguity in Fig. 3. There indeed are differences in their inputs, as explained in section 3.2 and **A1**. The garment deformer takes person $p$ and garment $g$ as inputs, while the auxiliary deformer takes the densepose descriptor $p_d$ of $p$, and $g$ as inputs. We have revised Fig. 3 to enhance its clarity. (2) The deformation field $f$ is directly generated by the $G_\\theta$, i.e., $G_\\theta(g,p)=f$, and $\\hat g$ is obtained by bilinear sampling of $g$ using $f$.\n---\n- **A4:** We deeply apologize for our inaccurate description. We intended to convey that USC-PFN can indeed be trained and inferred solely based on garment and person images **as input**.\n---\n### Questions:\n- **A1:** Thanks for your professional suggestion. Our auxiliary deformer $G_A$ is necessary, this is because we attempted to directly train the main deformer $G_\\theta$ using paired ($g$, $p$), but features extracted from $p$ contain abundant shape, color, and texture correlations with $g$, which hinders the generalizability of $G_\\theta$. Consequently, we had to separately train $G_A$ to eliminate latent shape, color, and texture correlations from $p$'s features, enabling feasible training with paired images. The specific process is detailed in above **A1**.\n---\n- **A2:** Thanks for your insightful comments. Firstly, we employ a global self-cycle consistency loss $L_{scyc}$, which does not introduce parsing errors. For skin loss, we utilize a pre-trained SR trained using perceptual loss to generate skin regions, it produces clean skin outputs even when there are cloth-related impurities in the input. This is fully in accord with the consistency of the distribution of skin in perceptual loss. For garment loss, it is only used in first phase. We segment the garment area of $\\hat g$ and $p$ using the mask of $\\hat g$. As our network's input is also segmented $\\hat g$, this loss ensures direct $\\hat g$ output by SIG, so it only possibly introduces white boundaries caused by parsing errors. We address this via $L_{scyc}$ in second phase, which enforces supervision via GT without such boundaries. Preserved content loss is used to preserve the fixed region, it is penalized by skin loss and garment loss for regions that are incorrectly segmented. Even if preserved content is incomplete, SIG's extensive training with massive data mitigates the impact.\n- If possible, please refer to Sec. 8.4 (link above) for a more comprehensive explanation.\n---\n- **A3:** Thanks for your constructive comments. During training of SIG, USC-PFN can leverage large amout of unpaired images from Internet. This facilitates the performance of USC-PFN. We augmented the VITON-HD dataset with additional VITON data, see the table below. The experimental results validate your point.\n\n|Methods|SSIM ↑|FID ↓|KID ↓|\n|:--:|:--:|:--:|:--:|\n|Ours (VITON-HD)|0.901|9.08|0.142|\n|Ours (VITON-HD + VITON)|0.906|9.01|0.131|\n---\n### Limitations:\n- **A1:** Thanks for your suggestion. We have added experiments on the high-resolution VTON-HD dataset , and specific qualitative and quantitative results can be found in https://github.com/anony-conf/results-USC-PFN , Sec. 1 to 5.\n---\n> We sincerely thanks for your dedicated efforts. We hope that our responses have provided you with new insights into our work. We look forward to you giving us a chance by increasing your approval of our paper. If you have more questions or encounter broken links, please comment, and we'll assist you quickly."}, {"review_id": "6DCb9PXU64", "reviewer": "Reviewer_99zV", "summary": "This work proposes a virtual try-on method. At inference time, no parser information (e.g., pose, segmentation map) is needed; given a person's image and a garment, the method is able to output a try-on image. To learn this model with a dataset of paired person and garment images, a deformation network is learned to warp the garment to match the person. Meanwhile, a generator is learned to combine the warped new garment and the person to create the try-on image. The generator is trained with a self-cycle-consistency objective, and the intuition is simple: a try-on image should be within the distribution of real images (adversarial loss), and trying on the person's original cloth on an already tried-on image should look like the original image (cycle-consistency loss). The method has multiple other components in it, making it hard to add this to the summary. The proposed method is compared with prior work and shows similar performance with the state of the art.", "questions": "1. The paper mentioned that prior work uses two CNNs to train models cyclically, while the proposed method uses one CNN to train a self-cycle. However, is there a reason why training one model directly with self-cycle consistency will not work? Also, I am guessing that the skin, content, and garment reconstruction loss is one of the important components to make training converge to a reasonable solution. It will be great to confirm this.\n\n2. In my opinion, the draft needs a major update, especially in the method section. If the revision explains the design choices more clearer, I am happy to change my ratings.", "limitations": "1. The paper mentions that it is challenging to achieve good convergence during end-to-end training. Does this mean that a two-stage training (pretraining NGD then training SIG) is required for good results? Or does this mean in general the current method does not attain good convergence?\n\n2. In line 290, what does it mean by closed-form solution here?\n\n-----------------\nRaised score to a borderline accept.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. From the qualitative comparisons, using ground truth deformation field as supervision instead of other alternatives (appearance flow, TPS) seems to help reduce artifacts. Especially in the case where the clothes have stripes.\n\n2. The work proposes a good idea to use an auxiliary deformer, that takes in only pose information (without RGB) of the person as input to predict the garment deformation. It makes sense that it will help generalizing perdiction when the garment is not paired with the person (the training set is all paired up). Since at inference time, only RGB input are given, this auxiliary deformer is used to generate pseudo ground truths for unpaired garment and person images. In short, auxiliary deformer \"teaches\" the final deformer to generalize better, which is a good idea.", "weaknesses": "1. In the related work section, it is mentioned that using parser information can introduce unreliable errors and artifacts. And even with teacher-student training, this issue can still lead to robustness issues. Meanwhile, I do not understand why the proposed method alleviates the issue. The auxiliary deformation network, which is essential to the method (see Fig. 5), is trained with parser information (dense pose). It will be great to clarify this.\n\n2. Writing is hard to understand. It prevents readers from understanding what exactly the algorithm is. See the points below:\n\n3. In line 189, the Lsec loss is refered to a citation without mentioning what exactly it is. Is it precisely the regularization term in Eq. 2?\n\n4. The skin region is mentioned in the method section (s_p, p _s). How do you locate, define, or detect these regions? Same question holds for the content preservation loss: how do you locate the invariant content region (head, trouser, etc.)?\n\n5. Is there any reason that the cycle-consistency loss does not need to be separated into skin, content and garment reconstruction loss? It seems compatible, and I don't understand why the loss function design cannot be applied symmetrically.\n\n6. In figure 5, I do not understand the visualization in the left column of SIG w/o MRF. Having an explanation of the visualization is nice. Also, although I understand other visualziations are representing the deformation field, it is still good to mention it in the caption.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Thank you for your diligent efforts and valuable suggestions.\n---\n### Weaknesses:\n- **W-A1:** Thanks for your comments. Datasets consist of paired garment-person images $(g,p)$, where $p$ wearing $g$. Current methods use the person representation $I_p$ and $g$ to find dense spatial correspondences between $g,p$. However, $I_p$ lacks $p$'s depth and spatial info, leading to rigid deformation and fitting. We attempted to directly train main deformer $G_\\theta$ using $(g,p)$, this is because extracted $p$'s features contain rich spatial and depth info, which flat $I_p$ does not possess. However, $p$ also introduce the same shape, color, and texture correlation features as those of $g$ (strong correlation), which hinders the generalizability of $G_\\theta$. \n- Therefore, we train $G_A$ and $G_\\theta$ using $(g,p)$, allowing the rich spatial and depth knowledge in $G_\\theta$ to guide the convergence of $G_A$. Note that the strong correlation in $G_\\theta$ is not learned by $G_A$, as $G_A$'s input ($I_p$, dense pose) does not contain clothing's shape, color, and texture info. Thereby alleviating the impact of irresponsible prior knowledge on $G_A$ stemming from erroneous $I_p$. Then, we retrain $G_\\theta$ using $G_A$ to eliminate ineffective strong correlations in the latent feature space of $G_\\theta$. In this way, we achieve an efficient $G_\\theta$.\n---\n- **W-A2:** We apologize for our writing issues. We previously lacked detailed explanations. Now, we have conducted extensive experiments and provided additional explanations to improve its readability.\n---\n- **W-A3:** We apologize for any confusion caused by us. Yes, on lines 188 and 189, we clarify that $R_{sm}$ is implemented by $L_{sec}$, which indeed corresponds to the regularization term in Eq. (2).\n---\n- **W-A4:** Thanks for your comments. Datasets contain garment $g$ and its mask $p_M$, person $p$ and its paring map $p_p$. $p_p$ includes several layers, the neck and arms layers are merged to form the skin layer, denoted as $p^n_p+p^a_p=p^s_p$. And the skin and garment layers are merged to form the agnostic layer, denoted as $p^s_p+p^c_p=p^u_p$. Then, $p_M$ that deforms along with $g$ is used as the new target clothing layer $\\hat p_M$, and subtracting $\\hat p_M$ from $p_p^u$ yields the new target skin layer $\\hat p_p^s$ corresponding to $g$, i.e., $\\hat p_p^s=p_p^u-\\hat p_M$. $s_p$ is fixed skin region before and after trying on, which can be obtained by intersecting $\\hat p_p^s$ and $p_p^s$, denoted as $s_p=(p_p^s \\cap \\hat p_p^s)\\times p$. $\\hat p'_s=\\hat p_p^s \\times \\hat p$, where $\\hat p$ is results generated in first stage. In this manner, the invariant content region (head, trouser, etc.) are also located by $p^h_p$, $p^t_p$ in $p_p$.\n- Note that during training and inference stages, we do not require the parser to estimate new human parsing as input. This perfectly aligns with the requirements of parser-free virtual try-on.\n---\n- **W-A5:** Thanks for your comments. As shown in Fig. 1 (https://github.com/anony-conf/results-USC-PFN), we initially define a cycle-consistency loss $L_{scyc}$, which is employed to supervise the reconstruction of GT $p$. This global loss is used to preserve the structural and distributional consistency between the reconstructed $p'$ and $p$, ensuring a harmonious coordination of generated skin, garment, and the overall appearance. However, it cannot specially focus on the generation of localized body regions, particularly in first stage. Since there is no GT available to calculate $L_{scyc}$, we introduce only the skin, content, and garment reconstruction losses in first stage to optimize these regions that can be supervised. Furthermore, due to the weight sharing between the two stages, designing symmetrical loss functions becomes unnecessary.\n---\n- **W-A6:** Yes, the left column of SIG w/o MRF is the deformation field $f$, indicating the effects after removing $f$ generated by NGD during the try-on synthesis stage of SIG.\n---\n### Questions:\n- **Q-A1:** Thanks for your comments. As shown in Fig. 1 (https://github.com/anony-conf/results-USC-PFN), in first stage, the try-on result $p'$ do not have corresponding GT in dataset. If we do not supervise this stage and directly use $p'$ as input for the second stage, it can be considered as a direct mapping from input $p$ to output $p$. However, our task requires achieving the cyclic mapping $p$→$p'$ and $p'$→$p$. Therefore, directly applying self-cyclic consistency training in the second stage will not work.\n- These reconstruction losses are crucial components, 1) there is a need to enforce a mapping of the outputs from the first stage into the domain of $p'$, requiring the formulation of local losses for the first stage; 2) the second reason is the same as **W-A5**.\n---\n- **Q-A2:** We sincerely appreciate your professional and responsible comments and suggestions. Based on the feedback from all the reviewers, we have made major revisions to our manuscript, including the addition of the required experiments. Therefore, we earnestly implore that you reconsider the ratings of our paper.\n---\n## Limitations:\n- **L-A1:** Thanks for your comments. To demonstrate the necessity of two-stage training, we conducted end-to-end training experiments to demonstrate that this manner cannot achieve the same favorable convergence as the two-stage approach, both NGD and SIG did not converge, resulting in an extremely low SSIM score of 0.37 (see Fig. 8.5 (a)). We also computed the FID and KID values on the test set during training to reflect the convergence of our manner (see Fig. 8.5 (b)). This successfully validates all the points in **Q-A1**.\n---\n- **L-A2:** The term \"closed-form solution\" in line 290 refers to our proposed self-cycle consistency pipeline.\n---\n> We sincerely thanks for your dedicated efforts. We hope that our responses have provided you with new insights into our work. We look forward to you giving us a chance by increasing your approval of our paper."}]

Author rebuttal:
> We sincerely appreciate the diligent efforts of the reviewers. We propose a novel parser-free self-cycle consistency framework, USC-PFN. To validate the effectiveness, robustness, and generalization of this architecture, we have added the following supplementary experiments:

1) Qualitative experiments of the clothing deformer NGD based on the VITON dataset.
2) Quantitative experiments of the clothing deformer NGD based on the VITON dataset.
3) Qualitative experiments of the clothing deformer NGD based on the high-definition **VITON-HD** dataset.
4) Quantitative experiments of the clothing deformer NGD based on the high-definition **VITON-HD** dataset.
5) Quantitative experiments of full network based on the augmented VITON dataset.
6) Qualitative experiments of full network on the high-definition **VITON-HD** dataset.
7) Quantitative experiments of full network on the high-definition **VITON-HD** dataset.
8) Computational complexity analysis of full network on the high-definition **VITON-HD** dataset.
9) Several other relevant ablation experiments.

- The tables of all the added quantitative experiments are presented below:
---
### Quantitative experiments of the clothing deformer NGD based on the VITON dataset.

|Methods|Publication|Warping|FID-P $\downarrow$|KID-P $\downarrow$|FID-UP $\downarrow$|KID-UP $\downarrow$|
|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
CP-VTON [3]|ECCV 2018|TPS|43.95|2.233|42.13|2.112|
ACGPN [4]|CVPR 2020|TPS|42.10|2.009|41.48|2.048|
DCTON [2]|CVPR 2021|TPS|42.80|2.170|42.19|2.126|
PFAFN [6]|CVPR 2021|AF|22.81|0.785|23.90|0.860|
SGAFN [7]|CVPR 2022|AF|20.07|0.552|20.38|0.481|
**Ours** (MRF)|**This Work**|**MRF**|**18.70**|**0.390**|**19.50**|**0.355**|
---
### Quantitative experiments of the clothing deformer NGD based on the high-definition VITON-HD dataset.

|Methods|Publication|Warping|FID-P $\downarrow$|KID-P $\downarrow$|FID-UP $\downarrow$|KID-UP $\downarrow$|
|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
|VITON-HD|CVPR 2021| TPS|32.968|1.407|32.93|1.353|
|HR-VITON|ECCV 2022|AF|25.499|0.926|24.826|0.759|
|**Ours** (MRF)|**This Work**|**MRF**|**19.060**|**0.418**|**22.861**|**0.504**|
---
### Quantitative experiments of full network based on the augmented VITON dataset.

|Methods|SSIM $\uparrow$|FID $\downarrow$|
|:--:|:--:|:--:|
|ACGPN|0.81|20.75|
|Cloth-flow|0.86|13.05|
|PF-AFN|0.87|12.19|
|**Ours**|**0.90**|**10.46**|
---
### Quantitative experiments of full network on the high-definition VITON-HD dataset.

|Methods|Publication|Parsing as input|SSIM $\uparrow$|FID $\downarrow$|KID $\downarrow$|
|:--:|:--:|:--:|:--:|:--:|:--:|
|CP-VTON|ECCV 2018|Y|0.791|30.25|4.012|
|ACGPN|CVPR2020|Y|0.858|14.43|0.587|
|DCTON|CVPR2021|Y|0.810|15.55|/|
|VITON-HD|CVPR2021|Y|0.843|11.64|0.300|
|HR-VITON|ECCV2022|Y|0.878|9.90|0.188|
|**Ours**|**This Work**|**N**|**0.899**|**9.10**|**0.159**|
---
### Computational complexity analysis of full network  on the high-definition VITON-HD dataset.

|Methods|Weight Size|#Params|FLOPs|FPS|
|:--:|:--:|:--:|:--:|:--:|
|VITON-HD|588M|154M|1689.7G|3.76|
|HR-VITON|586M|148M|1555.4G|4.09|
|**Ours**|**334M**|**87M**|**467.9G**|**22.19**|
---
### Computational complexity analysis of full network on the VITON dataset.

|Methods|#Params|FLOPs|FPS|FID-Clothing $\downarrow$|SSIM-TryOn $\uparrow$|
|:--:|:--:|:--:|:--:|:--:|:--:|
|ACGPN [4]|139M|206G|10|42.10|0.84|
|DCTON [2]|153M|194G|19|42.80|0.83|
|PF-AFN [6]|99M|69G|34|22.81|0.89|
|**Ours** (All ngf=64)|140M|46G|39|18.70|0.91|
|**Ours** (NGD ngf=32)|**87M**|**29G**|**40**|**18.85**|**0.91**|
---

> The remaining figures and tables for qualitative results, ablation experiments, and rebuttals related to the experiments are all included in the following link and attachment.
---
- https://github.com/anony-conf/results-USC-PFN

---

- https://github.com/anony-conf/USC-PFN/ 
- (The checkpoints will be made publicly available immediately.)

---
---
> We sincerely thank you for your dedicated efforts. We hope that our responses have provided you with new insights into our work. We look forward to you giving us a chance by increasing your approval of our paper. If you have more questions or encounter broken links, please comment, and we'll assist you quickly.

Meta-review: Four reviewers favor acceptance, while one is borderline. The reviewer recommending rejection (Reviewer RUsU) was initially concerned about the presentation of the MRF and its novelty over previous methods based on appearance flow. After the discussion, they suggested that these issues had mostly been addressed, but felt that the discussion of the MRF was not sufficiently detailed in the current version. The AC agrees that the paper has clarity issues (as well as a number of minor formatting issues, such as inconsistently capitalized section titles and writing issues. On balance, the AC recommends acceptance, but urges the authors to improve the clarity of the paper in a revision.
Decision: Accept (poster)