SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Self-cyclic Image Generator (SIG)  $\mathcal{G}_{\varphi}$ . The generator  $\mathcal{G}_{\varphi}$  can adopt arbitrary encoder-decoder networks. Our goal is to optimize  $\mathcal{G}_{\varphi}$  by taking unpaired images  $(\mathbf{p},\hat{\mathbf{g}}^{\prime})$ . The training process is divided into two steps: 1) generating the try-on result  $\hat{\mathbf{p}}^{\prime}$  by taking unpaired images  $(\mathbf{p},\hat{\mathbf{g}}^{\prime})$  as input, i.e.  $\hat{\mathbf{p}}^{\prime} = \mathcal{G}_{\varphi}(\mathbf{p},\hat{\mathbf{g}}^{\prime})$ . Immediately after, 2) taking  $\hat{\mathbf{p}}^{\prime}$  and warped garment  $\hat{\mathbf{g}}$  as input of  $\mathcal{G}_{\varphi}$  to generate try-on result  $\hat{\mathbf{p}}$ , i.e.  $\hat{\mathbf{p}} = \mathcal{G}_{\varphi}(\hat{\mathbf{p}}^{\prime},\hat{\mathbf{g}})$ . Different from the current methods that rely on an additional network to provide prior pseudo-label  $\hat{\mathbf{p}}^{\prime}$ , training  $\mathcal{G}_{\varphi}$  directly with self-cycle consistency is not work due to the absence of ground truth of  $\hat{\mathbf{p}}^{\prime}$ ,  $\mathcal{G}_{\varphi}$  is more inclined to learn the mapping:  $\mathbf{p} \mapsto \hat{\mathbf{p}}$  when we jointly perform the two steps. Therefore, we introduce additional supervision on the upper body of the generated  $\hat{\mathbf{p}}^{\prime}$  to take over the role of the label in [22]. In simple terms, the functionality of the person representation  $\mathcal{R}$  has shifted from the input side to the supervision side. In addition, the self-cycle consistency can be ensured by enforcing  $\hat{\mathbf{p}} \approx \mathbf{p}$  to establish self-supervision in step 2.
3.3 Self-Cycle Consistency Training Strategy
Training of NGD. As shown in Eq.(1), Eq.(2), and Eq.(8), to achieve the MRF energy minimization, we design multiple loss functions.  $E_{\text{dat}}$  is implemented with the pixel-level  $\mathcal{L}_1$  loss to encourage the similarity between  $\hat{\mathbf{f}}$  and  $\tilde{\mathbf{f}}_{\mathcal{A}}$ .  $E_{reg}$  is composed of the weights  $\mathbf{w}_{\phi}$  and the smoothing term  $R_{sm}$  (implemented as second-order smooth constraint  $\mathcal{L}_{\text{sec}}$  [6]).  $E_{pix}$  is implemented with  $\mathcal{L}_1$  loss and VGG perceptual loss  $\mathcal{L}_{per}$  [26] as the evaluation of structural similarity. Finally, we define the overall loss function of NGD as:
$$
\mathcal {L} _ {n g d} = \mathcal {L} _ {1} ^ {d a t} + \lambda_ {r} \mathcal {L} _ {s e c} + \lambda_ {p} \left(\mathcal {L} _ {1} + \mathcal {L} _ {p e r}\right), \tag {9}
$$
where  $\lambda_r$  and  $\lambda_d$  are hyperparameters that balances the relative contributions of sub-loss terms.
Training of SIG. To self-supervised cyclic optimization of SIG, we propose a self-cycle consistency loss consisting of  $\mathcal{L}_1$  loss and perceptual loss  $\mathcal{L}_{per}$ , respectively, to maintain consistency between the try-on result  $\hat{\mathbf{p}}$  and the real person  $\mathbf{p}$ :
$$
\mathcal {L} _ {s c y c} = \left\| \mathbf {p} - \hat {\mathbf {p}} \right\| _ {1} + \left\| \mathbf {p} - \hat {\mathbf {p}} \right\| _ {p e r}. \tag {10}
$$
To make  $\hat{\mathbf{p}}'$  as close as possible to the real distribution of person, we employ a discriminator  $\mathcal{D}_s$  to introduce adversarial loss:
$$
\mathcal {L} _ {a d v} ^ {D} = \mathbb {E} _ {\mathbf {p}} \left[ \log \left(\mathcal {D} _ {s} (\mathbf {p})\right) \right] + \mathbb {E} _ {\hat {\mathbf {g}} ^ {\prime}, \mathbf {p}} \left[ \log \left(1 - \mathcal {D} _ {s} \left(\mathcal {G} _ {\varphi} \left(\hat {\mathbf {g}} ^ {\prime}, \mathbf {p}\right)\right)\right) \right]. \tag {11}
$$
Additionally, the adversarial loss  $\mathcal{L}_{adv}^{G}$  of generator  $\mathcal{G}_{\varphi}$  is introduced simultaneously. In addition to the aforementioned losses, we propose a skin reconstruction loss  $\mathcal{L}_{sr}$  to forcefully supervise the skin reconstruction of  $\hat{\mathbf{p}}'$  in the arm and neck regions. To do this, we pre-trained a model SR, which can be fed into deformed garment  $\hat{\mathbf{g}}'$  and parts of the skin  $\mathbf{s_p}$  to reconstruct the envisaged arm and neck regions of  $\hat{\mathbf{p}}'$ :
$$
\mathcal {L} _ {s r} = \left\| \operatorname {S R} \left(\hat {\mathbf {g}} ^ {\prime}, \mathbf {s} _ {\mathbf {p}}\right) - \hat {\mathbf {p}} _ {\mathbf {s}} ^ {\prime} \right\| _ {1}, \tag {12}
$$
where  $\hat{\mathbf{p}}_{\mathrm{s}}^{\prime}$  denotes skin regions of  $\hat{\mathbf{p}}^{\prime}$ . Furthermore, to generate the garment part of  $\hat{\mathbf{p}}^{\prime}$  and preserve the invariant body content (e.g. head, trouser), we introduce the garment reconstruction loss  $\mathcal{L}_{gr}$  and content preservation loss  $\mathcal{L}_{cp}$ :
$$
\mathcal {L} _ {g r} = \left\| \hat {\mathbf {p}} _ {\mathbf {g}} ^ {\prime} - \hat {\mathbf {g}} ^ {\prime} \right\| _ {1}, \quad \mathcal {L} _ {c p} = \left\| \hat {\mathbf {p}} _ {\mathbf {c}} ^ {\prime} - \mathbf {p} _ {\mathbf {c}} \right\| _ {1}, \tag {13}
$$
where  $\hat{\mathbf{p}}_{\mathbf{g}}^{\prime}$  denotes garment part of  $\hat{\mathbf{p}}^{\prime}$ ,  $\hat{\mathbf{p}}_{\mathbf{c}}^{\prime}$  and  $\mathbf{p}_{\mathbf{c}}$  denote the same content of  $\hat{\mathbf{p}}^{\prime}$  and  $\mathbf{p}$ , respectively. Finally, we define the overall loss function of SIG as:
$$
\mathcal {L} _ {s i g} ^ {t} = \lambda_ {s c y c} \mathcal {L} _ {s c y c} + \lambda_ {a d v} ^ {G} \mathcal {L} _ {a d v} ^ {G} + \lambda_ {s r} \mathcal {L} _ {s r} + \lambda_ {g r} \mathcal {L} _ {g r} + \lambda_ {c p} \mathcal {L} _ {c p}. \tag {14}
$$
To prevent the unreliable result  $\hat{\mathbf{p}}^{\prime}$  generated in step 1 to wrongly guide step 2, we introduce an adjustable gated mechanism late in the training to prevent irresponsible backpropagation:
$$
\mathcal {L} _ {s i g} = \boldsymbol {\alpha} \mathcal {L} _ {s i g} ^ {t}, \text {w i t h} \boldsymbol {\alpha} = \left\{ \begin{array}{l l} 1, & \| \mathbf {p} - \hat {\mathbf {p}} ^ {\prime} \| _ {1} > \| \mathbf {p} - \hat {\mathbf {p}} \| _ {1}, \\ 0, & \text {o t h e r w i s e ,} \end{array} \right. \tag {15}
$$
where  $\alpha$  is the adjustable gated factor that determines whether backpropagation is enabled.
2We provide the means to obtain  $\mathbf{s_p}$ $\hat{\mathbf{p}}_{\mathrm{s}}^{\prime}$ $\hat{\mathbf{p}}_{\mathrm{g}}^{\prime}$  , and  $\hat{\mathbf{p}}_{\mathrm{c}}^{\prime}$  in the supplementary material.
Figure 4: Qualitative results of different methods (CP-VTON+[27], ACGPN[4], DCTON[2], PF-AFN[6], SDAFN[28], RT-VTON[13], FS-VTON[7], and ours) in the unpaired setting.
Dataset. We conduct experiments using the VITON dataset [8], which consists of 16,253 image groups. Each group includes a front-view female image  $\mathbf{p}$ , an in-shop garment image  $\mathbf{g}$  with its mask  $\mathbf{p}_{\mathbf{M}}$ , a reference semantic map  $\mathbf{p}_{\mathbf{p}}$ , and a pose heatmap  $\mathbf{p}_{\mathbf{h}}$ . The image size for each is 256 pixels  $\times$  192 pixels. The training set contains 14,221 groups, and the test set contains 2,032 groups. All evaluations and visualizations are performed using the test set. Note that the in-shop garment and the garment worn on the person are identical in the training set, while they differ in the test set.
Implementation Details. The USC-PFN is implemented in PyTorch and trained on a single Nvidia Tesla V100 GPU running Ubuntu 16.04. During training, a batch size of 16 is used for 100 epochs, and the Adam optimizer [29] is employed with parameters  $\beta_{1} = 0.5$  and  $\beta_{2} = 0.999$ , and the initial learning rate is set to  $1e^{-4}$  with linear decay after 50 epochs. The USC-PFN consists of NGD and SIG, both have the same structure as Res-UNet [30], and the discriminator is from pix2pixHD [16]. In the loss functions, the  $\lambda_r = 20$  and  $\lambda_p = 0.25$  in the  $\mathcal{L}_{ngd}$ . The  $\lambda_{scyc} = 1$ ,  $\lambda_{adv}^{G} = 0.1$ ,  $\lambda_{sr} = 50$ ,  $\lambda_{gr} = 1$ , and  $\lambda_{cp} = 10$  in the  $\mathcal{L}_{sig}^t$ .
Baselines. We perform comparative evaluations of our model's performance, which includes the deformer and generator, against publicly available state-of-the-art methods. To this end, we leverage ten popular methods, including VITON[8], CP-VTON[3], Cloth-flow[9], CP-VTON+[27], ACGPN[4], DCTON[2], PF-AFN[6], ZFlow[31], SDAFN[28], and RT-VTON[13], as baseline methods for quantitative evaluation. In particular, we select seven most cutting-edge methods [27, 4, 2, 6, 28, 13, 7] for qualitative evaluation.
Evaluation Metrics. Quantitative evaluations are carried out in both paired and unpaired settings to compare our approach with baseline methods. For the paired setting, we use Structure Similarity (SSIM) [32] to evaluate the visual quality and diversity of the generated images. In the unpaired setting, as there is no ground truth available, FID is directly employed to evaluate the distributional similarity between the generated and real images. Note that we do not use Inception Score (IS) [33]
Table 1: Quantitative results of different methods on VITON. 'Def.' represents different warping methods. 'Syn.' represents different synthesizing paradigms. 'Parser' indicates whether the parser is used in the model during inference. The best result are in bold and the second best result are in blue.
Methods	Publication	Def.	Syn.	Parser	SSIM ↑	FID ↓
VITON [8]	CVPR 2018	TPS	IP	Y	0.74	55.71
CP-VTON [3]	ECCV 2018	TPS	IP	Y	0.72	24.45
Cloth-flow [9]	ICCV 2019	AF	IP	Y	0.84	14.43
CP-VTON+ [27]	CVPRW 2020	TPS	IP	Y	0.75	21.04
ACGPN [4]	CVPR 2020	TPS	IP	Y	0.84	16.64
DCTON [2]	CVPR 2021	TPS	CC	Y	0.83	14.82
PF-AFN [6]	CVPR 2021	AF	KD	N	0.89	10.09
ZFlow [31]	ICCV 2021	AF	IP	Y	0.88	15.17
RT-VTON [13]	CVPR 2022	MLS	IP	Y	-	11.66
SDAFN [28]	ECCV 2022	AF	IP	N	0.88	12.05
USC-PFN (Ours)	This Work	MRF	SC	N	0.91	10.47
- : official code or data are not provided. IP: in-painting; CC: cycle consistency; KD: knowledge distillation; SC: self-cycle consistency.
in the evaluation [34, 35], as it is only effective in datasets similar to ImageNet [36]. Specifically, a lower score of FID indicates a higher quality of the result.
Qualitative Results. In Figure 4, we perform visual comparison with seven methods, including SOTA in-painting methods [27, 4, 13, 28], the SOTA cycle consistency method [2], and SOTA knowledge distillation methods [6, 7]. In particular, we specifically select four striped garments to visually showcase the deformation effects of each method. It indicates that all methods can achieve approximate garment alignment and try-on synthesis. However, in terms of garment alignment, the TPS-based methods
Table 2: Analysis of time cost and computational complexity between ACGPN[4], DCTON[2], PF-AFN[6], and ours.
Methods	Time	#Params	FLOPs	FPS
ACGPN [4]	~40h	139M	206G	10
DCTON [2]	~44h	153M	194G	19
PF-AFN [6]	-	99M	69G	34
Ours	~32h	140M	46G	39
(columns 3, 4, and 5) exhibit excessive local fabric deformation and misaligned spatial correspondence. The AF-based methods (columns 6, 7, and 8) demonstrate overly flexible and unconstrained deformation effects, which are particularly noticeable in the first row. The MRF-based method (column 9) shows semi-rigid deformation effects in highly non-rigid body poses, such as misalignment in the arm region. In terms of try-on synthesis, the in-painting methods (columns 3, 4, 7, and 9) exhibit mismatches between the given garments and the synthesized try-on results due to parsing errors, particularly noticeable in the arms region. The cycle consistency method (column 5), being modular in nature, shows artifacts along the module boundaries. The knowledge distillation methods (columns 6 and 8) occasionally suffer from similar issues as the in-painting methods, both in terms of clothing deformation and try-on synthesis, due to irresponsible teacher knowledge. In contrast, our proposed method (last column) can achieve the most realistic results benefits from two key factors. Firstly, it leverages MRF's perception of human spatial structure to faithfully simulate the process of cloth deformation after try-on, capturing realistic fabric draping effects. Secondly, it utilizes the self-cycle consistency paradigm to accurately reconstruct the appearance of the garment worn on the body, all without the need for additional parsers or complex designs.
Quantitative Results. We evaluate the quantitative results of different methods under the same configuration for a fair comparison. We use SSIM for measuring structural similarity in the paired setting and FID for measuring distributional similarity in the unpaired setting. Table 1 presents our quantitative results, including the SSIM and FID scores of baseline methods [8, 3, 9, 27, 4, 2, 6, 31, 28, 13] and the proposed USC-PFN, on the VITON dataset. In terms of SSIM, USC-PFN
Figure 5: Ablation studies on the effect of NGD and SIG in the unpaired setting.
surpasses the SOTA in-painting method SDAFN [28] by 0.03, the SOTA knowledge distillation method PF-AFN[6] by 0.02, and the SOTA cycle consistency method DCTON [2] by 0.08. In terms of FID, it surpasses the SOTA in-painting method RT-VTON [13] by 1.19 and the SOTA cycle consistency method DCTON by 4.35. Although PF-AFN obtains a lower FID score, some of their results appear to be worse than ours<sup>3</sup>. Additionally, PF-AFN requires complex training with the assistance of the teacher model. The result demonstrates the superiority of our self-cycle consistency approach while showcasing the significant advantage of generating high-quality try-on images.
Computational Complexity. In addition, to demonstrate the superior performance of our network, we calculate the computational costs of ACGPN[4], DCTON[2], PF-AFN[6], and our method as shown in Table 2. On the same Tesla V100 GPU, we have an appropriate number of parameters and the fewest FLOPs (floating point operations), yet our inference speed is nearly four times faster than ACPGN and about twice as fast as DCTON. This demonstrates the real-time capability of our method and its ability to achieve high-quality visual performance with fewer computational resources. Note that our architecture can employ any encoder-decoder network, allowing for flexible selection of the network based on computational resources.
Ablation Study. In the ablation study, we evaluate the effectiveness of NGD and SIG in USC-PFN. Table 3 and Figure 5 provide both quantitative and qualitative results. Firstly, we demonstrate that the auxiliary deformer  $\mathcal{G}_{\mathcal{A}}$  successfully disentangles the color, texture, and shape correlations in the pre-trained  $\mathcal{G}_{\theta}$ . The experiments further validate the necessity of the data term  $E_{dat}$  in MRF, as it plays a crucial role in ensuring the accurate deformation of the garment. Moreover, we conduct an experiment by removing the entire MRF module to emphasize its essential in USC-PFN. Finally, the efficacy of  $\mathcal{L}_{sr}$  in SIG is validated, emphasizing its critical role in supervising skin generation.
Limitations and Discussion. There are still limitations to USC-PFN that need to be addressed in future work. Firstly, due to the complex non-rigid deformation of garment images, it is challenging to achieve good convergence during end-to-end training; that is, integrating the garment deformer  $\mathcal{G}_{\theta}$  into the image generator  $\mathcal{G}_{\varphi}$  remains a difficult task. Secondly, accurately aligning garments with human poses is challenging due to the complexity and variability of human
poses. Thus, finding a more optimal closed-form solution is necessary to solve the garment deformation task. Finally, we still require extensive supervision during the training, and it is still an urgent issue to find a solution to train a high-quality network with less supervision.
Table 3: Ablation studies of NGD and SIG in the unpaired setting. Lower score of FID indicates higher quality of results.
NGD	FID ↓	SIG	FID ↓
w/o G_A	42.23	w/o MRF	14.28
w/o E_dat	30.53	w/o Lsr	15.44
Full NGD	25.44	Full SIG	10.47
In this paper, we present USC-PFN, a parser-free virtual try-on network that utilizes a self-cycle consistency pipeline, which only uses one model to cyclically learn trying on different styles of
See more visual comparisons in the supplementary material.
garments. To get a more natural and realistic garment alignment, USC-PFN first incorporates the Markov Random Field as a non-rigid deformation method, by enhancing the deformer's perception of human spatial structure, thereby mimicking the natural interaction between the garment and the person. In addition, unlike existing paradigms, our method does not require input information from parsers, teacher knowledge, or complex person representations for the generator. Instead, it solely relies on the garment and human images as input to train and infer try-on results. The result indicates that USC-PFN outperforms the state-of-the-art methods significantly in terms of subjective and objective evaluations, demonstrating its superior performance. In the future, we plan to extend this architecture to different tasks of image-to-image translation.
This work was in part supported by the National Key Research and Development Program of China (Grant No. 2022ZD0160604) and NSFC (Grant No. 62176194), and the Key Research and Development Program of Hubei Province (Grant No. 2023BAB083), the Project of Sanya Yazhou Bay Science and Technology City (Grant No. SCKJ-JYRC-2022-76, SKJC-2022-PTDX-031), and the Project of Sanya Science and Education Innovation Park of Wuhan University of Technology (Grant No. 2021KF0031).
[1] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2223-2232, 2017.
[2] Chongjian Ge, Yibing Song, Yuying Ge, Han Yang, Wei Liu, and Ping Luo. Disentangled cycle consistency for highly-realistic virtual try-on. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16928-16937, 2021.
[3] Bochao Wang, Huabin Zheng, Xiaodan Liang, Yimin Chen, Liang Lin, and Meng Yang. Toward characteristic-preserving image-based virtual try-on network. In Proceedings of the European conference on computer vision (ECCV), pages 589-604, 2018.
[4] Han Yang, Ruimao Zhang, Xiaobao Guo, Wei Liu, Wangmeng Zuo, and Ping Luo. Towards photorealistic virtual try-on by adaptively generating-preserving image content. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7850-7859, 2020.
[5] Ning Wang, Jing Zhang, Lefei Zhang, and Dacheng Tao. Gt-must: Gated try-on by learning the mannequin-specific transformation. In Proceedings of the 30th ACM International Conference on Multimedia, pages 2182-2190, 2022.
[6] Yuying Ge, Yibing Song, Ruimao Zhang, Chongjian Ge, Wei Liu, and Ping Luo. Parser-free virtual try-on via distilling appearance flows. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8485-8493, 2021.
[7] Sen He, Yi-Zhe Song, and Tao Xiang. Style-based global appearance flow for virtual try-on. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3470-3479, 2022.
[8] Xintong Han, Zuxuan Wu, Zhe Wu, Ruichi Yu, and Larry S Davis. Viton: An image-based virtual try-on network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7543-7552, 2018.
[9] Xintong Han, Xiaojun Hu, Weilin Huang, and Matthew R Scott. Clothflow: A flow-based model for clothed person generation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10471-10480, 2019.
[10] Ruiyun Yu, Xiaoqi Wang, and Xiaohui Xie. Vtnfp: An image-based virtual try-on network with body and clothing feature preservation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10511-10520, 2019.
[11] Thibaut Issenhuth, Jérémie Mary, and Clément Calauzenes. Do not mask what you do not need to mask: a parser-free virtual try-on. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XX 16, pages 619-635. Springer, 2020.
[31] Ayush Chopra, Rishabh Jain, Mayur Hemani, and Balaji Krishnamurthy. Zflow: Gated appearance flow-based virtual try-on with 3d priors. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5433-5442, 2021.
[32] Kalpana Seshadrinathan and Alan C Bovik. Unifying analysis of full reference image quality assessment. In 2008 15th IEEE International Conference on Image Processing, pages 1200-1203. IEEE, 2008.
[33] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
[34] Shane Barratt and Rishi Sharma. A note on the inception score. arXiv preprint arXiv:1801.01973, 2018.
[35] Mihaela Rosca, Balaji Lakshminarayanan, David Warde-Farley, and Shakir Mohamed. Variational approaches for auto-encoding generative adversarial networks. arXiv preprint arXiv:1706.04987, 2017.
[36] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.