SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chenghu Du<sup>1</sup> Junyin Wang<sup>1</sup> Shuqin Liu<sup>4</sup> Shengwu Xiong<sup>1,2,3,5,*</sup>
<sup>1</sup>Wuhan University of Technology, <sup>2</sup>Shanghai AI Laboratory
<sup>3</sup>Sanya Science and Education Innovation Park, Wuhan University of Technology
<sup>4</sup>Wuhan Textile University, <sup>5</sup>Qiongtai Normal University
{duch, wjy199708, xiongsw}@whut.edu.cn, liushuqingwtu@foxmail.com
https://du-chenghu.github.io/USC-PFN/
Image-based virtual try-on tasks remain challenging, primarily due to inherent complexities associated with non-rigid garment deformation modeling and strong feature entanglement of clothing within human body. Recent groundbreaking formulations, such as in-painting, cycle consistency, and knowledge distillation, have facilitated self-supervised generation of try-on images. However, these paradigms necessitate the disentanglement of garment features within human body features through auxiliary tasks, such as leveraging 'teacher knowledge' and dual generators. The potential presence of irresponsible prior knowledge in the auxiliary task can serve as a significant bottleneck for the main generator (e.g., 'student model') in the downstream task. Moreover, existing garment deformation methods lack the ability to perceive the correlation between the garment and the human body in the real world, leading to unrealistic alignment effects. To tackle these limitations, we present a new parser-free virtual try-on network based on unified self-cycle consistency (USC-PFN), which enables robust translation between different garments using just a single generator, faithfully replicating non-rigid geometric deformation of garments in real-life scenarios. Specifically, we first propose a self-cycle consistency architecture with a circular mode. It utilizes real unpaired garment-person images exclusively as input for training, effectively eliminating the impact of irresponsible prior knowledge at the model input end. Additionally, we formulate a Markov Random Field to simulate a more natural and realistic garment deformation. Furthermore, USC-PFN can leverage a general generator for self-supervised cycle training. Experiments demonstrate that our method achieves state-of-the-art performance on a popular virtual try-on benchmark.
To provide consumers with an online try-on experience similar to physical stores, researchers have focused on image-based virtual try-on (VTON), allowing individuals to virtually try on garments available in online shops. The mainstream image-based approach of VTON involves the replacement of a person's clothing area with an in-shop garment image. However, all current datasets are paired, meaning one person corresponds to only one piece of clothing they are wearing. To capture underlying correlations between unpaired images from random combinations in the dataset, three architectures have been used: in-painting methods [8, 3, 9, 10, 4, 11-14], knowledge distillation methods [11, 6, 7], and cycle consistency methods[15, 2] (see Figure 1).
*Shengwu Xiong is corresponding author.
Figure 1: Comparison of virtual try-on pipelines.  $"+"\mathcal{R}$  denotes that additional person representation (e.g. semantic map) may be required. (a) cycle consistency approaches [1, 2] require two generators to implement bidirectional translation between various garment styles. (b) in-painting approaches [3-5] are implemented by reproducing the masked person's garment. (c) knowledge distillation approaches [6, 7] use pre-trained "teacher knowledge" to optimize student model. (d) USC-PFN utilizes self-cycle consistency without the need for additional person representation, enabling a single generator to perform bidirectional translation between various garment styles.
The in-painting-based method, inspired by [16], addresses variations in garment styles by masking the upper body of the person image and training the model to learn the reproduction of masked regions. This paradigm uses a person representation (e.g., semantic map) as input to provide masked body information; however, slight errors in the person representation can result in occluded and unrealistic outcomes. Moreover, this paradigm faces challenges when dealing with cases where the same person tries on different garments due to their reliance on one-to-one paired garment-person images during training (see Figure 2). Therefore, the knowledge distillation-based structure [11] is proposed, which pretransmits an in-painting-based teacher model to provide prior try-on result images for training the student model. This paradigm introduced a new pipeline that enables self-supervised training using only unpaired images as input. However, the potential presence of irresponsible teacher knowledge can still reproduce some issues of in-painting methods. Additionally, a method is proposed in [1, 2] that employs two CNNs to achieve the cycle training. In this approach, the output of  $\mathrm{CNN}_1$  as input is fed to  $\mathrm{CNN}_2$  and then minimizes the difference between the output of  $\mathrm{CNN}_2$  and the input of  $\mathrm{CNN}_1$  to achieve cycle consistency. However, the complex dual structure poses challenges in network convergence, and reliance on person representation throughout the entire network hinders the generation of high-quality virtual try-on results.
In this task, Thin Plate Splines (TPS) [17, 18, 8, 3, 10, 4, 11, 2, 12] is commonly used to deform garments. However, TPS focuses only on pixel transformations around the control point, often causing excessive distortion. Another non-rigid deformation technique, called Appearance Flow (AF) [19, 9, 6, 7, 14], predicts a pixel-level displacement field of garments for pixel-by-pixel transformations. However, it lacks spatial prior knowledge, resulting in an inaccurate perception of the spatial relationship between the garment and the human body. Recently, the Moving Least Squares (MLS) [20, 13] has shown promise in handling deformation for simple poses. However, there are difficulties in handling complex poses. These issues can lead to misalignment, hindering accurate simulation of the natural non-rigid deformation that occurs during the interaction between the garment and the body (see Figure 2).
Therefore, we propose a virtual parser-free try-on network based on self-cycle consistency to address the aforementioned limitations. First, we explore the relationship between garments and the human body. We use an auxiliary network to disentangle irrelevant features such as texture, color, and shape in the latent feature space of the human body. This approach preserves valuable features like depth and lighting while removing entanglements with unrelated features. Consequently, the network can authentically simulate the interaction between different garments and the human body in real-life scenarios, without being affected by the garment it wears. Furthermore, it has been confirmed that the person representation appearing at the input end of the network has potential negative impacts on performance [11, 6, 7]. Moreover, we argue that there is no inherent translation between multiple
domains in virtual try-on tasks. Therefore, we introduce a shared-weight circular architecture where the parser is shifted from the input end to the output end, specifically for loss computation. This approach allows us to bypass the parser during the inference stage. The main contributions of our paper can be summarized as follows:
- We propose USC-PFN, a novel parser-free virtual try-on network that generates highly realistic try-on results using only unpaired data as input and a single CNN as the generator, offering a new perspective for the virtual try-on task.
- We first leverage the Markov Random Field to estimate a deformation field to establish precise dense pixel-level correspondences between the in-shop garment and the person. This is a new approach to address the misalignment caused by a lack of spatial perception, and simulate the non-rigid deformation and interaction of the garment in real-life scenarios.
- We propose a concise and effective self-cycle consistency paradigm for the parser-free virtual try-on task, and extensive experiments on the popular benchmark show that our framework outperforms state-of-the-art methods both qualitatively and quantitatively.
Parser-Free Virtual Try-On The formulations mentioned earlier (see Figure 1) can be subdivided based on whether the parser is used in image-to-image translation [21, 16]. It has been demonstrated that the parser can improve try-on results but may introduce unreliable semantic errors [11, 6, 7]. Initially, [8, 3] used pose heatmap and body mask as person representation to provide body's geometry information. Later, [10] introduced a parser to estimate an "after-try-on" semantic map instead of the above person representation. To avoid unreliable semantic information, [11, 6, 7] trained a "student model" without the parser using pretrained teacher knowledge. However, the potential presence of irresponsible teacher knowledge can still affect the "student model," making it challenging to train robust architectures without
the parser in the input end. To address this challenge, we present a parser-free pipeline called self-cycle consistency. Our goal aims to achieve high-quality virtual try-on results using a general network, minimum input, and simple but efficient architecture.
Figure 2: Comparisons of virtual try-on (DCTON [2], PF-AFN [6], FS-VTON [7], RT-VTON [13], and ours) and different deformation methods (TPS, AF, MLS, and our proposed MRF).
Non-rigid Deformation In virtual try-on tasks, common non-rigid image deformation methods include Thin Plate Spline (TPS) [17, 18], Moving Least Squares (MLS) [20], and Appearance Flow (AF) [19]. As shown in Figure 2, TPS is the most widely used due to its ability to provide high-quality deformation while preserving the garment's image structure [8, 3, 10, 4, 11, 2, 12]. However, TPS is limited by its local mode, which only significantly affects pixels near control points. MLS is another method introduced recently, but it is not suitable for large-scale deformation [13]. AF is the most flexible method, deforming the garment by estimating pixel displacement vectors between the source and target shapes [9, 6, 7, 14]. However, none of these methods can fully mimic the natural interaction between the garment and the human body due to a lack of spatial perception. Our proposed method, which is based on Markov Random Field system with unpaired images as input, can obtain realistic spatial deformation.
Self-Cycle Consistency CycleGAN [1] utilized a self-supervised learning framework for cyclically learning the translation between unpaired images from different domains. [2] extended this with a deconstruction method for different garment styles to address the failure of cycle consistency in virtual try-on tasks. However, the complex dual-model structure and reliance on the parser hinder the generation of realistic results. To break through these limitations, following StarGAN [22], we propose a single-model self-cyclic learning translation paradigm between different garment styles.
Figure 3: Overview of our self-cycle consistency network: paired garment-person images  $(\mathbf{g},\mathbf{p})$  and an arbitrary garment  $\mathbf{g}'$  serve as the training data. Left:  $\mathcal{G}_{\theta}$  is pre-trained using paired images to obtain the ground-truth deformation field  $\tilde{\mathbf{f}}$  for the auxiliary deformer  $\mathcal{G}_A$  to optimize  $\tilde{\mathbf{f}}_A$ . Then,  $\mathcal{G}_{\theta}$  is further trained by using  $\tilde{\mathbf{f}}_A$  of  $\mathcal{G}_A^*$  with unpaired images. Right:  $\mathcal{G}_{\varphi}$  takes unpaired  $(\hat{\mathbf{g}}',\mathbf{p})$  to generate a fake person  $\hat{\mathbf{p}}'$ . Then, the real person  $\mathbf{p}$  is reconstructed by re-feeding  $\hat{\mathbf{p}}'$  and  $\hat{\mathbf{g}}$  into  $\mathcal{G}_{\varphi}$ , ensuring self-cycle consistency  $\mathbf{p} \approx \hat{\mathbf{p}}$ .  $D_s$  is the discriminator that learns to distinguish real and fake images. EPS represents enhanced pixel-level supervision and Det. represents detaching tensor  $\hat{\mathbf{p}}'$  from the computation graph.
Notation. In this work, we use  $(\mathbf{g},\mathbf{p})$  to represent paired garment-person images where the person  $\mathbf{p}$  is wearing garment  $\mathbf{g}$ . Moreover, we randomly select the garment image  $\mathbf{g}'$  to form unpaired garment-person images  $(\mathbf{g}',\mathbf{p})$  with  $\mathbf{p}$ . The try-on result of  $(\mathbf{g},\mathbf{p})$  and  $(\mathbf{g}',\mathbf{p})$  are denoted as  $\hat{\mathbf{p}}$  and  $\hat{\mathbf{p}}'$ , respectively. In general, "\*" represents the target image, and "\*" denotes the transformation result. In addition, " $\mapsto$  " represents the mapping relationship of the entire architecture, while " $\Rightarrow$  " represents the mapping relationship of the sub-modules.
Overview. The objective of the virtual try-on task is to naturally dress an arbitrary in-shop garment image  $\mathbf{g}' \in \mathbb{R}^{3 \times H \times W}$  on a reference person image  $\mathbf{p} \in \mathbb{R}^{3 \times H \times W}$  to obtain a try-on result  $\hat{\mathbf{p}}' \in \mathbb{R}^{3 \times H \times W}$ . As illustrated in Figure 3, we first propose a self-cycle consistency framework, USC-PFN, a new virtual try-on paradigm that does not require the human parser and other person representation at the model input end for generating highly realistic try-on images. The framework is broken up into two sub-networks: (i) a non-rigid garment deformer  $\mathcal{G}_{\theta}$ , and (ii) a self-cyclic image generator  $\mathcal{G}_{\varphi}$ . They are mainly used to construct a cyclic mapping  $V_{t}: (\mathbf{p}, \mathbf{g}') \mapsto \hat{\mathbf{p}}', (\hat{\mathbf{p}}', \mathbf{g}) \mapsto \hat{\mathbf{p}}$ , where  $\mathbf{p} \approx \hat{\mathbf{p}}$ , i.e.,  $\hat{\mathbf{p}}$  is the same to the real image  $\mathbf{p}$ .
Garments Deformation. In  $(\mathbf{p},\mathbf{g}^{\prime})\mapsto \hat{\mathbf{p}}^{\prime}$ , the initially flat garment  $\mathbf{g}^{\prime}$  must be deformed to match the posture of the person  $\mathbf{p}$  before synthesizing the target try-on result  $\hat{\mathbf{p}}^{\prime}$ , denotes as  $(\mathbf{p},\mathbf{g}^{\prime})\stackrel {\theta}{\Rightarrow}\hat{\mathbf{g}}^{\prime},(\mathbf{p},\hat{\mathbf{g}}^{\prime})\stackrel {\varphi}{\Rightarrow}\hat{\mathbf{p}}^{\prime}$ . Our key idea involves using  $\mathcal{G}_{\theta}$  to model the non-rigid deformation based on Markov Random Field (MRF) [23], to seek an optimal spatial transformation solution  $\hat{\mathbf{f}}^{*}\in \mathbb{R}^{2\times H\times W}$  (i.e. optimal deformation field  $\hat{\mathbf{f}}^{*} = \mathcal{G}_{\theta}^{*}(\mathbf{p},\mathbf{g}^{\prime}))$  that minimizes the dissimilarity between the warped garment  $\hat{\mathbf{g}}^{\prime}$  and the garment  $\hat{\mathbf{p}}_{\mathbf{g}}^{\prime}$  worn on the target result  $\hat{\mathbf{p}}^{\prime}$ , where  $\hat{\mathbf{g}}^{\prime}$  is obtained by using bilinear interpolation  $(\delta)$ :  $\hat{\mathbf{g}}^{\prime} = \varrho (\hat{\mathbf{f}},\mathbf{g}^{\prime})$ . This can be formalized as minimizing the MRF energy function:
$$
\underset {<   \tilde {\mathbf {f}} _ {\mathcal {A}}, \hat {\mathbf {p}} _ {\mathbf {g}} ^ {\prime} > \in \mathcal {V} ^ {+}} {\text {m i n i m i z e}} \left\{E _ {d a t} \left(\tilde {\mathbf {f}} _ {\mathcal {A}}, \hat {\mathbf {f}}\right) + \lambda_ {r} E _ {r e g} \left(\hat {\mathbf {f}}\right) + \lambda_ {p} E _ {p i x} \left(\hat {\mathbf {p}} _ {\mathbf {g}} ^ {\prime}, \hat {\mathbf {g}} ^ {\prime}\right) \right\}, \tag {1}
$$
where  $\lambda_r$  and  $\lambda_p$  are trade-off hyper-parameters.  $\mathcal{V}^+$  represents the prior pseudo-label from the pre-trained garment deformer and  $\tilde{\mathbf{f}}_{\mathcal{A}}$  is the deformation field of  $\hat{\mathbf{p}}_{\mathbf{g}}'$ .
$$
E _ {d a t} = \operatorname {a r g m i n} _ {\hat {\mathbf {f}}} \left\| \tilde {\mathbf {f}} _ {\mathcal {A}} - \hat {\mathbf {f}} \right\|, E _ {r e g} = \mathbf {w} _ {\phi} R _ {s m} (\hat {\mathbf {f}}), E _ {p i x} = \operatorname {a r g m i n} _ {\hat {\mathbf {f}}} D [ \hat {\mathbf {p}} _ {\mathbf {g}} ^ {\prime}, \hat {\mathbf {g}} ^ {\prime} ], \tag {2}
$$
where  $E_{reg}$  is regularization term of  $\hat{\mathbf{f}}$ , which composed of spatial position weights  $\mathbf{w}_{\phi}$  of pixel points and a smoothing term  $R_{sm}$ .  $R_{sm}$  is used to ensure that each element in  $\hat{\mathbf{f}}$  and its neighborhood have similar values.  $D[\cdot, \cdot]$  indicates quantification of similarity or difference. During garment deformation, we find that the deformation range of the edge region is often larger than the central region. Therefore, we use the two-dimensional Gaussian distribution [24] with the centroid and variance of pixel coordinates in the cloth region, as  $\mathbf{w}_{\phi}$  to maintain the flexibility of the edge region. In  $(\hat{\mathbf{p}}', \mathbf{g}) \mapsto \hat{\mathbf{p}}$ , this process is similar to the one above, i.e.,  $(\hat{\mathbf{p}}', \mathbf{g}) \stackrel{\theta}{\Rightarrow} \hat{\mathbf{g}}, (\hat{\mathbf{p}}', \hat{\mathbf{g}}) \stackrel{\varphi}{\Rightarrow} \hat{\mathbf{p}}$ , but the difference is that we remove  $E_{\text{dat}}$  from Eq.(1) to ensure more accurate results.
Synthesis of Try-On Results. Recall that the goal of the virtual try-on task is to synthesize the target try-on result after warping the garment as if it was shot from realistic scenes. Therefore, reconstructing the real image  $\mathbf{p}$  is an essential task to form self-supervised training, which can be formalized as minimizing the following objective:
$$
\operatorname {m i n i m i z e} \sum_ {<   \hat {\mathbf {g}}, \mathbf {p} > \in \mathcal {V}} \operatorname {a r g m i n} _ {\hat {\mathbf {p}}} D [ \mathbf {p}, \hat {\mathbf {p}} ], \text {w i t h} (\hat {\mathbf {p}} ^ {\prime}, \hat {\mathbf {g}}) \stackrel {{\varphi}} {{\Rightarrow}} \hat {\mathbf {p}}, \tag {3}
$$
where  $\mathcal{V}$  is training set.  $\hat{\mathbf{p}}^{\prime}$  denotes  $\mathbf{p}$  wearing the garment  $\mathbf{g}^{\prime}$ . This differentiable generation appears to have completeness, however, the absence of  $\hat{\mathbf{p}}^{\prime}$  in  $\mathcal{V}$  forces the process to be compensated by extra person representation  $\mathcal{R}$  (semantic maps, pose heatmaps, etc.) as input. This means that results are heavily constrained by  $\mathcal{R}$ , and any errors that occur can directly affect the final try-on effect. To tackle this problem, [6, 7] utilized a pre-trained and parser-based "teacher network"  $\mathcal{T}$  to generate  $\hat{\mathbf{p}}^{\prime}$ :
$$
\hat {\mathbf {p}} ^ {\prime} = \mathcal {T} ^ {*} (\mathcal {R}, \mathbf {p}, \hat {\mathbf {g}} ^ {\prime}), \hat {\mathbf {p}} = \mathcal {S} (\hat {\mathbf {p}} ^ {\prime}, \hat {\mathbf {g}}). \tag {4}
$$
Yet they cannot handle that irresponsible teacher knowledge brings to the learning process of their "student network"  $\mathcal{S}$ . [2] introduced the cycle-consistency structure that employed and trained cyclically two CNNs (e.g.,  $\mathcal{N}_1$  and  $\mathcal{N}_2$ ), where let  $\mathcal{N}_1$  provide  $\hat{\mathbf{p}}'$  for  $\mathcal{N}_2$ :
$$
\hat {\mathbf {p}} ^ {\prime} = \mathcal {N} _ {1} (\mathcal {R}, \mathbf {p}, \hat {\mathbf {g}} ^ {\prime}), \hat {\mathbf {p}} = \mathcal {N} _ {2} (\mathcal {R}, \hat {\mathbf {p}} ^ {\prime}, \hat {\mathbf {g}}), \tag {5}
$$
where the structural difference between  $\hat{\mathbf{p}}$  and  $\mathbf{p}$  is minimized to ensure cycle-consistency. However, decomposing the body and then synthesizing it back together based on semantics leads to occlusion and artifacts due to some parsing errors. Relying on prior knowledge weakens the learning of the garment of adaptive spatial structure on the human body. Additionally, the dual-model structure is complex, resulting in longer training times and unstable convergence. Inspired by [22], we reformulate the synthesis process as an unpaired garment style translation problem. Specifically, we provide  $\hat{\mathbf{p}}'$  for ourselves  $(\mathcal{G}_{\varphi})$  by ourselves (weight-sharing structure):
$$
\hat {\mathbf {p}} = \mathcal {G} _ {\varphi} (\hat {\mathbf {p}} ^ {\prime}, \hat {\mathbf {g}}), \text {w i t h} \hat {\mathbf {p}} ^ {\prime} = \mathcal {G} _ {\varphi} (\mathbf {p}, \hat {\mathbf {g}} ^ {\prime}). \tag {6}
$$
Inference. During inference, our goal is to synthesize the desired try-on result  $\hat{\mathbf{p}}$  just by feeding arbitrary garment-person pairs  $(\mathbf{p},\mathbf{g}^{\prime})$  you need into  $\mathcal{G}_{\theta}^{*}$  and  $\mathcal{G}_{\varphi}^{*}$ :
$$
\hat {\mathbf {p}} = \mathcal {G} _ {\varphi} ^ {*} (\mathbf {p}, \hat {\mathbf {g}} ^ {\prime}), \text {w i t h} \hat {\mathbf {g}} ^ {\prime} = \varrho \left(\mathcal {G} _ {\theta} ^ {*} (\mathbf {p}, \mathbf {g} ^ {\prime}), \mathbf {g} ^ {\prime}\right). \tag {7}
$$
3.2 Network Architecture
Non-rigid Garment Deformer (NGD)  $\mathcal{G}_{\theta}$ . The deformer  $\mathcal{G}_{\theta}$  can adopt arbitrary encoder-decoder networks. As shown in Eq. (1), to fit Markov Random Field, we first pre-trained  $\mathcal{G}_{\theta}$  by feeding paired images  $(\mathbf{p},\mathbf{g})$  to learn a real deformation field  $\tilde{\mathbf{f}}$ , i.e.  $\tilde{\mathbf{f}} = \mathcal{G}_{\theta}(\mathbf{p},\mathbf{g})$ . At this point, pre-trained  $\mathcal{G}_{\theta}$  contains the full latent alignment information, including shape, color, perspective changes, texture variations, shadows and lighting, and depth of field, between the paired images  $(\mathbf{p},\mathbf{g})$ . However, features such as color, shape, and texture do not need to be learned, because the input garment varies greatly during inference, but  $\mathbf{g}$  and  $\mathbf{p}$  used for training are paired. Therefore,  $\mathcal{G}_{\theta}$  is necessarily invalid when unpaired images are fed. To disentangle the aforementioned feature entanglement, we employ an auxiliary deformer  $\mathcal{G}_{A}$ , which takes the garment  $\mathbf{g}$ , prior Densepose descriptor  $\mathbf{p_d} \in \mathbb{R}^{1\times H\times W}$  [25], and the pose heatmap  $\mathbf{p_h} \in \mathbb{R}^{18\times H\times W}$  as input to learn the same goal as pre-trained  $\mathcal{G}_{\theta}$ , i.e.,  $\tilde{\mathbf{f}}_{A} = \mathcal{G}_{A}(\mathbf{p_d},\mathbf{p_h},\mathbf{g})$ . Because  $\mathbf{p_d}$  and  $\mathbf{p_h}$  are not correlated with  $g$  in terms of color, texture, and shape,  $\mathcal{G}_{A}$  does not face the challenges mentioned above and can handle any type of garment effortlessly. However, unfortunately,  $\mathbf{p_d}$  and  $\mathbf{p_h}$  also lack perspective changes, shadows, lighting,
and depth of field. As a result, the deformation results obtained are often not realistic. Therefore, we use  $\tilde{\mathbf{f}}$  to supervise the training of  $\mathcal{G}_A$ ,  $\tilde{\mathbf{f}}_A \approx \tilde{\mathbf{f}}$ , supplementing its spatial perception abilities. After obtaining the optimal  $\mathcal{G}_A^*$ , we continue to train the deformer  $\mathcal{G}_{\theta}$  by taking unpaired  $(\mathbf{p}, \mathbf{g}')$  as input and  $\mathbf{f}_A$  as supervision, to disentangle the feature entanglement of  $\mathcal{G}_{\theta}$ :
$$
\tilde {\mathbf {f}} _ {\mathcal {A}} \approx \hat {\mathbf {f}}, \text {w i t h} \hat {\mathbf {f}} = \mathcal {G} _ {\theta} \left(\mathbf {p}, \mathbf {g} ^ {\prime}\right), \tilde {\mathbf {f}} _ {\mathcal {A}} = \mathcal {G} _ {\mathcal {A}} ^ {*} \left(\mathbf {p} _ {\mathbf {d}}, \mathbf {p} _ {\mathbf {h}}, \mathbf {g} ^ {\prime}\right). \tag {8}
$$