Towards Scalable Unpaired Virtual Try-On via Patch-Routed Spatially-Adaptive GAN
Year: 2021

Abstract:
Image-based virtual try-on is one of the most promising applications of humancentric image generation due to its tremendous real-world potential. Yet, as most try-on approaches fit in-shop garments onto a target person, they require the laborious and restrictive construction of a paired training dataset, severely limiting their scalability. While a few recent works attempt to transfer garments directly from one person to another, alleviating the need to collect paired datasets, their performance is impacted by the lack of paired (supervised) information. In particular, disentangling style and spatial information of the garment becomes a challenge, which existing methods either address by requiring auxiliary data or extensive online optimization procedures, thereby still inhibiting their scalability. To achieve a scalable virtual try-on system that can transfer arbitrary garments between a source and a target person in an unsupervised manner, we thus propose a texture-preserving end-to-end network, the PAtch-routed SpaTially-Adaptive GAN (PASTA-GAN), that facilitates real-world unpaired virtual try-on. Specifically, to disentangle the style and spatial information of each garment, PASTA-GAN consists of an innovative patch-routed disentanglement module for successfully retaining garment texture and shape characteristics. Guided by the source person keypoints, the patch-routed disentanglement module first decouples garments into normalized patches, thus eliminating the inherent spatial information of the garment, and then reconstructs the normalized patches to the warped garment complying with the target person pose. Given the warped garment, PASTA-GAN further introduces novel spatially-adaptive residual blocks that guide the generator to synthesize more realistic garment details. Extensive comparisons with paired and unpaired approaches demonstrate the superiority of PASTA-GAN, highlighting its ability to generate high-quality try-on images when faced with a large variety of garments (e.g. vests, shirts, pants), taking a crucial step towards real-world scalable try-on.

heading: 1 Introduction
text: Image-based virtual try-on, the process of computationally transferring a garment onto a particular person in a query image, is one of the most promising applications of human-centric image generation with the potential to revolutionize shopping experiences and reduce purchase returns. However, to fully exploit its potential, scalable solutions are required that can leverage easily accessible training data, handle arbitrary garments, and provide efficient inference results. Unfortunately, to date, most existing methods [35, 38, 12, 7, 37, 9, 10, 4, 36, 39] rely on paired training data, i.e., a person image
∗Xiaodan Liang is the corresponding author. Our code will be available at PASTA-GAN.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
and its corresponding in-shop garment, leading to laborious data-collection processes. Furthermore, these methods are unable to exchange garments directly between two person images, thus largely limiting their application scenarios and raising the need for unpaired solutions to ensure scalability.
While unpaired solutions have recently started to emerge, performing virtual try-on in an unsupervised setting is extremely challenging and tends to affect the visual quality of the try-on results. Specifically, without access to the paired data, these models are usually trained by reconstructing the same person image, which is prone to over-fitting, and thus underperform when handling garment transfer during testing. The performance discrepancy is mainly reflected in the garment synthesis results, in particular the shape and texture, which we argue is caused by the entanglement of the garment style and spatial representations in the synthesis network during the reconstruction process.
While this is not a problem for the traditional paired try-on approaches [35, 12, 37, 10], which avoid this problem and preserve the garment characteristics by utilizing a supervised warping network to obtain the warped garment in target shape, this is not possible in the unpaired setting due to the lack of warped ground truth. The few works that do attempt to achieve unpaired virtual try-on, therefore, choose to circumvent this problem by either relying on person images in various poses for feature disentanglement [23, 33, 32, 31, 1, 5], which again leads to a laborious data-collection process, or require extensive online optimization procedures [25, 17] to obtain fine-grain details of the original garments, harming the inference efficiency. However, none of the existing unpaired try-on methods consider the problem of coupled style and spatial garment information directly, which is crucial to obtain accurate garment transfer results in the unpaired and unsupervised virtual try-on scenario.
In this paper, to tackle the essential challenges mentioned above, we propose a novel PAtch-routed SpaTially-Adaptive GAN, named PASTA-GAN, a scalable solution to the unpaired try-on task. Our PASTA-GAN can precisely synthesize garment shape and style (see Fig. 1) by introducing a patch-routed disentanglement module that decouples the garment style and spatial features, as well as a novel spatially-adaptive residual module to mitigate the problem of feature misalignment.
The innovation of our PASTA-GAN includes three aspects: First, by separating the garments into normalized patches with the inherent spatial information largely reduced, the patch-routed disentanglement module encourages the style encoder to learn spatial-agnostic garment features. These features enable the synthesis network to generate images with accurate garment style regardless of varying spatial garment information. Second, given the target human pose, the normalized patches can be easily reconstructed to the warped garment complying with the target shape, without requiring a warping network or a 3D human model. Finally, the spatially-adaptive residual module extracts the warped garment feature and adaptively inpaints the region that is misaligned with the target garment shape. Thereafter, the inpainted warped garment features are embedded into the intermediate layer of the synthesis network, guiding the network to generate try-on results with realistic garment texture.
We collect a scalable UnPaired virtual Try-on (UPT) dataset and conduct extensive experiments on the UPT dataset and two existing try-on benchmark datasets (i.e., the DeepFashion [21] and the MPV [6] datasets). Experiment results demonstrate that our unsupervised PASTA-GAN outperforms both the previous unpaired and paired try-on approaches.

heading: 2 Related Work
text: Paired Virtual Try-on. Paired try-on methods [13, 35, 38, 12, 24, 37, 9, 10, 36] aim to transfer an in-shop garment onto a reference person. Among them, VITON [13] for the first time integrates a U-Net [29] based generation network with a TPS [2] based deformation approach to synthesize the try-on result. CP-VTON [35] improves this paradigm by replacing the time-consuming warping module with a trainable geometric matching module. VTNFP [38] adopts human parsing to guide the generation of various body parts, while [24, 37, 39] introduce a smooth constraint for the warping module to alleviate the excessive distortion in TPS warping. Besides the TPS-based warping strategy, [12, 36, 10] turn to the flow-based warping scheme which models the per-pixel deformation. Recently, VITON-HD [4] focuses on high-resolution virtual try-on and proposes an ALIAS normalization mechanism to resolve the garment misalignment. PF-AFN [10] improves the learning process by employing knowledge distillation, achieving state-of-the-art results. However, all of these methods require paired training data and are incapable of exchanging garments between two person images.
Unpaired Virtual Try-on. Different from the above methods, some recent works [23, 33, 32, 31, 25, 17] eliminate the need for in-shop garment images and directly transfer garments between two person images. Among them, [23, 33, 32, 31, 1, 5] leverage pose transfer as the pretext task to learn disentangled pose and appearance features for human synthesis, but require images of the same person with different poses.2 In contrast, [25, 17] are more flexible and can be directly trained with unpaired person images. However, OVITON [25] requires online appearance optimization for each garment region during testing to maintain texture detail of the original garment. VOGUE [17] needs to separately optimize the latent codes for each person image and the interpolate coefficient for the final try-on result during testing. Therefore, existing unpaired methods require either cumbersome data collecting or extensive online optimization, extremely harming their scalability in real scenarios.

heading: 3 PASTA-GAN
text: Given a source image Is of a person wearing a garment Gs, and a target person image It, the unpaired virtual try-on task aims to synthesize the try-on result I ′t retaining the identity of It but wearing the source garment Gs. To achieve this, our PASTA-GAN first utilizes the patch-routed disentanglement module (Sec. 3.1) to transform the garment Gs into normalized patches Pn that are mostly agnostic to the spatial features of the garment, and further deforms Pn to obtain the
2As the concurrent work StylePoseGAN [31] is the most related pose transfer-based approach, we provide a more elaborate discussion of the inherent differences in the supplementary.
warped garment Gt complying with the target person pose. Then, an attribute-decoupled conditional StyleGAN2 (Sec. 3.2) is designed to synthesize try-on results in a coarse-to-fine manner, where we introduce novel spatially-adaptive residual blocks (Sec. 3.3) to inject the warped garment features into the generator network for more realistic texture synthesis. The loss functions and training details will be described in Sec. 3.4. Fig. 2 illustrates the overview of the inference process for PASTA-GAN.

heading: 3.1 Patch-routed Disentanglement Module
text: Since the paired data for supervised training is unavailable for the unpaired virtual try-on task, the synthesis network has to be trained in an unsupervised manner via image reconstruction, and thus takes a person image as input and separately extracts the feature of the intact garment and the feature of the person representation to reconstruct the original person image. While such a training strategy retains the intact garment information, which is helpful for the garment reconstruction, the features of the intact garment entangle the garment style with the spatial information in the original image. This is detrimental to the garment transfer during testing. Note that the garment style here refers to the garment color and categories, i.e., long sleeve, short sleeve, etc., while the garment spatial information implies the location, the orientation, and the relative size of the garment patch in the person image, in which the first two parts are influenced by the human pose while the third part is determined by the relative camera distance to the person.
To address this issue, we explicitly divide the garment into normalized patches to remove the inherent spatial information of the garment. Taking the sleeve patch as an example, by using division and normalization, various sleeve regions from different person images can be deformed to normalized patches with the same orientation and scale. Without the guidance of the spatial information, the network is forced to learn the garment style feature to reconstruct the garment in the synthesis image.
Fig. 3 illustrates the process of obtaining normalized garment patches, which includes two main steps: (1) pose-guided garment segmentation, and (2) perspective transformation-based patch normalization. Specifically, in the first step, the source garment Gs and human pose (joints) Js are firstly obtained by applying [11] and [3] to the source person Is, respectively. Given the body joints, we can segment the source garment into several patches Ps, which can
be quadrilaterals with arbitrary shapes (e.g., rectangle, square, trapezoid, etc.), and will later be normalized. Taking the torso region as an example, with the coordinates of the left/right shoulder joints and the left/right hips joints in P is , a quadrilateral crop (of which the four corner points are visualized in color in P is of Fig. 3) covering the torso region of Gs can be easily performed to produce an unnormalized garment patch. Note that we define eight patches for upper-body garments, i.e., the patches around the left/right upper/bottom arm, the patches around the left/right hips, a patch around the torso, and a patch around the neck. In the second step, all patches are normalized to remove their spatial information by perspective transformations. For this, we first define the same amount of template patches Pn with fixed 64× 64 resolution as transformation targets for all unnormalized source patches, and then compute a homography matrixHis→n ∈ R3×3 [40] for each pair of P is and P in, based on the four corresponding corner points of the two patches. Concretely,His→n serves as a perspective transformation to relate the pixel coordinates in the two patches, formulated as: xinyin
1
 = His→n  xisyis
1  =  hi11 hi12 hi13hi21 hi22 hi23 hi31 h i 32 h i 33  xisyis 1  (1) where (xin, y i n) and (x i s, y i s) are the pixel coordinates in the normalized template patch and the unnormalized source patch, respectively. To compute the homography matrixHis→n, we directly leverage the OpenCV API, which takes as inputs the corner points of the two patches and is implemented by using least-squares optimization and the Levenberg-Marquardt method [8]. After obtainingHis→n, we can transform the source patch P is to the normalized patch P i n according to Eq. 1.
Moreover, the normalized patches Pn can further be transformed to target garment patches Pt by utilizing the target pose Jt, which can be obtained from the target person It via [3]. The mechanism
of that backward transformation is equivalent to the forward one in Eq. 1, i.e., computing the homography matrixHin→t based on the four point pairs extracted from the normalized patch P in and the target pose Jt. The recovered target patches Pt can then be stitched to form the warped garment Gt that will be sent to the texture synthesis branch in Fig. 2 to generate more realistic garment transfer results. We can also regardHs→t = Hn→t · Hs→n as the combined deformation matrix that warps the source garment to the target person pose, bridged by an intermediate normalized patch representation that is helpful for disentangling garment styles and spatial features.

heading: 3.2 Attribute-decoupled Conditional StyleGAN2
text: Motivated by the impressive performance of StyleGAN2 [15] in the field of image synthesis, our PASTA-GAN inherits the main architecture of StyleGAN2 and modifies it to the conditional version (see Fig. 2). In the synthesis network, the normalized patches Pn are projected to the style code w through a style encoder followed by a mapping network, which is spatial-agnostic benefiting from the disentanglement module. In parallel, the conditional information including the target head Ht and pose Jt is transferred into a feature map fid, encoding the identity of the target person by the identity encoder. Thereafter, the synthesis network starts from the identity feature map and leverages the style code as the injected vector for each synthesis block to generate the try-on result Ĩt ′ .
However, the standalone conditional StyleGAN2 is insufficient to generate compelling garment details especially in the presence of complex textures or logos. For example, although the illustrated Ĩt ′
in Fig. 2 can recover accurate garment style (color and shape) given the disentangled style code w, it lacks the complete texture pattern. The reasons for this are twofold: First, the style encoder projects the normalized patches into a one-dimensional vector, resulting in loss of high frequency information. Second, due to the large variety of garment texture, learning the local distribution of the particular garment details is highly challenging for the basic synthesis network.
To generate more accurate garment details, instead of only having a one-way synthesis network, we intentionally split PASTA-GAN into two branches after the 128× 128 synthesis block, namely the Style Synthesis Branch (SSB) and the Texture Synthesis Branch (TSB). The SSB with normal StyleGAN2 synthesis blocks aims to generate intermediate try-on results Ĩt ′ with accurate garment style and predict a precise garment mask Mg that will be used by TSB. The purpose of TSB is to exploit the warped garment Gt, which has rich texture information to guide the synthesis path, and generate high-quality try-on results. We introduce a novel spatially-adaptive residual module specifically before the final synthesis block of the TSB, to embed the warped garment feature fg (obtained by passing Mg and Gt through the garment encoder) into the intermediate features and then send them to the newly designed spatialy-apaptive residual blocks, which are beneficial for successfully synthesizing texture of the final try-on result I ′t. The detail of this module will be described in the following section.
3.3 Spatially-adaptive Residual Module
Given the style code that factors out the spatial information and only keeps the style information of the garment, the style synthesis branch in Fig. 2 can accurately predict the mean color and the shape mask of the target garment. However, its inability to model the complex texture raises the need to exploit the warped garment Gt to provide features that encode high-frequency texture patterns, which is in fact the motivation of the target garment reconstruction in Fig. 3.
However, as the coarse warped garment Gt is directly obtained by stitching the target patches together, its shape is inaccurate and usually misaligns with the predicted mask Mg (see Fig.4). Such shape misalignment in Gt will consequently reduce the quality of the extracted warped garment feature fg .
To address this issue, we introduce the spatially-adaptive residual module between the last two synthesis blocks in the texture synthesis branch as shown in Fig. 2. This module is comprised of a garment encoder and three spatially-adaptive residual blocks with feature inpainting mechanism to modulate intermediate features by leveraging the inpainted warped garment feature.
To be specific on the feature inpainting process, we first remove the part of Gt that falls outside of Mg (green region in Fig.4), and explicitly inpaint the misaligned regions of the feature map within Mg with average feature values (orange region in Fig. 4). The inpainted feature map can then help the final synthesis block infer reasonable texture in the inside misaligned parts.
Therefore given the predicted garment mask Mg , the coarse warped garment Gt and its mask Mt, the process of feature inpainting can be formulated as:
Malign =Mg ∩Mt, (2)
Mmisalign =Mg −Malign, (3) f ′g = Eg(Gt Mg), (4)
fg = f ′ g (1−Mmisalign) +A(f ′g Malign) Mmisalign, (5)
where Eg(·) represents the garment encoder and f ′g denotes the raw feature map of Gt masked by Mg . A(·) calculates the average garment features and fg is the final inpainted feature map. Subsequently, inspired by the SPADE ResBlk from SPADE [26], the inpainted garment features are used to calculate a set of affine transformation parameters that efficiently modulate the normalized feature map within each spatially-adaptive residual block. The normalization and modulation process for a particular sample hz,y,x at location (z ∈ C, y ∈ H,x ∈ W ) in a feature map can then be formulated as:
γz,y,x(fg) hz,y,x − µz
σz + βz,y,x(fg), (6)
where µz = 1HW ∑ y,x hz,y,x and σz = √ 1 HW ∑ y,x (hz,y,x − µz)
2 are the mean and standard deviation of the feature map along channel C. γz,y,x(·) and βz,y,x(·) are the convolution operations that convert the inpainted feature to affine parameters.
Eq. 6 serves as a learnable normalization layer for the spatially-adaptive residual block to better capture the statistical information of the garment feature map, thus helping the synthesis network to generate more realistic garment texture.
With the modulated intermediate feature maps produced by the spatially-adaptive residual module, the texture synthesis branch can effectively utilize the reconstructed warped garment and generate the final compelling try-on result with high-frequency texture patterns.

heading: 3.4 Loss Functions and Training Details
text: As paired training data is unavailable, our PASTA-GAN is trained unsupervised via image reconstruction. During training, we utilize the reconstruction loss Lrec and the perceptual loss [14] Lperc for both the coarse try-on result Ĩ ′ and the final try-on result I ′:
Lrec = ∑
I∈{Ĩ′,I′}
‖I − Is‖1 and Lperc = ∑
I∈{Ĩ′,I′}
5∑ k=1 λk ‖φk(I)− φk (Is)‖1 , (7)
where φk(I) denotes the k-th feature map in a VGG-19 network [34] pre-trained on the ImageNet [30] dataset. We also use the L1 loss between the predicted garment mask Mg and the real mask Mgt which is obtained via human parsing [11]:
Lmask = ‖Mg −Mgt‖1. (8)
Besides, for both Ĩ ′ and I ′, we calculate the adversarial loss LGAN which is the same as in StyleGAN2 [15]. The total loss can be formulated as
L = LGAN + λrecLrec + λpercLperc + λmaskLmask, (9)
where λrec, λperc, and λmask are the trade-off hyper-parameters.
During training, although the source and target pose are the same, the coarse warped garment Gt is not identical to the intact source garment Gs, due to the crop mechanism in the patch-routed disentanglement module. More specifically, the quadrilateral crop for Gs is by design not seamless/perfect and there will accordingly often exist some small seams between adjacent patches in Gt as well as incompleteness along the boundary of the torso region. To further reduce the training-test gap of the warped garment, we introduce two random erasing operations during the training phase. First, we randomly remove one of the four arm patches in the warped garment with a probability of α1. Second, we use the random mask from [19] to additionally erase parts of the warped garment with a probability of α2. Both of the erasing operations can imitate self-occlusion in the source person image. Fig. 5 illustrates the process by displaying the source garment Gs, the warped garment G′t that is obtained by directly stitching the warped patches together, and the warped garment Gt that is sent to the network. We can observe a considerable difference between Gt and Gs. An ablation experiment to validate the necessity of the randomly erasing operation for the unsupervised training is included in the supplementary material.

heading: 4 Experiments
text: Datasets. We conduct experiments on two existing virtual try-on benchmark datasets (MPV [6] dataset and DeepFashion [21] dataset) and our newly collected large-scale benchmark dataset for unpaired try-on, named UPT. UPT contains 33,254 half- and full-body front-view images of persons wearing a large variety of garments, e.g., long/short sleeve, vest, sling, pants, etc. UPT is further split into a training set of 27,139 images and a testing set of 6,115 images. In addition, we also pick out the front view images from MPV [6] and DeepFashion [21] to expand the size of our training and testing set to 54,714 and 10,493, respectively. Personally identifiable information (i.e. face information) has been masked out.
Metrics. We apply the Fréchet Inception Distance (FID) [27] to measure the similarity between real and synthesized images, and perform human evaluation to quantitatively evaluate the synthesis quality of different methods. For the human evaluation, we design three questionnaires corresponding to the three used datasets. In each questionnaire, we randomly select 40 try-on results generated by our PASTA-GAN and the other compared methods. Then, we invite 30 volunteers to complete the 40 tasks by choosing the most realistic try-on results. Finally, the human evaluation score is calculated as the chosen percentage for a particular method.
Implementation Details. Our PASTA-GAN is implemented using PyTorch [28] and is trained on 8 Tesla V100 GPUs. During training, the batch size is set to 96 and the model is trained for 4 million iterations with a learning rate of 0.002 using the Adam optimizer [16] with β1 = 0 and β2 = 0.99. The loss hyper-parameters λrec, λperc, and λmask are set to 40, 40, and 100, respectively. The hhyper-parameters for the random erasing probability α1 and α2 are set to 0.2 and 0.9, respectively. 3
Baselines. To validate the effectiveness of our PASTA-GAN, we compare it with the state-of-the-art methods, including three paired virtual try-on methods, CP-VTON [35], ACGPN [37], PFAFN [10], and two unpaired methods Liquid Warping GAN [20] and ADGAN [23], which have released the official code and pre-trained weights.4 We directly use the pre-trained model of these methods as their training procedure depends on the paired data of garment-person or person-person image pairs, which are unavailable in our dataset. When testing paired methods under the unpaired try-on setting, we extract the desired garment from the person image and regard it as the in-shop garment to meet the
3Additional details for the UPT dataset (e.g., data distribution, data pre-processing), the human evaluation, training details, and the inference time analysis, etc. are provided in the supplementary material.
4For all these prior approaches, research use is permitted according to the respective licenses. Note, we are unable to compare with [31], [17] and [25] as they have not released their code or pre-trained model.
need of paired approaches. To fairly compare with the paired methods, we further conduct another experiment on the paired MPV dataset [6], in which the paired methods take an in-shop garment and a person image as inputs, while our PASTA-GAN still directly receives two person images. See the following two subsections for detailed comparisons on both paired and unpaired settings.

heading: 4.1 Comparison with the state-of-the-art methods on unpaired benchmark
text: Quantitative: As reported in Table 1, when testing on the DeepFashion [21] and the UPT dataset under the unpaired setting, our PASTA-GAN outperforms both the paired methods [35, 37, 10] and the unpaired methods [23, 20] by a large margin, obtaining the lowest FID score and the highest human evaluation score, demonstrating that PASTA-GAN can generate more photo-realistic images. Note that, although ADGAN [23] is trained on the DeepFashion dataset, our PASTA-GAN still surpasses it. Since the data in the DeepFashion dataset is more complicated than the data in UPT, the FID scores for the DeepFashion dataset are generally higher than the FID scores for the UPT dataset.
Qualitative: As shown in Fig. 6, under the unpaired setting, PASTA-GAN is capable of generating more realistic and accurate try-on results. On the one hand, paired methods [35, 37, 10] tend to fail in deforming the cropped garment to the target shape, resulting in the distorted warped garment that is largely misaligned with the target body part. On the other hand, unpaired method ADGAN [23] cannot preserve the garment texture and the person identity well due to its severe overfitting on the DeepFashion dataset. Liquid Warping GAN [20], another publicly available appearance transfer model, heavily relies on the 3D body model named SMPL [22] to obtain the appearance transfer flow. It is sensitive to the prediction accuracy of SMPL parameters, and thus prone to incorrectly transfer the appearance from other body parts (e.g., hand, lower body) into the garment region in case of inaccurate SMPL predictions. In comparison, benefited by the patch-routed mechanism, PASTA-GAN can learn appropriate garment features and predict precise garment shape. Further, the spatially-adaptive residual module can leverage the warped garment feature to guide the network to synthesize try-on results with realistic garment textures. Note that, in the top-left example of Fig. 6, our PASTA-GAN seems to smooth out the belt region. The reason for this is a parsing error. Specifically, the human parsing model [18] that was used does not designate a label for the belt, and the parsing estimator [11] will therefore assign a label for the belt region (i.e. pants, upper clothes, background, etc). For this particular example, the parsing label for the belt region is assigned the background label. This means that the pants obtained according to the predicted human parsing will
Table 2: The FID score [27] and human evaluation score among different methods under their corresponding test setting on the MPV dataset [6].
Method CP-VTON [35] ACGPN [37] PFAFN [10] PASTA-GAN(Ours)
FID ↓ 37.72 23.20 17.40 16.48 Human Evaluation ↑ 8.071% 12.64% 28.71% 50.57%
Person In-shop
Garment Garment CP-VTON ACGPN PFAFN
PASTA-
GAN Person
In-shop Garment Garment CP-VTON ACGPN PFAFN PASTA-
GAN
Figure 7: Visual comparison among PASTA-GAN and the paired baseline methods under their corresponding test setting on the MPV dataset [6]. Please zoom in for more details.
not contain the belt, which will therefore not be contained in the normalized patches and the warped pants. The style synthesis branch then predicts the precise mask for the pants (including the belt region) and the texture synthesis branch inpaints the belt region with the white color according to the features of the pants.

heading: 4.2 Comparison with the state-of-the-art methods on paired benchmark
text: Quantitative: Tab. 2 illustrates the quantitative comparison on the MPV dataset [6], in which the paired methods are tested under the classical paired setting, i.e., transferring an in-shop garment onto a reference person. Our unpaired PASTA-GAN, nevertheless, can surpass the paired methods especially the state-of-the-art PFAFN [10] in both FID and human evaluation score, further evidencing the superiority of our PASTA-GAN.
Qualitative: Under the paired setting, the visual quality of the paired methods improves considerably, as shown in Fig. 7. The paired methods depend on TPS-based or flow-based warping architectures to deform the whole garment, which may lead to the distortion of texture and shape since the global interpolation or pixel-level correspondence is error-prone in case of large pose variation. Our PASTAGAN, instead, warps semantic garment patches separately to alleviate the distortion and preserve the original garment texture to a larger extent. Besides, the paired methods are unable to handle garments like sling that are rarely presented in the dataset, and perform poorly on full-body images. Our PASTA-GAN instead generates compelling results even in these challenging scenarios.

heading: 4.3 Ablation Studies
text: Patch-routed Disentanglement Module: To validate its effectiveness, we train two PASTA-GANs without texture synthesis branch, denoted as PASTA-GAN? and PASTA-GAN∗, which take the intact garment and the garment patches as input of the style encoder, respectively. As shown in Fig. 8, PASTA-GAN? fails to generate accurate garment shape. In contrast, the PASTA-GAN∗ which factors out spatial information of the garment, can focus more on the garment style information, leading to the accurate synthesis of the garment shape. However, without the texture synthesis branch, both of them are unable to synthesize the detailed garment texture. The models with the texture synthesis branch can preserve the garment texture well as illustrated in Fig 8.
Spatially-adaptive Residual Module To validate the effectiveness of this module, we further train two PASTA-GANs with texture synthesis branch, denoted as PASTA-GAN† and PASTA-GAN‡, which excludes the style synthesis branch and replaces the spatially-adaptive residual blocks with normal residual blocks, respectively. Without the support of the corresponding components, both
PASTA-GAN† and PASTA-GAN‡ fail to fix the garment misalignment problem, leading to artifacts outside the target shape and blurred texture synthesis results. The full PASTA-GAN instead can generate try-on results with precise garment shape and texture details. The quantitative comparison results in Fig. 8 further validate the effectiveness of our designed modules.

heading: 5 Conclusion
text: We propose the PAtch-routed SpaTially-Adaptive GAN (PASTA-GAN) towards facilitating scalable unpaired virtual try-on. By utilizing the novel patch-routed disentanglement module and the spatiallyadaptive residual module, PASTA-GAN effectively disentangles garment style and spatial information and generates realistic and accurate virtual-try on results without requiring auxiliary data or extensive online optimization procedures. Experiments highlight PASTA-GAN’s ability to handle a large variety of garments, outperforming previous methods both in the paired and the unpaired setting.
We believe that this work will inspire new scalable approaches, facilitating the use of the large amount of available unlabeled data. However, as with most generative applications, misuse of these techniques is possible in the form of image forgeries, i.e. warping of unwanted garments with malicious intent.
Acknowledgments and Disclosure of Funding
We would like to thank all the reviewers for their constructive comments. Our work was supported in part by National Key R&D Program of China under Grant No. 2018AAA0100300, National Natural Science Foundation of China (NSFC) under Grant No.U19A2073 and No.61976233, Guangdong Province Basic and Applied Basic Research (Regional Joint Fund-Key) Grant No.2019B1515120039, Guangdong Outstanding Youth Fund (Grant No. 2021B1515020061), Shenzhen Fundamental Research Program (Project No. RCYX20200714114642083, No. JCYJ20190807154211365), CSIG Youth Fund.

Review summaries:
This paper presents a human garment try-on pipeline. The key point of the proposed approach is to normalize each body part into a regular patch according to the body skeleton, and use this patch representation to extract the style and disentangle the spatial variation due to different body poses. The network is based on StyleGAN architecture, and adapted to accept style and pose conditions. Furthermore, to overcome the loss of the texture details, it also proposes to use the first-stage network to generate the clothed mask, and use a refining network to synthesize/inpaint the texture details based on the deformed patches aligned to the target skeleton.

This paper presents a GAN model for virtual try-on trained from unpaired image data. The main network is based on StyleGAN2. During the training stage, for each image, the extracted 2D skeleton pose is used to convert the parsed garment region into a set of rectified texture patches with the proposed Patch-routed disentanglement Module. These rectified patches are encoded to be a single vector fed into every stage of the generator. The head region and the 2D skeleton pose are encoded to be identity features as input to the generator. Because the detailed texture information is not well preserved in style patches, the generator first synthesize coarse-level image and garment mask. The garment region is further fused in later stage of the generator to synthesize the final output. The model is trained in a reconstruction way without using paired person images. During the inference stage, the garment region is warped by the target pose as input into the texture synthesis stage of the generator. The key idea is its patch-based garment disentangling. The part-wise patch rectification removes pose and shape related information but only keeps low-level texture information. Results are reported on DeepFashion and MPV and self-collected UPT datasets with comparisons to state-of-the-art methods. 

This paper focuses on virtual try-on based on unpaired images. The authors propose to disentangle the style and spatial information of each garment, and generate it under the condition of homography transformed normalized patches. The authors also introduce a new unpaired dataset. Compared with the paired and unpaired state-of-the-art methods, both quantitative and qualitative results prove the effectiveness of the proposed approach.

The paper creates realistic virtual try-on results by modifying the StyleGAN2 architecture and using the garment appearance as the style modulation. In particular, to preserve the details of the target garment, the garment patches are warped and used as reference for the output image. The proposed method significantly outperforms the existing baselines thanks to the photorealism brought by preserving the input patches. The improved quality is backed by FID, user study, and analysis via ablations.