Towards Scalable Unpaired Virtual Try-On via Patch-Routed Spatially-Adaptive GAN
Authors: Zhenyu Xie
Zaiyu Huang
Fuwei Zhao
Haoye Dong
Michael Kampffmeyer
Xiaodan Liang
Venue: NeurIPS 2021 Poster (2021)
Keywords: image manipulation
unpaired virtual try-on
patch-routed warping

Abstract: Image-based virtual try-on is one of the most promising applications of human-centric image generation due to its tremendous real-world potential. Yet, as most try-on approaches fit in-shop garments onto a target person, they require the laborious and restrictive construction of a paired training dataset, severely limiting their scalability. While a few recent works attempt to transfer garments directly from one person to another, alleviating the need to collect paired datasets, their performance is impacted by the lack of paired (supervised) information.  In particular, disentangling style and spatial information of the garment becomes a challenge, which existing methods either address by requiring auxiliary data or extensive online optimization procedures, thereby still inhibiting their scalability. To achieve a scalable virtual try-on system that can transfer arbitrary garments between a source and a target person in an unsupervised manner, we thus propose a texture-preserving end-to-end network, the PAtch-routed SpaTially-Adaptive GAN (PASTA-GAN), that facilitates real-world unpaired virtual try-on. Specifically, to disentangle the style and spatial information of each garment, PASTA-GAN consists of an innovative patch-routed disentanglement module for successfully retaining garment texture and shape characteristics.  Guided by the source person's keypoints, the patch-routed disentanglement module first decouples garments into normalized patches, thus eliminating the inherent spatial information of the garment, and then reconstructs the normalized patches to the warped garment complying with the target person pose. Given the warped garment, PASTA-GAN further introduces novel spatially-adaptive residual blocks that guide the generator to synthesize more realistic garment details. Extensive comparisons with paired and unpaired approaches demonstrate the superiority of PASTA-GAN, highlighting its ability to generate high-quality try-on images when faced with a large variety of garments(e.g. vests, shirts, pants), taking a crucial step towards real-world scalable try-on.

Towards Scalable Unpaired Virtual Try-On via Patch-Routed Spatially-Adaptive GAN
Zhenyu Xie Zaiyu Huang', Fuwei Zhao', Haoye Dong Michael Kampffmeyer', Xiaodan Liang!.4*
Shenzhen Campus of Sun Yat-Sen University, HHuya Inc
3 UIT The Arctic University of Norway, *Peng Cheng Laboratory
fxiezhy6, hww.guy225,s... 2hwoof)aaaaa222222 aysu.edu. donghaoye@huya.comm mi michael.c kwwpffueyerruit...mmmmm no, xdliang3280maiil.com
Abstract
Image-based virtual try-on is of the most promising applications of human- centric image generation due to its tremendous real- real-world potential. Yet, as most try-on approaches fit in-shop garments onto target person, they require the labori- ous and restrictive construction of a paired training dataset, severely limiting their scalability. While few recent works attempt transfer garments directly from person to another, alleviating the need to collect paired datasets, their performance is impacted by the lack of paired (supervised) information. In particular, disen- tangling style and spatial information of the garment becomes challenge, which existing methods either address by requiring auxiliary data or extensive online opti- mization procedures, thereby still inhibiting their scalability. To achieve a scalable virtual try-on system that can transfer arbitrary garments between a source and a target person in an unsupervised manner, we thus propose texture- -reuurrrrreeeving end-to-end network, the PAtch-routed OpaTially-Adapleee GAN SEASTA-GAN that facilitates real-world unpaired virtual try-on. Specifically, to disentangle the style and spatial information of each garment, PASTA-GAN consists an inno- vative patch-routed disentanglement module for successfully retaining garment texture and shape characteristics. Guided by the source person keypoints, the patch-routed disentanglement module first decouples garments into normalized patches, thus eliminating the inherent spatial information of the garment, and then reconstructs the normalized patches the warped garment complying with the target person pose. Given the warped garment, PASTA-GAN further introduces novel spatially- residual blocks that guide the generator to synthesize more realistic garment details. Extensive comparisons with paired and unpaired approaches demonstrate the superiority of PASTA-GAS generate high-quality try-on images when faced with a large variety of garments (e.g. vests, shirts, pants), taking crucial step towards real-world scalable try-on.
highlighting its ability
1 Introduction
Image-basee based virtual try-on, the process computationally transferring a garment onto particular person in query image, is one of the most promising applications of human-centrie image generation with the potential to revolutionize shopping experiences and reduce purchase returns. However, to fully exploit its potential, scalable solutions are required that can leverage easily accessible training data, handle arbitrary garments, and provide efficient inference results. Unfortunately, to date, most existing methods [35, 38, 12, 7, 37, 9, 10, 4, 36, 39] rely paired training data, i.c., person image
Xiaodan Liang is the corresponding author. Our code will be available at PASTA-GAN.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Person
Garment
Try-on
Person
Garment
Tiy-on
Person
Garment
Try-on
Upper- body Transfer
Lower- body Transfer
Full Body Transfer
Figure 1: Example virtual try-on results from our PASTA-GAN, which flexible for various try-on scenarios, e.g., garment transfer for the upper body, the lower body, and the full body.
and its corresponding in-shop garment, leading to laborious data-collection processes. Furthermore, these methods are unable to exchange garments directly between two person images, thus largely limiting their application scenarios and raising the need for unpaired solutions ensure scalability.
While unpaired solutions have recently started to emerge, performing virtual try-on in an unsupervised setting extremely challenging and tends to affect the visual quality of the try-on results. Specifically, without access to the paired data, these models are usually trained by reconstructing the same person image, which is prone to over-fitting, and thus underperform when handling garment transfer during testing. The performance discrepancy is mainly reflected in the garment synthesis results, in particular the shape and texture, which we argue is caused by the entanglement of the garment style and spatial representations in the synthesis network during the reconstruction process.
While this is not a problem for the traditional paired try-on approaches [35, 12, 37, 10], which avoid this problem and preserve the garment characteristics by utilizing a supervised warping network to obtain the warped garment in target shape, this is not possible in the unpaired setting due to the lack of warped ground truth. The few works that do attempt to achieve unpaired virtual try-on, therefore, choose to circumvent this problem by either relying disentanglement [23, 33, 32, 31, 1, 5], which again leads to a laborious data-collection process, or require extensive online optimization procedures [25, 17] obtain fine-grain details of the original garments, harming the inference efficiency. However, none of the existing unpaired try-on methods consider the problem of coupled style and spatial garment information directly, which is crucial to obtain accurate garment transfer results in the unpaired and unsupervised virtual try-on scenario.
person images in various poses for feature
In this paper, to tackle the essential challenges mentioned above, we propose novel PAtch-routed SpaTillyy-daptive GAN, named PASTA-GAN, a scalable solution to the unpaired try-on task. Our PASTA-GAN can precisely synthesize garment shape and style (see Fig. 1) by introducing patch-routed disentanglement module that decouples the garment style and spatial features, as well as novel ppatially-adaptee residual module to mitigate the problem of feature misalignment.
The innovation of our PASTA-GAT includes three aspects: First, by separating the garments into normalized patches with the inherent spatial information largely reduced, the patch-routed disen- tanglement module encourages the style encoder to learn spatial-agnostic garment features. These features enable the synthesis network to generate images with accurate garment style regardless of varying spatial garment information. Second, given the target human pose, the normalized patches can be casily reconstructed to the warped garment complying with the target shape, without requiring a warping network or a 3D human model. Finally, the apatially-adaptive residual module extracts the warped garment feature and adaptively inpaints the region that is misaligned with the target garment shape. Thereafter, the inpainted warped garment features are embedded into the intermediate layer of the synthesis network, guiding the network to generate try-on results with realistic garment texture.
We collect scalable UnPaired virtual Try-on (UPT) dataset and conduct extensive experiments on the UPT dataset and two existing try-on benchmark datasets (i.e., the DeepFashion [21] and the MPV [6] datasets). Experiment results demonstrate that our unsupervised PASTA-GAN outperforms both the previous unpaired and paired try-on approaches.

---

Style Syuthesis Branch
Texture Synthesis Branch
1,'
Mg
Dieenoineement Module
Patch rouned
Style Innodde
fa
Incoder
256x256 Syntheiis
Garment
Frenndr
Mappinz Network
60
axd Symthess Block
SxBl Syuthesis
128x128
Spenialyy
256x256
Synthesis
adaptive
Systhesis
Blocks
Block
Figure 2: Overview of the inference process. (a) Given the source and target images of person (Ts, It). extract the source garment G., the source pose J,, and the target pose J4- The three are then sent to the patch-routed disentanglement module to yield the normalized garment patches Pa and the warped garment Gr. (b) The modified conditional StyleGAN2 first collaboratively exploits the disentangled style code w, projected from Pn. and the person identity feature fia. encoded from target head and pose Je), to synthesize the coarse try-on result i, in the style synthesis branch along with the target garment mask Mg. It then leverages the warped garment feature fy in the texture synthesis branch to generate the final try-on result 4.
2 Related Work
Paired Virtual Try-on. Paired try-on methods [13, 35, 38, 12, 24, 37, 10, 36] aim to transfer in-shop garment onto reference person. Among them, VITON [13] for the first time integrates U-Net [29] based generation network with TPS [2] based deformation approach synthesize the try-on result. CP-VTON [35] improves this paradigm by replacing the time-consuming warping module with trainable geometric matching module. VTNFP [38] adopts human parsing guide the generation various body parts, while [24, 37, 39] introduce smooth constraint for the warping module to alleviate the excessive distortion in TPS warping. Besides the TPS-based warping strategy, [12, 36, 10] turn to the flow-based warping scheme which models the per-pixel deformation. Recently, VITON-HD [4] focuses high-resolution virtual try-on and proposes ALIAS normalization mechanism to resolve the garment misalignment. PF-AFN [10] improves the learning process by employing knowledge distillation, achieving state-of-the-ar results. However, all of these methods require paired training data and are incapable of exchanging garments between two person images.
Unpaired Virtual Try-on. Different from the above methods, some recent works [23, 33, 32, 31, 25, 17] eliminate the need for in-shop garment images and directly transfer garments between two person images. Among them, [23, 33, 32, 31, 5] leverage pose transfer the pretext task to learn disentangled pose and appearance features for human synthesis, but require images of the same person with different poses. In contrast, [25, 17) are more flexible and can be directly trained with unpaired person images. However, OVITON [25] requires online appearance optimization for each garment region during testing maintain texture detail of the original garment. VOGUE [17] needs to separately optimize the latent codes for each person image and the interpolate coefficient for the final try-on result during testing. Therefore, existing unpaired methods require either cumbersome data collecting or extensive online optimization, extremely harming their scalability in real scenarios.
3 PASTA-GAN
Given source image I, of person wearing garment G., and target person image I., the unpaired virtual try-on task aims to synthesize the try-on result 7 retaining the identity 7, but wearing the source garment Gg. To achieve this, our PASTA-GAN first utilizes the patch-routed disentanglement module (Sec. 3.1) to transform the garment G, into normalized patches P. that mostly agnostic to the spatial features of the garment, and further deforms P., to obtain the
2 As the concurrent work StylePoseGAN [31] is the most related pose transfer-based approach, we provide claborate discussion of the inherent differences the supplementary.

---

warped garment G₁ complying with the target person pose. Then, an attribute-decoupler conditional StyleGAN2 (Sec. 3.2) is designed synthesize try-on results in coarse-to-fine manner, where we introduce novel spatially-adaptive residual blocks (Sec. 3.3) inject the warped garment features into the generator network for more realistic texture synthesis. The loss functions and training details will be described in Sec. 3.4. Fig. 2 illustrates the overview of the inference process for PASTA-GAN.
3.1 Patch-routed Disentanglement Module
Since the paired data for supervised training is unavailable for the unpaired virtual try-on task, the synthesis network has to be trained an unsupervised manner via image reconstruction, and thus takes person image as input and separately extracts the feature of the intact garment and the feature of the person representation to reconstruct the original person image. While such a training strategy retains the intact garment information, which is helpful for the garment reconstruction, the features of the intact garment entangle the garment style with the spatial information in the original image. the garment transfer during testing. Note that the garment style here refers
This is detrimental to the garment color and categories, i.c., long sleeve, short sleeve, etc., while the garment spatial information implies the location, the orientation, and the relative size of the garment patch in the person image, in which the first two parts are influenced by the human pose while the third part is determined by the relative camera distance the person.
To address this issue, we explicitly divide the garment into normalized patches to remove the inherent spatial information of the garment. Taking the sleeve patch as example, by using division and normalization, various sleeve regions from different person images can be deformed to normalized patches with the same orientation and scale. Without the guidance of the spatial information, the network is forced to learn the garment style feature to reconstruct the garment in the synthesis image.
Fig. 3 illustrates the process of obtain- ing normalized garment patches, which in- cludes two main steps: (1) pose-guides garment segmentation, and (2) perspective transformation-basee patch normalization. Specifically, in the first step, the source gar- ment G, and human pose (joints) Jg are firstly obtained by applying [11] and [3] to
Ky
G,
the source person L,, respectively. Given the body joints, we can segment the source garment into several patches P., which can be quadrilaterals with arbitrary shapes (e.g.. rectangle, square, trapezoid, etc.). and will later be normalized, Taking the torso region as an example, with the coordinates of the left/right shoulder joints and the left/right hips joints in P quadrilateral crop (of which the four corner points are visualized in color in Pd of Fig. 3) covering the torso region of G, can be easily performed to produce an unnormalized garment patch. Note that we define eight patches for upper-body garments, i.e., the patches around the left/right upper/bottom arm, the patches around the left/right hips, patch around the torso, and patch around the neck. In the second step, all patches are normalized to remove their spatial information by perspective transformations. For this, we first define the same amount of template patches P, with fixed 64 x 64 resolution as transformation targets for all unnormalized source patches, and then compute a homography matrix Hism E R3xx3 [40] for each pair of Ps and Pa. based on the four corresponding corner points of the two patches. Concretely, Himn serves as perspective transformation to relate the pixel coordinates in the two patches, formulated as:
Figure 3: The process the patch-routed deformation. Please zoom in for more details.
x. y 1
hh
N12
14
yh 1
HC
h
y
hag
hgz
hgg
where (x),, v and (x), v are the pixel coordinates in the normalized template patch and the unnor- malized source patch, respectively. To compute the homography matrix H;... we directly leverage the OpenCV API, which takes inputs the corner points of the patches and is implemented by using least-squares optimization and the Wwrenber-MMaquarddh method [8]. After obtaining Hi Hi... we can transform the source patch Př to the normalized patch P according to Eq. 1.
Moreover, the normalized patches P., can further be transformed to target garment patches P. by utilizing the target pose Jr, which can be obtained from the target person L via [3]. The mechanism

---

of that backward transformation is equivalent to the forward one in Eq. 1, i... computing the homography matrix = based on the four point pairs extracted from the normalized patch Pá and the target pose J. The recovered target patches P. can then be stitched to form the warped garment G, that will be sent to the texture synthesis branch in Fig. to generate more realistic garment transfer results. We can also regard Ha-ي Hn Hy- as the combined deformation matrix that warps the source garment to the target person pose, bridged by an intermediate normalized patch representation that is helpful for disentangling garment styles and spatial features.
3.2 Attribute-decoupledd Conditional StyleGAN2
Motivated by the impressive performance of StyleGANZ [15] in the field of image synthesis, PASTA-GAN inherits the main architecture of StyleGAN2 and modifies it to the conditional version (see Fig. 2). In the synthesis network, the normalized patches Pn are projected to the style code through a style encoder followed by a mapping network, which is spatial-agnostic benefiting from the disentanglement module. In parallel, the conditional information including the target head He and pose Jr is transferred into feature map fid, encoding the identity of the target person by the identity encoder. Thereafter, the synthesis network starts from the identity feature map and leverages the style code as the injected vector for each synthesis block to generate the try-on result Iç'-
However, the standalone conditional StyleGAN2 is insufficient to generate compelling garment details especially in the presence of complex textures or logos. For example, although the illustrated
I in Fig. 2 can recover accurate garment style (color and shape) given the disentangled style code w, it lacks the complete texture pattern. The reasons for this are twofold: First, the style encoder projects the normalized patches into a one-dimensional vector, resulting in loss of high frequency information. Second, due the large variety of garment texture, learning the local distribution of the particular garment details is highly challenging for the basic synthesis network.
To generate more accurate garment details, instead of only having one-way synthesis network, we intentionally split PASTA-GAN into two branches after the 128 x 128 synthesis block, namely the Style Synthesis Branch (SSB) and the Texture Synthesis Branch (TSB). The SSB with normal StyleGAN2 synthesis blocks aims to generate intermediate try-on results T," with accurate garment style and predict precise garment mask Mg that will be used by TSB. The purpose of TSB is to exploit the warped garment G,, which has rich texture information to guide the synthesis path, and generate high-quality try-on results. We introduce novel spatially-adaptive residual module specifically before the final synthesis block of the TSB, to embed the warped garment feature fg (obtained by passing Mg and G, through the garment encoder) into the intermediate features and then send them to the newly designed spatialy-apaptive residual blocks, which are beneficial for successfully synthesizing texture of the final try-on result I/. The detail of this module will be described in the following section.
3.3 Spatially-adaptive Residual Module
Given the style code that factors out the spatial informa- tion and only keeps the style information of the garment, the style synthesis branch in Fig. 2 can accurately predict the mean color and the shape mask of the target garment. However, its inability to model the complex texture raises the need to exploit the warped garment G₃ to provide fea- tures that encode high-free freeueegyy texture patterns, which is in fact the motivation of the target garment reconstruction in Fig. 3.
Warped Garment
Tageet G Garment Mask
Misaligned Region
Figure 4: Illustration of misalignment between the warped garment and target garment shape. The orange and green re- gion represent the region to be inpainted and to be removed, respectively.
However, as the coarse warped garment G. is directly obtained by stitching the target patches together, its shape inaccurate and usually misaligns with the predicted mask Mg (see Fig.4). Such shape misalignment in G, will consequently reduce the quality of the extracted warped garment feature fg-
5

---

To address this issue, we introduce the centiall--daptiing residual module between the last two synthesis blocks in the texture synthesis branch shown in Fig. This module is comprised of garment encoder and three watially-adaptive residual blocks with feature inpainting mechanism to modulate intermediate features by leveraging the inpainted warped garment feature.
To be specific on the feature inpainting process, we first remove the part of G. that falls outside of M, (green region in 4)g.4), and explicitly inpaint the misaligned regions of the feature map within Ma with average feature values (orange region in Fig. 4). The inpainted feature map can then help the final synthesis block infer reasonable texture the inside misaligned parts.
Therefore given the predicted garment mask Mg. the coarse warped garment G2 and its mask Mr. the process of feature inpainting can be formulated as:
Malign = My n 2r,
(2)
M Mmisalign Mg Malign:
(3)
E Ey(G, My),
(4)
f- 3 Mmisalign) + A(f, @ Malign) Mmisalign.
(5)
where (.) represents the garment encoder and f' denotes the raw feature map of G, masked by Mg. A()) calculates the average garment features and fg is the final inpainted feature map.
Subsequently, inspired by the SPADE ResBlk from SPADE [26], the inpainted garment features are used calculate a set of affine transformation parameters that efficiently modulate the normalized feature map within each spatially-adaptiv residual block. The normalization and modulation process for a particular sample fg, at location (= E C,y E H E W) in feature map can then be formulated as:
Tz.y (Fg) kem - He + (fa)
(6)
where fı= HW Σg. Dz.g.a and = V HW Ey,e (hzyy. - =2)² are the mean and standard deviation the feature map along channel C. (..m.ol)) and (e are the convolution operations that convert the inpainted feature to affine parameters.
Eq. serves as a learnable normalization layer for the spatially-adage residual block better capture the statistical information of the garment feature map, thus helping the synthesis network to generate more realistic garment texture.
With the modulated intermediate feature maps produced by the spatially-adaptive residual module, the texture synthesis branch effectively utilize the reconstructed warped garment and generate the final compelling try-on result with high-frequency texture patterns.
3.4 Loss Functions and Training Details
As paired training data is unavailable, our PASTA-GAN is trained unsupervised via image recon- struction. During training, we utilize the reconstruction loss Cree and the perceptual loss [14] Cpere for both the coarse try-on result 7' and the final try-on result l':
Lrec
Σ 1.11 and Epere IE(P.P)
Σ [E(T,P) k=1
Σ de (l du (I.)I
where áv((I) denotes the k-th feature map in a VGG-19 network [34] pre-trained on the ImageNet [30] dataset. We also use the L, loss between the predicted garment mask M, and the real mask Mgr which is obtained via human parsing [11]:
Lmask = MMM Moxll1-
(8)
Besides, for both 7' and 1', we calculate the adversarial loss CCAN which the same in Style- GAN2 [15]. The total loss can be formulated as
L LGAN Arechrec Apere + AmaskLLmask
where Xrecs Apere. and Amask the trade-off syper-parametent

---

ls
Gs
Ge
Ge
1
Gs
G
Gr
Figure 5: Comparison among the source garment and different warped garments.
During training, although the source and target pose the same, the coarse warped garment G2 is not identical to the intact source garment Gs. due to the crop mechanism in the patch-routed disentan- glement module. More specifically, the quadrilateral crop for G, is by design not seamless/perfect and there will accordingly often exist some small seams between adjacent patches in G, as well as incompleteness along the boundary of the torso region. To further reduce the training-test gap of the warped garment, we introduce two random erasing operations during the training phase. First, we randomly remove one of the four arm patches the warped garment with a probability a11 Second, we use the random mask from [19] to additionally erase parts of the warped garment with probability of ag. Both of the erasing operations can imitate self-occlusion in the source person image. Fig. 5 illustrates the process by displaying the source garment G,, the warped garment G that obtained by directly stitching the warped patches together, and the warped garment G, that is sent to the network. We can observe considerable difference between G2 and G., An ablation experiment to validate the necessity of the randomly erasing operation for the unsupervised training is included in the supplementary material.
4 Experiments
Datasets. We conduct experiments two existing virtual try-on benchmark datasets (MPV [6] dataset and DeepFashion [21] dataset) and our newly collected large-scale benchmark dataset for unpaired try-on, named UPT. UPT contains 33,254 half- and full-body front-view images of persons wearing a large variety of garments, e.g., long/short sleeve, vest, sling, pants, etc. UPT is further split into training of 27,139 images and testing set of 6, 6,115 images. In addition,
also pick out the front view images from MPV [6] and DeepFashion [21] to expand the size of our training and testing to 54,714 and 10,493, respectively. Personally identifiable information (i.e. face information) has been masked out.
Metrics. We apply the Fréchet Inception Distance (FID) [27] to measure the similarity between real and synthesized images, and perform human evaluation to quantitatively evaluate the synthesis quality of different methods. For the human evaluation, design three questionnaires corresponding to the three used datasets. In each questionnaire, we randomly select 40 try-on results generated by PASTA-GAN and the other compared methods. Then, invite 30 volunteers to complete the 40 tasks by choosing the most realistic try-on results. Finally, the human evaluation score calculated as the chosen percentage for particular method.
Implementation Details. Our PASTA-GAN is implemented using PyTorch [28] and is trained on Tesla V100 GPUs During training. the batch size is set to 96 and the model is trained for 4 million iterations with a learning rate of 0.002 using the Adam optimizer [16] with 31 = and 32 = 0.99. to 40, 40, and 100, respectively. The hhyper--prameters for the random erasing probability ai and a2 set to 0.2 and 0.9, respectively.
The loss syyper-parameeee
Avecs Apere and Amask
Baselines. To validate the effectiveness of our PASTA-GAN, we compare it with the state-of-the-art methods, including three paired virtual try-on methods, CP-VTON [35], ACGPN [37]. PFAFN [10], and two unpaired methods Liquid Warping GAN [20] and ADGAN [23], which have released the official code and pre- pre-trained weights. We directly use the pre-trained model of these methods as their training procedure depends on the paired data of garment- pennt-personnno or person-person image pairs, which are unavailable in our dataset. When testing paired methods under the unpaired try-on setting, we extract the desired garment from the person image and regard the in-shop garment meet the
Additional details for the UPT dataset (e.g., data distribution, data pre- processing). the human evaluation. training details, and the inference time analysis. etc. are provided in supplementary material. For all these prior approaches, research use is permitted according to the respective licenses. Note, we are unable to compare with [31]. [17] and [25] as they have not released their code pre-trained model.
7

---

Table 1:
The FID score [27] and human evaluation score among different methods under the unpaired setting on the DeepPashion dataset [21] and our UPT dataset.
DcepFashion
UPT
Method
FID 4
Human Evaluation 1
FID +
Human Evaluation 1
CP-VTON [35]
69.46
2.177%
70.76
1.551 %
ACGPN [37]
44,41
4.597%
37.99
3.448%
PFAFN [10]
46.19
4.677%
36.69
4.224%
ADGAN [23]
37.36
21.29%
39.60
7.241 %
Liquid Warping GAN [20]
42.18
12.98%
33.18
9.310%
PASTA-GAN (Ours)
21.58
54.27% %
7.852
74.22%
Liquidd Warping GAN
Liiuid
Person
Gameent CP-YTON ACGPN
ADCAN
BASIA-
Person Gament CP-VTON ACGPN
PRAEN ADGAN
Warzing GAN
GAN
4
4
A4
Figure 6: Visual comparison among PASTA-GAN and the baseline methods under the unpaired setting on the UPT dataset. Please zoom in for more details.
need of paired approaches. To fairly compare with the paired methods, we further conduct another experiment on the paired MPV dataset [6]. in which the paired methods take an in-shop garment and a person image as inputs, while our PASTA-GAN still directly receives two person images. See the following two subsections for detailed comparisons on both paired and unpaired settings.
4.1 Comparison with the state-of-the-ar methods on unpaired benchmark
Quantitative: As reported in Table 1, when testing on the DeepFashion [21] and the UPT dataset under the unpaired setting, our PASTA-GAN outperforms both the paired methods [35, (3,,37, 10] and the unpaired methods [23, 20] by a large margin. obtaining the lowest FID score and the highest human evaluation score, demonstrating that PASTA-GAN can generate more photo-realistic images. Note that, although ADGAN [23] is trained on the DeepFashion dataset, our PASTA-GAN still surpasses it. Since the data the DeepFashion dataset more complicated than the data in UPT, the FID scores for the DeepFashion dataset are generally higher than the FID scores for the UPT dataset.
Qualitative: As shown in Fig. 6, under the unpaired setting. PASTA-GAN is capable of generating more realistic and accurate try-on results. On the one hand, paired methods [35, 37, 10] tend to fail in deforming the cropped garment to the target shape, resulting in the distorted warped garment that is largely misaligned with the target body part. On the other hand, unpaired method ADGAN [23] cannot preserve the garment texture and the person identity well due its severe overfitting on the DeepFashion dataset. Liquid Warping GAN [20], another publicly available appearance transfer model, heavily relies the 3D body model named SMPL [22] obtain the appearance transfer flow. It is sensitive to the prediction accuracy of SMPI. parameters, and thus prone to incorrectly transfer the appearance from other body parts (c.g.. hand, lower body) into the garment region in case of inaccurate SMPL predictions. In comparison, benefited by the patch-routed mechanism, PASTA-GAN can learn appropriate garment features and predict precise garment shape. Further, the spatially- residual module can leverage the warped garment feature to guide the network to synthesize try-on results with realistic garment textures. Note that, in the top-left example of Fig. 6, our PASTA-GAN seems to smooth out the belt region. The reason for this is a parsing error. Specifically, the human parsing model [18] that was used does not designate label for the belt, and the parsing estimator [11] will therefore assign a label for the belt region (i.e. pants, upper clothes, background, etc). For this particular example, the parsing label for the belt region is assigned the background label. This means that the pants obtained according to the predicted human parsing will
8

---

Table 2: The FID score [27] and human evaluation score among different methods under their
corresponding test setting on the MPV dataset [6].
Method
CP-N [35]
ACGPN [37]
PFAFN [10]
MASTA-GAN(OUU)
FID +
37.72
23.20
17.40
16.48
Human Evaluation f
8.071%
12.64%
28.71%
50.57%
Person
In-shop Carment
Garment CP-VTON ACGPN
PEAFN
PASTA GAN
Person
In-shop
Gament CP-VTON ACGPN
PRAFN
PASTA- GAN
&
Figure 7: Visual comparison among PASTA-GAN and the paired baseline methods under their corresponding test setting on the MPV dataset [6]. Please zoom in for more details.
not contain the belt, which will therefore not be contained the normalized patches and the warped pants. The style synthesis branch then predicts the precise mask for the pants (including the belt region) and the texture synthesis branch inpaints the belt region with the white color according the features of the pants.
4.2 Comparison with the state-of-the-art methods on paired benchmark
Quantitative: Tab. 2 illustrates the quantitative comparison on the MPV dataset [6], in which the paired methods are tested under the classical paired setting, i.e., transferring an in-shop garment onto a reference person. Our unpaired PASTA-GAN, nevertheless, can surpass the paired methods especially the state-of-the-art PFAFN [10] in both FID and human evaluation score, further evidencing the superiority of our PASTA-GAN.
Qualitative: Under the paired setting, the visual quality of the paired methods improves considerably, as shown in Fig. 7. The paired methods depend on TPS-based or flow-based warping architectures to deform the whole garment, which may lead to the distortion of texture and shape since the global interpolation or pixel-level correspondence error- prone in case of large pose variation. Our PASTA- GAN, instead, warps semantic garment patches separately alleviate the distortion and preserve the original garment texture to larger extent. Besides, the paired methods are unable to handle garments like sling that rarely presented in the dataset, and perform poorly full-body images. Our PASTA-GAN instead generates compelling results even in these challenging scenarios.
4.3 Ablation Studies
Patch-routed Disentanglement Module: To validate its effectiveness, we train two PASTA-GANs without texture synthesis branch, denoted as PASTA-GAN and PASTA-GANs, which take the intact garment and the garment patches as input of the style encoder, respectively, As shown in Fig. 8, BASTA-GAN+ fails generate accurate garment shape. In contrast, the PASTA-GAN= which factors out spatial information of the garment, can focus more on the garment style information, leading to the accurate synthesis of the garment shape. However, without the texture synthesis branch, both of them are unable to synthesize the detailed garment texture. The models with the texture synthesis branch can preserve the garment texture well illustrated in Fig 8.
Spatially-adaptive Residual Module To validate the effectiveness of this module, we further train two PASTA- with texture synthesis branch, denoted as PASTA-GANÍ and PASTA-GAN which excludes the style synthesis branch and replaces the apatially-adapel residual blocks with normal residual blocks, respectively. Without the support of the corresponding components, both

---

BASIA- D
PASTA- GAN=
PASTA- GANi
BASTA- GAN
Person
Garment
GAN:
Method
SSB
TSB
GP
NRB
SRB
PASTA GAN+
21.99
PASTA-GAN:
/
/
X
X
17.69
PASTA-GANI PASTA-GAN!
X
V
X
V
11.27
/
10.08 x
PASTA-GAN
x
/
7.851
Figure 8: Qualitative results and quantitative results of the ablation study with different configurations, in which SSB, TSB, GP, NRB. SRB refer to style synthesis branch, texture synthesis branch, garment patches, normal residual blocks, and spatially-adaje adaptive residual blocks, respectively.
PASTA-GANi and PASTA-GAN; fail to fix the garment misalignment problem, leading to artifacts outside the target shape and blurred texture synthesis results. The full PASTA-GAN instead can generate try-on results with precise garment shape and texture details. The quantitative comparison results in Fig. further validate the effectiveness of designed modules.
Conclusion
We propose the PAtch-routed SpalTially-Adaptive GAN (PASTA-GAN towards facilitating scalable unpaired virtual try-on. By utilizing the novel patch-routed disentanglement module and the spatially- adaptive residual module, PASTA-GAN effectively disentangles garment style and spatial information and generates realistic and accurate virtual-try on results without requiring auxiliary data extensive online optimization procedures. Experiments highlight PASTA-GAN's ability to handle a large variety of garments, outperforming previous methods both the paired and the unpaired setting.
We believe that this work will inspire new scalable approaches, facilitating the use the large amount of available unlabeled data. However, with most generative applications, misuse of these techniques possible in the form of image forgeries, i.e. warping of unwanted garments with malicious intent.
Acknowledgments and Disclosure of Funding
We would like thank all the reviewers for their constructive comments. Our work supported in part by National Key R&D Program of China under Grant No. 1018AAA0100300 National Natural Science Foundation of China (NSFC) under Grant No. U19A2073 and No.61 6o.61976233, Guangdong Province Basic and Applied Basic Research (Regional Joint Fund- Key) Grant No.201991551120055 Guangdong Outstanding Youth Fund (Grant No. 2021B151502006 JCC12000099...1 JCY/2019080715221 1365), CSIG
Shenzhen Fundamental
Research Program (Project No. RCYX202007141 14442000,,,,,,,,,,, No.
Youth Fund.
References
[1] Badour AlBahar, Jingwan Lu, Jimei Yang, Zhixin Shu, Eli Shechtman, and Jia-Bin Huang. Pose with Style: Detail-preserving pose-get pose-guided image synthesis with conditional stylegan. ACM Transactions Graphics, 2021.
[2] F.I.. Bookstein. Principal warps: thin-plate splines and the decomposition of deformations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1166):677-585 1989.
[3] Z. Cao, G. Hidalgo, T. Simon, S. Wei, and Y. Sheikh, Openpose: Realtime multi-person 2d pose estimation using part affinity fields, IEEE Transactions on Pattern Analysis and Machine Intelligence, 13(01): 2021.
[4] Seunghwan Choi, Sunghyun Park, Minsoo Lee, and Jaegul Choo. Viton-hd: High-resolution vir- tual try-on via aaaa normalization. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 14131-14140, 2021.
10

---

[5] Aiyu Cui, Daniel McKee, and Svetlana Lazebnik. Dressing in order: Recurrent person image generation for pose transfer, virtual try-on and outfit editing, In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 14638-14647, 2021.
[6] Haoye Dong, Xiaodan Liang, Xiaohui Shen, Bochao Wang, Hanjiang Lai, Jia Zhu, Zhiting Hu, and Jian Yin. Towards multi-pose guided virtual try-on network. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9026-9035 2019.
[7] Haoye Dong, Xiaodan Liang, Xiaohui Shen, Bowen Wu, Bing-Cheng Chen, and Jian Yin. Fw-gan: Flow-navigated warping gan for video virtual try-on. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1161-1170, 2019.
[8] Henri P. Gavin. The eevenberg-maqquare algorithm for nonlinear least squares curve-fitting problems. 2013.
[9] Chongjian Ge. Yibing Song, Yuying Ge, Han Yang, Wei Liu, and Ping Luo. Disentangled cycle consistency for highly- virtual try-on. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16928-16937, 2021.
[10] Yuying Ge, Yibing Song, Ruimao Zhang, Chongjian Ge, Wei Liu, and Ping Luo. Parser-free virtual tny-on via distilling appearance flows. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8485-8493, 2021.
[11] Ke Gong, Yiming Gao, Xiaodan Liang, Xiaohui Shen, Meng Wang, and Liang Lin. Graphon- omy: Universal human parsing via graph transfer learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7450-7459, 2019.
[12] Xintong Han, Xiaojun Hu, Weilin Huang, and Matthew R. Scott. Clothflow: A flow-based model for clothed person generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10471-10480, 2019.
[13] Xintong Han, Zuxuan Wu, Zhe Wu, Ruichi Yu, and Larry S Davis. Viton: An image-based virtual try-on network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 7543-7552, 2018.
[14] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision (ECCV), pages 694-711, 2016.
[15] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 81 8110-8119, 2020.
[16] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
[17) Kathleen M Lewis, Srivatsan Varadharajan, and Ira Kemellmacher-Shhinemmnn Vogue: Try-on by stylegan interpolation optimization. In arXiv preprint arXiv: 101.02288, 2021.
[18] Xiaodan Liang, Ke Gong, Xiaohui Shen, and Liang Lin. Look into person: Joint body parsing pose estimation network and a new benchmark. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(4):871 1008-1110 885, 2018.
[19] Guilin Liu, Fitsum A Reda, Kevin J Shih, Ting-Ch Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro. Image inpainting for irregular holes using partial convolutions. In European Conference on Computer Vision (ECCV), page 85-100, 2018
[20] Wen Liu, Zhixin Piao, Jie Min, Wenhan Luo, Lin Ma, and Shenghua Gao. Liquid warping gan: A unified framework for human motion imitation, appearance transfer and novel view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1904-5913 2019.
[21] Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 1096-1104, 2016.
11

---

[22] Matthew Loper. Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. Smpl: a skinned multi-person linear model. ACM Transactions on Graphics, 34(6). 2015.
[23] Yifang Men, Yiming Mao, Yuning Jiang, Wei- Ying Ma. and Zhouhui Lian. Controllable person image synthesis with attribute -ebbbuue-dcooposedd gan. In Proceedings of the /EEE/CVFConference on Computer Vision and Pattern Recognition (CVPR), pages 5084-5093, 2020.
[24] Matiur Rahman Minar, Thai Thanh Tuan, Heejune Ahn, Paul Rosin, and Yu- Kun Lai. Cp- vton+: Clothing shape and texture preserving image-based virtual try-on. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2020.
[25] Assaf Neuberger, Eran Borenstein, Bar Hilleli, Eduard Oks, and Sharon Alpert. Image based virtual try-on network from unpaired data. In Proceedings of the IEEE/CVFConference Computer Vision and Pattern Recognition (CVPR), pages 5184-5193, 2020.
[26] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. In Proceedings of the IBLE/CVFConference on Computer Vision and Pattern Recognition (CVPR),, pages 2337-2346, 2019.
[27] Gaurav Parmar, Richard Zhang, and Jun- Yan Zhu. On buggy resizing libraries and surprising subtleties fid calculation. In arXiv preprint arXiv:2104. 1222, 2021.
[28] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zach Vetoto Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, 2019.
[29] Olaf Ronneherger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Inter- vention (MICCAI), pages 234-241, 2015.
[30] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115:)) 2015.
[31] Kripasindhu Sarkar, Vladislav Golyaník, Lingjie Liu, and Christian Theobalt. Style and pose control for image synthesis of humans from single monocular view. In arXiv preprint arXiv:2102.1 / 1263, 2021.
[32] Kripasindhu Sarkar, Lingjie Liu, Vladislav Golyanik, and Christian Theobalt. Humangan: A generative model of humans images. In arXiv preprint arXiv:2103.06900 2021.
[33] Kripasindhu Sarkar, Dushyant Mehta, Weipeng Xu, Vladislav Golyanik, and Christian Theobalt. Neural re-rendering of humans from single image. In European Conference onComputer Vision (ECCV), pages 596-613, 2020.
[34] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun, editors, International Conference on Learning Representations (ICLR), 2015.
[35] Bochao Wang, Huabin Zheng, Xiaodan Liang, Yimin Chen, Liang Lin, and Meng Yang. Toward characteristie-prssevvee image-based virtual try-on network. In European Conference onComputerVision (ECCV), pages 589-604, 2018.
[36] Zhenyu Xie, Xujie Zhang, Fuwei Zhan, Haoye Dong, Michael C. Kampffmeyer, Haonan Yan, and Xiaodan Liang. Was-vton: Warping architecture search for virtual try-on network. In Proceedings of the 29th ACM International Conference on Multimedia (ACM MM), page
3350-3359, 2021.
12

---

[37] Han Yang. Ruimao Zhang, Xiaobao Guo, Wei Liu, Wangmeng Zuo, and Ping Luo. Towards photo-realistic virtual try-on by adaptively generating preserving image content. Proceedings of the IEF#/CVFCCnference on ComputerVisse Vision and Pattern Recognition (CVPR), pages 7850- 7859, 2020).
[38] Ruiyun Yu, Xiaoqi Wang, and Xiaohui Xie. Vuip:: An image-bare based virtual try-on network with body and clothing feature preservation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10511-10520, 2019.
[39] Fuwei Zhao, Zhenyu Xie, Michael Kampffmeyer, Haoye Dong, Songfang Han, Tianxiang Zheng, Tao Zhang, and Xiaodan Liang. M3d-vton: A monocular-to-3d virtual try-on network. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 13239-13249, October 2021.
[40] Qiang Zhou and Xin Li. Stn- hunnhomggraphy: estimate homography parameters directly. In arXiv preprint arXiv: 02539, 2019.
13


Reviews:
[{"review_id": "eYwe2R0aGIN", "reviewer": "Reviewer_4awU", "summary": "The paper creates realistic virtual try-on results by modifying the StyleGAN2 architecture and using the garment appearance as the style modulation. In particular, to preserve the details of the target garment, the garment patches are warped and used as reference for the output image. The proposed method significantly outperforms the existing baselines thanks to the photorealism brought by preserving the input patches. The improved quality is backed by FID, user study, and analysis via ablations. ", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I am generally satisfied with the clarity of writing, several insights and conceptual contributions, practicality of the application, and evaluation. \n\nWhile the synthesis models based on neural networks produce nice results, it is very challenging to use it for retaining the precise details and patterns of the input image in the process of image manipulation, such as logos on t-shirts. For example, many GAN inversion works (Image2StyleGAN Abdal et al, 2020, StyleGAN encoder Richardson et al., CVPR 2021) can't embed complex patterns on hats (see Figure 11 of https://arxiv.org/pdf/2008.00951.pdf). The proposed method nicely handles this by first warping the garment to retain the details, and then using deep networks to refine the final output. I think this combination of non-parametric sampling and parametric approach will inspire future works in image manipulation using deep networks. \n\nThe writing is clear, and the authors comprehensively listed the details of the method, including the precise network architecture, insights (L175-181), details about implementing baselines (L271-281), failure cases and ethical concern. \n\nThe evaluation clearly supports the improved quality of the proposed method, both qualitatively and quantitatively. The FID is significantly better, likely because the input patches are recycled in the output. \n\nHaving said that, here are some weaknesses and suggestions.\n\nResolution is 256px. Can it be higher?\n\nThe quality of the final output depends on the quality of the warped garment. Therefore, when the pose change is too drastic to find a nice warp, the quality of the output will degrade as in Figure 3 of Supp Mat. Likely because of this reason, most results of the submitted paper are frontal-facing pose. Still, I think this limitation is not important, as it is reasonable to assume that the target garment will contain at least one frontal facing picture in the virtual try-on task. \n\nThe human evaluation protocol could be performed at larger scale, because 30 volunteers may not represent unbiased population. \n\nThe labeling of ablated models was too difficult to follow. Could you use more intuitive names? Such as “-TSB, intact-garment”.  \n\nIt dismisses a line of work [19, 27, 26, 25] because their methods require images of the same person in different poses. However, this may not be a harsh requirement on the fashion dataset, as these types of images can be readily found on the Internet. In particular, the authors already evaluate their method on the DeepFasion dataset, and StylePoseGAN[25] shows nice results on this dataset. Ideally, some side-by-side comparison to the SOTA methods such as [25] would be nice, while clearly stating that the training requires different amount of supervision. This is merely a suggestion for future revisions, since [25] is still only on arxiv. \n\nIn particular, it does share some similarity to StylePoseGAN[25]. Both inherit StyleGAN2 and modify it to a conditional version. The garment style is used as style modulation. The garment style information is normalized to some canonical pose in both works. Considering StylePoseGAN is a recent work and still only on Arxiv, similarity to StylePoseGAN does not affect my rating, but it would be nice to further discuss similarities and differences, because the authors already cited the work. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discussed both positive and negative implications of the work in Conclusion. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "LPqeUu2GH88", "reviewer": "Reviewer_BGd4", "summary": "This paper presents a GAN model for virtual try-on trained from unpaired image data. The main network is based on StyleGAN2. During the training stage, for each image, the extracted 2D skeleton pose is used to convert the parsed garment region into a set of rectified texture patches with the proposed Patch-routed disentanglement Module. These rectified patches are encoded to be a single vector fed into every stage of the generator. The head region and the 2D skeleton pose are encoded to be identity features as input to the generator. Because the detailed texture information is not well preserved in style patches, the generator first synthesize coarse-level image and garment mask. The garment region is further fused in later stage of the generator to synthesize the final output. The model is trained in a reconstruction way without using paired person images. During the inference stage, the garment region is warped by the target pose as input into the texture synthesis stage of the generator. The key idea is its patch-based garment disentangling. The part-wise patch rectification removes pose and shape related information but only keeps low-level texture information. Results are reported on DeepFashion and MPV and self-collected UPT datasets with comparisons to state-of-the-art methods. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1) It is an interesting idea of garment disentangling by rectifying the corresponding garment region on each skeleton bone into a standard square patch. The rectification process makes sure the resulting patches are normalized in terms of pose and shape. It is also the key to enable unpaired learning.\n2) The StyleGAN2 based generator is well constructed. Having a separate texture synthesis branch with warped garment as input makes sure the generator can synthesis high-quality texture with the aid of spatially-adaptive residual module. \n3) It presents impressive visual results. Its high-quality synthesis is also verified with significant improvements in FID scores and user preferences compared to the related works. \n\nCons:\n1) The proposed patch-routed disentanglement module is based on 2D skeleton pose, which may be only feasible to normalize/rectify garment w.r.t. in-plane pose changes. The presented visual results are all about near frontal views, which seem to agree with this observation. The visual results sometimes look 3D implausible, especially when large patterns/logos/designs are present in the garment. In a recent work by Kripasindhu Sarkar et al. (StylePoseGAN), the DensePose is used to extract UV texture map, which is meant to be 3D pose invariant. How would this paper compare with DensePose-based disentangling?\n2) In Line 140, it'd be better to explain how to come up with the numbers of valid patches for different kinds of garments. \n3) It'd be also desirable to generate higher-resolution results, 512x512. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and societal impact are mostly addressed. It'd be better to provide additional results for viewpoint limitation resulted from 2D skeleton pose as mentioned above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "Kv8IPD8X2Ix", "reviewer": "Reviewer_5Zj3", "summary": "This paper presents a human garment try-on pipeline. The key point of the proposed approach is to normalize each body part into a regular patch according to the body skeleton, and use this patch representation to extract the style and disentangle the spatial variation due to different body poses. The network is based on StyleGAN architecture, and adapted to accept style and pose conditions. Furthermore, to overcome the loss of the texture details, it also proposes to use the first-stage network to generate the clothed mask, and use a refining network to synthesize/inpaint the texture details based on the deformed patches aligned to the target skeleton.", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "It is not new to have a canonical representation for body appearance and garment to neutralize the pose variations, including using 3D UV space like DensePose or 2D patches like this paper. This paper does not provide enough insight why this simple patch extracting and normalization is better than previous solutions. Also, the normalization step is confusing, by mapping a rectangle (expanded vector between two joints) to a patch, a simple closed-form rotation+scaling should suffice, why there is a need to do least-squares optimization to get the matrix?\n\nRegarding the network architecture, it looks pretty similar to the StylePoseGAN work (Style and Pose Control for Image Synthesis of Humans from a Single Monocular View). Although I guess it is still an unpublished work, why that method can produce better quality results without the need for the texture synthesis branch as the second stage? And in the texture branch, the feature inpainting method does not sound very convincing to me. What if the warped patches are not accurate, and how would it be to simply ask the network to generate the image with the warped patches as the soft condition input?\n\nAnd my biggest concern is on the training losses. Why is it reasonable to have reconstruction loss between the source image and the result? They wear the same garment, but the body poses are not aligned.\n\n---------------------\n\nAddition to the final review:\n\nI would like to thanks the authors for participating in the long discussion. While the discussion helped achieve a better understanding of most technical details, I am not fully convinced that this paper is ready for acceptance. My major concerns still exist:\n1. The way how self-supervised training is done in the proposed method introduces a huge training-testing gap that is not negligible. The proposed augmentation approach does help improve robustness against occlusion but still suffers from unseen mismatches and other defects that are not seen during training. Please see the detailed comments.\n2. The way how the paper handles patch cropping and normalization/deformation is presented as is. Many choices should be justified: Why optimization is needed for normalization? Why the proposed way to generate the cropping patch is the best? Why the patch does not cover the whole garment? But the paper does not provide enough experiments to convince the reader.\n3. Lack of significant technical breakthrough. The proposed method is somehow similarly studied by other works on cloth try-on or human retargeting. It does have some interesting ideas, but I am concerned that how these ideas can really inspire follow-up works.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Most of the limitations are listed in the main review. Including unclear technical details and lack of explanation of the proposed network design.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "I4cvUrg9ejD", "reviewer": "Reviewer_vykN", "summary": "This paper focuses on virtual try-on based on unpaired images. The authors propose to disentangle the style and spatial information of each garment, and generate it under the condition of homography transformed normalized patches. The authors also introduce a new unpaired dataset. Compared with the paired and unpaired state-of-the-art methods, both quantitative and qualitative results prove the effectiveness of the proposed approach.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed pose-guided patch-routed disentanglement module is interesting, and the method is technically reasonable. Detailed ablation studies have proven the effectiveness of each design choice. The authors also propose a new dataset that can facilitate further research. Both quantitative and qualitative results are promising. I also appreciate the detailed descriptions in the supplemental material. In general, I am inclined to accept. I have some concerns about the experiment and the dataset, and I want to hear the authors' response to re-evaluate my score.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- In Table 1, I am surprised that the proposed method performs very well on the UPT dataset (~7.8 FID, compared to 21.58 in DeepFashion). Other methods show similar performance on the two datasets. It is better to discuss the root cause here. I am a little worried that the proposed design choices are particularly suitable for UPT, but they may not always be very helpful in other datasets.\n- It is better to have an apple-to-apple comparison to further demonstrate the advantages of the proposed dataset. Figure 1(b) in the supplement is a good example. I want to know how the 25 clothing categories are distributed in the new dataset compared to the other two. Since the authors have proposed a new dataset, it is better to display detailed statistics to convince readers of the benefits of the dataset.\n- Figure 5, 1st row, left example. It seems that the proposed method smoothed out the black belt. Is it because of the error in the predicted garment mask? It's better to discuss.\n- It is better to show the running time compared with other methods, because the speed of inference is also a consideration for virtual try-on.\n- Minor issue: L40, the main text is too close to the caption of the figure 1.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper was reviewed by four expert reviewers in the community. Most reviewers appreciate the novelty on unpaired training for virtual try-on (although paired data is not a critical restriction in this problem domain). The conditional extension of StyleGAN2 is somewhat similar to StylePoseGAN. All reviewers are aware that StylePoseGAN is an unpublished work so the reviewer assessment of this paper is not affected by an arXiv paper. There were an extensive discussions between the authors and Reviewer 5Zj3. The AC reads the reviews, authors' rebuttal, and the discussions. While there are still some clarifications required for the method exposition and several other limiting factors (e.g., low resolution results only, similar approach as concurrent work), the AC thinks the paper has sufficient merits and could inspire future research work. The AC thus recommends to accept.
Decision: Accept (Poster)