Towards Scalable Unpaired Virtual Try-On via Patch-Routed Spatially-Adaptive GAN

1 Introduction . Image-based virtual try-on , the process of computationally transferring a garment onto a particular person in a query image , is one of the most promising applications of human-centric image generation with the potential to revolutionize shopping experiences and reduce purchase returns . However , to fully exploit its potential , scalable solutions are required that can leverage easily accessible training data , handle arbitrary garments , and provide efficient inference results . Unfortunately , to date , most existing methods [ 35 , 38 , 12 , 7 , 37 , 9 , 10 , 4 , 36 , 39 ] rely on paired training data , i.e. , a person image ∗Xiaodan Liang is the corresponding author . Our code will be available at PASTA-GAN . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . and its corresponding in-shop garment , leading to laborious data-collection processes . Furthermore , these methods are unable to exchange garments directly between two person images , thus largely limiting their application scenarios and raising the need for unpaired solutions to ensure scalability . While unpaired solutions have recently started to emerge , performing virtual try-on in an unsupervised setting is extremely challenging and tends to affect the visual quality of the try-on results . Specifically , without access to the paired data , these models are usually trained by reconstructing the same person image , which is prone to over-fitting , and thus underperform when handling garment transfer during testing . The performance discrepancy is mainly reflected in the garment synthesis results , in particular the shape and texture , which we argue is caused by the entanglement of the garment style and spatial representations in the synthesis network during the reconstruction process . While this is not a problem for the traditional paired try-on approaches [ 35 , 12 , 37 , 10 ] , which avoid this problem and preserve the garment characteristics by utilizing a supervised warping network to obtain the warped garment in target shape , this is not possible in the unpaired setting due to the lack of warped ground truth . The few works that do attempt to achieve unpaired virtual try-on , therefore , choose to circumvent this problem by either relying on person images in various poses for feature disentanglement [ 23 , 33 , 32 , 31 , 1 , 5 ] , which again leads to a laborious data-collection process , or require extensive online optimization procedures [ 25 , 17 ] to obtain fine-grain details of the original garments , harming the inference efficiency . However , none of the existing unpaired try-on methods consider the problem of coupled style and spatial garment information directly , which is crucial to obtain accurate garment transfer results in the unpaired and unsupervised virtual try-on scenario . In this paper , to tackle the essential challenges mentioned above , we propose a novel PAtch-routed SpaTially-Adaptive GAN , named PASTA-GAN , a scalable solution to the unpaired try-on task . Our PASTA-GAN can precisely synthesize garment shape and style ( see Fig . 1 ) by introducing a patch-routed disentanglement module that decouples the garment style and spatial features , as well as a novel spatially-adaptive residual module to mitigate the problem of feature misalignment . The innovation of our PASTA-GAN includes three aspects : First , by separating the garments into normalized patches with the inherent spatial information largely reduced , the patch-routed disentanglement module encourages the style encoder to learn spatial-agnostic garment features . These features enable the synthesis network to generate images with accurate garment style regardless of varying spatial garment information . Second , given the target human pose , the normalized patches can be easily reconstructed to the warped garment complying with the target shape , without requiring a warping network or a 3D human model . Finally , the spatially-adaptive residual module extracts the warped garment feature and adaptively inpaints the region that is misaligned with the target garment shape . Thereafter , the inpainted warped garment features are embedded into the intermediate layer of the synthesis network , guiding the network to generate try-on results with realistic garment texture . We collect a scalable UnPaired virtual Try-on ( UPT ) dataset and conduct extensive experiments on the UPT dataset and two existing try-on benchmark datasets ( i.e. , the DeepFashion [ 21 ] and the MPV [ 6 ] datasets ) . Experiment results demonstrate that our unsupervised PASTA-GAN outperforms both the previous unpaired and paired try-on approaches . 2 Related Work . Paired Virtual Try-on . Paired try-on methods [ 13 , 35 , 38 , 12 , 24 , 37 , 9 , 10 , 36 ] aim to transfer an in-shop garment onto a reference person . Among them , VITON [ 13 ] for the first time integrates a U-Net [ 29 ] based generation network with a TPS [ 2 ] based deformation approach to synthesize the try-on result . CP-VTON [ 35 ] improves this paradigm by replacing the time-consuming warping module with a trainable geometric matching module . VTNFP [ 38 ] adopts human parsing to guide the generation of various body parts , while [ 24 , 37 , 39 ] introduce a smooth constraint for the warping module to alleviate the excessive distortion in TPS warping . Besides the TPS-based warping strategy , [ 12 , 36 , 10 ] turn to the flow-based warping scheme which models the per-pixel deformation . Recently , VITON-HD [ 4 ] focuses on high-resolution virtual try-on and proposes an ALIAS normalization mechanism to resolve the garment misalignment . PF-AFN [ 10 ] improves the learning process by employing knowledge distillation , achieving state-of-the-art results . However , all of these methods require paired training data and are incapable of exchanging garments between two person images . Unpaired Virtual Try-on . Different from the above methods , some recent works [ 23 , 33 , 32 , 31 , 25 , 17 ] eliminate the need for in-shop garment images and directly transfer garments between two person images . Among them , [ 23 , 33 , 32 , 31 , 1 , 5 ] leverage pose transfer as the pretext task to learn disentangled pose and appearance features for human synthesis , but require images of the same person with different poses.2 In contrast , [ 25 , 17 ] are more flexible and can be directly trained with unpaired person images . However , OVITON [ 25 ] requires online appearance optimization for each garment region during testing to maintain texture detail of the original garment . VOGUE [ 17 ] needs to separately optimize the latent codes for each person image and the interpolate coefficient for the final try-on result during testing . Therefore , existing unpaired methods require either cumbersome data collecting or extensive online optimization , extremely harming their scalability in real scenarios . 3 PASTA-GAN . Given a source image Is of a person wearing a garment Gs , and a target person image It , the unpaired virtual try-on task aims to synthesize the try-on result I ′t retaining the identity of It but wearing the source garment Gs . To achieve this , our PASTA-GAN first utilizes the patch-routed disentanglement module ( Sec . 3.1 ) to transform the garment Gs into normalized patches Pn that are mostly agnostic to the spatial features of the garment , and further deforms Pn to obtain the 2As the concurrent work StylePoseGAN [ 31 ] is the most related pose transfer-based approach , we provide a more elaborate discussion of the inherent differences in the supplementary . warped garment Gt complying with the target person pose . Then , an attribute-decoupled conditional StyleGAN2 ( Sec . 3.2 ) is designed to synthesize try-on results in a coarse-to-fine manner , where we introduce novel spatially-adaptive residual blocks ( Sec . 3.3 ) to inject the warped garment features into the generator network for more realistic texture synthesis . The loss functions and training details will be described in Sec . 3.4 . Fig . 2 illustrates the overview of the inference process for PASTA-GAN . 3.1 Patch-routed Disentanglement Module . Since the paired data for supervised training is unavailable for the unpaired virtual try-on task , the synthesis network has to be trained in an unsupervised manner via image reconstruction , and thus takes a person image as input and separately extracts the feature of the intact garment and the feature of the person representation to reconstruct the original person image . While such a training strategy retains the intact garment information , which is helpful for the garment reconstruction , the features of the intact garment entangle the garment style with the spatial information in the original image . This is detrimental to the garment transfer during testing . Note that the garment style here refers to the garment color and categories , i.e. , long sleeve , short sleeve , etc. , while the garment spatial information implies the location , the orientation , and the relative size of the garment patch in the person image , in which the first two parts are influenced by the human pose while the third part is determined by the relative camera distance to the person . To address this issue , we explicitly divide the garment into normalized patches to remove the inherent spatial information of the garment . Taking the sleeve patch as an example , by using division and normalization , various sleeve regions from different person images can be deformed to normalized patches with the same orientation and scale . Without the guidance of the spatial information , the network is forced to learn the garment style feature to reconstruct the garment in the synthesis image . Fig . 3 illustrates the process of obtaining normalized garment patches , which includes two main steps : ( 1 ) pose-guided garment segmentation , and ( 2 ) perspective transformation-based patch normalization . Specifically , in the first step , the source garment Gs and human pose ( joints ) Js are firstly obtained by applying [ 11 ] and [ 3 ] to the source person Is , respectively . Given the body joints , we can segment the source garment into several patches Ps , which can be quadrilaterals with arbitrary shapes ( e.g. , rectangle , square , trapezoid , etc . ) , and will later be normalized . Taking the torso region as an example , with the coordinates of the left/right shoulder joints and the left/right hips joints in P is , a quadrilateral crop ( of which the four corner points are visualized in color in P is of Fig . 3 ) covering the torso region of Gs can be easily performed to produce an unnormalized garment patch . Note that we define eight patches for upper-body garments , i.e. , the patches around the left/right upper/bottom arm , the patches around the left/right hips , a patch around the torso , and a patch around the neck . In the second step , all patches are normalized to remove their spatial information by perspective transformations . For this , we first define the same amount of template patches Pn with fixed 64× 64 resolution as transformation targets for all unnormalized source patches , and then compute a homography matrixHis→n ∈ R3×3 [ 40 ] for each pair of P is and P in , based on the four corresponding corner points of the two patches . Concretely , His→n serves as a perspective transformation to relate the pixel coordinates in the two patches , formulated as : xinyin 1 = His→n xisyis 1 = hi11 hi12 hi13hi21 hi22 hi23 hi31 h i 32 h i 33 xisyis 1 ( 1 ) where ( xin , y i n ) and ( x i s , y i s ) are the pixel coordinates in the normalized template patch and the unnormalized source patch , respectively . To compute the homography matrixHis→n , we directly leverage the OpenCV API , which takes as inputs the corner points of the two patches and is implemented by using least-squares optimization and the Levenberg-Marquardt method [ 8 ] . After obtainingHis→n , we can transform the source patch P is to the normalized patch P i n according to Eq . 1 . Moreover , the normalized patches Pn can further be transformed to target garment patches Pt by utilizing the target pose Jt , which can be obtained from the target person It via [ 3 ] . The mechanism of that backward transformation is equivalent to the forward one in Eq . 1 , i.e. , computing the homography matrixHin→t based on the four point pairs extracted from the normalized patch P in and the target pose Jt . The recovered target patches Pt can then be stitched to form the warped garment Gt that will be sent to the texture synthesis branch in Fig . 2 to generate more realistic garment transfer results . We can also regardHs→t = Hn→t · Hs→n as the combined deformation matrix that warps the source garment to the target person pose , bridged by an intermediate normalized patch representation that is helpful for disentangling garment styles and spatial features .

Summary: This paper presents a human garment try-on pipeline. The key point of the proposed approach is to normalize each body part into a regular patch according to the body skeleton, and use this patch representation to extract the style and disentangle the spatial variation due to different body poses. The network is based on StyleGAN architecture, and adapted to accept style and pose conditions. Furthermore, to overcome the loss of the texture details, it also proposes to use the first-stage network to generate the clothed mask, and use a refining network to synthesize/inpaint the texture details based on the deformed patches aligned to the target skeleton.