Shape-Guided Clothing Warping for Virtual Try-On

Authors: xiaoyu  han
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-02 17:06:34

Abstract: Image-based virtual try-on aims to seamlessly fit in-shop clothing to a person image while maintaining pose consistency. Existing methods commonly employ the thin plate spline (TPS) transformation or appearance flow to deform in-shop clothing for aligning with the person's body. Despite their promising performance, these methods often lack precise control over fine details, leading to inconsistencies in shape between clothing and the person's body as well as distortions in exposed limb regions. To tackle these challenges, we propose a novel shape-guided clothing warping method for virtual try-on, dubbed SCW-VTON, which incorporates global shape constraints and additional limb textures to enhance the realism and consistency of the warped clothing and try-on results. To integrate global shape constraints for clothing warping, we devise a dual-path clothing warping module comprising a shape path and a flow path. The former path captures the clothing shape aligned with the person's body, while the latter path leverages the mapping between the pre- and post-deformation of the clothing shape to guide the estimation of appearance flow. Furthermore, to alleviate distortions in limb regions of try-on results, we integrate detailed limb guidance by developing a limb reconstruction network based on masked image modeling. Through the utilization of SCW-VTON, we are able to generate try-on results with enhanced clothing shape consistency and precise control over details. Extensive experiments demonstrate the superiority of our approach over state-of-the-art methods both qualitatively and quantitatively.

Main paper: 
 related work 2.1 appearance flow: 
 Appearance flow is applied to predict a 2D vector field and warp the source image to the target based on the similarity in appearance, which is proposed by Zhou et al. [61] to solve the problem of novel view synthesis. In recent years, appearance flow has been widely applied in various fields of computer vision. In image inpainting tasks [39, 52] , appearance flow is used to propagate pixels from source to missing regions, enhancing realism in generated contents. Additionally, appearance flow also enables human pose transfer [4, 29, 38, 58] , synthesizing novel poses by warping the feature representations of the human body. Besides, virtual tryon has attracted widespread attention in recent years and many methods [1, 10, 14, 16] introduce the appearance flow to deform the in-shop clothing to achieve the alignment with the person's body.Clothing Warping. Clothing Warping is typically a key processing step of most image-based virtual try-on methods. Early methods [15, 33, 45, 55] primarily achieve this through thin-plate spline (TPS) transformation. However, TPS-based methods are unable to accurately handle large geometric deformation due to the limited degrees of freedom. Therefore, ClothFlow [14] proposes an appearance flow based method to perform clothing warping with pixel-level displacements. Then, Zflow [6] adopts gated appearance flow to further stabilize the deformation. SAL-VTON [53] links the clothing with the person's body with semantically associated landmarks and proposes the local flow to alleviate the misalignment. However, considering the potential drastic deformation caused by appearance flow, we introduce extra global shape constraints based on a cross-attention mechanism, which warrants a more stable flow and makes the warped clothing match a consistent shape. Try-on Synthesis. Another important step is try-on synthesis, which refers to synthesizing the target clothing and the person image to produce the try-on result. Benefiting from the powerful generation capability of the diffusion model [20, 40] , some existing methods [2, 11, 26, 35, 62] introduce it as the final component of tryon synthesis. DCI-VTON [11] treats virtual try-on as an inpainting task and incorporates the diffusion model to refine a coarse result. LaDI-VTON [35] employs textual inversion technique in virtual try-on and proposes EMASC modules to enhance synthesis quality. StableVITON [26] proposes a zero cross-attention block used in the pre-trained diffusion decoder to learn the semantic correspondence between the clothing and the person for achieving better detail preservation. However, most methods require the synthesis network to reconstruct limb textures in the absence of corresponding reference cues from the person image, leading to the degradation of performance and distortions in try-on results. We attempt to alleviate this problem by creating an additional limb reconstruction network based on masked image modeling. 
proposed method: 
 As shown in Figure 2 , SCW-VTON consists of three modules: 1) a dual-path clothing warping module, 2) a semantic-replacement layout estimation module, and 3) a limb-weighted try-on synthesis module. The dual-path clothing warping module produces the warped clothing 𝐶 𝑤 and the warped clothing mask 𝑀 𝑤 while the semantic-replacement layout estimation module generates the target semantic layout 𝑆 𝑡 . Based on these acquired results, the limbweighted try-on synthesis module ultimately generates the try-on result 𝐼 𝑡𝑟 𝑦 through a diffusion model.Figure 2 (a) shows the schematic of the dual-path clothing warping module, which can be divided into a shape path, shape-guided cross-attention blocks, and a flow path. We begin by introducing the shape characteristics, which will be used in the shape path and shape-guided cross-attention blocks. Shape Characteristics. The first consideration is how to obtain the shape characteristics of clothing pre-and post-deformation, as this is essential for constructing the mapping between them. In this section, we design a color normalization strategy to obtain the shape characteristics of clothing before deformation firstly, which explicitly normalizes each color channel of in-shop clothing 𝐶 to discard its raw color while keeping the overall gradient difference. Specifically, as shown in Figure 3 , we first calculate the average color value 𝜉 𝑘 for each channel 𝑘 of in-shop clothing 𝐶. Then we subtract 𝜉 𝑘 from each pixel value within the clothing regions of 𝐶.Finally, we add in-shop clothing mask 𝑀 to the subtracted result and get the normalized clothing Ĉ, which can reflect the shape characteristics of in-shop clothing more intuitively while excluding interfering information. The above process can be expressed as:EQUATIONEQUATIONwhere ⊙ is the element-wise multiplication, 𝑖 and 𝑗 is the position of a sample pixel, 𝐻 𝐶 and 𝑊 𝐶 are the height and width of 𝐶, respectively. T (•) is a truncated function ensuring the output value falls within the range of zero and one. Shape Path. The subsequent goal is to obtain the shape characteristics of clothing after deformation, which are then used to match the normalized clothing Ĉ. We first adopt a CNN-based encoder to extract human pose features 𝐹 𝑝𝑜𝑠𝑒 from the skeleton map 𝑃 𝑠 and dense pose estimation 𝑃 𝑑 , where 𝑃 𝑠 is obtained by connecting 9 key points of the person's body and 𝑃 𝑑 is from [13] . Although 𝐹 𝑝𝑜𝑠𝑒 already contains basic information that matches the person's body, it is also essential to ensure that the design and style of the generated clothing are consistent with the in-shop clothing 𝐶. Inspired by [23] , we adopt a multilayer perceptron (MLP) to embed in-shop clothing 𝐶 into the latent space as a global style code 𝑠, which is used to calculate a set of affine transformation parameters in style blocks to adjust the pose features 𝐹 𝑝𝑜𝑠𝑒 and get the stylized shared features 𝐹 𝑠𝑡 𝑦𝑙𝑒 for the shape decoder and the subsequent flow decoder. This process can be formulated as:EQUATIONwhere 𝐹 𝑘,𝑖,𝑗 𝑠𝑡 𝑦𝑙𝑒 is a particular sample of 𝐹 𝑠𝑡 𝑦𝑙𝑒 at location (𝑘, 𝑖, 𝑗), 𝛾 (•) and 𝛿 (•) are the convolution operations that convert the input into affine parameters, 𝜇 𝑘 and 𝜎 𝑘 are the mean and standard deviation of pose features 𝐹 𝑘 𝑝𝑜𝑠𝑒 , respectively. 𝜇 𝑘 is calculated as: where 𝐻 𝐹 and 𝑊 𝐹 are the height and width of 𝐹 𝑝𝑜𝑠𝑒 , respectively.EQUATION𝜎 𝑘 is calculated as:EQUATIONThen, we adopt a shape decoder to up-sample the shared features 𝐹 𝑠𝑡 𝑦𝑙𝑒 and generate a shape-guided clothing map 𝐶 𝑠 , where we employ our color normalization strategy again to get the ground truth of 𝐶 𝑠 in the training phase (please refer to Section 3.4 for more details). Note that 𝐶 𝑠 is not the binary mask of the warped clothing, as shown in Figure 4 , compared to the binary mask 𝑀 𝑤 obtained by directly applying flow to the in-shop clothing mask 𝑀, 𝐶 𝑠 encompasses detailed shape characteristics of clothing after deformation such as the basic appearance, shadows, and wrinkles. Shape-guided Cross-attention Blocks. After obtaining the normalized clothing Ĉ and the shape-guided clothing map 𝐶 𝑠 that respectively reflect the shape characteristics of clothing before and after deformation, we adopt cross-attention blocks to explore the mapping between them, which is subsequently used to constrain the estimation of appearance flow. As shown in Figure 3 , 𝐶 𝑠 and Ĉ are fed into three separate encoders to get 𝑄,𝐾,𝑉 , respectively:EQUATIONwhere 𝐸 𝑞 , 𝐸 𝑘 , 𝐸 𝑣 are encoders to get the corresponding components of the cross-attention mechanism, and 𝐹𝑙𝑎(•) is the flattening operation for extracted feature maps. Next, for the 𝑖-th cross-attention block, its output can be represented as:EQUATIONwhere 𝐴 𝑞 , 𝐴 𝑘 , 𝐴 𝑣 represent linear layers, and 𝑑 is the dimension of𝐾𝐴 𝑖 𝑘 .Finally, all the outputs of cross-attention blocks are integrated to form the cross-attention features 𝐹 𝑎𝑡𝑡 . Flow Path. The essence of appearance flow is the pixel-wise displacement, which does not involve the degradation of image quality caused by down-sampling, so a warped result applied by appearance flow has the ability to retain detailed textures. However, directly estimating appearance flow without global shape constraints may lead to unnatural drastic deformations. Therefore, instead of directly decoding the shared features 𝐹 𝑠𝑡 𝑦𝑙𝑒 to get the appearance flow, we introduce 𝐹 𝑎𝑡𝑡 as extra global shape constraints to perform the shape-guided warping. Specifically, the inputs to the flow decoder consist of two parts: one is the shared features 𝐹 𝑠𝑡 𝑦𝑙𝑒 , and the other is the cross-attention features 𝐹 𝑎𝑡𝑡 . They are concatenated and fed into the flow encoder together to perform pixel-wise regression and estimate the appearance flow. Note that at the training stage, the weight parameters of the shape decoder and the flow decoder are updated simultaneously. This enables the flow decoder to collaborate with the shape decoder seamlessly and implicitly, thereby predicting the appearance flow with a more consistent shape. Finally, the appearance flow is applied to the in-shop clothing 𝐶 and the in-shop clothing mask 𝑀, getting the warped clothing 𝐶 𝑤 and the corresponding warped clothing mask 𝑀 𝑤 .Given the warped clothing mask 𝑀 𝑤 , the skeleton map 𝑃 𝑠 , and the source semantic layout 𝑆 𝑠 , the goal of the semantic-replacement layout estimation module is to produce the target semantic layout 𝑆 𝑡 , which describes the person wearing new clothing. 𝑆 𝑡 is utilized to provide the location information for limb reconstruction in the following process of try-on synthesis.Since it is almost unavailable to acquire the training data about a person wearing two different clothing in a fixed pose, it is a necessary and common practice to acquire person representations that discard the person's original clothing information to train the network in a self-supervised way. We use a semantic-replacement strategy to perform the above discarding during predicting 𝑆 𝑡 . As shown in Figure 2 (b), the source semantic layout 𝑆 𝑠 is separated into a multi-channel binary parsing map and each channel corresponds to clothing or a part of the person's body. We utilize the warped clothing mask 𝑀 𝑤 and a portion of the skeleton map 𝑃 𝑠 to replace the original clothing and limb channels to get the target semantic guidance 𝑆 𝑟𝑒𝑝 , which ensures semantic continuity and pose consistency while preventing negative interference with the network training caused by the person's original clothing. Subsequently, we use a UNet [41] model as the semantic layout estimator, which takes 𝑆 𝑟𝑒𝑝 as the input and predicts the target semantic layout 𝑆 𝑡 . More detailed descriptions of this module can be found in the supplementary material.After obtaining the warped clothing 𝐶 𝑤 and the target semantic layout 𝑆 𝑡 , our final goal is to generate the try-on result. However, during this process, reconstructing limb textures is a challenging task, as the lack of sufficient reference information may lead to inconsistency between the distribution of the reconstructed result and the original image. Limb Reconstruction. Therefore, we propose a separate limb reconstruction network, which focuses on learning compact latent representations of the person's limbs through masked image modeling, aiming to enhance the realism and consistency of the reconstructed limb textures with the original data. As depicted in Figure 2 (c), we first extract the source limb map 𝐿 𝑠 from the person image 𝐼 . Inspired by [17] , we randomly mask 𝐿 𝑠 by 20%∼75% and obtain the masked limb map before an autoencoder. On the one hand, data pairs with the asymmetric input and output are created in the training stage, enhancing the ability of the autoencoder to reconstruct limb textures. On the other hand, random masking encourages the network to learn more compact limb representations. Note that the mask operation is only set in the training phase, while during testing, the autoencoder directly takes 𝐿 𝑠 as input. Besides, the target semantic layout 𝑆 𝑡 is another input of the autoencoder, which is used to provide location information for limb reconstruction. The autoencoder is pre-trained individually and is capable of outputting the reconstructed limb map 𝐿 𝑟 and the limb weight map 𝐿 𝑤 , where the latter is a correlation weight to adaptively modify the limb regions that may be prone to performance degradation and distortions (as shown in Figure 5 ). Try-on Synthesis. Based on 𝐿 𝑤 , 𝐿 𝑟 is then combined with the occluded person image 𝐼 𝑜𝑐𝑐 (obtained by occluding the clothing and limb regions of 𝐼 ) and the warped clothing 𝐶 𝑤 to get the combined representation 𝐼 𝑐𝑜𝑚 . Inspired by [11] , we adopt the pre-trained diffusion model as the final try-on synthesis network, utilizing the combined representation 𝐼 𝑐𝑜𝑚 as the input to produce the tryon result 𝐼 𝑡𝑟 𝑦 . Besides, a pre-trained CLIP [37] image encoder is employed to extract additional conditions, which are injected into the Denoising UNet through the cross-attention mechanism to guide the generation of the diffusion model.Dual-path Clothing Warping. To train the dual-path clothing warping module, we first adopt the reconstruction loss to constrain the pixel-wise value of 𝐶 𝑠 and 𝐶 𝑤 , which is formulated as:EQUATIONwhere 𝐶 𝑔𝑡 is the ground truth of the warped clothing 𝐶 𝑤 , it is extracted from the person image 𝐼 . Ĉ𝑔𝑡 is obtained by applying the color normalization strategy to 𝐶 𝑔𝑡 . Besides, we adopt the perceptual loss proposed in [25] to calculate the distance of the features extracted by the VGG-19 [44] network:EQUATIONwhere 𝜙 𝑖 (•) denotes the feature maps of the 𝑖-th layer in the pretrained perception network. Furthermore, we adopt a mask loss to constrain the warped clothing mask 𝑀 𝑤 , which is formulated as:EQUATIONwhere 𝑀 𝑔𝑡 is the mask of 𝐶 𝑔𝑡 . Overall, the loss of the dual-path clothing warping module is represented as:EQUATIONwhere 𝜆 𝑝𝑒𝑟 is used to balance the weights of these losses. Semantic-replacement Layout Estimation. We use a weighted cross-entropy loss to supervise the training process of the semantic layout estimator, which is expressed as:EQUATIONwhere 𝑛 denotes the number of samples, 𝑐 is the number of channels of 𝑆 𝑠 and 𝑆 𝑡 , and 𝑤 𝑗 is the loss weight in the 𝑗-th class channel.Limb-weighted Try-on Synthesis. Finally, following [11] , the loss of the limb-weighted try-on synthesis 𝑙 𝑠𝑦𝑛 is designed in two parts, which can be represented as:EQUATION)𝑙 𝑣𝑔𝑔 is similar to 𝑙 𝑝𝑒𝑟 :EQUATIONand 𝑙 𝑙𝑑𝑚 is formulated as:EQUATION)where E is a pre-trained encoder belonging to the diffusion model, which embeds the images from image space to latent space. 𝜏 (•) is the operation of adding noise, 𝑚 is the mask used to occlude the person image 𝐼 , and 𝑡 is the timestamp. 
experiment 4.1 datasets: 
 We conduct main experiments on the public virtual try-on benchmark dataset VITON [15] , which contains 14,221 data pairs for training and 2,032 data pairs for testing. Also, we carry out the comparative experiments under the higher resolution on the VITON-HD [5] dataset, which contains 11,647 data pairs for training and 2,032 data pairs for testing.We train three modules of SCW-VTON independently. For the dualpath clothing warping module and the semantic layout estimation module, they are both trained for 30 epochs and optimized by Adam [27] with 𝛽 1 =0.5 and 𝛽 2 =0.999. The learning rate is fixed at 0.0001 in the first half of training and then linearly decays to zero in the remaining steps. We set the hyper-parameters as 𝜆 𝑝𝑒𝑟 = 5, 𝑤 0 = 𝑤 1 = 𝑤 2 = 𝑤 6 = 1, and 𝑤 3 = 𝑤 4 = 𝑤 5 = 3. For the limbweighted try-on synthesis module, following [11] , we use AdamW [32] optimizer with the learning rate of 0.00001 to train this module for 50 epochs, and the hyper-parameters 𝜆 𝑣𝑔𝑔 is set to 0.0001.We first compare our SCW-VTON with existing virtual try-on methods ACGPN [55] , SDAFN [1] , RMGN [31] , and DOC-VTON [56] on the VITON [15] dataset qualitatively. In Figure 6 , we divide the comparison into two groups, where the left group focuses on the clothing and the right group focuses on the person's limbs. On the one hand, most baseline methods struggle to handle clothing with dense and complex textures. Take the first row of the left group in Figure 6 as an example, the logo after deformation presents a large range of distortions in the results of RMGN [31] and DOC-VTON [56] , while the results obtained by ACGPN [55] and SDAFN [1] appear blurry. In comparison, benefiting from the proposed shape constraints on clothing warping, the global shape and pose consistency of clothing is warranted in our results, leading to remarkable realism. On the other hand, it is also challenging to fit in-shop clothing into a person image with a complex pose, which mainly involves limb occlusion and rotation. The right group of Figure 6 showcases the inferior performance of baseline methods in these difficult cases. For instance, in the first row of the right group, ACGPN [55] and SDAFN [1] generate distorted and unnatural limbs, while RMGN [31] and DOC-VTON [56] disrupt the continuity of limb textures. Compared to baseline methods, we achieve more reasonable effects, which are attributed to the consistent clothing shape and realistic limb textures acquired by our method. Qualitative experiments on the VITON-HD [5] dataset are conducted in Figure 7 , where the results of HR-VTON [28] , SAL-VTON [53] , DCI-VTON [11] , StableVITON [26] , and our SCW-VTON are represented. Similarly, most baseline methods struggle to align clothing with the person's body while maintaining the integrity of the clothing design. For example, in the first row on the left side of Figure 7 , most baseline methods fail to accurately estimate the correct position of the waistband, and even lose the texture information of the waistband after the clothing deformation. In contrast, our method can preserve the waistband intact and deform it to the appropriate position according to the person's posture. These results validate the robust performance of our method even with an increased resolution. Additional qualitative experimental results are provided in the supplementary material.We conduct quantitative experiments both in paired data setting and unpaired data setting, denoting that a person wears original clothing and new clothing after try-on, respectively. We use Structural Similarity (SSIM) [46] and Peak Signal to Noise Ratio (PSNR) [21] as Table 1 : Quantitative comparisons on the VITON [15] and VITON-HD [5] datasets. The percentage results in the last column are displayed as a/b, with a and b representing the preference for the baseline method and our method, respectively. [19] for the unpaired data setting. To perform a more comprehensive evaluation, we compute metrics both for the process of clothing warping and overall try-on. Table 1 summarizes the quantitative results of our method and baseline methods on two datasets, which indicates that our SCW-VTON outperforms all the baseline methods in SSIM, PSNR, and FID, both in terms of the warped clothing and try-on result.We conduct a user study with 30 recruited volunteers to evaluate the visual effect of our method, where SCW-VTON is compared with each baseline method in an A/B manner. Specifically, 200 in-shop clothing images and 200 person images are randomly selected from the testing set in each comparison, which are used to output the try-on results by SCW-VTON and another baseline method to form 200 pairwise result groups. The volunteers are asked to consider the rationality of two try-on results from each group carefully and choose the better one. The results are summarized in the last column Table 1 , it is evident that our SCW-VTON always provides a better visual experience in each A/B comparison.Global Shape Constraints. To assess the impact of the crossattention features 𝐹 𝑎𝑡𝑡 as global shape constraints, we design a variant SCW-VTON * that removes shape-guided cross-attention blocks, where the flow decoder only takes the shared features 𝐹 𝑠𝑡 𝑦𝑙𝑒 as the input. We first compare the variant and our full model in Figure 8 Limb Reconstruction Network. Finally, we propose a variant SCW-VTON † to evaluate the role of the limb reconstruction network in the process of try-on synthesis. We ablate the autoencoder that outputs the reconstructed limb map 𝐿 𝑟 , so the final try-on result is generated only based on the combined representation 𝐼 𝑐𝑜𝑚 without additional limb texture references. We compute the quantitative metrics for SCW-VTON † , which is also shown in Table 2 .In addition, the qualitative comparison between this variant and SCW-VTON is presented in Figure 8 (b), which illustrates that SCW-VTON can effectively solve the problems of performance degradation and distortions in SCW-VTON † , and make the distribution of reconstructed limb textures more consistent with the ground truth.Besides the classic virtual try-on, we also try to explore other applications of our method. By introducing extra global shape constraints, our method is liberated from the restriction of generating try-on results solely based on specific input clothing shapes. For example, as shown in Figure 9 , we can seamlessly transfer textures from a shapeless texture map to the person's body while preserving the original distribution. This versatility highlights that our method extends beyond conventional virtual try-on applications, potentially sparking innovative ideas for novel computer vision tasks in the fashion and clothing domain. 
conclusion: 
 In this paper, we propose SCW-VTON, a novel shape-guided clothing warping method for virtual try-on. Based on a dual-path clothing warping module, our method incorporates additional global shape constraints on clothing deformation, resulting in realistic warped clothing that conforms accurately to the person's body. Besides, we design a limb reconstruction network based on masked image modeling to learn the compact latent limb representations and generate additional limb textures to refine the details in these regions through adaptive weighting. With the implementation of SCW-VTON, we produce try-on results with improved clothing shape consistency and precise control over details. Extensive experiments are conducted to demonstrate the superiority of our SCW-VTON over existing state-of-the-art methods. 
: 
 Figure 1 : Virtual try-on typically deforms clothing to fit the person's body, which is then combined with the person image to obtain try-on results. Compared with existing TPS-based methods (e.g., VITON-HD [5] ) and appearance flow based methods (e.g., HR-VTON [28] ), our method excels in capturing the clothing shape conforming to the person's body.Image-based virtual try-on aims to seamlessly fit in-shop clothing to a person image while maintaining pose consistency. Existing methods commonly employ the thin plate spline (TPS) transformation or appearance flow to deform in-shop clothing for aligning with the person's body. Despite their promising performance, these methods often lack precise control over fine details, leading to inconsistencies in shape between clothing and the person's body as well as distortions in exposed limb regions. To tackle these challenges, we propose a novel shape-guided clothing warping method for virtual try-on, dubbed SCW-VTON, which incorporates global shape constraints and additional limb textures to enhance the realism and consistency of the warped clothing and try-on results. To integrate global shape constraints for clothing warping, we devise a dual-path clothing warping module comprising a shape path and a flow path. The former path captures the clothing shape aligned with the person's body, while the latter path leverages the mapping 1 In recent years, the e-commerce industry has experienced rapid development, leading to an increasing number of consumers purchasing clothing online. To provide online customers with a shopping experience that rivals in-store try-on, significant attention has been devoted to exploring the virtual try-on technology. Virtual try-on can be broadly categorized into 3D-based [12, 34, 36, 42, 43, 47] and image-based methods [7-10, 14, 15, 18, 22, 24, 30, 45, 48-51, 54, 55, 57, 59, 60] , with the latter garnering more interest due to its lightweight data and wider applicability. This study will specifically delve into image-based virtual try-on.Among the key processing steps in image-based virtual try-on, clothing warping stands out as particularly challenging. Its objective is to deform the target clothing to align with the person's body, which is vital for maintaining consistency between the clothing shape and the person's posture. Early methods [15, 33, 45, 55] adopt the thin plate spline (TPS) transformation [3] to achieve this deformation. However, these TPS-based methods, constrained by limited degrees of freedom, prove inadequate when significant geometric deformations are needed [6, 14] , such as the misaligned clothing generated by VITON-HD [5] in Figure 1 . To overcome this limitation, recent studies [1, 6, 14, 31, 53] introduce appearance flow [61] as a solution, which provides dense pixel-level predictions for deforming from the source clothing to the target one, thereby enhancing the accuracy of the resulting warped clothing. While these methods have demonstrated promising performance in virtual try-on, a notable limitation remains in their current inability to exert precise control over the details of the obtained warped clothing. Firstly, the lack of the appropriate constraint on shape during pixel-level prediction may lead to inconsistencies in the shape of the warped clothing compared to the person's body. For instance, as shown in the HR-VTON [28] column of Figure 1 , the generated clothing edges appear torn and inconsistent. Secondly, distortions may occur partially in the try-on result, especially in the limb regions not covered by clothing. This is attributed to the absence of detailed information guidance for these parts, as existing methods often use the person image with masked clothing and limb regions as the input for try-on synthesis.In this study, we direct our attention towards addressing these underexplored challenges. We present SCW-VTON, a shape-guided clothing warping method for virtual try-on, which incorporates global shape constraints and additional limb texture references to enhance the realism of generated warped clothing and try-on results. Initially, we leverage global shape constraints to facilitate the clothing warping phase, alleviating discrepancies between the warped clothing shape and the person's body. We design a dual-path clothing warping module comprising a shape path and a flow path. The shape path first predicts the shape of the target clothing, facilitating the subsequent exploration of the mapping between the preand post-deformation clothing shapes using a set of shape-guided cross-attention blocks. The acquired mapping is then integrated into a flow path as global shape constraints to steer the estimation of appearance flow. This flow is subsequently applied to in-shop clothing to create the target warped clothing. It is worth mentioning that introducing additional global shape constraints frees our method from reliance on the original input shape, which implies that our method is also capable of transferring textures from a shapeless texture map to the person image while preserving their distribution. Furthermore, we employ a co-training strategy for the two paths during training, which positively affects the weight update of the flow estimation model and further enhances the stability and accuracy of appearance flow. In the subsequent process of tryon synthesis, to alleviate distortions in limb regions, we develop an additional limb reconstruction network based on the masked image modeling method. Specifically, we first derive a limb map from the source person image. Then, we employ an autoencoder to take the masked partial limb component as input to learn latent limb representations, from which we reconstruct realistic limb textures at specific locations. Through the combination of shape-guided appearance flow and reconstructed limb textures, our approach can ultimately produce try-on results with enhanced clothing shape consistency and precise control over details.The contributions of this paper are summarized as follows:• We introduce a shape-guided clothing warping method for virtual try-on that incorporates global shape constraints on clothing deformation, resulting in realistic warped clothing that conforms accurately to the person's body.  
 
 Appendix:  


Decision: This paper presents a new approach for virtual try-on called SCW-VTON. The reviewers recognize the clarity of the presentation, the thoroughness of the methodology, and the comprehensive experimental validation, although some concerns were raised regarding specific implementation details and the extent of the experiments. The suggestion to include additional datasets and scenarios, as well as making the source code publicly available, should be considered by the authors in their preparation of revision.

Reviews: review: paper_summary: paper_summary: This paper propose a try-on pipeline with Stable Diffusion. To handle clothes distortion and achieve better generation results, the author propose a dual-path clothing warping module, a semantic-replacement layout estimation module and a limb-weighted try-on synthesis module. Experiments results demonstrate that the methods surpass the SOTA methods.
strength_weakness: strength_weakness: (1) The paper is well-written and easy to understand.

(2) The proposed solution demonstrates a certain degree of feasibility, with a clear and detailed elucidation of the technical aspects, which enhances the reader's comprehension of the methodology and its potential applications.

(3) The explanation of the formula is basically clear, and the logic is acceptable, providing a solid foundation for understanding the underlying principles and I can follow the mathematical reasoning without difficulty.
limitations: limitations: Weaknesses:

(1) In the Limb-weighted Try-on Synthesis module, the reconstructed limb map and limb weight map are unnecessary. The task of try-on doesn't require altering the limb; the only concern regarding this part is whether the limb is visible and how many parts of the limb are visible.

(2) In Sec 3.2, Semantic-replacement Layout Estimation has already generate the mask of both the target limb and other parts.  Why not simply use the generated mask to mask out the target limb and then send the result into the network, or just input the original limb?  Why is the output from the previous module not used as input for the next network, but instead regenerated through a new network?  In my view, this is redundant and may cause confusion.

(3) The method based on semantic correspondences heavily depends on the accuracy and generalization performance of the task-oriented semantic segmentation/prediction model, which the authors have not adequately considered. 

(4) The experiments conducted in the paper are inadequate, as the authors solely focused on VITON/VITON-HD without including any experiments on Dress Code.
paper_summary: paper_summary: This paper proposes a new shape-guided clothing warping method for virtual try-on, called SCW-VTON. It is an optimized virtual try-on model, which adopts the approach of "Clothing Warping-Diffusion Synthesis." The main contents can be summarized in the following three aspects: 
a) This paper designs a dual-path clothing warping module, including a shape path and a flow path. Shape path captures the clothing shape aligned with the human body, while the flow path guides the appearance flow estimation by utilizing the mapping between the pre-deformation and post-deformation clothing shapes. Compared to previous efforts, it achieves superior clothing warping results (e.g., folds, shadows).
b) Additionally, this paper designs a limb reconstruction network, A pre-trained autoencoder is used to generate limb weight maps to guide the network in producing more refined and natural human body parts. Compared to existing work, SCW-VTON demonstrates advantages in generating complex human poses (e.g., crossed arms). 
c) SCW-VTON achieves state-of-the-art (SOTA) performance on the VITON and VITON-HD datasets. 
Although SCW-VTON performs well on existing datasets, it may have limitations in generalization compared to recent virtual try-on methods based on diffusion pre-trained models. For more details, please refer to the limitation section.
strength_weakness: strength_weakness: 1. This article has accomplished thorough work.
2. The proposed Shape-guided Clothing Map for clothing warping is meaningful, addressing the issue of unnatural clothing deformation in existing methods.
3. The article demonstrates that the Limb Weight Map can effectively handle human body region generation under complex poses, which is beneficial for virtual try-on under complex poses.
limitations: limitations: 1. The method proposed in this article relies on using appearance flow for clothing warping, which may limit the model's applicability in complex try-on scenarios, such as garments with intricate patterns (e.g., coats, elaborately designed dresses) or the clothing in images are laid flat.
2. Including commonly used metrics like LPIPS and KID in the experimental section might enhance the persuasiveness of the findings.
3. The experimental section of this article lacks comparative results with representative studies, such as GP-VTON.
4. In the visualization results of VITON-HD in this article, there are relatively few types of garments. Could you provide more samples of different garment types (e.g., short type, big logo)?
5. The concepts of shape-guided cloth deformation presented in this paper are not particularly novel. It is recommended to enhance the discussion of the innovative aspects and contributions of this work to better distinguish it from existing research.
6. Lines 524-528 provide an introduction to the output of the pre-trained autoEncoder. I would like to know how this autoEncoder is designed and trained, and how it is constrained for the limb weight map.
paper_summary: paper_summary: This paper proposes a shape-guided clothing warping method for virtual try-on, dubbed SCW-VTON, which incorporates global shape constraints and additional limb textures to enhance the realism and consistency of the warped clothing and try-on results. Extensive experiments
demonstrate the superiority of the approach over state-of-the-art methods both qualitatively and quantitatively.
strength_weakness: strength_weakness: This article is technically sound, and it has a clearer motivation and a more complete experiment. Furthermore, this paper is written with a high degree of readability.
limitations: limitations: 1. Whether or not the body part detection tool has a large impact on the performance of the proposed algorithm. Did the authors try different body part detection tools, and please provide a specific analysis.

2.In addition to simulating real clothing deformation, please elaborate on what gaps still exist between the method and real clothing changes. For example, texture differences, color differences, etc., such fine-grained differences are key factors in simulating clothing versus real clothing, however causing a great sense of virtuality.

3.The article visualization image shows women, is it equally valid for men's clothing? This is because there are certain differences between male and female clothing, such as between dresses and shorts. In addition to the question of whether further effective analysis of children's clothing can be achieved, the visualization results only show that women's clothing is a very inadequate simulation of the real world.

4.Furthermore, the paper is missing an analysis of the different forms of clothing, which vary greatly from one body type to another, such as height differences, fat and thin. Once again, the clothes of different seasons are not effectively presented.

5.Whether the encoder-decoder structure used by the model incurs additional training costs, the authors need to perform a richer parameter volume analysis.

6. Since human body postures are varied, back and side garment simulations are necessary, and the authors only show the front situation in the visualization example diagrams, which lacks richer information about the postures.

7.There are still some grammar errors, and the English usage can be improved in terms of grammar and clarity.
paper_summary: paper_summary: To address the issue that existing virtual try-on methods often lack precise control over fine details, leading to inconsistencies between the clothing shape and the person’s body, this paper proposes a novel shape-guided clothing warping method. This method incorporates global shape constraints on clothing deformation to ensure a better fit. Additionally, the paper introduces a limb reconstruction network to provide precise guidance in generating the limb regions of the try-on result, effectively mitigating performance degradation and distortions during the synthesis process. Experiments demonstrate the superior performance of the proposed method. The paper addresses an intriguing topic, capturing the reader's interest with its depth and relevance. Moreover, the figures presented in the paper exhibit exceptional clarity and high quality, enhancing the overall visual impact and comprehension of the research findings.
strength_weakness: strength_weakness: The paper's meticulous organization and logical flow aid reader understanding. The authors' clear exposition of the methodologies instills trust in the reliability of their findings. Furthermore, the paper offers comprehensive explanations of the experimental protocols and interpretation of results, providing profound insights into the effectiveness and relevance of the proposed methodologies. These detailed explanations contribute to a nuanced understanding of the research outcomes and enhance the paper's impact in the field.
limitations: limitations: 1. It is highly recommended that the authors consider making the source code publicly available to promote transparency and reproducibility in their research. Access to the source code would enable other researchers to validate the findings, replicate the experiments, and potentially build upon the proposed methodologies. By sharing the code, the authors contribute to the advancement of the scientific community and facilitate further innovation in the field.
2. The paper would benefit from a thorough proofreading to correct grammatical errors that detract from the clarity of the writing. For example, in the sentence “where Cgt is the ground truth of the warped clothing Cw, it is extracted from the person image I.” there is a grammatical inconsistency and lack of clarity in the phrasing.