GPD-VVTO: Preserving Garment Details in Video Virtual Try-On

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-01 13:54:38

Abstract: Video Virtual Try-On aims to transfer a garment onto a person in the video. Previous methods typically focus on image-based virtual try-on, but directly applying these methods to videos often leads to temporal discontinuity due to inconsistencies between frames. Limited attempts in video virtual try-on also suffer from unrealistic results and poor generalization ability. In light of previous research, we posit that the task of video virtual try-on can be decomposed into two key aspects: (1) single-frame results are realistic and natural, while retaining consistency with the garment; (2) the person's actions and the garment are coherent throughout the entire video. To address these two aspects, we propose a novel two-stage framework based on Latent Diffusion Model, namely Garment-Preserving Diffusion for Video Virtual Try-On (GPD-VVTO). In the first stage, the model is trained on single-frame data to improve the ability of generating high-quality try-on images. We integrate both low-level texture features and high-level semantic features of the garment into the denoising network to preserve garment details while ensuring a natural fit between the garment and the person. In the second stage, the model is trained on video data to enhance temporal consistency. We devise a novel Garment-aware Temporal Attention (GTA) module that incorporates garment features into temporal attention, enabling the model to maintain the fidelity to the garment during temporal modeling. Furthermore, we collect a video virtual try-on dataset containing high-resolution videos from diverse scenes, addressing the limited variety of current datasets in terms of video background and human actions. Extensive experiments demonstrate that our method outperforms existing state-of-the-art methods in both image-based and video-based virtual try-on tasks, indicating the effectiveness of our proposed framework.

Main paper: 
 related work 2.1 image-based virtual try-on: 
 GAN-based methods. Most GAN-based methods [15, 26, 34, 42] follow a two-stage generation framework, first deforming the target garment to fit the person's body, and then fusing the deformed garment with the reference person. HR-VITON [26] proposes a tryon condition generator that predicts simultaneously the warped garment and the segmentation map, removing the misalignment and handling occlusions of clothes by body parts naturally. GP-VTON [42] employs local flows to warp garment parts individually and assembles the local warped results via global garment parsing, avoiding the garment squeezing and texture distortion problems during warping. However, the generational capacity of GANs significantly restricts the performance of GAN-based methods.Diffusion-based methods. Recently, diffusion models have emerged as a promising alternative to GANs due to their exceptional performance in generating high-quality images at high resolutions. Consequently, there has been growing interest in exploring the application of diffusion models in virtual try-on tasks [6, 12, 23, 28, 43, 52] . TryOnDiffusion [52] introduces a try-on framework with parallel U-Nets to handle garment warping and person blending simultaneously. Despite its promising performance, this approach is computationally intensive due to the requirement for multiple diffusion models in concatenation. DCI-VTON [12] and LaDI-VTON [28] treat virtual try-on as an exemplar-based inpainting task, and finetunes the pre-trained inpainting diffusion model on virtual try-on datasets. StableVITON [23] eliminates the warping module and employs zero cross-attention blocks to learn semantic correspondence between person and garment in an end-to-end manner. However, it is still difficult for these methods to preserve detailed textures of the garment. Our method employs a pre-trained DINOv2 [30] encoder to provide global semantic features and a parallel U-Net-based garment encoder to extract local texture features of the garment, which are integrated into the SCA and JSA modules of the main U-Net, leading to a better preservation of garment details.Compared with image virtual try-on, video virtual try-on [8, 21, 51] offers a more user-friendly and natural try-on experience. FW-GAN [8] enhances spatiotemporal smoothing in generated videos by incorporating optical flow with a warping net for person and garment manipulation. MV-TON [51] proposes a memory refinement operation to improve the details of initially generated results by referring to the previous frames. ClothFormer [21] adopts Transformer [39] architecture and proposes a new warping module and tracking module to warp the garment with temporal consistency. However, previous methods commonly utilize GAN-based frameworks and require separate warping modules, leading to a complex processing workflow. In this work, we explore the capability of diffusion models in video virtual try-on. We forgo the standalone warping module and enable the model to implicitly learn fine-grained spatial correspondence between the person and garment within the JSA module.Given the notable advancements of diffusion models in image generation [27, 31-33, 37, 45, 48] , numerous studies have extended text-to-image (T2I) diffusion models to video-related tasks, including text-to-video (T2V) [2, 7, 9, 14] , image-to-video (I2V) [20, 44, 49] and video editing [4, 41, 53] . AnimateDiff [14] introduces a plugand-play motion module that can be seamlessly integrated into personalized T2I models, forming an animation generator. Pixel-Dance [49] proposes straightforward modifications on the T2V U-Net, incorporating the latent of image condition with input noise to facilitate I2V generation. Tune-A-Video [41] enhances self-attention layers to enable each frame to attend to both the previous and first frames. Cut-and-Paste [53] integrates an extra reference image with the text condition to achieve more precise and fine-grained video editing. As a specific case of video editing, video virtual try-on necessitates maintaining the temporal consistency of the video while ensuring the fidelity to the garment. Our proposed GTA module incorporates features of the garments into temporal attention mechanisms, enabling the model to simultaneously address temporal continuity and garment consistency. 
method 3.1 preliminary: stable diffusion: 
 Our proposed GPD-VVTO builds upon Stable Diffusion (SD), an advanced image generation model derived from Latent Diffusion Model (LDM) [32] . LDM operates the diffusion process in the latent space to mitigate computational complexity.During training, a latent encoder [25, 38] compresses the input image x into a latent code z 0 = E (x) using an encoder E. SD performs forward diffusion process by adding Gaussian noise to the latent:EQUATIONwhere 𝑡 stands for the number of diffusion timesteps and {𝛼 𝑖 } 𝑡 𝑖=1 control the diffusion schedule. The denoising U-Net is trained to reverse this process by predicting the added noise. The optimization objective of this process is expressed as:EQUATIONwhere 𝜖 𝜃 represents the function of U-Net, 𝑡 stands for the number of diffusion timesteps and 𝑐 is the embedding of conditional information. In the original SD, conditional text is embedded using a CLIP-based transformer. Each typical block in denoising U-Net contains three types of computation: 2D convolution, self-attention and cross-attention with conditional text embedding. During inference, z 𝑡 is randomly sampled from a Gaussian distribution, and progressively denoised to obtain z 0 following a predefined sampling schedule [17, 36] . Finally, a latent decoder D decodes the result back into the image space.An overview of GPD-VVTO is illustrated in Figure 2 (a) . Given a segment of human video {x} 𝑇 1 = {x 1 , ..., x 𝑇 } ∈ R 𝑇 ×3×𝐻 ×𝑊 and a garment image c ∈ R 3×𝐻 ×𝑊 , where 𝐻 and 𝑊 denote the height and width of each frame and 𝑇 denotes the length of the segment, GPD-VVTO aims to synthesis a realistic and natural video sequence {x} 𝑇 1 ∈ R 𝑇 ×3×𝐻 ×𝑊 that presents the person in {x} 𝑇 1 wearing the garment c.GPD-VVTO contains two U-Net branches. On the one hand, the person video {x} 𝑇 1 and the cloth-agnostic video {a} 𝑇 1 are encoded to the latent space by the encoder of VAE E. A Gaussian noise is then added to E ({x} 𝑇 1 ) to form the noisy latent {z} 𝑇 1 ∈ R 𝑇 ×4×ℎ×𝑤 , where ℎ = 𝐻 /8 and 𝑤 = 𝑊 /8. The main U-Net is an inpainting network that takes a 9-channel tensor as input, with 4 channels of the noisy latent of the video, 4 channels of the latent of the clothagnostic video and 1 channel of the binary agnostic mask sequence. On the other hand, the garment encoder is a standard text-to-image diffusion model with the latent of the garment E (c) ∈ R 4×ℎ×𝑤 as input. Both the main U-Net and the garment encoder receive the embedding of garment image extracted by the DINO encoder for cross-attention. To further enhance the preservation of the body shape and pose of the person, the DensePose sequence {p} 𝑇 1 is embedded by a zero-initialized lightweight fully convolutional pose encoder and incorporated into the main U-Net via element-wise addition.To construct the cloth-agnostic video {a} 𝑇 1 and agnostic mask sequence {m} 𝑇 1 , we first extract the human parsing results using Graphonomy [10] in a frame-by-frame manner, and mask out regions related to the garment (e.g., clothes, coat, and body skin for upper-body garments). Subsequently, we utilize DWPose [47] results to mask the arms, leaving the hands exposed for better realism of the generation results. Finally, we take the bounding rectangle of the mask to eliminate the influence of the original garment style on the results. Areas unrelated to clothing are left exposed to better preserve the identity of the person. We utilize [13] to extract the DensePose sequence frame by frame as the input of pose encoder.In the original SD, cross-attention is performed using text embeddings extracted by CLIP text encoder. To better adapt to the virtual try-on task, we replace these embeddings with features extracted from the garment image by DINOv2 [30] , which contain rich global semantic information (e.g., style). Previous works [23, 46] only utilize the last hidden state for all cross-attention modules. However, since different blocks in the U-Net have varying degrees of spatial dimension compression, semantic features of different abstraction levels are required.To make full use of semantic features of different abstraction levels, we propose a Semantic-enhanced Cross-Attention (SCA) module that applies multiple intermediate features from the DINO encoder to different blocks of the two U-Net branches. Specifically, we define a mapping function 𝜎 : {1, ..., 𝑁 } → {1, ..., 𝑀 }, where 𝑁 denotes the number of cross-attention modules in the U-Net, 𝑀 denotes the number of layers in DINO encoder, and 𝜎 (𝑖) represents which layer of semantic features should be utilized in the 𝑖-th cross-attention module. The operations in the SCA module can be expressed as follow:EQUATIONwhere f𝑖 stands for the feature map before the 𝑖-th cross-attention module in the U-Net, s denotes the semantic features extracted by DINO encoder, and 𝜙 𝑖 𝑞 , 𝜙 𝑖 𝑘 , 𝜙 𝑖 𝑣 are linear projection layers for queries, keys and values respectively. By incorporating semantic features from different abstraction levels into corresponding blocks, the model can better preserve the overall style of the garment.Although injecting global semantic information of the garment helps preserve the overall style, the features extracted by the DINO encoder do not attach importance to local texture features, resulting in the loss of many local details (e.g., text, lines and patterns). Hence, we leverage a parallel garment encoder to extract multi-level dense features of the garment, which are integrated with the features from main U-Net to perform self-attention in the Joint Self-Attention (JSA) modules. To ensure compatibility between garment features and human features, the garment encoder adopts the same architecture as the main U-Net, but is initialized using the weights of pre-trained text-to-image Stable Diffusion model. In each JSA module, the human feature map from the main U-Net f 𝑖 is concatenated with the corresponding garment feature map from the garment encoder g 𝑖 to jointly compute self-attention. Concretely, the 𝑖-th JSA module in the main U-Net is computed as follow:EQUATIONwhere h 𝑖 = Concat(f 𝑖 , g 𝑖 ) is the concatenation of human feature map and garment feature map in the spatial dimension, and 𝜓 𝑖 𝑞 , 𝜓 𝑖 𝑘 , 𝜓 𝑖 𝑣 are linear projections. Afterwards, only the half from the main U-Net undergoes further computation. By integrating the dense garment feature maps abound of local texture information into self-attention modules, the main U-Net can selectively extract local features from the garment feature map, achieving the effective transfer of detailed garment appearance. Furthermore, through the joint self-attention of human and garment features, we establish a soft spatial correspondence between the human image and the garment image, implicitly deforming the garment to fit the pose of the person more naturally.Both the SCA and JSA modules focus on the realism and fidelity of individual frame results without establishing correlations between frames, thus unable to address the issue of temporal inconsistency. To efficiently transform an image generation model into a video generation model, AnimateDiff [14] proposes a plug-and-play motion module that involves temporal attention to improve the smoothness of generated videos. However, since other parameters of the model are not updated during the training of motion modules, there is a stagnation or even degradation in the quality of single-frame results. Moreover, fine-tuning the entire model on video data requires abundant computational resources and time.To overcome this issue, we devise a novel Garment-aware Temporal Attention (GTA) based on the motion module, which preserves fidelity to the garment by referencing garment features during the temporal modeling process. As illustrated in the right branch of Figure 2 (b), we conduct an extra spatial cross-attention in parallel with the JSA module, with the human features from the main U-Net as queries and the garment features from the garment encoder as keys and values:EQUATIONIt is worth noting that the feature maps used in this operation (i.e., f 𝑖 and g 𝑖 ), as well as the weights of the linear projection layers (i.e., 𝜓 𝑖 𝑞 , 𝜓 𝑖 𝑘 and 𝜓 𝑖 𝑣 ), are the same as those in the JSA module. Therefore, the learned spatial correspondence between the person and the garment in the JSA module remains effective. In this way, the obtained feature map ĝ𝑖 represents the garment features corresponding to each spatial position in the human image, and it is spatially aligned with the human feature map f 𝑖 . Subsequently, the human feature maps { f𝑖 } 𝑇 1 are concatenated with { ĝ𝑖 } 𝑇 1 in the temporal dimension to jointly compute temporal self-attention. Similar to the JSA module, only the sequence from the main U-Net is retained for further operations. By incorporating aligned garment features into the temporal attention, we simultaneously enhance the temporal continuity of the video and the quality of individual frames.Training scheme. The training scheme of GPD-VVTO comprises two stages:In the first stage, we take single frames as input and initialize the main U-Net as a 2D inpainting model, improving the model's ability of generating high-quality try-on images. All parameters except for the pre-trained DINO encoder are updated. To further align basic features of the results, we compute perceptual loss [22] utilizing a pre-trained VGG [35] model. The total optimization objective of the first stage is expressed as follow:EQUATIONwhere ẑ| 𝑡 = Concat(z| 𝑡 , m, a) is the input of the main U-Net at timestep 𝑡 and 𝜆 is the weight of the perceptual loss.In the second stage, we fine-tune the model on video data by inflating the 2D convolutions to spatial-only pseudo-3D convolutions [14, 19] , and insert GTA modules to enhance the temporal consistency. We only update the parameters of GTA modules and linear layers that project the features extracted by DINO encoder to the same dimensions as the U-Nets. The total optimization objective of the second stage is expressed as follow:EQUATION)Classifier-free guidance. We apply classifier-free guidance [18] on the garment image during inference to obtain more robust results. Specifically, we randomly set c = Ø during the whole training phase, with all elements in Ø ∈ R 3×𝐻 ×𝑊 equal to zero. In this way, the model is trained on both conditional and unconditional settings. During inference, we utilize a scalar guidance scale 𝑠 𝑔 ≥ 1 to combine the conditional and unconditional results. For instance, the final noise predicted by the model in the first stage is as below:ε𝜃 (ẑ| 𝑡 , p, c, 𝑡) = 𝑠 𝑔 • 𝜖 𝜃 (ẑ| 𝑡 , p, c, 𝑡) + (1 -𝑠 𝑔 ) • 𝜖 𝜃 (ẑ| 𝑡 , p, Ø, 𝑡). (8)Inference with sliding windows. To enhance the temporal coherence and smoothness for long videos, we follow [44] to adopt  
experiments 4.1 datasets: 
 Video Virtual Try-On. We conduct the experiments of video virtual try-on using VVT dataset [8] and our collected dataset.The VVT dataset contains 791 video clips with a resolution of 256 × 192. Following previous works [8, 21] , we partition it into a training set of 661 clips and a test set of 130 clips. However, the VVT dataset lacks diversity in terms of video backgrounds and human actions, leading to significant disparities from real-world scenarios. Hence, we collect a video virtual try-on dataset from well-known e-commerce platforms, which contains high-resolution videos of numerous models performing various actions in diverse scenes, together with the corresponding garment image. Moreover, our dataset possesses a greater diversity of garment styles and more intricate patterns on the garments compared to VVT dataset. Visual comparisons between VVT dataset and our dataset are illustrated in the supplementary materials. Our dataset contains 3, 156/1, 295/7, 631 pairs of human video clips and upperbody garment/lower-body garment/dress images. We randomly split the dataset into training and testing sets according to a ratio of 0.85 : 0.15 for each garment category.Image-based Virtual Try-On. To further validate the effectiveness of our method, we also conduct experiments on two public image-based virtual try-on datasets (i.e., VITON-HD [5] and Dress Code [29] ) using our first-stage model. The VITON-HD dataset contains 13, 679 pairs of upper-body model and garment images, with 2, 032 pairs utilized for testing. The Dress Code dataset contains 15, 363/8, 951/2, 947 pairs of full-body model and upper-body garment/lower-body garment/dress images, with 1, 800 pairs of each category utilized for testing.We initialize the main U-Net with pre-trained Stable Diffusion 2 inpainting 1 model and initialize the garment encoder with Stable 1 https://huggingface.co/stabilityai/stable-diffusion-2-inpainting Diffusion v2.1 [32] . The pose encoder consists of three convolutional blocks, which down-sample the input DensePose to match the latent's spatial shape. The weights of its final layer are initialized to zero following previous works [20] . We apply data augmentation with random horizontal flipping at a probability of 0.5 to enhance the robustness of the model. For image-based virtual try-on task, our model is trained at a resolution of 512 × 384 and 1024 × 768 separately, while for video virtual try-on task, we only train at 512 × 384 resolution. We adopt Adam [24] optimizer with a constant learning rate of 5 × 10 -5 . All the experiments are conducted using 16 NVIDIA A100 GPUs. In the first stage, we utilize a batch size of 64 for high-resolution and 256 for low-resolution models. In the second stage, we sample 16-frame sequences and set batch size to 64. The model is trained for 20, 000 iterations during the first stage and 10, 000 iterations during the second stage. The weight of perceptual loss 𝜆 is set to 10 -3 . During inference, we set the guidance scale 𝑠 𝑔 = 1.5. We utilize the DDIM [36] sampler, sampling on a single NVIDIA A100 GPU for 50 steps.Video Virtual Try-On. We evaluate the performance of video virtual try-on in two test settings. In the paired setting, the model is provided with both a person video and their original garment for reconstruction, while the unpaired setting involves substituting the garment of a person video with a different garment. Following previous work [21] , for the paired setting, we utilize two image-level metrics, SSIM [40] and LPIPS [50] , to measure the model's ability of reconstructing original videos. For the unpaired setting, we employ the Video Fréchet Inception Distance (VFID) by replacing the 2D image backbone in FID [16] with a 3D backbone I3D [3] to measure the realism and fidelity of generated videos.Table 1 presents the quantitative results of video virtual try-on task on VVT [8] and our dataset. To validate the effectiveness of our models of both two stages, and highlight the limitations of imagebased try-on methods in video try-on task, we compare our method with both image-based virtual try-on and video virtual try-on methods. Our first-stage model surpasses the other two image-based VTON methods and even outperforms video virtual try-on methods, demonstrating its capability to generate high-quality single-frame images. After training in the second stage, our final model surpasses all other methods in terms of all metrics on both datasets. Compared Image-based Virtual Try-On. To further demonstrate the effectiveness of our proposed method, we also evaluate our first-stage model trained at a resolution of 512 × 384 on image-based virtual try-on task. Similar to video virtual try-on, we evaluate the performance of image-based virtual try-on under the paired and unpaired settings. We utilize SSIM [40] and LPIPS [50] metrics for the paired setting, and FID [16] and KID [1] for the unpaired setting.We compare with both GAN-based and diffusion-based methods. Table 2 displays the qualitative results on the VITON-HD [15] dataset. Our proposed GPD-VVTO achieves the best performance across all metrics, demonstrating that our method can generate realistic and natural images while preserving the structure of original image. We notice that the FID score of diffusion-based methods is generally lower than that of GAN-based methods, indicating that diffusion models can generate images with more fidelity. We also conduct experiments on the Dress Code [29] dataset, which contains a greater variety of garment types and presents more challenging scenarios. The results are shown in Table 3 . By adjusting the mask strategy, our method can be applied to garments of different categories. Our method outperforms previous methods by a large margin in all the garment categories as well as overall evaluation (-2.57 FID score and -1.88 KID score compared with the second best result), showcasing its capability to handle complex cases.Video Virtual Try-On. The left panel of Figure 1 illustrates the video virtual try-on results of our method on the VVT dataset and our collected dataset. The videos in the VVT dataset primarily feature a model walking towards and away from the camera. Our generated results maintain both the fidelity to the target garment and the consistency between frames as the model walks and turns. In our dataset, the video backgrounds and human poses are more complex. Despite this, our generated results still manage to maintain consistency in both background and garment, demonstrating the robustness of our method. More video results are available in the supplementary materials. Image-based Virtual Try-On. The right panel of Figure 1 illustrates the image-based virtual try-on results of our method on VITON-HD and Dress Code datasets. Our approach is capable of preserving highly intricate patterns and text on garments, and demonstrates strong robustness in challenging human pose scenarios. We also conduct qualitative comparisons between previous and our method. The qualitative comparison on VITON-HD dataset is shown in Figure 3 . In the first row, GP-VTON [42] and DCI-VTON [12] employ separate warping modules, resulting in unnatural blending between the garment and the person. StableVI-TON [23] achieves a natural fitting, but fails to preserve the text on the garment. In contrast, our method ensures the seamless blending between the garment and the person while retaining the text on the garment. Moreover, our method restores the fabric of the garment more faithfully. In the second row, the target garment is an offshoulder top. While GP-VTON and DCI-VTON manage to preserve the appearance of the garment, they exhibit artifacts around the arms and neck. StableVITON changes the style of the target garment and introduces color discrepancies. In contrast, our method preserves the style of the garment while retaining its appearance. Similar phenomenons can be observed in the qualitative comparison on Dress Code dataset as shown in Figure 4 . GP-VTON [42] and LaDI-VTON [28] fail to preserve the local texture or overall style of the target garment and exhibit artifacts in the blending of the person and the garment. On the contrary, our method excels in both the naturalness and fidelity of the garment.Different components of GPD-VVTO. The quantitative results of ablating different components in GPD-VVTO on the VVT dataset are reported in Table 4 . In the 1-st row, we replace the garment features used in the cross-attention modules with the final output of the DINO encoder instead of multiple intermediate features.The performance exhibits some decline compared with our full model, indicating that the SCA modules enhance the quality of the generated results by leveraging semantic features from different abstraction levels. In the 2-nd row, we discard the garment encoder and substitute the JSA modules with standard self-attention. We observe a significant decrease in the model's performance, demonstrating the crucial impact of multi-scale dense garment features on the try-on results. In the 3-rd row, we replace the GTA modules with normal temporal attention without referring to garment features. The LPIPS and VFID scores remain relatively consistent with the full model, but there is a notable discrepancy in SSIM, indicating that concatenating garment feature sequences in the GTA module benefits the preservation of garment information. Overall, the modules proposed in our method enhance the fidelity to the garment and temporal consistency of the generated results. Effects of hyperparameters. We also investigate the effects of hyperparameters on the performance of our model, including the guidance scale 𝑠 𝑔 and the weight of perceptual loss 𝜆. Table 5 (a) presents the ablation study on 𝑠 𝑔 . When 𝑠 𝑔 = 1, the classifier-free guidance is not activated, resulting in relatively inferior performance. As 𝑠 𝑔 increases, the performance of the model first improves and then declines. We find that a larger guidance scale can preserve more complete patterns, but there is a significant color discrepancy of the garment. Taking into account the overall performance of the model under different values of 𝑠 𝑔 , we set 𝑠 𝑔 = 1.5 according to experimental results. Table 5 (b) shows the effects of changing the value of 𝜆. As 𝜆 becomes larger, the weight of the perceptual loss gradually increases. We observe that the model's performance first improves and then declines. This may be because when 𝜆 is small, the perceptual loss has little effect; and when 𝜆 is large, the update of model's parameters relies more on the perceptual loss rather than the MSE loss, leading to a degradation of the quality of generated results. Given the ablation results, we set 𝜆 = 10 -3 in our experiments. 
conclusion: 
 In this paper, we focus on the task of video virtual try-on and propose a LDM-based two-stage framework for the first time. Our method leverages both global semantic features and local texture features of the garment to preserve garment details as much as possible. We devise a novel Garment-aware Temporal Attention module that integrates features of the garment into temporal attention, improving the temporal consistency of the generated videos while maintaining the fidelity to the garment. Furthermore, we collect an in-the-wild video virtual try-on dataset to address the limited variety in terms of video background and human actions of current datasets. Extensive experimental results demonstrate that our method outperforms previous state-of-the-art methods on both image-based and video-based virtual try-on datasets, indicating the effectiveness of our proposed framework.  
: 
 the specific person wearing the provided garment. With the rise of e-commerce, this technology has garnered widespread attention in the industry, offering consumers an immersive and interactive online shopping experience.Previous virtual try-on methods [6, 12, 15, 23, 26, 28, 34, 42, 43, 52] typically focus on image-based operations, employing Generative Adversarial Networks [11] (GANs) or Diffusion Models [17] as the foundational architecture. For instance, GP-VTON [42] warps local garment parts individually and assembles them via global parsing. StableVITON [23] introduces zero cross-attention modules to learn the semantic correspondence between the human and the garment. However, these methods still struggle to balance the naturalness of person-garment blending and the fidelity to the garment's appearance. Moreover, the experience provided by imagebased virtual try-on is far less immersive for consumers compared to video virtual try-on. Even though we can directly apply these image-based methods to video virtual try-on with a frame-by-frame method, this often leads to discontinuity in the generated videos due to inconsistencies between frames.Several efforts have been made in the domain of video virtual try-on. FW-GAN [8] integrates optical flow prediction to warp the preceding frames for subsequent frame generation. MV-TON [51] introduces a memory refinement module that aggregates spatiotemporal information of multiple frames to enhance the generated frames. ClothFormer [21] proposes a novel warping and tracking module to ensure temporal consistency in garment warping. Despite the improvements in the spatiotemporal consistency of generated videos, these methods rely on GAN-based frameworks and necessitate separate warping modules, resulting in poor realism of generated videos and limited generalization of the models.In light of previous research, we posit that the task of video virtual try-on can be decomposed into two key aspects. First, every single frame in the generated video is realistic and natural, while keeping the background regions unchanged and preserving the details of the garment as much as possible. Second, the actions of the person and the appearance of the garment are consistent and coherent throughout the entire video.To address the aforementioned two aspects, we propose a novel two-stage video virtual try-on method based on Latent Diffusion Model [32] , namely Garment-Preserving Diffusion for Video Virtual Try-On (GPD-VVTO). In the first stage, the model is trained on image-based virtual try-on task to ensure the high quality of individual frames. To retain more garment details, we extract both high-level semantic features and low-level texture features of the garment and inject them into the denoising network. On the one hand, global semantic features of the garment at different abstraction levels are extracted by a pre-trained DINOv2 [30] encoder, which are utilized to conduct cross-attention with feature maps of different spatial resolution in our proposed Semantic-enhanced Cross-Attention (SCA) modules. On the other hand, we employ a U-Net-based garment encoder to extract multi-level dense features of the garment. The obtained garment features are incorporated into the main U-Net within our proposed Joint Spatial Attention (JSA) modules and jointly perform self-attention with the features of the main U-Net. Through the combination of global and local garment information, the model is capable of generating realistic try-on images while preserving the overall style and fine-grained details of the garment. Furthermore, the JSA module implicitly establishes a soft spatial correspondence between the human and the garment, facilitating their natural blending. In the second stage, the model is fine-tuned on video data to enhance the temporal consistency of the generated videos with additional temporal modules. To maintain the fidelity to the garment during temporal modeling, we devise a novel Garment-aware Temporal Attention (GTA) module, which leverages features extracted from the target garment to perform temporal attention. Specifically, we introduce an extra spatial cross-attention operation that shares weights with the JSA module, while the human video features serve as queries, and the garment features serve as keys and values. Therefore, the person-garment correspondence established by the JSA module enables the resulting feature maps to represent the garment features spatially aligned to the human video. Subsequently, we append these features to the feature sequence of the human video in the temporal dimension for temporal self-attention. In this way, the model can not only improve the continuity of the generated videos by referencing features from preceding and subsequent frames, but also preserve more garment details by leveraging features from the garment encoder.In addition, current video virtual try-on datasets [8] lack diversity in terms of video backgrounds and human actions, diverging significantly from real-world scenarios. To address the limitation of data, we collect a high-resolution video virtual try-on dataset from well-known e-commerce platforms, which contains videos of numerous models performing various actions in diverse scenes and the corresponding garment images. Extensive experiments demonstrate that our method outperforms previous state-of-theart methods on both public and our collected datasets. We also evaluate our model of the first stage on two image-based virtual try-on benchmarks [5, 29] . Our model surpasses existing imagebased virtual try-on methods in all metrics, further validating the effectiveness of our proposed modules.In summary, our contributions are as follows:• We explore the capability of diffusion models in video virtual try-on for the first time, and propose a two-stage framework that leverages both global semantic features and local texture features of the garment to preserve garment details. • We devise a novel Garment-aware Temporal Attention module that integrates features of the garment into temporal attention, improving the temporal consistency of the generated videos while maintaining the fidelity to the garment. • We collect a video virtual try-on dataset with more diversity to resolve the limited variety of current datasets in terms of video background and human actions. Extensive experiments demonstrate that our method outperforms previous stateof-the-art methods on both image-based and video-based virtual try-on datasets. 
 
 Appendix:  


Decision: The paper proposes a Latent Diffusion-based method for virtual try-ons for videos.  The goal is to accurately and realistically transfer garments onto a person in a video.  The method is based on a two-stage approach, first focusing on a single-frame image to ensure high quality and then extending it to video data to maintain temporal consistency.
The method is scientifically sound, effectively exploring the capabilities of diffusion models for virtual try-on.  The experimental setup is well-designed, and the results and ablation studies show that the method outperforms previous SOTA methods (including a concurrent one, IDM-VTON).  Moreover, the authors plan to make the dataset available, which can, which can significantly aid the community in further exploring this important topic.  Some of the reviewers raised concerns, in particular, in terms of novelty compared to the concurrent work IDM-VTON.  During the rebuttal, the authors provided more information on the dataset collection, further compared its method with IDM-VTON, and provided additional quantitative results, showing that their method outperforms additional baselines.

Given the importance of the work, the perfect fit for the conference, and the technical contribution together with the release of the dataset, the recommendation is to 'accept' the paper.

Reviews: review: paper_summary: paper_summary: This paper presents a new method for video virtual try-on called Garment-Preserving Diffusion for Video Virtual Try-On (GPD-VVTO). The method tackles two key issues in Video Virtual Try-On by a two-stage framework based on a latent diffusion model: the realistic naturalness of the single-frame results and the consistency of the character's movements with the clothing throughout the video. In the first stage, the model is trained on single-frame data to generate high-quality fitting images and maintains clothing details by integrating low-level texture features and high-level semantic features. In the second phase, the model is trained on video data to enhance temporal consistency, and a new garment-aware temporal attention module is introduced to maintain garment coherence. In addition, the authors collected a high-resolution video virtual fitting dataset containing multiple scenarios. Experimental results show that the method outperforms existing state-of-the-art methods in both image and video virtual fitting tasks.
strength_weakness: strength_weakness: 1. the paper is well expressed and can help the reader to understand it better.
2. the author clearly describes the methodology designed in the paper and the configuration of the experiment is described in great detail.
3. the ablation study of the paper is detailed and the key components mentioned in the paper are ablated. The work is experimentally adequate.
limitations: limitations: 1. The paper aims to accomplish the Video Virtual Try-On task, but lacks a comparative analysis of the processing frame rate with the baseline method. 
2. What are the significant performance and structural differences of the proposed method compared to IDM-VTON (Improving Diffusion Models for Virtual Try-on)? What are the different design concepts?
3. lack of comparison with more baseline methods on Video Virtual Try-On.
4. please analyse specifically the dataset you have collected.
5. Table 1 is missing key quantitative data for Video VTON on its own dataset.
6. the paper aims to solve the Video Virtual Try-On task, but does not show the crucial visualisation results in the qualitative experimental part. Instead, the results presented in Figure 1 lack a comparison with the baseline.
7. The visualisation results corresponding to Table 4 are missing in the ablation experiments.
paper_summary: paper_summary: With the rise of e-commerce, video virtual try-on has garnered widespread attention in the industry, offering consumers an immersive and interactive online shopping experience. This work introduces an LDM-based two-stage framework, GPD-VVTO, to ensure realistic and coherent garment transfers in videos. By integrating garment features into the denoising network, this method preserves garment details and maintains temporal consistency throughout the video. Additionally, to address the lack of diversity in existing datasets, this method collects a video virtual try-on dataset from e-commerce platforms, containing high-resolution videos depicting diverse scenes and human actions, which is more beneficial for real-world applications. The proposed method outperforms state-of-the-art approaches in both image-based and video-based virtual try-on tasks.
strength_weakness: strength_weakness: -1. This work explores the capability of diffusion models in video virtual try-on, and proposes a two-stage framework that leverages both global semantic features and local texture features of the garment to preserve garment details.

-2. This work devises a novel Garment-aware Temporal Attention module that integrates features of the garment into temporal attention, improving the temporal consistency of the generated videos while maintaining fidelity to the garment.

-3. This work collects a video virtual try-on dataset with more diversity to resolve the limited variety of current datasets in terms of video background and human actions.
limitations: limitations: -1. From Figure 2, it appears that, apart from the JSA and GTA modules, this architecture is almost identical to  IDM-VTON [1] and OOTDiffusion [2]. What are the fundamental differences in principle? 

[1] Improving Diffusion Models for Virtual Try-on.

[2] OOTDiffusion: Outfitting Fusion based Latent Diffusion for Controllable Virtual Try-on.

-2. This paper uses existing methods such as [10][47][13] to preprocess the training samples. If these methods ([10][47][13]) generate unreliable results, would they negatively impact the outcomes? How should this be addressed?

-3. If $m$ incorrectly segments $x$, the resulting output will lose identity information. As shown in Figure 3, both GP-VTON and DCI-VTON faithfully preserve the shape of the hair, whereas the method in this paper is clearly at a disadvantage in this regard.

-4. How is perceptual loss $L_{𝑝𝑒𝑟𝑐𝑒𝑝𝑡𝑢𝑎l}$ applied? Please explain in detail.

-5. Is there a plan to open-source the collected dataset? If not, how can the effectiveness of this method be fairly validated?

-6. There is little information about the collected dataset. Please add an analysis.

-7. In Table 2, the values for the StableVITON method seem to differ from those in the original paper. Why does this discrepancy exist?

-8. Please add more baseline methods to the quantitative results.

-9. Please add continuous comparison results reflecting video virtual try-on in the main text, not just single-frame results.

-10. From Table 4, it can be seen that the performance w/o GTA is almost identical to the performance w/ GTA, indicating that GTA does not provide significant performance gains.

-11. Providing visualized ablation experiment results can help the authors intuitively understand the contributions of each module.
paper_summary: paper_summary: This paper focuses on improving video virtual try-on (VVT) technologies, where a garment from a still image is realistically animated onto a person in a video. The authors introduce a two-stage Latent Diffusion Model framework called Garment-Preserving Diffusion for Video Virtual Try-On (GPD-VVTO). This method first trains on single-frame images to ensure high-quality garment visualization while preserving details. It then extends to video data to enhance temporal consistency using a Garment-aware Temporal Attention module, which integrates garment features into the temporal modeling process.
strength_weakness: strength_weakness: 1.  The final experimental outcomes for GPD-VVTO demonstrate robust performance in both Image-based and Video Virtual Try-On comparisons.
2.  The GTA Module can improve the temporal consistency of the generated video and is corroborated by experimental results.
3.  This paper is well-organized and easy to follow.
limitations: limitations: 1.  In terms of novelty, IDM-VTON[1] have similarities with the SCA and JSA modules, which somewhat limit the novelty of this paper.
2.  Apart from the GTA module, the main differences between this paper and IDM-VTON lie in the use of the DINO Encoder (as opposed to IDM-VTON's IP-Adapter) and the addition of a Pose Encoder. Conducting ablation experiments on these modules will help verify their effectiveness.
3.  Table 3 lacks SSIM and LPIPS metrics for the three subcategories of the Dresscode dataset.
4.  The video virtual try-on dataset is one of the key contributions of this paper. Therefore, I would like to ask if there are plans to release the code and dataset, and if so, do you have a tentative date in mind?

[1] Choi Y, Kwak S, Lee K, et al. Improving Diffusion Models for Virtual Try-on[J]. arXiv preprint arXiv:2403.05139, 2024.
paper_summary: paper_summary: The paper focuses on enhancing the realism and consistency of virtual try-on technologies for videos using a novel two-stage framework based on Latent Diffusion Models (LDMs). The main goal is to accurately transfer garments onto a person in a video, maintaining high-quality single-frame results and ensuring temporal consistency throughout the video sequence. The paper involves a two-stage approach. The first stage focuses on generating high-quality images for individual frames using a diffusion model that integrates both low-level texture and high-level semantic features of garments. The second stage enhances the model with video data to maintain temporal consistency using a novel Garment-aware Temporal Attention (GTA) mechanism. The paper also highlights the creation of a new video virtual try-on dataset that features diverse backgrounds and human actions to better reflect real-world applications. Extensive experiments show that this method outperforms existing state-of-the-art techniques in both image-based and video-based virtual try-on tasks.
strength_weakness: strength_weakness: 1. The approach introduces a two-stage framework utilizing Latent Diffusion Models (LDMs) specifically tailored for video virtual try-on applications. This is a novel adaptation as previous works predominantly focus on static image-based virtual try-on. 
2. The introduction of the Garment-aware Temporal Attention (GTA) module is particularly innovative, addressing the challenge of maintaining garment fidelity throughout the motion in videos, a feature not extensively explored in prior research.
3. The video try-on data set proposed in this paper makes up for the lack of data in this field.
limitations: limitations: 1.	The author's representation of the attention mechanism using Attn() in equations (3) to (5) is not sufficiently formalized. It is advisable to either explicitly detail the attention calculation formula or define Attn() precisely, e.g., Attn(Q,K,V) = ...
2.	The manuscript omits a discussion on the methodological limitations. For robustness, performance analysis under atypical scenarios, such as individuals wearing unconventional clothing, should be included.
3.	The performance evaluation of the algorithm relies on LPIPS and VFID metrics to assess temporal consistency, which are primarily designed to evaluate consistency in video frame quality. The paper lacks an assessment of motion coherency. Given the assertion in line 152 regarding the consistency and coherence of actions and garment appearance throughout the video, incorporating a metric for evaluating coherence, such as dtSSD[1], is recommended.
4.	The documentation concerning the dataset used is inadequate. The authors need to provide additional details about the data collection and preprocessing procedures, selection criteria, data filtering, handling of personally identifiable information, ethical considerations, dataset accessibility (public or private), and other pertinent information.

[1] M. Erofeev, Yury Gitman, D. Vatolin, Alexey Fedorov, and J. Wang. Perceptually motivated benchmark for video matting. In BMVC, 2015.