Enhancing Virtual Try-On with Synthetic Pairs and Error-Aware Noise Scheduling

Authors: Nannan Li, Kevin J. Shih, Bryan A. Plummer
Primary area: generative models
Keywords: Virtual Try-On, Human Image Generation, Diffusion Models

Abstract:
Given an isolated garment image in a canonical product view and a separate image of a person, the virtual try-on task aims to generate a new image of the person wearing the target garment. Prior virtual try-on works face two major challenges in achieving this goal: a) the paired (human, garment) training data has limited availability; b) generating textures on the human that perfectly match that of the prompted garment is difficult, often resulting in distorted text and faded textures.  Our work addresses these issues through a dual approach. First, we introduce a garment extraction model that generates (human, synthetic garment) pairs from a single image of a clothed individual. The synthetic pairs can then be used to augment the training of virtual try-on. Second, we propose an Error-Aware Refinement-based Schr\"odinger Bridge (EARSB) that surgically targets localized generation errors for correcting the output of a virtual try-on model. To identify likely errors, we propose a weakly-supervised error classifier that localizes regions for refinement, subsequently augmenting the Schr\"odinger Bridge's noise schedule with its confidence heatmap. Experiments on VITON-HD and DressCode-Upper demonstrate that our synthetic data augmentation enhances the performance of prior work, while EARSB improves the overall image quality. In user studies, our model is preferred by the users in an average of 59\% of cases.

Decision: reject
Reviewer scores: 5, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730029751777

thread_type:
official_review

<thread_object>
creator: reviewer_uiak

<content>
summary:
This paper introduces two methods to enhance virtual try-on performance. The first method utilizes a GAN-based approach to generate synthetic data for training try-on models, creating garment data from single human images drawn from multiple datasets. The second method focuses on training the try-on model itself, which again consists of two stages. The first stage is a GAN-based method that produces intermediate results, while the second stage is an image-to-image model based on I2SB, which refines the initial outputs. Unlike the naive I2SB approach, the proposed method incorporates an error map module to emphasize low-quality regions.

strengths:
1. The paper is well-structured and easy to follow, with clear presentation.
2. The proposed method outperforms several baseline methods in virtual try-on tasks.
3. The synthetic data generation approach could be beneficial for future research.

weaknesses:
1. The paper does not offer significant technical contributions, primarily combining existing methods, such as combining GAN outputs with I2SB for post-processing.
2. According to Table 3, combining existing methods with synthetic data sometimes yields better performance than the proposed method.
3. Minor formatting issues. For example, some equations (e.g., Eq 2 and Eq 8) are missing proper punctuation.
4. Inconsistency in terminology, with both "error-map" and "error map" used throughout the paper.

questions:
1. I feel the training process of EARSB is not clearly explained. How are data pairs prepared for this stage? What initialization is used for the U-Net, and what is the training loss?
2. In I2SB, training assumes that the posterior, given a boundary pair, follows a Gaussian distribution. Does WSC affect the Gaussian properties?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730365190517

thread_type:
official_review

<thread_object>
creator: reviewer_rzb7

<content>
summary:
This paper aims to address two main challenges in the virtual try-on field:

1. Difficulty in obtaining paired data (i.e., garment images and images of people wearing the corresponding garment) for use in datasets.
2. Existing virtual try-on models struggle to generate accurate texture details and patterns of the garment.

To tackle the first issue, this paper proposes a model that generates corresponding garment images based on images of people wearing the garment. The generated garment images and original people images are used to train virtual try-on models.

For the second issue, the paper employs a weakly supervised classifier to generate an error map, allowing the model to identify regions that need refinement, which is achieved using a Schrödinger bridge.

strengths:
1. Extensive experiments demonstrate the effectiveness of the method proposed in this paper.
2. The data augmentation approach introduced in this paper can be easily applied to other virtual try-on models, and this paper have verified this on both Stable-VITON and CAT-DM.
3. Compared to adding random noise directly to GAN-generated images in CAT-DM, using a Schrödinger bridge to refine erroneous areas appears to be more effective.

weaknesses:
1. The performance improvement of the model does not seem very significant; as shown in Table 3, Stable-VTON + H2G-UH appears to achieve better performance. Perhaps the single U-Net structure limits the model’s generative capability? However, as stated in this paper, SD-based methods like Stable-VTON use pretrained weights, so this is not a major issue.
2. The baselines compared in this paper are somewhat outdated; more recent virtual try-on models such as IDM-VTON [1], OOTDiffusion [2], and CATVTON [3] should perhaps be included in the comparison experiments.
3. The results of LaDI-VTON on VITON-HD and Stable-VITON on DressCode-Upper are missing in Table 1.
4. Although there are some visualized examples of the synthetic garment images, I am somewhat concerned about the quality of the generated images. Could you provide some quantitative result to support this?
5. Although this paper compares the results of different try-on GAN models for generating the initial image and demonstrates that EARSB can refine GAN-generated images, I am curious whether EARSB could have a similar effect on diffusion-generated images as well. 

[1] Choi Y, Kwak S, Lee K, et al. Improving diffusion models for virtual try-on[J]. arXiv preprint arXiv:2403.05139, 2024.

[2] Xu Y, Gu T, Chen W, et al. Ootdiffusion: Outfitting fusion based latent diffusion for controllable virtual try-on[J]. arXiv preprint arXiv:2403.01779, 2024.

[3] Chong Z, Dong X, Li H, et al. CatVTON: Concatenation Is All You Need for Virtual Try-On with Diffusion Models[J]. arXiv preprint arXiv:2407.15886, 2024.

questions:
1. To obtain the error map, this paper manually annotated 5% of the GAN-generated images. Could you analyze the impact of the annotation ratio on model performance?
2. I would like to confirm which dataset the human-to-garment model was trained on. Was it VITON-HD or DressCode?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730688447555

thread_type:
official_review

<thread_object>
creator: reviewer_jb7x

<content>
summary:
This paper explores virtual fitting techniques that aim to demonstrate the effect of a person wearing a specific garment by generating new images. The authors propose a garment extraction model that generates (person, synthetic garment) pairs from a single image of a wearing individual, which helps to address the problem of insufficient training data. An Error-Aware Refinement-based Schrodinger Bridge(EARSB) is proposed, which is capable of targeting and correcting local generation errors in the output of the virtual try-on model. This error-aware noise scheduling-based approach is an innovative technical tool.

strengths:
+ 1. The abstract and introduction sections clearly outline the motivation, methodology, and contributions of the study. The methodology section provides a detailed description of the technical details.

+ 2. The paper employs an advanced SB method for denoising training, which can better accomplish personalized design tailored for the VTON task.

+ 3. The author compares the performance of EARSB with existing methods and discusses the impact of synthetic data augmentation, which helps demonstrate the effectiveness of EARSB.

weaknesses:
- 1. The error map-reweighted diffusion process appears to be a relatively universal technique, but the author has not provided further introductions to its applications in this section. Therefore, there is no way to prove whether this method can be applied to other methods and bring about similar performance improvements.

- 2. More visualizations are needed to show how good regions are assigned less noise and bad regions are assigned more noise. Simply relying on Figure 4 does not adequately convey the precise meaning of this innovation.

- 3. The Diffusion-VTON series of methods lacks comparisons with some necessary approaches, such as quantitative and qualitative comparisons with IDM-VTON, DCI-VTON, and other methods.

- 4. The explanation of the EARSB model in the paper is not sufficient, especially regarding the details of how the model identifies and corrects local errors. Can the optimization of local details be achieved solely through weakly supervised masks?

- 5. The paper mentions that SD-based methods require large-scale pretraining, while EARSB does not. How does EARSB ensure the effectiveness and generalization ability of VTON in scenarios beyond the dataset?

-6. I did not find more introductions related to the code of the paper, which reduces the reproducibility of this paper.

questions:
Please refer to the "Weaknesses."
</content>
</thread_object>

</thread>
<thread>
creation date:
1730708070696

thread_type:
official_review

<thread_object>
creator: reviewer_d9nt

<content>
summary:
The authors propose a framework for improved generation of virtual try-on images by leveraging synthetic data and an error correction module. The synthetic data is created from images of clothed individuals by using a garment extraction module. These synthetic pairs are then used to as augmented data during training of the virtual try on module. The authors also propose error aware refinement based Schrödinger bridge that aim to correct localized errors in the generated image. State of the art performance is demonstrated with a 59% user preference for try-on results generated by the proposed approach.

strengths:
1. **Claity**: The paper is well written with adequate details and explanation provided for all design components. In particular, the specific details regarding the training of garment extraction network and error detection network is helpful to reproduce the approach. 
2. **Comparisions**: The framework shows state of the art quantitative performance compared to a number of recent baselines. The user study additionally strengths the claim of user preference and highlights the efficacy of the proposed approach. 
3. **Ablation**: A number of detailed ablations are provided to highlight the need and effect of data augmentation and the use of manually annotated data for training the error detection network. 
4. **Garment-Extraction network**: The idea of reducing the task of virtual try-on to a simpler problem by training a network to generate garments in canonical pose for an input dressed individual in any arbitrary pose is an interesting way of obtaining useful training data.
5. **Error-detection network**: Using manually annotated data to train a network to recognize error regions, and then using the detected error regions to encourage local refinement is an interesting idea. 
6. **Result Quality**: A number of qualitative results are provided on a wide variety of example that demonstrates that high frequency textures and details of lines/ logos are maintained better with the proposed approach.

weaknesses:
1. **Novelty**: Although the authors present an interesting approach to improve the quality of generated virtual try on images, the novelty is somewhat limited to the synthetic pair generation component. In particular, the other components such as $I^2SB$ have been previously introduced and have only been adapted to this framework. Highlighting the main novel components and differences from prior work would aid in addressing this concern.  
2. **Relevance**: Although virtual try-on represents an important problem in the space of computer vision, the findings in the paper are primarily described in context of improving quality for try on task. Briefly commenting about how some of these components can be adapted to other general settings would be helpful. In particular, the finding that 5% of generated images can be manually annotated to identify error prone areas and used to train a error prediction network is potentially useful in many pipelines that would use a refinement network.
3. **Details about the manual annotation**: The authors mention that 5% of generated images are manually annotated with error regions. Providing a small ablation showing the effect of training the network with lesser and more annotated data would be helpful to highlight the importance of this finding.
4. **Target clothing shape**: Although the generated images maintain the texture of the target clothing item, most times the generated image does not fully respect the shape of the target clothing item (eg. ll345-349 makes the dress longer than the product, ll328-333 makes the generated clothing shorter than the product. The results mostly appear like the texture of the target clothing is transferred in detail into the shape of the source posed image. Providing more examples of (short source, long target) or (long source, short target) pairs, would be helpful to understand its limitations and generalizability.

questions:
1. The authors use a GAN based model for the first stage and then use a Schrödinger bridge for the refinement network. What is the motivation behind using a GAN based base network as opposed to a diffusion based base network? (particularly in light of the fact that the next best performant networks are SD based). Providing a brief explanation regarding the choice of base network would be insightful.
2. How many total number of images are 5% of generated images?
</content>
</thread_object>

</thread>


Metareview: