GS-VTON: Controllable 3D Virtual Try-on with Gaussian Splatting

Authors: Yukang Cao, Masoud Hadi, Liang Pan, Ziwei Liu
Primary area: generative models
Keywords: 3D virtual try-on, 3D Gaussian Splatting, diffusion model

Abstract:
Diffusion-based 2D virtual try-on (VTON) techniques have recently demonstrated strong performance, while the development of 3D VTON has largely lagged behind. Despite recent advances in text-guided 3D scene editing, integrating 2D VTON into these pipelines to achieve vivid 3D VTON remains challenging. The reasons are twofold. First, text prompts cannot provide sufficient details in describing clothing. Second, 2D VTON results generated from different viewpoints of the same 3D scene lack coherence and spatial relationships, hence frequently leading to appearance inconsistencies and geometric distortions. To resolve these problems, we introduce an image-prompted 3D VTON method (dubbed GS-VTON) which, by leveraging 3D Gaussian Splatting (3DGS) as the 3D representation, enables the transfer of pre-trained knowledge from 2D VTON models to 3D while improving cross-view consistency. **(1)** Specifically, we propose a personalized diffusion model that utilizes low-rank adaptation (LoRA) fine-tuning to incorporate personalized information into pre-trained 2D VTON models. To achieve effective LoRA training, we introduce a reference-driven image editing approach that enables the simultaneous editing of multi-view images while ensuring consistency. **(2)** Furthermore, we propose a persona-aware 3DGS editing framework to facilitate effective editing while maintaining consistent cross-view appearance and high-quality 3D geometry. **(3)** Additionally, we have established a new 3D VTON benchmark, 3D-VTONBench, which facilitates comprehensive qualitative and quantitative 3D VTON evaluations. Through extensive experiments and comparative analyses with existing methods, the proposed GS-VTON has demonstrated superior fidelity and advanced editing capabilities, affirming its effectiveness for 3D VTON.

Decision: reject
Reviewer scores: 6, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730307893597

thread_type:
official_review

<thread_object>
creator: reviewer_w66j

<content>
summary:
This paper introduces an image-prompted 3D VTON method (dubbed GS-VTON) which, by leveraging 3D Gaussian Splatting (3DGS) as the 3D representation, enables the transfer of pre-trained knowledge from 2D VTON models to 3D while improving cross-view consistency. Specifically, they propose a personalized diffusion model that utilizes low-rank adaptation (LoRA) fine-tuning to incorporate personalized information into pre-trained 2D VTON models. Moreover, they introduce a reference-driven image editing approach that enables the simultaneous editing of multi-view images while ensuring consistency. Furthermore, they propose a persona-aware 3DGS editing framework to facilitate effective editing while maintaining consistent cross-view appearance and high-quality 3D geometry. Additionally, they proposed a new 3D VTON benchmark, 3D-VTONBench, which facilitates comprehensive qualitative and quantitative 3D VTON evaluations. The experiments demonstrate the superior fidelity and advanced editing capabilities of GS-VTON.

strengths:
1. The paper is well-written and easy to understand.
2. This paper introduces a new perspective on persona-aware editing which effectively improves the performance of the 3D VTON task.
3. The limitations of the proposed method are well-discussed.

weaknesses:
1. There are concerns about the contribution of the proposed module, as it mainly adopts LoRA and a pre-trained diffusion model. For example, the X_train is not explained clearly in Line 339. Does it only refer to the results of the pre-trained IDM-VTON model?
2. I noticed that the paper does not provide evaluation metrics (e.g., LPIPS, FID). Including these metrics would improve the evaluation of the GS-VTON method.
3. There are some writing errors: a) Line 31: "GS-VTONhas" is missing a space. b) Missing commas in Functions 7 and 10.

questions:
1. It is better to discuss the reproducibility.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730366486265

thread_type:
official_review

<thread_object>
creator: reviewer_bmem

<content>
summary:
The paper introduces a novel image-prompted 3D virtual try-on method that leverages 3D Gaussian Splatting for fine-grained editing of human garments within a 3D scene. The authors propose a personalized diffusion model adapted via LoRA fine-tuning to incorporate personalized information into pre-trained 2D VTON models. They also introduce a persona-aware 3DGS editing framework to maintain consistent cross-view appearance and high-quality 3D geometry. The paper establishes a new benchmark, 3D-VTONBench, for comprehensive 3D VTON evaluations and demonstrates through extensive experiments that the proposed GS-VTON method outperforms existing techniques in terms of fidelity and editing capabilities.

strengths:
* The paper presents a groundbreaking method for 3D virtual try-on by extending pre-trained 2D VTON models to 3D using 3DGS, addressing the challenge of cross-view consistency and spatial relationships in 3D scenes.
* The establishment of the 3D-VTONBench dataset is a valuable resource for the research community, facilitating more comprehensive evaluations and fostering further advancements in 3D VTON.
* The method demonstrates superior performance over existing techniques.

weaknesses:
* In some cases, such as the first row in Fig. 1, there are noticeable artifacts on the sleeves and edges of the garments.

* The statements about the effects of persona-aware 3DGS editing are inconsistent between the abstract and introduction. The abstract states "maintain consistent cross-view appearance," while the introduction says "enhancing multi-view consistency."
* What are the differences or advantages of your methods compared to RelFill in "Personalized Diffusion Model via LoRA fine-tuning"? Some experiments may be needed to demonstrate this.
* The hyperparameter in persona-aware 3DGS editing seems tricky.
* To demonstrate the ability to maintain 3D consistency, the following works should be discussed.

  * Geometry-Aware Score Distillation via 3D Consistent Noising and Gradient Consistency Modeling
  * MaTe3D: Mask-guided Text-based 3D-aware Portrait Editing
  * ConsistNet: Enforcing 3D Consistency for Multi-view Images Diffusion
* Need more detailed descriptions of the proposed benchmark.

questions:
* In line 110, does LoRA "extend its learned distribution"? Are there any citations?
* I need more details about ControlNet. Is it from the original repository?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730570668753

thread_type:
official_review

<thread_object>
creator: reviewer_2fdr

<content>
summary:
This study proposes GS-VTON, a novel and potentially first-of-its-kind 3D virtual try-on method, based on a conditional diffusion model and 3D Gaussian Splatting. Unlike traditional 2D virtual try-on methods, GS-VTON enables users to visualize how clothing would appear on their bodies from different viewing angles, making it particularly promising for VR/AR applications. Moreover, the proposed reference-driven image editing and persona-aware 3D Gaussian Splatting techniques improves multi-view consistency in the virtual try-on experience in 3D.

strengths:
1. GS-VTON is the first 3D virtual try-on method, showing more diverse real-world applications compared to 2D virtual try-on. It holds promising potential to transform online shopping and create positive social impact.
2. GS-VTON utilizes Reference-driven Image Editing and 3D Gaussian editing to ensure the try-on scene is consistent in both texture and geometry across multiple views. The design seems sound.

weaknesses:
1) **Reference-driven Image Editing**: The authors propose this method to ensure texture consistency across multi-view images by integrating attention features from a reference image. However, if the reference image has incorrect textures, it may negatively affect the consistency of subsequent images.

2) **Questionable Experimental Setting**: 
   - All benchmark methods use text as input, while GS-VTON uses an image as a prompt. However, the user study criterion of clothing image similarity may not be ideal for comparing these approaches.
   - Though the authors provide qualitative examples comparing GS-VTON with a baseline 2D VTON method, including this baseline in the user study for comprehensive quantitative analysis is essential. Most of the compared methods were not natively designed for virtual try-on applications, raising concerns about experimental fairness.

3) **Limited View Coverage**: GS-VTON mainly shows the front of the clothing, without displaying how the back of the body would appear.

4) **Pipeline Presentation Issue**: Figure 2 shows Reference-driven Image Editing as the first step, followed by the Personalized Diffusion Model via LoRA Fine-tuning. However, the main text introduces these components in reverse order, which caused some confusion initially.

questions:
1. Could the authors provide more details on how the $G_{src}$ point cloud was collected in Figure 2? Since RGB-D sensors or other 3D sensors are less accessible compared to standard cameras, this may limit the method's real-world applicability. A more practical scenario might involve capturing multi-view images with a camera, but in that case, the lack of camera pose information could limit the feasibility of training the 3D GS model.

2. Could the authors provide more information about the response processing in the user study? Was a crowdsourcing platform used for data collection?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731386486626

thread_type:
official_review

<thread_object>
creator: reviewer_23cd

<content>
summary:
This paper introduces a novel approach for achieving 3D virtual try-on (VTON) that addresses current limitations in consistency and spatial relationships when extending 2D VTON methods to 3D. The method, GS-VTON, leverages 3D Gaussian Splatting (3DGS) as a 3D representation framework, combined with personalized diffusion model adaptation using LoRA (Low-Rank Adaptation) fine-tuning. This allows for multi-view image editing with consistency across different viewpoints and high-quality geometric and texture fidelity. Additionally, the paper introduces a benchmark, 3D-VTONBench, to support quantitative and qualitative evaluations for 3D VTON methods. The experiments show that GS-VTON outperforms state-of-the-art techniques, establishing a new benchmark for 3D VTON performance.

strengths:
+ Effectively bridges the gap between 2D VTON and 3D applications by incorporating 3D Gaussian Splatting, which ensures consistency across multi-view images.
+ Uses a personalized diffusion model with LoRA fine-tuning, improving adaptability and customization for different subjects and garments.
+ Presents a new benchmark, 3D-VTONBench, which is an important addition for the comprehensive evaluation of 3D VTON performance.
They also demonstrates superior performance over existing methods, particularly in areas of realism, garment detail accuracy, and editing consistency.

weaknesses:
- The model is a very straightforward follow-up of 2D VTON models and inherits some biases from pre-trained 2D VTON models.

questions:
From line 62-68, is the pink stuff on the rightmost edited images because of the artifacts of the proposed method? 

Personally, I think this paper is a good follow-up on 2D virtual try-on method and will be beneficial to this research area. Extensive experiments and comparisons with state-of-the-art techniques demonstrate the superiority of GS-VTON in terms of realism and multi-view consistency. This is validated not only through quantitative metrics but also through user studies. Therefore,  I don't have any major concerns.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733067396546

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank all reviewers for their time and effort in reviewing our paper!

Our method presents the first method for text-guided 3D virtual try-on (VTON) methods via diffusion models. Our method proposes two major components to realize this goal:

**(1)** We observe that images offer richer and more precise information compared to text prompts. To this end, we propose to leverage low-rank adaptation (LoRA) to integrate image priors. Additionally, we introduce a reference-driven image editing technique that enables the simultaneous editing of multi-view images while maintaining consistency, thereby enhancing the training of the LoRA module.

**(2)** We propose a persona-aware 3DGS editing framework for 3D virtual try-on. This framework involves adjusting the attention module to ensure a consistent appearance across multiple views during the editing process.

**Below, we summarize the changes made according to the reviews:**

1. We explore the impact of biases inherited from the pre-trained diffusion model on our method. Furthermore, we discuss how fine-tuning the LoRA module in our framework can help mitigate and improve this scenario (`#23CD`).

2. We discuss the pink artifacts presented in Fig.1 and how we can address this issue (`#23CD`, `#bmEM`).

3. We discuss the definition of the reference image used in Eq. (6) and how our reference-drive image editing will positively improve the consistency of the subsequent images (`#2fdr`).

4. We conduct user studies to further compare with the baseline method and include the metric of text similarity. We also provide more discussion about the user studies. (`#2fdr`).

5. We analyze and discuss our data with only frontal views (`#2fdr`).

6. We update the pipeline figure to make it more consistent with the writing (`#2fdr`). 

7. We provide more details about how we obtain the 3D Gaussian points from multi-view input images (`#2fdr`).

8. We update the statement of the claims in the paper to make them more consistent (`#bmEM`).

9. We show comparisons with RealFill (`#bmEM`).

10. We analyze the hyper-parameters used in the persona-aware 3DGS editing (`#bmEM`).

11. We discuss more related works suggested by the reviewer to ensure comprehensive discussion(`#bmEM`).

12. We illustrate more details about the LoRA and ControlNet used in our framework (`#bmEM`).

13. We further clarify how we obtain the X_train via IDM-VTON (`#W66J`).

14. We conduct quantitative evaluations via CLIP to further demonstrate our method (`#W66J`).

15. We improve our writing to rectify the typos and discuss more about the implementation details (`#W66J`).

We sincerely thank all reviewers and the AC(s) again for their valuable suggestions, which have significantly contributed to enhancing the quality of our paper.

If you have any further questions, we would be happy to discuss them!
</content>
</thread_object>

</thread>


Metareview:
metareview:
This submission proposes a 3D virtual try-on method that leverages 3D Gaussian Splatting (3DGS) and diffusion model adaptation to address limitations in consistency and spatial relationships, present in existing 2D methods. The approach enables multi-view image editing towards improving consistency across different viewpoints. The paper also establishes a new benchmark, 3D-VTONBench, for evaluating 3D VTON techniques. Experimental work evidences that the introduced approach can outperform alternative methods in terms of performance and editing capabilities.

Due to the extent of wide-spread proposed changes, resulting from the rebuttal, the manuscript likely benefits from deeper edits (and polish; typographical errors still exist). After reviewing the paper, rebuttal and resulting discussion AC believes that this submission can be strengthened by further refinement and subsequent round of reviews and here recommends rejection. The data contribution is likely of value and may be well suited for a dedicated dataset benchmark track.