Title: VTON-VLLM: Aligning Virtual Try-On Models with Human Preferences
Venue: NeurIPS 2025
Accepted: yes
Keywords: virtual try-on
Abstract: Diffusion models have yielded remarkable success in virtual try-on (VTON) task, yet they often fall short of fully meeting user expectations regarding visual quality and detail preservation. To alleviate this issue, we curate a dataset of synthesized VTON images annotated with human judgments across multiple perceptual criteria. A vision large language model (VLLM), namely VTON-VLLM, is then learnt on these annotations. VTON-VLLM functions as a unified ``fashion expert'' and is capable of both evaluating and steering VTON synthesis towards human preferences. Technically, beyond serving as an automatic VTON evaluator, VTON-VLLM upgrades VTON model through two pivotal ways: (1) providing fine-grained supervisory signals during the training of a plug-and-play VTON refinement model, and (2) enabling adaptive and preference-aware test-time scaling at inference. To benchmark VTON models more holistically, we introduce VITON-Bench, a challenging test suite of complex try-on scenarios, and human-preference–aware metrics. Extensive experiments demonstrate that powering VTON models with our VTON-VLLM markedly enhances alignment with human preferences. Code is publicly available at: [https://github.com/HiDream-ai/VTON-VLLM/](https://github.com/HiDream-ai/VTON-VLLM/).


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper pointed our existing evaluate metrics e.g. FID, LPIPS has limited effectiveness in fine-grained visual attributes under Virtual Try-On task. Then it proposed a more human-preference-aligned VLM metric that can evaluate images and use as a reward signal in the Virtual Try-On task setting. In detail it trained a VLM called VTON-VLLM as a \"fashion expert\" which can provide feedback signal to improve existing VTON models performance and align them better with human preferences. The paper also curated VITON-Bench as a hard benchmark. Extensive experiments show the effectiveness of VTON-VLLM in improving existing Virtual Try-On models is demonstrated through comprehensive experiments and ablation studies, also the reliability of VTON-VLLM as a metric is supported by a preference agreement study.\n\nSTRENGTHS AND WEAKNESSES:\nS1) Virtual Try-On is an important topic with a significant gap between current research and commercial applications. The observation that existing evaluation metrics are unreliable in fine-grained visual attributes is well-motivated, and the use of an evaluation model to align Virtual Try-On models with human preferences strengthens the contribution of this work.\n\nS2) The refinement module (VRM-Instruct) can enhance existing VTON models, which is a strong practical contribution can be widely adapted.\n\nS3) The effectiveness of VTON-VLLM in improving existing Virtual Try-On models is convincingly demonstrated through comprehensive experiments and ablation studies.\n\nW1) I believe the paper has no major flaws. One potential weakness is that the dataset focuses exclusively on women, while Virtual Try-On is inherently a task that should generalize across all genders and diverse demographics. For example, the model's performance on male subjects, different body types, or various skin tones remains unclear. Evaluating its robustness in these scenarios would make the work more comprehensive. The current setup also focuses mainly on upper-body clothing and frontal-view images.\n\nQUESTIONS:\nIntroduction (1) and Motivations (3.3) feel overlapped. Consider merging them together?\n\nQuestion.\nWhy the faces in Figure 2b are not masked, but in Figure 2a they are masked?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes VTON-VLLM, a vision large language model designed to enhance virtual try-on (VTON) image synthesis by aligning model outputs more closely with human preferences. It introduces a dataset annotated with human feedback, enabling VTON-VLLM to evaluate and guide the synthesis process. Two main approaches are presented: (1) a plug-and-play refinement model (VRM-Instruct) providing fine-grained supervision during training, and (2) adaptive test-time scaling guided by learned human preferences. Additionally, the authors introduce VITON-Bench, a comprehensive benchmark set of challenging scenarios. Experimental results demonstrate improved performance and alignment with human judgments.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Introducing a VLM explicitly aligned to human preferences for VTON task is original and well-motivated.\n\n2. The new benchmark and human-centric metrics provide valuable resources for future research and enable more meaningful evaluation of VTON quality.\n\n3. VTON-VLLM has a significant improvement over the baseline on the proposed benchmark.\n\nWeaknesses:\n\nMy primary concern of this paper is the novelty. While the paper presents a well-integrated pipeline, many of its technical components—including supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF), and test-time scaling—are established techniques in the broader LLM and VLM literature. The main novelty appears to stem from applying these existing methods to the specific domain of VTON. Given this, I am somewhat uncertain about the proper positioning of the paper.\n\nQUESTIONS:\n1. In Tables 1 and 2, the experiments only report results for the combined setting of VRM-Instruct and TTS, without separating the effects of each component. This limits our ability to understand which part of the pipeline contributes most to the performance improvements over the baselines.\n\n2. Given that VTON is a practical, real-world application, I am also concerned about the feasibility of TTS, as it is known to incur significant computational and time costs. Could the authors clarify whether substantial performance gains could still be achieved using only VRM-Instruct, without TTS, to make the approach more practical for deployment?\n\nLIMITATIONS:\nSee weakness above.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
The paper collects human feedback on VTON output to train a VLLM tailored to provide image quality assessment tailored for VTON. Then, it uses the instruction of the VLLM model to train a diffusion-based VTON refinement model. One can achieve test-time scaling by iteratively applying the VLLM and the refinement model.

STRENGTHS AND WEAKNESSES:
Strength:
1. The proposed method can improve the eventual VTON output on top of existing VTON methods, even SOTA ones.
2. The trained VLLM itself can achieve a high preference agreement rate with humans.
3. There's sufficient ablation showcasing the effect of test time scaling.

I didn't notice any particularly important weakness in the paper. While the paper doesn't provide a particularly striking empirical finding, it presents solid results.

QUESTIONS:
1. How important is it to train the VLLM on the human feedback dataset? Can we simply rely on the generalization capabilities of the base VLLM? Does the refinement model and iterative refinement framework still work if the VTON-VLLM is replaced with an off-the-shelf VLLM and the refinement model is retrained on top of the output of the off-the-shelf VLLM?
2. How many iterations are used for the iterative refinement? How does the compute for the iterative refinement process compare to the compute for inferencing the base model?

LIMITATIONS:
The limitations are adequately addressed in the paper.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-03

comments: SUMMARY:
This paper studies the problem of improving the human alignment of virtual try on models. Particularly, the virtual try on models (VTON) is adopting diffusion models to synthesizing the images given an character and an image of clothes, to generate the character in the provided clothes. And the VTON-VLLM is trained over their collected human preference data as a fashion expert. This expert further used for VTON modle refinmenet through fine-grained supervision and test-time scaling.

STRENGTHS AND WEAKNESSES:
Strengths
1. The virtual try on task is of important application and research value. The proposed approach brings the latest techs to solve this problem better, and achieve consistent improvement over ViTON-HD.
2. In general, the paper is well-structured, and easy to follow the idea.

Weaknesses
1. I'm concerned of the reproducible of this appraoch as the only verified effectiveness is on their own curated VITON-Bench.
2. It is unclear how the inpainting task can transfer the low-quality image to high-quality one even if assuming the refinement instruction generated from VTON-VLLM is correct, how the inpainting model ensure it's following refinement instruction at a high quality bar.

QUESTIONS:
NA

LIMITATIONS:
As in the weaknesses
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-06