VTON-VLLM: Aligning Virtual Try-On Models with Human Preferences
Authors: Siqi Wan
Jingwen Chen
Qi Cai
Yingwei Pan
Ting Yao
Tao Mei
Venue: NeurIPS 2025 poster ()
Keywords: virtual try-on
Primary area: applications

Abstract: Diffusion models have yielded remarkable success in virtual try-on (VTON) task, yet they often fall short of fully meeting user expectations regarding visual quality and detail preservation. To alleviate this issue, we curate a dataset of synthesized VTON images annotated with human judgments across multiple perceptual criteria. A vision large language model (VLLM), namely VTON-VLLM, is then learnt on these annotations. VTON-VLLM functions as a unified ``fashion expert'' and is capable of both evaluating and steering VTON synthesis towards human preferences. Technically, beyond serving as an automatic VTON evaluator, VTON-VLLM upgrades VTON model through two pivotal ways: (1) providing fine-grained supervisory signals during the training of a plug-and-play VTON refinement model, and (2) enabling adaptive and preference-aware test-time scaling at inference. To benchmark VTON models more holistically, we introduce VITON-Bench, a challenging test suite of complex try-on scenarios, and human-preference–aware metrics. Extensive experiments demonstrate that powering VTON models with our VTON-VLLM markedly enhances alignment with human preferences. Code is publicly available at: [https://github.com/HiDream-ai/VTON-VLLM/](https://github.com/HiDream-ai/VTON-VLLM/).

VTON-VLLM: Aligning Virtual Try-On Models with Human Preferences
Siqi Wan's Jingwen Chen', Qi Cai?, Yingwei Pan", Ting Yao?, Tao Mei? University of Science and Technology of Chinal, 2 HiDreemm.ai
wansiqi4789 mail.ustc.cdu.cn. (chenjingwen, cqeaiqi, pandy, tiyao, tmei @ @hidream.ai
Abstract
Diffusion models have yielded remarkable success in virtual try-on (VTON) task, yet they often fall short of fully meeting user expectations regarding visual quality and detail preservation. To alleviate this issue, we curate a dataset of synthesized VTON images annotated with human judgments across multiple perceptual criteria. A vision large language model (VLLM), namely V'OO-VLLM, is then learnt on these annotations. VTON-VLLM functions as a unified "fashion expert" and ca- pable of both evaluating and steering VTON synthesis towards human preferences. Technically, beyond serving as an automatic VTON evaluator, VTON-VLLM upgrades VTON model through two pivotal ways: (1) providing fine- areegrained pervisory signals during the training of a plug-and aane--nd--payyy VTON refinement model, and (2) enabling adaptive and preference-awee To benchmark VTON models more holistically, we introduce VTTON-Bench a challenging test suite of complex try-on scenarios, and cuuma--rreerence-avaan metrics. Extensive experiments demonstrate that powering VTON models with our VTON-VLLM markedly enhances alignment with human preferences. Code is publicly available at: http:///gitbbbbom//tttaaaaaaav/p
test-time scaling at inference.
Introduction
amsge-based Virtual Try-On (VTON) is an increasingly influential task in computer vision that aims to synthesize realistic images of person wearing specified garment, given the input images of both the individual and the clothing item. By allowing users virtually preview apparel without the need for physical trials, VTON holds significant potential for transforming online shopping experiences and enabling new applications fashion e-commerce.
typically adopt a two-stage pipeline involving pose-guided garment warping and person-garmen blending. These approaches often yield visually implausible garment details and struggle with complex human poses due to inaccurate warping process. Motivated by the success of diffusion models
Early GAN-based VTON methods 8 12
13
15
16
24
42 45
22 B3
35 51 53), recent VTON approaches (9] 14 29 34 41 leverage the pre-trained diffusion priors to enhance generation quality. These approaches generally incorporate surment-specing features through specialized modules (e.g., ControlNet [49 and ReferenceNet [46)). ) and synthesize high- fidelity images progressive manner.
Despite achieving low FID [8] and LPIPS [50] scores by these diffusion-based models, the resultant try-on images often fall short of fully meeting user expectations (c.g., the inconsistent garment details in the right example of Figure T (a)). These metrics are seemingly effective for measuring general image quality, but fail adequately capture fine-grained visual attributes (e.g., texture preservation, and modd-clotthing coherence) that are critical for realistic and satisfactory virtual try-on experiences.
"hii work was performed at HiDream.ai.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(4) Human Preference on VTON Results
Good!
BBd!
(b) Human Feedback Collection
Qui Describe characters upper-body cloth each of two soporately.
QI Desoribe shape on upper-badl cloth each of two images seporctely.
Al: Left image festures the word 'adides" written white letters below Trefoil logo. Right image shows the white "adides" text and Trafcil loyo.
Al: Left image slim-fit. long-slauve with wide nacklnne. Right image shows clessic, sleeve fitted with neckline.
Q?: Bosed the description, rate character consistency upper-body cloth in images from to
Q2: Bosed the description, rate shope consistency the upper-bote -odd cloth two from
A2: Score
A2: Score is
Q3: Review your previous responses conrect
Q3 Review your previous responses is correct or not.
Correct.
A3: Correct.
Judge whether the arswer of VLLM correct
Judge whether the answer of VLLM correct
Correct!
Wrong! Shape of sleeves inconsistent.
Though low LPIPS scores are achieved, not all synthesized images generated by state-of-the-art VTON models fully meet user expectations. (b) Human Feedback Collection: A pre-trained VLLM is first leveraged to automatically assess the synthesized high-quality carment-person images step-by-step through multi-turn conversations. These assessments are subsequently reviewed and validated by human annotators to ensure reliability.
Figure (a) Human Preference on VTON Results:
To mitigate these limitations, introduce VTON-VLLM. human wwwptepprreeeeeeeeeeeee aware vision large language model to both evaluate and steer VTON towards human preferences. We begin by collecting dataset of human feedback VTON outputs (see Figure T (b)): each sample consists of a reference garment image Ig. a synthesized try-on image J,. and fine-grained annotations across multiple perceptual criteria (e.g.. visual pattern preservation, textual character correctness, and image quality). These annotations are organized into multi-turn conversational prompts for the fine-tuning of VTON-VLLM reinforce chain-of-thought reasoning. Consequently, the leart VTON-VLLM is able to meticulously analyze the visually grounded evidences in In, and score how well I3 matches the reference garment I as expected by humans step-by-step.
As unified "fashion expert", VTON-VLLM then aligns VTON models with human preferences through two pivotal ways. (1) Fine-grained supervision: Given a low-quality output I from any base VTON model, VTON-VILM generates detailed refinement instruction by comparing reference garment Lg and L. Conditioned on such refinement instruction fy. plug-and-play VTON Refinement Model ( (VRM-Instruct) is trained to transform the low-quality output 1 into its high- quality counterpart H By doing so, any VTON models be elegantly upgraded in cascade manner without retraining primary VTON models. (2) Adaptive and preference-aware test-time scaling: At inference, VTON-VLLA
further serves dynamic reward model, akin to test-time scaling in LLM [37], to iteratively steer VRM-Instruct's refinements based on the learnt human preferences. Such design adapts refinement strength on-the-fly, and ensures that each refinement incrementally improves VTON results.
In summary, main contributions threcfold:
We present VLLM, reliable human-pres pwwpepenceeaaigineeeeeee evaluator, which is learnt by tuning VLLM with newly collected human feedback synthesized VTON data. Our VTON-VLLM

---

goes beyond typical perceptual metrics and aligns better with human preference in image quality assessment tailored for VTON.
By taking STON-VLLM, as unified fashion expert. we further trigger human-preeereene pwwferencee enhancement in any VTON models via two ways: 1) providing fine-grat line-grained supervision for training plug-and-play VTON refinement model that elevates low-quality outputs to high-quality standards, and 2) triggering adaptive and preference-aware test-time scaling to iteratively guide VTON refinements at inference.
We construct newly collected test set that features challenging try-on scenarios, dubbed as IITON-Bench, alongside novel human- peeeaaa--peeeeeee-aaee metrics powered by VTON-VLLM. This benchmark enables more holistic assessment of VTON models, ensuring that quantitative evaluations align with real user expectations well.
Related Work
Virtual Try-on. Early virtual try-on (VTON) methods [5
mainly capitalize two- stage pipeline: first, deforming the garment to match the target body shape, and then synthesizing the final image using GAN-basee generator, conditioned the warped garment and human pose. Most recently, advanced approaches have explored the integration of powerful diffusion generation models into VTON pipelines better preserve garment appearance and achieve realism. [14 [9] applies
1 42 45
straightforward approach by overlaying the warped garment on the target person to guide the diffusion model. However, these methods heavily rely on deformed garments and thus easily introduce artifacts due to imperfect warping. To address this. Stable- VTON (34) extends ControlNet [49] to learn
implicit garment deformation. OOTDiffusion [46] and IDM-VTON [9] adopt dual-UNet framework to better leverage pre- pre-trained image priors within diffusion models for representation learning of
garment. Furthermore, SPM-Diff [40] incorporates the visual correspondence between the garment and the human body into the dual-UNet architecture, which enhances the preservation of fine-grained details and clothing outline. Although promising results are achieved, these diffusion- -baeed VTON models still fail to generate images that fully meet user expectations.
Human Preference Alignment in VLLMs. In recent years, vision large language models (VLLMs) have exhibited remarkable performance in tasks involving cross-modal understanding 28 52
reasoning [5 32 and interaction [9 48). However, the modality gap between vision and language leads to significant vision hallucinations in VLLMs, wherein the generated text is inconsistent with the associated images. Inspired by the reinforcement learning from human preference large language models [3 38] advanced works 33 47 propose collecting human feedback on vision hallucinations, such as visual instructions or error corrections, address the issue of cross-modal misalignment. However, integrating human preference alignment into VTON remains largely unexplored.
3 VTON-VLLM Human Preference-Aware VLLM for VTON
In this section, we first introduce VTON-VLLM, human peeraan--refeeeeee-aaeee vision large language model (VLLM) for virtual try-on (VTON). Given a reference garment image and a synthesized person image wearing this garment, the proposed "TON-VLLM can comprehensively assess whether the synthesized image aligns with human preferences in both image quality and detail preservation.
3.1 Motivation
The recent advancements in VTON techniques have significantly improved the quality of synthesized images, enabling practical applications in fashion e-commerce and digital styling. Although promising results are achieved, existing methods still fail to generate images that are fully aligned with human preferences. For example, as shown in Figure (a), inconsistencies and incoherence often arise in detail preservation and body alignment, respectively. And these types of defects may not be effectively captured by traditional metrics such as the Learned Perceptual Image Patch Similarity (LPIPS) and Frechet Inception Distance (FID) in one-to-man image synthesis. Thus, it is essential to accurately evaluate whether a synthesized image fully aligns with human preferences.
Recent works 25 have explored using vision large language model (VLLM) proxies for human judgment, leveraging their powerful capabilities in multi-modal understanding. However, the efficacy of these methods in evaluating the quality of VTON image remains unsatisfactory, particularly when
3

---

synthesized fashion images exhibit inconsistent fine tne-ggained attributes with the reference garment. To address this issue, develop a human- preeen-preeeeeaaaeee VLLM by first collecting comprehensive human feedback on synthesized VTON data across multiple perceptual criteria in VTON, which are then organized into multi-turn conversational prompts. These data are further used to fine-tune a pre-trained VLLM, facilitating chain-of-thoughe reasoning for quality assessment.
3.2 Human Feedback Dataset
To hetter investigate human judgments in VTON, we establish an evaluation framework that includes garment consistency and image quality. The two metrics encompass multiple sub-aspects follows:
Garment Consistency (GC). This criterion evaluates whether the generated deformed garment retains the original 1) visual pattern (i.e., texture and logo), 2) text characters, 3) sleeve style (e.g.. wave or puff sleeves) and 4) holistic garment shape.
Image Quality (IQ). This criterion assesses 1) edge artifacts near garment-body intersections and 2) pose plausibility of the target person in the synthesized image.
When constructing the human feedback dataset, we first employ a pre-trained VLLLM to roughly
annotate 40K synthesized images according the pre-defined evaluation specifications. 15 annotators from different educational backgrounds are then invited to review and validate these annotations. As result, we obtain high-quality dataset consisting of clothing-model pairs, fine-grained human feedback and quality scores across the above six fine-grained sub-aspects. A multi-turn conversa- tional prompt [X]] _..... X.M X is then constructed for each pair, where X3 and X denote question string and an answer string, respectively. This way, the trained model is capable of decom- posing the complex evaluation into intermediate reasoning steps (i.e., chain-of soiinn-f--hhoo of-thought reasoning) for improved VTON understanding, based on which faithful quality scores are computed.
3.3 Training
We adapt a pre-trained VLLM [ to comprehensively assess image quality in alignment with human judgments for VTON by minimizing the cross-entropy loss on the collected human feedback dataset. Specifically, we form all the multi-turn conversational instruction strings into a single input sequence with T. tokens. Hence, given input instruction Xq. we compute the probability of X,
p(Xallo X²)
II pa (2 Xg Xa.ci)
where 0 represents the trainable parameters, and I is the concatenated clothing-model image. Let X, Xg,ci and Xa.<i denote the historical instructions and answer tokens in all turns before the current prediction token I;, respectively.
4 Align VTON models with Human Preferences via VTON-VLLM
This section details how VTON-VLLM aligns VTON models with human preferences by: 1) providing fine-graid gue-ggrined supervision for plug-and-play VTON refinement model (Section 4.1 and 2) facilitating test-time scaling in inference with maeaan--rffeeeneeeee aware reward (Section
4.1 VTON Refinement Model with Fine-grained Supervision
Though existing VTON approaches achieve strong quantitative results on standard benchmarks, their generated outputs often fail meet the user expections. To address this limitation, we introduce VTON refinement model optimized with fine-grained supervision from the "TON-VLLAY dubbed as VRM-Instruct. The refinement model can be cascaded with arbitrary VTON models in plug play manner, consistently improving their generation quality and alignment with human preferences.
Recall that our proposed VTON-VILM can not only score garment-person pair from six fine- grained aspects but also excel in reasoning for interpretable visual evidences to justify the scoring decisions in Section Taking advantage of this capability, we first construct triplets (Ig. I4.0 .I!( consisting of a reference garment 4g. a low-quality person image 1 and high-quality version

---

Letters on garment are not correct; they should generate the letters "LEVIS"
VTON-VLLM
VTON Refinement model
laad
kmnk
(a) VTON Refinement Model
Letters on garment are not correct: they should generate the letters "LEVIS"
XN
VTON Refine-model
Score: wwt.teeeennooolsseeeeeecc/orrerr
VTON-VLLM
Shape(4.7)/Edge Artiae/acccracc((aaaaaa) P.5e(4.8)
Letters on garment
correct
Shapo(477VEde Artifact(4 Artifact((.8)/Hummn Pose(4.8)
(b) Test- Scaling with Human- Prerraaeeeeeeeeeeewwweeeeeee Reward
Figure 2: Illustration of our proposed VTON refinement model (VRM-Instruct) and test-time scaling (TTS), both of which are powered by our VTON-VLLM, (a) Given sample generated by any VTON model, "TON-VLLM, reasoning and then provide fine- seee-graine instruction to guide VRM-Instruct for generation of better results that meet user expectations. (b) TTS can readily triggered by collaborating VTON-VILMM and VRM-Instruct, further iteratively enhancing the outputs to align human preferences.
first employed to evaluate the synthesized image via chain-of-thous
HQ For each triplet, our VTON- VLLM generates a fine-granne refinement instruction t, that pinpoints specific regions in 2 that do not align human preferences in either garment consistency or image quality by comparing Ig and 1 Then, IRM-Instruct is steered to enhance 1/-9 into IHI following the instruction. In this work, I,, is sampled from high-quality datasets (e.g.. VITON- HD) while I is synthesized by less capable VTON model. Inspired by [2] 10], propose incorporating the intrinsic in-context visual priors of a pre-trained text-to- imate diffusion transformer (i.e., FLUX-FFII [] into VRM-Instruct, which frames VTON synthesis as conditional image inpainting. Specifically, we concatenate the garment image Ig E TR. Hx with the low-quality image 1/a² RR#x W x3 resulting in visual condition Leond 1,01, Mcanwhile, the ground -ruth image is defined as Iyt I,Or, "Q The inpainting mask Iaskk Io is derived by combining black image Io and white image I, indicating the untouched area and the inpainting region, respectively. The objective function for training IRM-Instruct can formulated as:
LDr (o. ([x,, Xcond, Xmassk) ), f,) (E xo)
where Xo. Xcond and Xmask are the latent codes of Igt. Leand and masi, respectively, x, is obtained by adding Gaussian noises e to the latent code Xo at timestep 7 denotes the trainable parameters.
At inference time, our VTON- VLLM is first utilized to identify perceptually defective regions in the initial output from an arbitrary VTON model and generate the refinement instruction f,. The low- quality initial image and the instruction are then fed into our VRM-Instruct to produce high-fidelity image that better aligns with learnt human preferences, as illustrated Figure (a).
4.2 Test-Time Scaling with Human Preference-Aware Reward
Test-time scaling (TTS) has demonstrated significant effectiveness in both linguistic and multi-modal understanding tasks 7 37 By further investigating VTON-VILM and VRM-Instruct, found that TTS is naturally supported through the mutual collaboration of these two models. Specifically, following the prediction of VRM-Instrus reward model to assess the quality of the refined image. Then, another forward pass of VRM-Instruct executed if this image does not fully align with human preferences. conditioned on the refined image and the updated refinement instruction, yielding an incrementally improved output. This
VTON-VLLM functions as a mamnaa-prefeeence--ee
5

---

iterative process continues until the output image meets the pre-defined quality criteria or a maximum of N iterations reached. TTS for VTON is shown in Figure (b).
5 Experiments
5.1 Experimental Setups
Datasets. VITON-HI 8 contains 13,679 frontal-view image pairs of women and upper garments. Following previous works [14] 29). we split the dataset into 11,647 training pairs and 2,032 testing pairs. DressCode [3] consists of 53,795 image pairs, divided into three categories: 15,366 upper- body clothes, 8,951 lower-body clothes, and 29,478 dresses. We adopt the official split, using 1,800 pairs from each category for testing and the remaining pairs for training.
Evaluation. Generally, two evaluation settings are adopted in VTON: paired and unpaired. Specif- ically, the paired setting involves reconstructing the ground-trune truth person image using the original garment, while the unpaired setting requires replacing the clothing of a person image with dif- ferent garment. To benchmark VTON models more holistically, test set that features realistic and complex VTON scenarios, namely VrTON-Bench. Particularly, VITON-Bench consists of 1,000 paired garment-persoe person images and 2,300 unpaired images manually collected from the Internet, VITON-HD and CosmicMan [26]. All the methods are evaluated on three test sets: VITON-HD, DressCode, and our VITON-Bench. For the paired setting, SSIM [K] and LPIPS [50] are commonly adopted to measure visual similarity between the generated images and the ground- truth images. Furthermore, we leverage our proposed VTON-VLLM as a "fashion expert" to compute human- human-preferent preference-aayeeeeee aware metrics, which evaluate both garment consistency (GC) and image quality (IQ) across set of fine-grained attributes, including visual patterns, text characters, sleeve style, garment shape, edge artifacts, and human pose. For the unpaired setting where ground-truth references are unavailable, we employ FID [8, KID 3 GC and IQ to assess generation quality. is worth noting that our GC metric can evaluate the visual consistency between the synthesized images and input garments in the unpaired setting, while FID and KID primarily measure the perceptual realism and distribution similarity of the generated images with respect to a target dataset.
newly curate challenging
Implementation Details. (1) VLLM-VLLM: Our VTON-VLLM is initialized from Pixtral12B M and further fine-tune tuned on the collected human feedback dataset. The model is trained for 2 epochs with a batch size of 64. The learning rate is set to 0.0001. and Adam W 227 is employed as the optimizer. We incorporate Low-Rank Adaptation (LoRA) 20 with rank of 16 for training efficiency.
(2) VTON Refinement Model: We also employ Adam W to optimize the model over 50,000 training steps. The learning rate is set to 0.00005 with a warmup over 500 iterations, and the batch size is 1.
5.2 Results
Quantitative Results on VITON-HD and DressCode. Table I presents the comparisons among different methods on VITON-HD. Note that GC,/IO, and II, are human- peeeraan-rrffreee-awaaar metrics driven by our VTON-VLLM for paired and unpaired settings, respectively. Although state-of- the-art models such as POM- and SPM-Diff have already achieved promising results, further improvements are realized through our InM--nsttuut and TTS module, both of which are powered by VTON- Specifically, in the paired setting. IDM-VTON w/ VRM-Instruct and TTS not only significantly reduces LPIPS from 0.082 to 0.051 but also increases GCp from 4.665 to 4.873, demonstrating better visual consistency compared to its baseline counterpart IDM-VTON. Moreover, in the unpaired setting. IDM-VTON w/ TRM-Iustruct and TTS demonstrates consistent performance gains, as evidenced by a decrease in KID from 0.741 to 0.634 and an increase in IQ, from 4.392 to 4.711, yielding absolute improvements of 0.107 and 0.319, respectively. These observations highlight the advantages of leveraging our VRM-Instruce and TTS to steer VTON models towards human preferences in both image quality and garment consistency, which is achieved through iterative refinement guided by fine- -rained and preference-aligee refer to the Appendix for more results on DressCode.
instructions from VTON-VLLM, Please
Quantitative Results on VITON-Bench. We quantitatively evaluate different VTON models on the newly introduced VITON-Bench, and 2 summarizes the results. The increased complexity of VrrON-Bench leads to substantial performance degradation in most metries, with particularly pronounced drops in SSIM, FID, and IQ,. Conversely, GC., exhibits relatively mild decreases. Upon

---

Table 1: Quantitative results on VITON-HD. GC. and IQ, denotes the proposed human- -uman-preference- aware metries Garment Consistency and Image Quality, respectively. VRM-Instruct and TTS are short for VTON refinement model with fine-grained instructions and test-time scaling, respectively.
Train/Test
FTTON-DDVITON-IHH
Average Preference-Awan Scores
Method
SSIM
LPIPS
FID L KID 4
GCp f
IQp 1
GC2 f
IQu
LaDI-VTON
29
0.864
0.096
10,531
2.303
4.542
4.621
4,320
4.291
Stable-VTON
23
0.852
0.084
10.200
1.921
4.625
4.660
4.331
4.405
OOTDiffusion
0.881
0.071
10.153
0.892
4.619
4.649
4.400
4.388
IDM-VTON
9
0.877
0.082
9.372
0.741
4.665
4.750
4,445
4.392
w/ VRM--Insrruc
0.880
0.054
9.012
0.667
4.870
4.852
4.712
4.677
w/ VRMMInstruct and TTS
0.882
0.051
8,902
0.634
4.873
4.920
4.724
4.711
SPM-Diff (40)
0.911
0.063
9.034
0.720
4.742
4.775
4,502
4,411
w/ VRM-Instruct
0,890
0.054
9.012
0.667
4.891
4.850
4.715
4,680
w/ VRM-Instruct and TTS
0,892
0,055
9.010
0.663
4.900
4.930
4.745
4.725
CAT-VTON [TO VRM-Instruct
0.870
0.057
9.015
0,670
4.713
4.765
4,510
4,420
0.885
0.055
8.915
0.660
4.832
4.834
4.673
4.661
w/ VRM-Instruct and TTS
0.891
0.052
8.923
0.652
4.895
4.900
4.733
4.738
Table 2: Quantitative results the newly introduced VITON-Bench. GC, and IQ. denotes the proposed human-prec aainaa--reference-awarr metrics Garment Consistency and Image Quality, respectively.
Train/Test
VITON-EDIVITON-Benddd
Average Preference- Scores
Method
SSIM
LPIPS + FID 4
KID 4
GCp f
IQp" 1
GC,," 1
IQ,
LaDI-VTON
0.702
0.173
14.032
2.03
4.107
3.692
3.631
3.021
Stable-
23
0.725
0.110
13.566
1.67
4.280
4.010
3.894
3.083
OOTDiffusion
46
0.733
0.097
13.087
1.43
4.357
4.036
3.926
3.447
IDM-VTON
9
0.750
0.083
12.562
1.02
4.374
4.069
4.026
3.825
w/ VRM-Instruct
0.787
0.064
10.931
0.92
4.566
4.398
4,341
4.126
w/ VRM-Instruct and TTS
0.794
0.060
10.784
0.87
4.612
4.429
4,442
4.302
SPM-Diff 440 w/ VRM-Instruct
0.765
0.084
12.208
1.34
4.519
4,230
4.235
3,804
0.782
0,069
11.011
0.94
4,570
4,401
4,288
4.027
w/ VRM-IIRMMuuu and TTS
0.797
0.062
10.627
0.87
4.623
4,429
4,407
4,280
CAT-YTON 10
0.751
0.078
12.663
1.23
4.518
4.238
4.150
3.991
w/VRM-Instruet
0.772
0.063
10.721
4.542
4.579
4.262
4.156
w/ VRM-Instruct and TTS
0.788
0.060
10.232
0.85
4.555
4.685
4.390
4.353
examining the synthesized results, we observed that these state-of-the-an VTON models are capable of preserving parts of the visual characteristics in the garment by learning robust garment features. However, they struggle with handling garment warping for complex human poses in real-world scenarios, resulting in notable artifacts and incorrect fine-grained details. Despite these challenges, the integration of our proposed VRMM-Inttruct and TTS further enhances the performance of all the methods. Particularly, SPM-Diff w/ VRM-Instruct and TTS outperforms the baseline SPM-Diff, achieving improvements in both GC, (from 4.235 to 4.407) and IQ, (from 3.804 to 4.280). These results clearly demonstrate that VRM-Instruct and TTS can effectively improve synthesized outputs, even in intricate real-world cases, by capitalizing on human preferences learnt by VTON-VLLM.
Qualitative Results. We conducted qualitative evaluation to compare the results synthesized by different methods on VITON-Bence Some examples are illustrated in Figure 3 It can be observed that our proposed VRM-Instruct and TTS module consistently improve the state-of-the-a baselines, i.e., LaDI-VTON, IDM-VTON and CAT-VTON. For instance, in the first row, the enhanced images exhibit significantly better visual quality compared to those generated by the baseline models in the challenging case with complex human poses. Moreover, in the second row, the images synthesized by baseline models suffer blurred or repeated text rendering and severe edge artifacts, whereas the counterparts refined by incorporating our VRM-Instruct and TTS achieve superior text fidelity. This underscores the effectiveness of our proposed designs (i.e., VRM-Instruct and TTS), which iteratively enhance the synthesized images with fine- teeeggr and human preference-aligeeeeeeeeeeeee aligned instructions from VTON- Please refer to the Appendix for more examples.
5.3 Discussions
Effect of VTON-VLLL VTON Synthesis. As previously noted, our VTON-VI.LM can upgrade arbitrary VTON models in cascade manner by employing plug-and-play VTON refinement model
7

---

LaDI-VTON
Enhanced
IDM-VTON
Enhanced
CAT-VTON
Enhanced
TIE
STA
Figure 3: Qualitative results on VITON-Bench. The first and second rows show results under unpaired and paired settings, respectively. For each pair, the left image is generated by baseline model (i.e., LaDI-VTON, IDM-VTON, CAT-VTON), and the right image enhanced by incorporating our proposed VRM-Instruct and TTS module powered by VTON-VLLM.
Table 3: Ablation study of VRM-Instruct and TTS module on VITON-HD. VRM-Instruct denotes the proposed VTON refinement model with fine-grained instructions from VTON-VLLM, TTS denotes test-time scaling.
Garment Consistency
Image Quality
Model
Pattern
Characters
Sleeve
Shape
Edge Artifact
Human Pose
SSIM
LPIPS 1
Base
4.645
4.698
4.679
4.833
4.650
4.821
0.870
0.057
Base VRM-Instruct
4.873
4.800
4.782
4.865
4.803
4.865
0.885
0.055
Base VRM-Insttuett+ TTS
4.931
4.913
4.865
4.870
4.889
4.913
0.891
0.052
that utilizes fine-grained refinement instructions (VRM-Instruut), and further function as human- preference-awart reward model iteratively guide VRM-Instruce based on learnt human preferences. akin to test-time scaling (TTS). In this part, we conduct an ablation study to evaluate individual contributions of these two core designs by sequentially adding VRM-Instruct and TTS Base VTON model. The scores across all six fine-grained dimensions of garment consistency (GC) and image quality (IQ) are shown. CAT- VTON adopted as the Base model here. As shown in Table 3 by incorporating the fine- grained instructions from VTON-VLLM to identify imperfect regions in the garment and guide the refinement process towards human preferences, Base VRM-Instruct achieves notable improvements over Base model in garment consistency. Specifically, compared with Base model, Base + VRM-Instruct boosts "Characters" and "Pattern" from 4. 4.698 to 4.800 and from 4.645 to 4.873, respectively. Furthermore, we integrate TTS into Base + VRM-Instruut to enable iterative enhancement with human cemman--rfference--awe awwar reward provided by VTON-VLLM in the full run Base + VRM-Instruce TTS, surpassing all the other settings. Compared to Base model, the "Edge Artifact" and SSIM scores increase from 4.650 to 4.889 and from 0.870 to 0.891, respectively.
Preference Agreement. In this part, we evaluate the aligmment between human preferences and our VLOM-VLLM. Specifically, we adopt held-out set from our collected human feedback data as ground-truth annotations, and compute the percentage of consistent judgements between human and VL.MM as a measure of inter-rater agreement. Chain-of-thought reasoning is employed in VTON-VILM to generate the fine-grained assessments of the synthesized images. Our VTON-VILMM attains an agreement rate of 84.52% demonstrating strong alignment with human preferences.
FID vs VTON-VLLM Score. In Table 4 we further investigate the correlation between the no- reference evaluation metric FID/VTON- COVYOO--VLLLS VLLM and human judgments by dividing images from the held-out human feedback data into subsets based on whether the generated images are preferred by humans or not. FID and the averaged VTON-VLLM scores are calculated for each subset. Note that no-reference evaluation refers to image quality assessment performed in the absence of ground-truth images. As shown, there is no significant difference in FID scores between the two subsets. This suggests that FID may not effectively capture the subjective dimensions underlying human judgments. In contrast, VTON-VLLM yields higher scores for the numan-preferee subset compared to the non-preferred one, confirming its better consistency with human perceptual criteria.

---

Table 4: Analysis of the alignment between our VTON-VLLM and human preferences. (1) FID and VTON-VLLM scores are calculated for subsets of preferred and non- non-preferred images by human. (2) Preference agreement refers to the inter-rater agreement between human and VTON-VLLM,
Method FID
Preferred Non-preferred Preference Agreement
6.84
6.99
/
VTON-VLLM
4.92
3,94
84.52%
Table 5: Analysis of our VTON-VLLM in high-quality dataset construction. VITON-SYN is a synthesized dataset by leveraging VTON model for generation and VTON-VLLM for data filtering.
Carment Consistency
Image Quality
Training Data
Puttern
Characters
Sleeve
Shape
Edge Artifuct
Human Pose
SSIM
LPIPS 1
VITON-HD
4.861
4.724
4.734
4.676
4.714
4.588
0.875
0.045
VITON-SYN
4,907
4,868
4.782
4.832
4.856
4.672
0.898
0,044
VITON-HD + VITON-SYN
4.931
4.912
4.822
4.872
4.890
4.726
0.903
0.043
High-quality Dataset Construction with VTON-VLLM. We conduct an ablation study to explore the intrinsic ability of our VTON- VLLM as an image quality evaluator high-quality dataset construction. Specifically, we first train a VTON model that is architecturally similar to VRM-Instruct on the VITON-HD dataset, where the garment image Iy and blank white image 11 are combined as the visual condition Loand during training. More implementation details are provided in Appendix pairs, using garment images sampled from existing datasets 8 30 36 and textual prompts describing diverse scenes and poses from a predefined pool. Finally, the generated images undergo rigorous data filtering by our VTON-VLLM, resulting in
See A.2 The trained model is then employed to synthesize garment-
aligned training dataset VITON-SYN In 5 we compare our VTON models trained on VITON-HD, VITON-SYN, and the combination of both. Surprisingly, training the VTON model solely on VITON-SYN outperforms the in-domain run VITON-HD. We attribute this improvement to two key factors: (1) VITON-SYN exhibits comparable visual quality to the manually collected VITON-HD and scalability in data quantity, and (2) the VTON model has gained enhanced capability in handling diverse scenarios when trained on VITON-SYN. This demonstrates the effectiveness of VTON- VLLM in performing quality assessment for the construction of high-quality datasets. Further improvements in all metrics are achieved by leveraging both sets for training. Please refer to the Appendix for details of VITON-SYN.
Conclusion
In this paper, we have presented VLLM-, a vision large language model fine-tuned with human feedback on synthesized VTON data, which can comprehensively assess whether synthesized images meet user expectations. As a unified "fashion expert", our VTON-VLLM can elegantly upgrade VTON models and align them with human preferences through two pivotal ways: 1) providing fine-grained supervision for the training of a plug-and-play VTON refinement model, and 2) enabling test-time scaling with mumaa-preference-awart reward for adaptive inference. To benchmark VTONN models more holistically, we curate a challenging test set (i.e., VITON-Bench), featuring realistic and complex VTON scenarios, and introduce guman-prefereneee by our VTON-VLLM. Extensive experiments on VITON-HD, DressCode, VITON-Bench validate that incorporating our WTON-VLLN, leads to significant improvements in VTON synthesis.
evaluation metries powered
Limitations. Despite the above advantages, our method still has several limitations. First, it may encounter diffieulties in handling rare long-tail clothing items with distinctive patterns or structures. Second, objects held by the person and significantly overlapping with the target garment regions may not be accurately reconstructed, as our method primarily emphasizes garment preservation. Finally, similar to test-time scaling techniques adopted in large language models, our method necessitates additional computational overhead during the inference.
Border Impacts. Similar to popular generative models for visual content creation, our proposed designs allow anyone to produce high-fidelity person images, which holds great potential impact to revolutionize the shopping experience in the e-commerce industry. However, these synthesized person images risk exacerbating issues of misinformation and deepfakes. To mitigate the misuse of our models, we will enforce strict adherence to usage guidelines and integrate an additional pipeline automatically embed digital watermarks into the synthesized outputs.
Acknovledgement. This work was supported in part by the Beijing Municipal Science and Technol- ogy Project No. Z241100001324000 and Beijing Nova Program No. 20240484681.
9

---

References
[1] Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, et al. Pixtral 12b. arXiv preprint arXiv:2410.07073 2024.
[2] Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In CVPR, 2022.
[3] Mikolaj Binikowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. In ICLR, 2018.
[4] Qi Cai, Jingwen Chen, Yang Chen, Yehao Li, Fuchen Long, Yingwei Pan, Zhaofan Qiu, Yiheng Zhang. Fenghin Gao, Peihan Xu, et Hidream-il: high-efficient image generative foundation model with sparse diffusion transformer. arXiv preprint arXiv:2505. 22705, 2025.
[5] Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvim: Endowing vision-language models with spatial reasoning capabilities. In CVPR, 2024.
[6] Yang Chen, Yingwei Pan, Yehao Li, Ting Yao, and Tao Mei. Control3d: Towards controllable text-to-3d generation. In ACM MM, 2023.
[7] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open- multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271,, 2024.
[8] Seunghwan Choi, Sunghyun Park, Minsoo Lee, and Jaegul Choo. Viton-hd: High-resolution virtual try-on via misalignment-awwre normalization. In CVPR, 2021.
(9] Yisol Choi, Sangkyung Kwak, Kyungmin Lee, Hyungwon Choi, and Jinwoo Shin. Improving diffusion models for authentie virtual try-on in the wild. In ECCV, 2024.
[10] Zheng Chong, Xiao Dong, Haoxiang Li, shiyue Zhang, Wenging Zhang, Hanqing Zhao, xujic zhang, Dongmei Jiang, and Xiaodan Liang. CatVTON: Concatenation is all you need for virtual try-on with diffusion models. In ICLR, 2025.
[11] Haoye Dong, Xiaodan Liang, Xiaohui Shen, Bochao Wang, Hanjiang Lai, Jia Zhu, Zhiting Hu, and Jian Yin. Towards multi-pose guided virtual try-on network. In CVPR, 2019.
[12] Haoye Dong, Xiaodan Liang, Xiaohui Shen, Bowen Wu, Bing-Cheng Chen, and Jian Yin. Fw-gan: Flow-navigated warping gan for video virtual try-on. ICCV, 2019.
[13] Benjamin Fele, Ajda Lampe, Peter Peer, and Vitomir Strue. C-vton: Context-driven image-based virtual try-on network. In WACV, 2022.
[14] Junhong Gou, Siyu Sun, Jianfu Zhang, Jianlou Si, Chen Qian, and Liqing Zhang. Taming the power of diffusion models for high-quality virtual try-on with appearance flow. In ACM MM, 2023.
[15] Xintong Han, Zuxuan Wu, Zhe Wu, Ruichi Yu, and Larry $ Davis. Viton: An image-based virtual try-on network. In CVPR, 2018.
[16] Sen He, Yi-Zhe Song, and Tao Xiang. Style-based global appearance flow for virtual try-on. In CVPR, 2022.
[17] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yacl Pritch, and Daniel Cohen-Or Prompt-to-prontt prompt image editing with cross attention control. In ICLR, 2023.
[18] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In NeurIPS, 2017.
10

---

[19] Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang. Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. In CVPR, 2024.
[20] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
[21] Lianghua Huang, Wei Wang, Zhi-Fan Wu, Yupeng Shi, Huanzhang Dou, Chen Liang. Yutong Feng. Yu Liu, and Jingren Zhou. In-context lora for diffusion transformers. arXiv preprint arxiv:2410.23775, 2024.
[22] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagie: Text-based real image editing with diffusion models. In CVPR, 2023.
[23] Jeongho Kim, Guojung Gu, Minho Park, Sunghyun Park, and Jaegul Choo. Stableviton: Learning semantic correspondence with latent diffusion model for virtual try-on. In CVPR, 2024.
[24] Sangyun Lee, Gyojung Gu, Sunghyun Park, Seunghwan Choi, and Jaegul Choo. High-resolution virtual try-on with misalignment and occlusion- occlusion-handled handled conditions. In ECCV, 2022.
[25] Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiayao Li, Yixin Fei, Kewen Wu, Xide Xia, Pengchuan Zhang, Graham Neubig, and Deva Ramanan. Evaluating and improving compositional text-to- visual generation. In CVPR, 2024.
[26] Shikai Li, Jianglin Fu, Kaiyuan Liu, Wentao Wang, Kwan- Yee Lin, and Wayne Wu. Cosmicman: text-to-image foundation model for humans. In CVPR, 2024.
[27] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
[28] Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, al. Deepseck-vl: towards real-world vision-language under- standing. arXiv preprint arXiv:2403.05525, 2024.
[29] Davide Morelli, Alberto Baldrati, Giuseppe Cartella, Marcella Comnia, Marco Bertini, and Rita Cucchiara. Ladi-vton: Latent diffusion textual-inversion enhanced virtual try-on. In ACM MM, 2023.
[30] Davide Morelli, Matteo Fincato, Marcella Cornia, Federico Landi, Fabio Cesari, and Rita Cucchiara. Dress code: High-resolution multi-category virtual try-on. In CVPR, 2022.
[31] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. NeurIPS, 2022.
[32] Renjie Pi, Jiahui Gao, Shizhe Diao, Rui Pan, Hanze Dong, Jipeng Zhang, Lewei Yao, Jianhua Han, Hang Xu, Lingpeng Kong, and Tong Zhang. Detgpt: Detect what you need via reasoning. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, EMNLP, 2023.
[33] Yurui Qian, Qi Cai, Yingwei Pan, Yehao Li, Ting Yao, Qibin Sun, and Tao Mei. Boosting diffusion models with moving average sampling frequency domain. In CVPR, 2024.
[34] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR. 2022.
[35] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023.
[36] Fei Shen, Xin Jiang, Xin He, Hu Ye, Cong Wang, Xiaoyu Du, Zechao Li, and Jinhui Tang. Imagdressing-vE. Customizable virtual dressing. arXiv preprint 240.12200...7055 2024.
2024.
11

---

[37] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute opti- mally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314 2024.
[38] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. NeurIPS, 2020.
[39] Zhiqing Sun, Sheng Shen, Shengeao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprini arXiv:2309. 14525, 2023.
[40] Siqi Wan, Jingwen Chen, Yingwei Pan, Ting Yao, and Tao Mei. Incorporating visual correspon- dence into diffusion model for virtual try-on. In ICLR, 2025.
[41] Siqi Wan, Yehao Li, Jingwen Chen, Yingwei Pan, Ting Yao, Yang Cao, and Tao Mei. Improving virtual try-on with garment-fooee focused diffusion models. In ECCV, 2024.
[42] Bochao Wang, Huabin Zheng, Xiaodan Liang, Yimin Chen, Liang Lin, and Meng Yang. Toward pnnneterristi--peeeevii image- based virtual try-on network. In ECCV, 2018.
[43] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE TIP, 2004.
[44] Tong Wu, Guandao Yang, Zhibing Li, Kai Zhang, Ziwei Liu, Leonidas Guibas, Dahua Lin, and Gordon Wetzstein. Gpt-4v (ision) is human-aligned evaluator for text-to-3d generation. In CVPR, 2024.
[45] Zhenyu Xie, Zaiyu Huang, Xin Dong, Fuwei Zhao, Haoye Dong, Xijin Zhang, Feida Zhu, and Xiaodan Liang. Gp-vton: Towards general purpose virtual try-on via collaborative local-flow global-parsing learning. In CVPR, 2023.
[46] Yuhao Xu, Tao Gu, Weifeng Chen, and Chengeai Chen. Ootdiffusion: Outfitting fusion based latent diffusion for controllable virtual try-on. arXiv preprint arXiv:2403.01779 2024.
[47] Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine-grained correctional human feedback. In ICCV, 2024.
[48] Simon Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Peter Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, al. Fine-tuning large vision-language models as decision- agents via reinforcement learning. NeurlPS, 2024.
[49] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, 2023.
[50] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features perceptual metric. In CVPR, 2018.
[51] Zhongwei Zhang, Fuchen Long, Yingwei Pan, Zhaofan Qiu, Ting Yao. Yang Cao, and Tao Mei. Trip: Temporal residual learning with image noise prior for image-to-vided diffusion models. In CVPR, 2024.
[52] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhanc- ing lasion-language understanding with advanced large language models. In ICLR, 2024.
[53] Rui Zhu, Yingwei Pan, Yehao Li, Ting Yao, Zhenglong Sun. Tao Mei, and Chang Wen Chen. Sd-dit: Unleashing the power of self-supervised discrimination in diffusion transformer. In CVPR, 2024.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please check our contribution and scope in abstract and introduction sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation of our work at the end of the main paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
13

---

Justification:
Our work does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We describe all necessary implementation details in Section S Moreover, provide the source code in the Supplementary Material to further ease the reproduction of our work.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
14

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide the code and data in the Supplementary Material, with sufficient instructions to faithfully reproduce the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ httpppscccppullccguuudddeo Cotee//ddeeeSbmmmcma ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all the experimental details in Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We follow most existing VTON works and report performances on the public standard benchmarks without statistical significance for evaluation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
15

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We describe the computer resources in Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in our paper conforms, in every respect, with the NeurlPS Code of Ethics httppp/////errppsss.cppplc/c///e
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the potential broader impacts at the end of the main paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
16

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We discuss some possible safeguards at the end of the main paper.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The used code and data have been credited in the released code in supplemen- tary material.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
17

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, papersai
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We have curated new and challenging test set, introduced in Section 5.1 which will be released upon acceptance of the paper for research only.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
18

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: Our core method development this research does involve LLMs.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy bettts://neurre /0tps//eeccccooeeeeee/////////// for what should or should be described.
19

---

A Appendix
A.1 More Quantitative and Qualitative Results
The quantitative results on DressCode across three macro- categories are summarized in Table and Table? 7 Table SSIM, LPIPS, GCp, and IQ₂ under the paired setting, while Table 7 presents resulfs under the unpaired setting, including FID, KID, GC and -Q With the integration of our proposed VRM-Instruct and TTS modules, state-of-the-ar VTON models achieve further improvements on the DressCode benchmark. Specifically, in the paired setting for the upper-body category, OOTDiffusion w/ VRM-Instruct + TTS increases SSIM from 0.902 to 0.928 and GCp from 3.59 to 4.14, indicating enhanced detail preservation. Meanwhile, in the unpaired setting, the same recipe boosts IQ, from 4.05 to 4.15 and reduces FID from 11.03 to 9.43, which demonstrates improved visual realism. These results on the DressCode dataset further validate the effectiveness of our VRM-Instruct and TTS modules in guiding VTON models towards better alignment with human preferences in both image quality and garment fidelity.
Quantitative Results on CressCode.
we present additional qualitative comparisons of synthesized results produced by different methods on VITON-HD, Furthermore, more results for other clothing categories (i.e., lower-body garments and dresses) on DressCode are presented in Figure
More Qualitative Results. In Figure 4
Table 6: Quantitative results on DressCode in paired setting of three Categories(D.c... Upper, D.C.Lower, D.C.Dress). VRM-Instruct and TTS are short for VTON refinement model with fine- grained instructions and test rest-time scaling, respectively.
Train/Test
D.C.IpeetB..CUpppp
DCC.oweeDCC.LLorr
Method
SSIM
LPIPS 1 GC,
IQ,
SSIM 1
LPIPS
GC,,
IQ,
SSIM
LPIPS
GC,,1
IQ,
LONN--TONN [20
0.904
3.48
3.32
0.862
0.122
3.36
1.22
0.762
0.129
3:25
VRM--nsttuct ITS
0.930
0.042
4.12
3,99
0,923
0,043
4.06
3.81
0.912
0,069
3.97
3.76
OOTDiffusion (4)
0.902
0.050
3.59
3.43
0.882
0,103
3.42
3.26
0.824
0.100
3.40
3.15
w/ VRM-Instruct and TTS
0.928
0.039
4.14
3,92
0,918
0.044
4.01
3.76
0.898
0.064
3.92
3.74
IDLYTON g
0.911
0.000
3.70
3.57
0.913
0.055
3.60
3.33
0.863
0.082
3.62
3.43
w/ VRM-Instruct and TTS
0.931
0.034
4.08
3.98
0.917
0.040
3.92
3.72
0,883
0.062
3.94
3.72
SPM.DiI
0.927
0.050
0:073
w/
WRM-llsssuuc
0.928
4,03
0.046
3.79
0.890
0.069
3.91
CAT-VION T0
0.919
0.015
3.68
3.58
0.897
0,063
3.65
3.40
0.900
0,065
3.51
3.41
w/ VRMM-nstruct TTS
0.927
0.039
4,00
0,918
0,042
3,99
3.74
0.892
0,065
3.93
3.76
Table 7: Quantitative results on DressCode in unpaired setting of three categories(D.cm Upper,
L.wer. Dre.S,,s), VRM-Instruct and TTS are short for VTON refinement model with fine- grained instructions and test-tine test-time scaling, respectively.
Train/Test
D.C.UppeDDD..Uprr
D.C.wwrrDC.LLoee
C...ressD...DDres
Method
FID
KID 4 GC,
1Q.
FID.
KID
GC,, 1
1Q), f
F1D 1
KID 1 GC
1Q.
LaDI-VTON
20
14.26
3.33
4.02
3.78
13.38
1.98
2,86
2.64
13.12
1.85
3.21
2.99
w/ IRM--nstruct and TTS
9,38
0.52
4.34
4.19
9,80
0.85
3.27
3,10
10.65
0.84
3,40
3.55
OOTDiffusion N6
11.03
0.86
4.10
4.05
9.62
0.84
2.99
2.92
10.65
0.84
3.40
3.29
w/ VRM-Instruct
TTS
9.43
0.42
4.31
4.15
9.66
0.50
3.19
3,08
9.69
0.52
3,76
3,58
IDM-VTON
4.13
4.11
12.05
0.93
3.02
12.33
1.41
3:48
3.29
w/ VRM-Instruct
9.55
4.30
4.18
0.77
3,03
9.92
0.54
3,80
3.51
SPM.Diff 40
10.56
4:23
4.15
0.80
10.17
0.50
3.55
1.49
w/ VRMM-IIsstuut and TTS
9.46
0.45
4.32
4,20
9.18
0.78
3.26
3.11
9.79
0.51
3.78
3.54
CAT-VTON LO
8.92
0.51
4.16
4.12
9.21
0.94
3.06
2.98
9.76
0.66
3.50
3.41
w/ VRM-Instruct and TTS
9.23
0.44
4.35
4.14
9.20
0.76
3.19
3.12
9.68
0.57
3.83
3.61
A.2 More Discussions
Effect of Iteration Count N in Test-Time Scaling, We analyze the sensitivity of our proposed TTS with respect to the iteration count N, which is guided by human- buman-preferencce preference-aligned reward from VTON-VLLM. The results are reported in Table 8 and the best results are achieved with N 3.
VTON-VLLM vs Base VLLM. To further evaluate the effectiveness of our proposed VTON-VLLM. we compare its capability in VTON assessment with that of the base VLLM (i.e., Pixtral- 12B (]) without fine-tuning on human feedback data. We sample 100 challenging garment-persee pairs from VITON-HD, where OOTDiffusion [66 struggles to produce satisfactory outputs, and generates 10 candidate images for each pair using different random seeds. Human annotators then compare the highest-rated images selected by both models from the 10 candidates to determine which is better. As a result, our VTON-VILM achieves a win rate 74.17% compared to the base VI.LM. Some examples are shown in Figure 7 It is evident that our VTON-VLLM consistently selects the
20

---

LaDI-VTON
Enhanced
IDM-VTON
Enhanced
CAT-VTON
Enhanced
Figure 4: Qualitative results VTON-HD dataset in unpaired settings. For each pair, the left image generated by a baseline model (i.e., LaDI-VTON, IDM-VTON, CAT-VTON) and the right image is enhanced by incorporating our proposed VRM-Instruct and TTS module powered by VTON-VLLM.
LaDI-VTON
Enhanced
IDM-VTON
Enhanced
CAT-VTON
Enhanced
Figure 5: Qualitative results on DressCode dataset in unpaired settings. For each pair, the left image is generated by baseline model (i.e., LaDI-VTON, IDM- IO-N,TON, CAT-VTON), and the right image is enhanced by incorporating proposed Instrrsttuct and TTS module powered by VTON-VLLM.
perceptually superior image, whereas the base VLLM often overlooks subtle garment inconsistency and visual artifacts. This highlights the effectiveness of our fine-tuned VTON-VIL VLLM in aligning VTON evaluations with human judgments. On the other hand, to verify the importance of training VLLMs on human feedback datasets, we conduct experiments on the VITON-HH dataset under two settings: (I) BLSM-VLLM: VTON refinement model is trained and guided by the instructions generated by the Base-VI.L without human feedback fine-tuning; (2) VTON- VLLM: VTON refinement model is trained on and guided by the instructions generated by our VTON-VILMM which fine-tuned with human feedback. Both models are trained and evaluated on the VITON-HI dataset. The quantitative results are summarized in Table 9 It can be observed that directly leveraging the VLLM to prompt our proposed VTON refinement model can, to some extent, improve the VTON results compared with the base VTON model. Further empolying our VTON- the refinement with human preferences consistently outperform those with the Base- VLLM across all the metrics, which validates the effectiveness of training VTON-VLLM with human feedbacks VTON images.
to align
Performances across fine-grained categories. We reported the performances across different genders, body types and skin tones. Gender, body type, and skin tone are roughly categorized into several classes. The guantitative results VITON-Bench are summarized in Table I0
21

---

System Mysttem-Messagl (Xa)) You are professional digital artist. You will have evaluate the effectiveness of the Al- Al-generated image based on given rules.
(uman(XX.) )) Please describe characters on hoper-hody garment in each of two images separately. Assistant x ): In the first image, the upper-body garment is borizontally striped red and white T-shirt. The word "FIORUCCI" printed bold, black and red letters across the chest area, centered and slightly tilted. In second image, T-shirt
image, featuring the horizontally striped red and white pattern. The word "FIORUCCT" also printed bold, black and red letters across the chest area, centered and slightly tilted.
Human X Now, based on the characters on the upper-body garments the two images, please determine how well the characters are consistent in the two images, output a score in [0, 51.
Assistant stant(X))::: 5.
Human X..) Review your previous responses step-hy-step. you completely confident that your reasoning and conclusions are correct, respond with "Correct". If you find any errors or are uncertain, respond with "Wrong".
Assistant( (X"): Correct.
Figure 6: An example of a multi-turn conversational instruction for text character consistency from our human feedback dataset.
Bave VLLM
VTON-VILM
Base VLLM
VION-VILM
Base VILM
VTON-VLLM
FILA
FILA
Misie
VTON-VILM
74.37%
25.83%
Baes VILM
Human Evaluation of Highest Rated Image
Figure 7: Comparisons between top-ranked images selected by our VTON-VLLM and the base VLLM from 10 candidate synthesized images for each garment-hume pair.
A.3 Dataset Details
Human Feedback Data. During the collection of human feedback data, a pre-trained VLLM is first employed to roughly assess garment-person images across various fine- mne-grained dimensions. Human annotators then review and validate these evaluations to ensure accuracy. These annotations further structured into multi-turn conversational instructions for training our VTON-VLLM. An example instruction for text character consistency is illustrated in Figure 6
Synthetic VITON-SYN. We adopt similar network
VRMM-Instruct (described in Section 4.1) as the VTON model and train on VITON-HD to generate VTON data Section 5.3. Different from VRM-Instruct, the inpainting condition in this VTON model is obtained by combining the garment image I and a blank white image /,, leading to Foond I, 11, To obtain VITON-SYN, sample garment images from existing datasets [8] 30 36 and scene prompts from a pre-defined pool as reference conditions, and synthesize images of persons wearing the corresponding garments
Table 8: Ablation study of iteration count N in our TTS VITON-HD.
Garment Consistency
Image Quality
Model
Pattern
Characters
Sleeve
Shape
Edge Artifact
Human Pose
SSIM
LPIPS |
2
4.817
4.803
4.720
4.688
4.736
4.450
0.825
0.064
3
4.931
4.912
4.822
4.872
4.890
4.726
0.903
0.043
5
4.896
4.882
4.807
4.750
4.820
4.601
0.883
0.058
22

---

with the trained VTON model. The derived garment-perse person pairs are further rigorously filtered by the proposed VTON-VLLM, resulting high-quality and camman-pefferenc--alee aligned training dataset. Figure 8 illustrates the data construction pipeline. Overall, VITON-SYN consists of 100,000 garment-person pairs spanning ten different scenarios, which is significantly larger and more diverse than VITON-HD dataset (13,679 indoor samples). Figure TO showcases various prompts along with their corresponding synthesized images from VITON-SYN.
New Test Set VITON-Bench. Unlike existing benchmarks such as VITON-HD and DressCode, our VITON-Bener emphasizes more on challenging VTON scenarios. These include uniquely styled clothing, complex human poses (e.g., sitting or lying down), and diverse, intricate outdoor environments. Figure Dillustrates some examples from our VITON- MrrONBBench, highlighting its greater scene diversity and increased difficulty compared to previous benchmarks (see Figure 4 and 5
Table 9: Quantitative comparisons between Base-VLLM and VTON-VLLM on VITON-HD dataset.
Train/Test
VITON-HD/VITON-II
Average Preference-Awar Scores
Method
SSIM t LPIPS
FID
KID
GCp f
IQp f
GC,,
IQ.
Base- VTON ITO
0.870
0.057
9.015
0.670
4.713
4.765
4.510
4,420
Base- VLLM
0.880
0.054
8.976
0.660
4.843
4.880
4.698
4.625
w/ VTON-VLLM
0.891
0.052
8.923
0.652
4.895
4.900
4.733
4.738
Table 10: Quantitative comparisons across fine-grained categories on VITON-Bence VRM-Instruct and TTS are short for VTON refinement model with fine- fine- grained instructions and test-time scaling.
Categories
Gender
Body Type
Male SSIM
Female SSIM
Black SSIM
White
Yellow
Brown
Slim SSIM
Averuge SSIM
Overweight SSIM
Method
SSTM
SSIM
SSIM
Base- -TON 10
0.7500
0.7514 0.7963
0.7562 0.7700
0.7538
0.7565 0.7570
0.7241
0.7623 0.7970
0.7211 0.7521
0.7603 0.7830
Base VRM-Instruct- TTS
0.7763
0.7815
0.8206
Provide a score based on hittem/ccchractrrssleleeeessssio The some cloth is consistency/edgee artifact/humen by a model
[IAAGE]] produces
clothing, [IMAGE2]
cond
pose between the two images.
Text Encoder
mask
DiT
VAF
VTON-VLLM
High Quality
gt
Figure 8: Data construction pipeline of synthetic VITON-SYN. The instruction provided to VTON-
VLLM simplified here.
23

---

Figure 9: Examples from our VITON-Bench.
FILA
FILA
serinng atting in beeiattre
setting. Sriting in dring roce
setting watling on the beach
seting, starding Inttors
setting madiigg stree?
seitly, watingg in park
settay stawling i librery
setting. welkin in wal
FRE
The see cleth wore mobkl Meeslle
none a model Afuryde morkertaac
by model in Marryle
saus ciats warn lifertyl
settley. attinng
Figure 10: Different prompts along with their corresponding synthesized images from VITON-SYN.
24


Reviews:
[{"review_id": "6uH6bKakfE", "reviewer": "Reviewer_5M4J", "summary": "This paper studies the problem of improving the human alignment of virtual try on models. Particularly, the virtual try on models (VTON) is adopting diffusion models to synthesizing the images given an character and an image of clothes, to generate the character in the provided clothes. And the VTON-VLLM is trained over their collected human preference data as a fashion expert. This expert further used for VTON modle refinmenet through fine-grained supervision and test-time scaling.", "questions": "NA", "limitations": "As in the weaknesses", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The virtual try on task is of important application and research value. The proposed approach brings the latest techs to solve this problem better, and achieve consistent improvement over ViTON-HD.\n2. In general, the paper is well-structured, and easy to follow the idea.\n\nWeaknesses\n1. I'm concerned of the reproducible of this appraoch as the only verified effectiveness is on their own curated VITON-Bench.\n2. It is unclear how the inpainting task can transfer the low-quality image to high-quality one even if assuming the refinement instruction generated from VTON-VLLM is correct, how the inpainting model ensure it's following refinement instruction at a high quality bar.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer 5M4J for recognizing the merits in our paper (e.g., latest techs and consistent improvements in VTON, well-structured paper), and your kind patience in engaging in this rebuttal. Below, we provide a point-by-point response. We hope the information here may further supplement our main submission and address your concerns.\n\n**Q1: Effectiveness of the proposed method on different datasets.**\n\n**A1:** Thanks. To be clear, the mentioned newly curated VITON-Bench (Table 2 in the main paper) is adopted to evaluate our proposal under challenging VTON scenarios. In addition to VITON-Bench, we actually have conducted experiments on two widely adopted benchmarks (i.e., VITON-HD and DressCode), which consistently demonstrates the effectiveness of our approach. Specifically, (1) Table 1 in the main paper reports quantitative comparisons across methods on the VITON-HD dataset, (2) Table 1 and 2 in the supplementary material summarize the results on the DressCode dataset. To support reproducibility and advance the research in VTON, we will release all the training data, source codes and model weights.\n\n**Q2: Instruction following in VTON refinement model.**\n\n**A2:** Appreciate this comment. Technically, to enforce the VTON refinement model to effectively refine images from any VTON model by adhering to instructions, we supervise the training of VTON refinement model with well-constructed training triplets $(I_g, I_h^{LQ}, I_h^{HQ})$, which consists of a reference garment $I_g$, a low-quality person image $I_h^{LQ}$ and a corresponding high-quality version $I_h^{HQ}$. For each triplet, our VTON-VLLM generates a fine-grained instruction $t_r$ to steer the refinement. Please refer to lines 144--166 in the main paper for more training details.\n\nAs suggested, we additionally validated the capability of instruction following in our trained VTON refinement model. Specifically, we randomly sampled 2,000 ground-truth garment-person pairs from the test datasets. For each pair, we first generated a low-quality try-on image and a corresponding refinement instruction using a base VTON model and our VTON-VLLM, respectively. These low-quality images were then refined by our VTON refinement model (VRM-Instruct) based on the provided instructions. Finally, we employed both our VTON-VLLM and GPT-4o as automated evaluators to assess whether the executed image refinement aligns with the given instruction. The success rate of instruction following evaluated by VTON-VLLM and GPT-4o is 98.79\\% and 98.32\\%, respectively. We will add the discussion in revision."}, {"review_id": "hyBokxxxwc", "reviewer": "Reviewer_TTMh", "summary": "The paper collects human feedback on VTON output to train a VLLM tailored to provide image quality assessment tailored for VTON. Then, it uses the instruction of the VLLM model to train a diffusion-based VTON refinement model. One can achieve test-time scaling by iteratively applying the VLLM and the refinement model.", "questions": "1. How important is it to train the VLLM on the human feedback dataset? Can we simply rely on the generalization capabilities of the base VLLM? Does the refinement model and iterative refinement framework still work if the VTON-VLLM is replaced with an off-the-shelf VLLM and the refinement model is retrained on top of the output of the off-the-shelf VLLM?\n2. How many iterations are used for the iterative refinement? How does the compute for the iterative refinement process compare to the compute for inferencing the base model?", "limitations": "The limitations are adequately addressed in the paper.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. The proposed method can improve the eventual VTON output on top of existing VTON methods, even SOTA ones.\n2. The trained VLLM itself can achieve a high preference agreement rate with humans.\n3. There's sufficient ablation showcasing the effect of test time scaling.\n\nI didn't notice any particularly important weakness in the paper. While the paper doesn't provide a particularly striking empirical finding, it presents solid results.", "was_revised": true, "final_justification": "I did not notice any particularly major weakness to the proposed method, and the authors have provided reasonable experimental results regarding both of my questions. Therefore, I keep my positive score.", "rebuttal": "We sincerely thank Reviewer TTMh for recognizing the merits in our paper (e.g., consistent improvements on top of SoTA VTON models, high preference agreement rate with humans), and your kind patience in engaging in this rebuttal. Below, we provide a point-by-point response. We hope the information here may further supplement our main submission and address your concerns.\n\n**Q1: Importance of training VLLM on human feedback dataset: comparisons between Base-VLLM and our VTON-VLLM.**\n\n**A1:** Appreciate the comment. As suggested, we provide quantitative comparisons between two settings: (1) **Base-VLLM**: The VTON refinement model is trained on and guided by the instructions generated by the Base-VLLM without human feedback fine-tuning, (2) **VTON-VLLM**: The VTON refinement model is trained on and guided by the instructions generated by our VTON-VLLM, which is fine-tuned using human feedbacks. Both models are trained and evaluated on VITON-HD dataset. This table lists the quantitative results:\n|Method|$\\mathbf{SSIM\\uparrow}$|$\\mathbf{LPIPS\\downarrow}$|$\\mathbf{FID\\downarrow}$|$\\mathbf{KID\\downarrow}$|$\\mathbf{GC_p\\uparrow}$|$\\mathbf{IQ_p\\uparrow}$|$\\mathbf{GC_u\\uparrow}$|$\\mathbf{IQ_u\\uparrow}$|\n|-|-|-|-|-|-|-|-|-|\n|LaDI-VTON|0.864|0.096|10.531|2.303|4.542|4.621|4.320|4.291|\n|**+Base-VLLM**|0.878|0.059|9.431|1.257|4.779|4.801|4.653|4.573|\n|**+VTON-VLLM**|0.899|0.050|8.916|0.922|4.873|4.920|4.765|4.723|\n|Stable-VTON|0.852|0.084|10.200|1.921|4.625|4.660|4.331|4.405|\n|**+Base-VLLM**|0.870|0.067|9.553|1.323|4.779|4.760|4.624|4.690|\n|**+VTON-VLLM**|0.894|0.052|9.021|1.020|4.873|4.920|4.671|4.750|\n|CAT-VTON|0.870|0.057|9.015|0.670|4.713|4.765|4.510|4.420|\n|**+Base-VLLM**|0.880|0.054|8.976|0.660|4.843|4.880|4.698|4.625|\n|**+VTON-VLLM**|0.891|0.052|8.923|0.652|4.895|4.900|4.733|4.738|\n\nIt can be observed that directly leveraging the Base-VLLM to prompt our proposed VTON refinement model can, to some extent, improve the VTON results compared with the base VTON model. Further empolying our VTON-VLLM to align the refinement with human preferences consistently outperform those with the Base-VLLM across all the metrics, which validates the effectiveness of training VTON-VLLM with human feedbacks on VTON images. We will add this discussion in the revised version.\n\n**Q2: Refinement iterations $N$ and computation overhead.**\n\n**A2:** Thanks for the comment. We have conducted ablation study on ''Effect of Iteration Count $N$ in Test-Time Scaling'' in the supplementary material, and the best results were achieved when $N = 3$ (please refer to lines 17-19 and Table 3 in the supplementary material for more details). Therefore, the iterative refinement proceeds until the output image meets a predefined quality threshold (e.g., high scores across all the sub-aspects defined in the paper) or a maximum of 3 iterations is reached. The computation time for the base VTON model and a single pass of refinement is about 7 seconds and 13.6 seconds (0.35 seconds for instruction generation with VTON-VLLM and 13.25 seconds for image generation with VTON refinement model), respectively. The total computation time for the refinement process with $N$ iterations is approximately $N$ times that of a single pass. The refinement process can be further accelerated by over 50\\% through the use of caching techniques and model distillation in practical applications. Also, users can manually terminate the refinement process if the current output meets their requirements, even after a single refinement step, through an interactive UI."}, {"review_id": "FPZ414QajC", "reviewer": "Reviewer_hywb", "summary": "This paper proposes VTON-VLLM, a vision large language model designed to enhance virtual try-on (VTON) image synthesis by aligning model outputs more closely with human preferences. It introduces a dataset annotated with human feedback, enabling VTON-VLLM to evaluate and guide the synthesis process. Two main approaches are presented: (1) a plug-and-play refinement model (VRM-Instruct) providing fine-grained supervision during training, and (2) adaptive test-time scaling guided by learned human preferences. Additionally, the authors introduce VITON-Bench, a comprehensive benchmark set of challenging scenarios. Experimental results demonstrate improved performance and alignment with human judgments.", "questions": "1. In Tables 1 and 2, the experiments only report results for the combined setting of VRM-Instruct and TTS, without separating the effects of each component. This limits our ability to understand which part of the pipeline contributes most to the performance improvements over the baselines.\n\n2. Given that VTON is a practical, real-world application, I am also concerned about the feasibility of TTS, as it is known to incur significant computational and time costs. Could the authors clarify whether substantial performance gains could still be achieved using only VRM-Instruct, without TTS, to make the approach more practical for deployment?", "limitations": "See weakness above.", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Introducing a VLM explicitly aligned to human preferences for VTON task is original and well-motivated.\n\n2. The new benchmark and human-centric metrics provide valuable resources for future research and enable more meaningful evaluation of VTON quality.\n\n3. VTON-VLLM has a significant improvement over the baseline on the proposed benchmark.\n\nWeaknesses:\n\nMy primary concern of this paper is the novelty. While the paper presents a well-integrated pipeline, many of its technical components—including supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF), and test-time scaling—are established techniques in the broader LLM and VLM literature. The main novelty appears to stem from applying these existing methods to the specific domain of VTON. Given this, I am somewhat uncertain about the proper positioning of the paper.", "was_revised": true, "final_justification": "I still have concern on the novelty of this paper, the main contribution is a VTON benchmark and a trained VTON VLM. I firmly stand by my opinion: this paper simply applies all existing post-training techniques for VLMs to a specific, already-established domain. If it were submitted to the Dataset and Benchmark track, I would consider accepting it. However, based on its current form, I cannot identify any technical contribution that would justify accepting this paper.", "rebuttal": "We sincerely thank Reviewer hywb for recognizing the merits in our paper (e.g., well-motivated approach, valuable resources such as new benchmark and human-centric metrics for future VTON research, significant improvement over the baselines on VITON-Bench), and your kind patience in engaging in this rebuttal. Below, we provide a point-by-point response. We hope the information here may further supplement our main submission and address your concerns.\n\n**Q1: Technical contribution.**\n\n**A1:** Though the latest techniques (SFT, RLHF, TTS) established in LLM/VLM literature are adapted to our proposed well-integrated pipeline for VTON, the success of our work is more than that. Our contributions are from three aspects:\n* **(1) The adaptation of VLLM to VTON synthesis is not trivial.** In our experiments, we found that the efficacy of the pre-trained base VLLM in predicting human judgments on VTON images is unsatisfactory. That's why we devise a new VTON-VLLM (acting as ''fashion expert'') particularly for VTON task. Specifically, we collected high-quality human feedbacks on 40K synthesized VTON images across six fine-grained sub-aspects, which is utilized to fine-tune the base VLLM and align it with human judgments for VTON. Moreover, in contrast to general image quality estimators (e.g., HPS, AES, CLIPScore, ImageReward) that provide only scalar scores, our VTON-VLLM can generate detailed and human-aligned assessments of VTON outputs, which is further utilized to steer the VTON refinement process.\n* **(2) The VTON refinement model guided by fine-grained instruction is new.** Existing VTON methods mostly adopt one-stage inference, which often exhibit limited detail preservation and generation fidelity. In contrast to existing works, our proposed approach novelly combines these VTON models and a plug-and-play VTON refinement model in a two-stage framework, which significantly boosts VTON by transforming the low-quality results to high-quality and preference-aligned ones. Moreover, different from general-purpose image refinement models (e.g., SDXL-Refiner) using generic prompts, our VTON refinement model is guided by the reference garment and fine-grained instructions generated by the trained VTON-VLLM, which provides the visual details of the garment and explicitly identifies local regions in the synthesized image that fail to meet user expectations, respectively.\n* **(3) The human-centric metrics for VTON are new.** Traditional metrics, such as LPIPS and FID, often fail to capture fine-grained inconsistencies in detail preservation and demonstrate poor correlation with human judgments in VTON. To address these limitations, we propose a human-centric evaluation paradigm based on the trained VTON-VLLM, which demonstrates high agreement with human preferences. These new metrics further function as reward signals to guide the VTON refinement model in test-time scaling (TTS), which converts the initial VTON outputs into human-preferred outcomes through iterative refinement process.\n\n\n**Q2: Ablation study on VRM-Instruct and TTS using more baseline VTON models.**\n\n**A2:** Thanks for this point. In the current version, we actually have conducted ablation study on the effect of these two components using a base VTON model (i.e., CAT-VTON) in Table 3 of the main paper. As suggested, we further evaluated the individual contributions of each component using more VTON models presented in Table 1 and 2 by sequentially adding our proposed VRM-Instruct and TTS to them. Please note that TTS is triggered during the iterative refinement process with our VRM-Instruct model as the execution model, and thus cannot be solely integrated into the base VTON models by excluding VRM-Instruct.\n\nAs shown in the revised Table 1 and 2 below, by incorporating the fine-grained instructions from VTON-VLLM to identify imperfect regions in the synthesized garment and guide the refinement process towards human preferences, **base VTON model+VRM-Instruct** achieves notable improvements compared to the base models. Furthermore, we incorporate TTS into **base VTON model+VRM-Instruct** to enable iterative enhancement with human-preference-aware reward provided by VTON-VLLM in the full run (**base VTON model+VRM-Instruct+TTS**), surpassing all the other settings. We will add the discussion in revision.\n\nThis table is the revised version of Table 1 in the main paper.\n|Method|$\\mathbf{SSIM\\uparrow}$|$\\mathbf{LPIPS\\downarrow}$|$\\mathbf{FID\\downarrow}$|$\\mathbf{KID\\downarrow}$|$\\mathbf{GC_p\\uparrow}$|$\\mathbf{IQ_p\\uparrow}$|$\\mathbf{GC_u\\uparrow}$|$\\mathbf{IQ_u\\uparrow}$|\n|-|-|-|-|-|-|-|-|-|\n|IDM-VTON|0.877|0.082|9.372|0.741|4.665|4.750|4.445|4.392|\n|**+VRM-Instruct**|0.880|0.054|9.012|0.667|4.870|4.852|4.712|4.677|\n|**+VRM-Instruct+TTS**|0.882|0.051|8.902|0.634|4.873|4.920|4.724|4.711|\n|SPM-Diff|0.911|0.063|9.034|0.720|4.742|4.775|4.502|4.411|\n|**+VRM-Instruct**|0.890|0.054|9.012|0.667|4.891|4.852|4.712|4.677|\n|**+VRM-Instruct+TTS**|0.892|0.055|9.010|0.663|4.900|4.930|4.745|4.725|\n|CAT-VTON|0.870|0.057|9.015|0.670|4.713|4.765|4.510|4.420|\n|**+VRM-Instruct**|0.885|0.055|8.915|0.660|4.832|4.834|4.673|4.661|\n|**+VRM-Instruct+TTS**|0.891|0.052|8.923|0.652|4.895|4.900|4.733|4.738|\n\n\nThis table is the revised version of Table 2 in the main paper.\n|Method|$\\mathbf{SSIM\\uparrow}$|$\\mathbf{LPIPS\\downarrow}$|$\\mathbf{FID\\downarrow}$|$\\mathbf{KID\\downarrow}$|$\\mathbf{GC_p\\uparrow}$|$\\mathbf{IQ_p\\uparrow}$|$\\mathbf{GC_u\\uparrow}$|$\\mathbf{IQ_u\\uparrow}$|\n|-|-|-|-|-|-|-|-|-|\n|IDM-VTON|0.750|0.083|12.562|1.02|4.374|4.069|4.026|3.825|\n|**+VRM-Instruct**|0.787|0.064|10.931|0.92|4.566|4.398|4.341|4.126|\n|**+VRM-Instruct+TTS**|0.794|0.060|10.784|0.87| 4.612|4.429|4.442|4.302|\n|SPM-Diff|0.765|0.084|12.208|1.34|4.519|4.230|4.235|3.804|\n|**+VRM-Instruct**|0.782|0.069|11.011|0.94|4.570|4.401|4.288|4.027|\n|**+VRM-Instruct+TTS**|0.797|0.062|10.627|0.87|4.623|4.429|4.407|4.280|\n|CAT-VTON|0.751|0.078|12.663|1.23|4.518|4.238|4.150|3.991|\n|**+VRM-Instruct**|0.772|0.063|10.721|0.90|4.542|4.579|4.262|4.156|\n|**+VRM-Instruct+TTS**|0.788|0.060|10.232|0.85|4.555|4.685|4.390|4.353|\n\n\n**Q3: Practical deployment without TTS.**\n\n**A3:** According to the ablation study in Table 3 of the main paper and the additional results provided in the above answer **A2**, substantial performance gains can be attained through single-pass refinement using only VRM-Instruct. Specifically, compared with the base model CAT-VTON in the improved Table 2 from **A2**, **CAT-VTON+VRM-Instruct** boosts $\\mathbf{SSIM}$ and $\\mathbf{IQ_p}$ from 0.751 to 0.772 and from 4.238 to 4.579, respectively, by refining the VTON results with human-preference-aligned instructions via VRM-Instruct. Employing TTS can further improve the generation quality. Compared with **CAT-VTON+VRM-Instruct**, the best scores in $\\mathbf{SSIM}$ (0.788) and $\\mathbf{IQ_p}$ (4.685) are achieved in the full run **CAT-VTON+VRM-Instruct+TTS**."}, {"review_id": "OBVO9pjlf6", "reviewer": "Reviewer_ZRwi", "summary": "This paper pointed our existing evaluate metrics e.g. FID, LPIPS has limited effectiveness in fine-grained visual attributes under Virtual Try-On task. Then it proposed a more human-preference-aligned VLM metric that can evaluate images and use as a reward signal in the Virtual Try-On task setting. In detail it trained a VLM called VTON-VLLM as a \"fashion expert\" which can provide feedback signal to improve existing VTON models performance and align them better with human preferences. The paper also curated VITON-Bench as a hard benchmark. Extensive experiments show the effectiveness of VTON-VLLM in improving existing Virtual Try-On models is demonstrated through comprehensive experiments and ablation studies, also the reliability of VTON-VLLM as a metric is supported by a preference agreement study.", "questions": "Introduction (1) and Motivations (3.3) feel overlapped. Consider merging them together?\n\nQuestion.\nWhy the faces in Figure 2b are not masked, but in Figure 2a they are masked?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "S1) Virtual Try-On is an important topic with a significant gap between current research and commercial applications. The observation that existing evaluation metrics are unreliable in fine-grained visual attributes is well-motivated, and the use of an evaluation model to align Virtual Try-On models with human preferences strengthens the contribution of this work.\n\nS2) The refinement module (VRM-Instruct) can enhance existing VTON models, which is a strong practical contribution can be widely adapted.\n\nS3) The effectiveness of VTON-VLLM in improving existing Virtual Try-On models is convincingly demonstrated through comprehensive experiments and ablation studies.\n\nW1) I believe the paper has no major flaws. One potential weakness is that the dataset focuses exclusively on women, while Virtual Try-On is inherently a task that should generalize across all genders and diverse demographics. For example, the model's performance on male subjects, different body types, or various skin tones remains unclear. Evaluating its robustness in these scenarios would make the work more comprehensive. The current setup also focuses mainly on upper-body clothing and frontal-view images.", "was_revised": true, "final_justification": "I will keep my score at 5. The authors provided experiments demonstrating that the method generalizes well. While the work is solid, it still feels like it lacks a key element of novelty to warrant a score of 6.", "rebuttal": "We sincerely thank Reviewer ZRwi for recognizing the merits in our paper (e.g., well-motivated approach, strong practical contribution that can be widely adapted to existing VTON models, convincing experiments and ablation studies), and your kind patience in engaging in this rebuttal. Below, we provide a point-by-point response. We hope the information here may further supplement our main submission and address your concerns.\n\n**A1: Performances across fine-grained categories.**\n\n**Q1:** Appreciate the comment. The reviewer pointed out that it is crucial to evaluate the model generalization ability of VTON techniques across diverse demographics. As suggested, we reported the performances across different genders, body types and skin tones. In the evaluation, gender, body type, and skin tone are roughly categorized into several classes. The quantitative results on VITON-Bench are summarized as:\n\n**(1) Gender: Female, Male**\n||Male||Female||\n|------|------|------|------|------|\n||$\\mathbf{SSIM\\uparrow}$|$\\mathbf{LPIPS\\downarrow}$|$\\mathbf{SSIM\\uparrow}$|$\\mathbf{LPIPS\\downarrow}$|\n|IDM-VTON|0.8636|0.0775|0.6316|0.0860|\n|**+VRM-Instruct+TTS**|0.8772|0.0525|0.6954|0.0750|\n|SPM-Diff|0.7573|0.0930|0.7683|0.0800|\n|**+VRM-Instruct+TTS**|0.7910|0.0648|0.7992|0.0608|\n|CAT-VTON|0.7500|0.0887|0.7514|0.0734|\n|**+VRM-Instruct+TTS**|0.7763|0.0525|0.7963|0.0632|\n\n**(2) Skin Tone: Black, White, Yellow, Brown**\n||Black||White||Yellow||Brown||\n|------|------|------|------|------|------|------|------|------|\n||$\\mathbf{SSIM\\uparrow}$|$\\mathbf{LPIPS\\downarrow}$|$\\mathbf{SSIM\\uparrow}$|$\\mathbf{LPIPS\\downarrow}$|$\\mathbf{SSIM\\uparrow}$|$\\mathbf{LPIPS\\downarrow}$|$\\mathbf{SSIM\\uparrow}$|$\\mathbf{LPIPS\\downarrow}$|\n|IDM-VTON|0.7191|0.0877|0.7598|0.0881|0.7331|0.0676|0.7357|0.0820|\n|**+VRM-Instruct+TTS**|0.7673|0.0697|0.7788|0.0543|0.7992|0.0508|0.8180|0.0752|\n|SPM-Diff| 0.7543|0.0840|0.7643|0.0874|0.7476|0.0877|0.7376|0.0670|\n|**+VRM-Instruct+TTS**|0.8220|0.0550|0.7734|0.0640|0.7950|0.0580|0.7870|0.0610|\n|CAT-VTON|0.7562|0.0754|0.7538|0.0794|0.7565|0.0775|0.7241|0.0743|\n|**+VRM-Instruct+TTS**|0.7700|0.0500|0.7815|0.0641|0.7570|0.0629|0.8206|0.0520|\n\n**(3) Body Type: Slim, Average, Overweight**\n||Slim||Average||Overweight||\n|------|------|------|------|------|------|------|\n||$\\mathbf{SSIM\\uparrow}$|$\\mathbf{LPIPS\\downarrow}$|$\\mathbf{SSIM\\uparrow}$|$\\mathbf{LPIPS\\downarrow}$|$\\mathbf{SSIM\\uparrow}$|$\\mathbf{LPIPS\\downarrow}$|\n|IDM-VTON|0.7501|0.0834|0.7343|0.0823|0.7986|0.0795|\n|**+VRM-Instruct+TTS**|0.8702|0.0506|0.8306|0.0648|0.8223|0.0558|\n|SPM-Diff|0.7628|0.0837|0.7729|0.0846|0.7680|0.0851|\n|**+VRM-Instruct+TTS**|0.7942|0.0611|0.8070|0.0650|0.8022|0.0638|\n|CAT-VTON|0.7623|0.0879|0.7211|0.0868|0.7603|0.0860|\n|**+VRM-Instruct+TTS**|0.7970|0.0610|0.7521|0.0560|0.7830|0.0627|\n\nFor the quantitative and qualitative results on **lower-body clothing and dresses**, please refer to Table 1 & 2 and Figure 2 in the supplementary material for more details.\n\n\n**Q2: Some duplicate presentation of motivations in paper.**\n\n**A2:** Appreciate this point. For our initial purpose, the discussion in Section 3.1 serves to recap key challenges and motivations for the later introduction of the proposed approach. As suggested, we will combine Sec 3.1 into the introduction section in the revised version.\n\n**Q3：Face mask in figures.**\n\n**A3:** Sorry for the confusion. To clarify: the human images in Fig. 1(a) and Fig. 3 are collected from real-world fashion data and depicted as real individuals, and face masks were applied to these images solely for privacy preservation. In contrast, the images in Fig. 1(b) and Fig. 2 are synthesized by generative models and therefore do not require masking. Moreover, it is worth noting that no face-masking operations were involved during model training or inference. We will add this clarification in the revised version."}]

Author rebuttal:

Meta-review: (a) Summary: the paper focuses on addressing the problem of improving the human alignment of virtual try-on (VTON) models. Existing VTON models, which often utilize diffusion models to synthesize images of a character wearing provided clothes, have limitations in capturing fine-grained visual attributes, and traditional evaluation metrics like FID and LPIPS are often found to be unreliable in this regard.
To overcome these challenges, the authors propose VTON-VLLM, a vision large language model (VLLM) specifically tailored for the VTON task. The authors also introduce a new benchmark dataset, VITON-Bench, which focuses on challenging VTON scenarios, and human-centric metrics for VTON evaluation that better reflect user preferences. 

(b) Strengths:
- The paper addresses an important problem in VTON—the gap between current research and commercial applications and the limitations of existing evaluation metrics in capturing fine-grained visual attributes and human preferences. The explicit alignment to human preferences is considered original and well-motivated.
- The method consistently improves the output of existing VTON models, even state-of-the-art ones, across various datasets and metrics.
- The introduction of VITON-Bench (a comprehensive benchmark for challenging scenarios) and novel human-centric metrics is a significant contribution that provides valuable resources for future research and more meaningful evaluation.
- The effectiveness of VTON-VLLM and its components is convincingly demonstrated through extensive experiments and ablation studies.
- VTON-VLLM itself achieves a high preference agreement rate with humans, validating its role as a "fashion expert".

(c) Weaknesses:
- The technical components leverage existing methods.
- Concerns were raised about the feasibility and practical deployment of TTS due to its known significant computational and time costs.

(d) Reasons to accept: 
The reviewers are generally positive.
- Strong practical impact
- Well-motivated approach to align VTON models with human preferences, identifying the weakness of VTON models of limiting alignment with human preferences and the inadequacy of traditional evaluation metrics. It introduces a human-centric evaluation paradigm and a "fashion expert" VLLM, which is crucial for practical applications.
- While the technical components leverage existing methods, the non-trivial adaptation and specific designs tailored for the VTON domain are significant contributions.

(e) Rebuttal:
- Concerns about the reproducibility of results (effectiveness only verified on VITON-Bench) and the capability of the inpainting model to follow refinement instructions at high quality. Authors clarified that experiments were conducted on three datasets (VITON-Bench, VITON-HD, DressCode) and promised to release training data, source codes, and model weights for reproducibility. They added an evaluation quantifying instruction-following capability for the VTON refinement model (VRM-Instruct), reporting success rates of 98.79% (VTON-VLLM) and 98.32% (GPT-4o).
- Questions about the necessity of training VLLM on human feedback were raised (vs. relying on base VLLM's generalization) and the computational overhead of iterative refinement (TTS). Authors provided quantitative comparisons showing that VTON-VLLM, fine-tuned with human feedback, consistently outperforms a Base-VLLM across all metrics, validating the importance of their training strategy. For TTS, they detailed that the best results were achieved with N=3 iterations and provided a breakdown of computation times, noting potential accelerations and interactive termination.
- Generalization across demographics and minor presentation issues were raised by Reviewer ZRwi. Authors provided extensive additional results on VITON-Bench, demonstrating the robustness and effectiveness of their method across different genders (Male, Female), skin tones (Black, White, Yellow, Brown), and body types (Slim, Average, Overweight). They agreed to combine the overlapping sections into the introduction and clarified that face masks were used solely for privacy preservation in real-world images, not during model training or inference. The reviewer is convinced.
- A question raised about how important to train the VLLM on the human feedback dataset and if it is possible to rely on the generalization capabilities of the base VLLM. Does the refinement model and iterative refinement framework still work if the VTON-VLLM is replaced with an off-the-shelf VLLM and the refinement model is retrained on top of the output of the off-the-shelf VLLM? The authors provided empirical comparisons to address the concern.
Decision: Accept (poster)