CatVTON: Concatenation Is All You Need for Virtual Try-On with Diffusion Models
Authors: Zheng Chong
Xiao Dong
Haoxiang Li
shiyue Zhang
Wenqing Zhang
Hanqing Zhao
xujie zhang
Dongmei Jiang
Xiaodan Liang
Venue: ICLR 2025 Poster (2025)
Keywords: diffusion models
virtual try-on
parameter-efficient training
Primary area: generative models
TL;DR: A high-quality virtual try-on diffusion model with parameter efficiency and simplified inference.

Abstract: Virtual try-on methods based on diffusion models achieve realistic effects but often require additional encoding modules, a large number of training parameters, and complex preprocessing, which increases the burden on training and inference. In this work, we re-evaluate the necessity of additional modules and analyze how to improve training efficiency and reduce redundant steps in the inference process. Based on these insights, we propose CatVTON, a simple and efficient virtual try-on diffusion model that transfers in-shop or worn garments of arbitrary categories to target individuals by concatenating them along spatial dimensions as inputs of the diffusion model. The efficiency of CatVTON is reflected in three aspects: (1) Lightweight network. CatVTON consists only of a VAE and a simplified denoising UNet, removing redundant image and text encoders as well as cross-attentions, and includes just 899.06M parameters. (2) Parameter-efficient training. Through experimental analysis, we identify self-attention modules as crucial for adapting pre-trained diffusion models to the virtual try-on task, enabling high-quality results with only 49.57M training parameters. (3) Simplified inference. CatVTON eliminates unnecessary preprocessing, such as pose estimation, human parsing, and captioning, requiring only a person image and garment reference to guide the virtual try-on process, reducing over 49% memory usage compared to other diffusion-based methods. Extensive experiments demonstrate that CatVTON achieves superior qualitative and quantitative results compared to baseline methods and demonstrates strong generalization performance in in-the-wild scenarios, despite being trained solely on public datasets with 73K samples.

Published as a conference paper at ICLR 2025
CATVTON: CONCATENATION Is ALL YOU NEED FOR VIRTUAL TRY-ON WITH DIFFUSION MODELS
Zheng Chong!44, Xiao Dong', Haoxiang LF, Shiyue Zhang', Wenqing Zhang', Xujie Zhang', Hanqing Zhao Dongmei Jiang* & Xiaodan I LianglA.7 1,,7+
Shenzhen Campus of Sun Yat-sen University, Shenzhen, Guangdong 518107, P.R. China
School of Artificial Intelligence, Sun Yat-sen University, Zhuhai Campus, Zhuhai 519082, China
Ppixocill Labs *Pengcheng Laboratory *LavieAl
'Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China "Guangdong Key Laboratory of Big Data Analysis and Processing, Guangzhou, 510006, China Jchongzheng98, dx. icandolt com, haoxlang@pixociaa com, { (zhangshy, zh6ngwq76, zhangxj59 1maal2..........mm.m...... sys..s..ss.uuuu cn, hg. =ha 90gmaigmm........m com, jiangdm@pcl en, xdliang328@@m 3m
Garment-
Try-On
Person- To-Person
In-The-Wild Try-On
Upper
Lower
Upper
Lower
Costume
Cropped
Drese
Overal!
Dress
Overall
Animate
Overall
Figure 1: CatVTON enables transferring in-shop or worn garments to the target person across var- ious categories. With lightweight architecture and efficient training (49.57M parameters, trained on 73K samples), our model allows inference without additional preprocessing, delivering high- quality virtual try-ons with fine-grained consistency in challenging scenarios like comics, complex backgrounds, special garments, and cropped images.
ABSTRACT
Virtual try-on methods based on diffusion models achieve realistic effects but of- ten require additional encoding modules, large number of training parameters, and complex preprocessing, which increases the burden on training and infer- In this work, re-evaluate the necessity of additional modules and analyze how to improve training efficiency and reduce redundant steps in the inference process. Based on these insights, we propose CatVTON, simple and efficient virtual try-on diffusion model that transfers in-shop or worn garments of arbitrary categories target individuals by concatenating them along spatial dimensions as inputs of the diffusion model. The efficiency of CatVTON is reflected in three aspects: (1) Lightweight network. CatVTON consists only of VAE and sim- plified denoising UNet, removing redundant image and text encoders as well cross-attentions, and includes just 899.06M parameters. (2) Parfmeter-efficicnt training. Through experimental analysis, identify self-attention modules crucial for adapting pre-trained diffusion models to the virtual try-on task, en- abling high-quality results with only 49.57M training parameters. (3) Simplified inference. CaIVTON eliminates unnecessary preprocessing, such as pose estima- tion, human parsing, and captioning, requiring only person image and garment reference to guide the virtual try-on process, reducing 49%+ memory usage com-
"Corresponding author. Project page: http://tttaa.ccm///hogcCCaaaanaa

---

Published as a conference paper at ICLR 2025
pared to other diffusion based methods. Extensive experiments demonstrate that CatVTON achieves superior qualitative and quantitative results compared to base- line methods and demonstrates strong generalization performance in in-the-wild scenarios, despite being trained solely on public datasets with 73K samples.
INTRODUCTION
Warping Madule
Reference
Try-on
Try-on
FID
DOTDifask
Try-oe
Module
MY-VTON
Warping-bused Try-on;
Diffusion-bused Try-on
(Cau))
Memory (G)
(a) Structure Comparison
(b) Efficiency Comparison
Figure 2: (a) Structure comparison of different try-on methods. CatVTON eliminates the need for garment warping additional ReferenceNet resulting in simple structure. (b) Efficiency compar- ison with diffusion where the outer circle denotes the total parameters and the inner circle indicates the trainable pa- rameters. CatVTON achieves lower FID on the VITON-HD dataset with fewer total parameters, trainable parameters, and memory usage.
try-on methods. Each method represented by two concentric circles,
Virtual Try-On (VTON), which transfers specific garments onto user photos, has attracted consider- able interest due to its potential applications o-commerce. Early try-on methods (Han et al. 2018 Wang et al.) 2018 Han et al. 2019 Minar et al.) 2020 Ge al.) 2021 Xie et al. (2021b) employ a two-stage process of pose-guided garment warping followed by blending with the target person, illustrated in the left of Figure 2 (a). However, these methods often result in unnatural fits and struggle with complex poses due to the limited warping process.
Benefitting from the success of diffusion models (Rombach al. 202 many diffusion-based
try-on methods (Zhu et al.) 2023 Kim et al. 20233] Xu et al.] [2024] Morelli et al. 2023 Choi et al.,
2024 Wang et al. | xujie zhang al. 2023 Sun al. (2024) have emerged and achieved more
natural try-on results. As shown in the middle of Figure these methods adopt structure called
Dual-UNet or ReferenceNet for processing garment images. Some methods (Kim et al. 2023 Choi et al.
2024 Xu et al.) 2024 [Sun et al.] 2024) also integrate image encoders, such as CLIP (Radford ct al. 2021 and DINOv2 (Oquab et al. 2023). to capture additional garment features. However, these encoders contribute to a more complex and computationally intensive network architecture, increasing the burdens both training and inference.
To integrate diffusion models into virtual try-on systems without sacrificing efficiency, it is essen- tial to discuss the role of extra image encoders and ReferenceNet. Pre-trained image encoders like DINOv2 and CLIP cations. In contrast, ReferenceNet, by replicating the structure and weights of the backbone UNet, allows for the generation of multi-scale garment features that naturally share latent spaces with the backbone layers. This feature- fcature-sharing facilitates a seamless link between garment and person repre- sentations, improving the overall accuracy of the virtual try-on process. Based on this shared latent space mechanism, we realized that the model architecture could be further simplified. If the gar- ment and person features can be efficiently integrated within the shared latent space, is it possible to use a single UNet model to process both person and garment images simultaneously? Such an approach would not only eliminate redundant encoders but also enhance try-on system efficiency by streamlining the model.
optimized for detail preservation-a crucial factor in virtual try-on appli-
Building on this, we propose CatVTON, a simple and efficient Liffusion-based virtual try-on model. Our Cat removes unnecessary encoders, and streamlines the garment and person interaction,
2

---

Published as a conference paper at ICLR 2025
thereby enabling efficient training and inference. Specifically, shown in Figure model com- prises only a VAE for mapping images to the latent space and a simplified UNet for denoising from LDM (Rombach al. 202I). We further remove the text encoder and the cross-attention modules as text conditions are not essential for try-on, simplifying the architecture to total of 899.06M param- eters. To optimize training efficiency, we investigated the effective modules in UNet to interact with garment and person features. By progressively adjusting the trainable modules in experiments, we find that self-attention modules with a global receptive field (Dosovitskiy et al. 2021) 1 are the most critical part for try-on task with diffusion models, and achieve realistic try-on results by training only 49.57M parameters. Furthermore, we explored more straightforward and efficient inference
process. Numerous try-on methods (Sun et al., 2024: Zhang et al. 2024b, Wang et al.) 2024c) Choi et al. 2024: Kim al. 2023) depend on extra preprocessing such as pose estimation, human parsing, and captioning to guide the try-on process, thereby increasing the computational burden during inference. Hence, we think that the garment and person images contain sufficient informa- tion for try-ons, and removing additional conditions can simplify the model while achieving efficient try-ons without compromising quality. By integrating these enhancements, CatVTON outperforms other diffusion-basse try-on methods in both effectiveness and efficiency, as shown in Figure 2 (b) and T In summary, the contributions of this work include:
We propose CatVTON, lightweight virtual try-on diffusion model with only 899.06M parameters, that achieves high-quality results by simply concatenating garment and person images inputs, eliminating the need for extra image encoders, ReferenceNet, and text- conditioned modules.
We introduce parameter-efiiee efficiient training strategy to transfer pre-trained diffusion models to virtual try-on tasks while preserving prior knowledge by training necessary modules with only 49.57M parameters.
We simplify the inference process by eliminating the need for extra pre-processing of input images and leverage the robust priors from pre-trained diffusion models to infer all nec- essary information, reducing memory usage by 49%+ compared other diffusion-based baselines.
Extensive experiments on the VITON-HD and DressCode datasets demonstrate that our method produces high-quality virtual try-on results with consistent details, outperforming state-of-the-arn art baselines in qualitative and quantitative analyses, and performs well in in- the-wild scenarios.
2 RELATED WORK
2.1 SUBJECT-DRIVEN IMAGE GENERATION
Subject-driven image generation is a hot topic in the field of image generation, focusing on inte- grating the target subject into new scenes or perspectives while maintaining consistency with the subject. LoRA (Hu et al.) (02T)) and DreamBooth (Ruiz et al.) (022)) train individual models for each subject, achieving consistent subject-driven generation, but the frequent training incurs a high cost. Paint by Example (Yang et al., 2022) and IP-Adapter (Ye et al. 2023) leverage CLIP (Rad-) ford et al. 202T) image encoders to extract subject features and inject them into diffusion models via cross attention,,, enabling convenient subject-driven generation. However, they fall short of pre- serving details. contrast, AnyDoor (Chen et al.) 2023) employs DINOv2 (Oquab et al.] (003)) and ControlNet (Zhang et al.) 2023) to jointly extract subject features to achieve more accurate subject-driven image generation. PCDMs (Shen et al., 2024) achieve high consistency in transfer- ring persons different perspectives through three progressive diffusion models. InstantID (Wang et al., (024b) introduces an additional IdentifyNet to encode facial information, achieving high- fidelity facial stylization. Similarly, MimicBrush (Ju al. 2024) proposes a dual-branch model that leamss from video data and masked image modeling to accomplish subject-driven generation. While these methods achieve high-quality subject-driven generation, they also lead complex network architectures and a large number of trainable parameters, which limit their applications.
2.2 IMAGE BASED VIRTUAL TRY-ON
In image- based virtual try-on, the goal is to create composite image of person wearing a spec- ified garment while maintaining identity and consistency. Warping- based methods typically de- compose the task into stages: garment warping and fusion based on warped garments. Some
3

---

Published as a conference paper at ICLR 2025
Concatioatt in Space
Self Attention
Concatonato in Channd]
Split Space
10% dropout
Skip Cruss Attn
Emboed
Learnable
Frozen
FFN
Transformer Block
Blacks
VAE Encoder
Transformer Blacks
VAE Decoder
Mask-free
Mask-bused
Denoising UNet
Figure 3: Overview of CatVTON. Our method achieves high-quality try-ons by simply concatenat- ing the conditional image (garment or reference person) with the target person image in the spatial dimension, ensuring they remain in the same feature space during denoising. Only self-attention pa- rameters, which provide global interaction, are learnable, while cross-attention for text interaction omitted. No additional conditions (pose, parsing) are needed, resulting in lightweight network with minimal trainable parameters and simplified inference.
warping- based methods (Wang et al. 2018 Han et al. 2018 Choi et al. (021) utilize geometric deformation like TPS (Bookstein] 1989) warp garment, while others (Han et al. 2019 Ge
et al. 2021: Xie et al. 2021b 2023 Gou et al. 2023) estimate appearance flow map to model non-rigid deformation for more complex garment warping. Besides, PASTA-GANs (Xie al. 2022 a0 propose patch-routed disentanglement module for pose-guided garment warping. However, warping-based methods often struggle with alignment issues caused by inaccurate TPS or flow es- timation. Diffusion-basee methods leverage the generation capacity of pre-trained diffusion models to avoid the limitations of garment warping. LaDI- VTON (Morelli et al.. 2023) and VITON (Kim al. (203) employ ControlNet-like structure encode additional information. TryOnDif- fusion (Zhu et al.) 2023) designs two UNets for feature extraction of garment and person images, respectively, and achieves impressive results. BoowYTON (Zhang et al., 2024a) utilizes generated pseudo data to train the diffusion model and employs clothing encoder to provide garment infor- mation, achieving mask-free virtual try-on. OOTDiffusion (Xu al. 2024). StableGarment (Wang) et al. 2024c), IDM-VTON (Choi et al. 2024). and OutfitAnyone (Sun et al. 2024) utilize a Ref- erenceNet structure, similar the denoising UNet from pre-trained models, process garment images, with slight structural variations. However, these methods often require complex network structures, numerous trainable parameters, and various conditions assist inference, which inspires exploration towards efficient virtual try-on diffusion models.
METHODS
CatVTON aims to streamline diffusion- based virtual try-on methods by climinating redundant com- ponents, focusing key modules, and simplifying preprocessing requirements.
3.1 LIGHTWEIGHT NETWORK
Our lightweight structure arises from the consideration of image representations for garments and persons and their effective interaction. Recent studies (Ye et al. 2023 Chen et al. 2023) have demonstrated that existing pre-trained encoders, such as DINOv2 (Qquab et al.) 2023 and CLIP (Radford et al., 2021), struggle to preserve fine details for subject-driven image generation. This indicates that using these encoders to encode garment images for try-on purposes is insufficient, hence we remove all additional image encoders in our method. Methods with ReferenceNet en- hance detailed alignment diffusion-based try-on by replicating weights from a denoising UNet and performing fine-tuning. However, this approach introduces additional trainable modules and increases the computational load. To address this, we concatenate person and garment images along the spatial dimension inputs to the original denoising UNet to avoid importing any modules.
4

---

Published as a conference paper at ICLR 2025
Mask-based Duta Generation
Mask-free Training
Mask-hased Try-on Model
Mask-free Try-on Model
Random
Generated
Real
Generated Pairs
Mask-free Result
Real
Sampled Pair
Penson
Gamment
Set Ancesions
Person
Figure 4: Overview of the mask-free training pipeline. We first use the trained mask-based model to generate synthetic person image from randomly sampled person-garment pairs. These synthetic person images, along with their corresponding original person and garment images, form the training data for the mask-free model.
As shown in Figure 3 CatVTON features a lightweight network structure comprising only two es- sential modules: (1) VAE. The VAE encoder encodes the person and garment images into the latent space, optimizing computational efficiency during diffusion. Once encoded, the latent garment and person are concatenated in the spatial dimension as inputs to the denoising UNet. Then, the VAE decoder reconstructs the output latent into the original pixel space after denoising. (2) Simplified Denoising UNet. As the text condition is necessary for image-based try-on tasks and our exper- iment reveals that training with text conditions leads to a detrimental impact on try-on performance (demonstrated in experiments section). we remove the text encoder and cross- cosssatttnnioon modules in the UNet to further simplify the network and reduce 167.02M parameters. The simplified denoising UNet accepts concatenated garments and persons as conditions, along with noise and masks, and generates the predicted try-on latent. Integrating these two modules, the proposed lightweight try-on diffusion model has only 899.06M parameters, representing a reduction over 44% compared to other diffusion-basee methods.
3.2 CEAMETER-EFFICIIE TRAINING
CatVTON aims to optimize the interaction between garment and person features with the fewest trainable modules in LDMs (Rombach et al.] (021) for parameter-efficient training. Diffusion-baser methods typically train the entire U-Net to adapt pre-trained models to the virtual try-on task. How- since LDMs have undergone extensive pre-training on large-scale datasets, they already possess robust prior knowledge. When transferring LDMs to the try-on task, is only necessary to fine-tune the parameters related to the interaction between person and garment features.
As shown in Figure 3 the denoising UNet comprises alternating ResNet (He et al. 2013) and trans- former (Vaswani al. (023) blocks. The transformer blocks, equipped with self-attention layers for global interaction, complement the ResNet's local feature capture, which stems from its convo- lutional architecture. We conduct experiments gradually find the most relevant modules. We set the trainable components to I) the entire U-Net, 2) the transformer blocks, and 3) the self-attention layers. The results indicate that despite a significant disparity in the number of trainable parame- ters (815.45M, 267.24M, and 49.57M, respectively), all three variants produced satisfactory virtual try-on results, and no substantial differences are observed in visual quality and metrics among them (detailed in experiments section).
Consequently, we adopted parameter-efficiet training strategy by finetuning only the self-attentio layers with 49.57M parameters. For the training of the mask-free try-on model, first leverage the already trained mask-based model to infer generated person images from randomly sampled person- garment pairs in the same datasets. These generated person images, along with their corresponding original person and garment images, form the training data for the mask-free model, as shown in Figure For both the mask-based and mask-free try-on models, (MSE) Toss for training. Additionally, we adopt a 10% conditional dropout to support classifier guidance (CFG) (Ho & Salimans, 2022) and employ the DREAM (Zhou al. 2024] strategy during training. The ablation studies of CFG and DREAM are illustrated in the experiments section.
employ Mean Squared Error
3.3 SIMPLIFIED INFERENCE
Besides training, also explored more straightforward and more efficient inference process for image-based try-on. We simplified the inference by eliminating the need for any preprocessing conditional information. The whole process can be completed with only the person image and
5

---

Published as a conference paper at ICLR 2025
SudiCCrmeet DOSSTOO
GPATTON
LEBUCTON
MOVIOON
OOTDITuses IDASNTON
GRVTON OOTDifes
EMAVION
(a) VITONHD
(b) DressCode
Figure 5:
Qualitative comparison on the VITON-HD and DressCode dataset. CatVTON demon-
strates a distinct advantage in handling complex patterns and text. Please zoom in for more details.
garment reference for the mask-free model and additional binary mask for the mask-based model. Specifically, given a target person image Ip E 3x3 and a binary cloth-agnostic mask M E RxW xW an input person image I, is obtained by:
Lp &
if mask-free
=
else
where & represents the element-wiss wise (Hadamard) product. Then input person image IR Hx and the garment reference (either in-shop garment worn person image) 1n |3x/XW is encoded into the latent space by the VAE encoder e:
X; = (1,0g)
(2)
where @ denotes the concatenation operation along the spatial dimension and X; x For mask-based model, M is also concatenated with all-zero masks and then interpolated match size latent space, resulting m, R x
M Interpolate
(3)
where O represents the all-zero mask with the same size as M. At the beginning of the denoising, the input conditions and random noise 27 ~ N0 1) E R4x x of the same size as X; are concatenated along the channel dimension and input to the denoising UNet to get predicted zT- and this process is repeated for T times to predict the final latent 2g. For denoising step this process can be written as:
UNet(z+ O X.) UNet(z; O M, O X²)
if mask-free
else
where @ denotes the concatenation operation along the channel dimension, finally, E B x is then split across the spatial dimension to extract the person part = E R # 4, we use the VAE decoder D to transform the denoised latent representation = back into the image space, producing the final output image Ip 3x HxW.
T D (Split (zo, W)),
(5)
where Split(,, W) means split across the spatial dimension in width.
4
EXPERIMENTS
4.1 DATASETS
Our experiments are conducted on three public datasets: VITON-HD (Choi et al. 2021). Dress- Code (Moeell al. 2022). and DeepFashion (Ge et al. 2019 VITON-HD comprises 13,679

---

Published as a conference paper at ICLR 2025
IDM-VTON
OOTDifissis OuttiAAyyue
Our
MOVE
RQIST
(a) In-the-wild Results
(b) In-the-wild Comparisons
Figure 6: Qualitative results and comparisons in in-the-wit wild scenarios. OutfitAnyone (Sun al. 2024) only supports inference on its provided person images. Our method combines background, person, and garment more naturally in complex scenarios. Please zoom in for more details.
image pairs of upper, 11,647/2,032 training/testing pairs. DressCode is composed of 48,392/5,400 training/testing pairs with full-body person images and in-shop upper, lower, and dresses. Besides, we select 13, 33,098/1,899 training/testing image pairs for the garment transfer task from the in-shop clothes retrieval benchmark of the DeepFashion dataset, which includes 52,712 high-resolution per- son images. For DressCode and DeepFashion datasets, we process clothing-agnoste masks using human parsing results from DensePose (Güler et al.) 2018) and SCHP (Li et al. 2020) of LIP (Gong ct al. 2017) and ATR (Liang et al. 2015) versions.
4.2 IMPLEMENTATION DETAILS
We utilize the inpainting and InstructPix2Pix (Brooks et al., 2023) version of StableDiffusion v1.5 (Rombach et al., 2021) as the base models for the mask-based and mask-free try-on mod- els, respectively. We train two models for each version on the VITON-HD (Choi et al., 202[ and DressCode (Morelli et al.) (202)) datasets separately for fair quantitative comparisons with previous methods. The AdamW (Loshchilov & Hutter 2019) optimizer is employed with a batch size of 128 and constant learning rate of 1e 5 for 16, 000 steps training under 512x384 resolution and DREAM À = 10. Additionally, multi-task models are trained on the three datasets (~ 73K image pairs) under x024x resolution for 48, 000 steps with an identical setup but a batch size of 32. All experiments are conducted on 8 NVIDIA A800 GPUs, which takes approximately 10 hours for 16K training steps.
4.3 METRICS
For paired try-on settings with ground truth in test datasets, we employ four widely used metrics to evaluate the similarity between synthesized images and authentic images: Structural Similar- ity Index (SSIM) (Wang et al., 20044) Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al.) 2018). Frechet Inception Distance (FID) (Seitzer, (0200) and Kernel Inception Distance (KID) (Binkowski et al., 2021). For unpaired settings, we use FID and KID to measure the dis- tribution of the synthesized and real samples.
4.4 QUALITATIVE COMPARISON
Figure 5 (a) presents the try-on results of garments with complex patterns from the VITON-HD
Choi et al., 202I) dataset. While other methods often exhibit artifacts, loss of detail, and blurry text logos, CatVTON demonstrates its superiority by effectively handling texture positioning and occlusions and producing more photo-realistic results. Figure 5 (b) illustrates the comparison for different garment types (upper, lower, and dress) full-body person images from the DressCode (Morelli et al. 2022) dataset. Our approach can generate results that are more consistent with the garment textures, length, and semi-transparent materials. We provided additional qualitative
7

---

Published as a conference paper at ICLR 2025
Table 1: Quantitative comparison with other methods. We compare the metrics under paired and unpaired settings on the VITON-HD and DressCode datasets. The best and second-best results are demonstrated in bold and underlined, respectively.
VITON-HI
DressCode
Methods
Paired
Unpaired
Paired
Unpaired
SSIM
FID
1
KID.
LPIPS &
FID KID SSIN FID, J KID
LPIPS 1 FID 1 KIDY
DCI VTON (Gou al. 2023
0.8620
9.408
4.547
0.0606
12.531
5.251
StableVTON Iklm ar 2025
0.8543
6.439
0.942
0.0905
11.054
3,914
StableGarment
2024g
0,8029
15.567
8.519
0.1042
17.115
8.851
MV-VTON (Wang ereir craie 20z40
0.8083
15.442
7.501
0.1171
17.900
8.861
GP-
0.8701
8.726
3.944
0.0585
11.844
4.310
0.7711
9.927
4,610
0.1801
12.791
6.627
LaDI-VTON (Morelil erair 2023
0.8603
11,386
7,248
0.0733
14.648
8.754
0.7656
9.555
4.683
0.2366
10.676
5.787
IDM. VTON
0,8499
5.762
0.732
0.0603
9.842
1.123
0.8797
6.821
2.924
0.0563
9.546
4.320
OOTDiffusiol
ou or 2024
0.8187
9.305
4.086
0.0876
12,408
4.689
0.8854
4.610
0.955
0.0533
12.567
6.627
CaVYTON (Mask-Free)
0.8701
0.513
0.0613
4.779
1.297
0.0452
7.400
CaIVTON (Inpainling)
0.8704
5.425
0.411
0.0565
9.015
1,091
0.8922
3.992
0.818
0.0455
6.137
1.403
Table 2: Comparison of GFLOPs, inference time, and memory usage across different methods.
GFLOIs
Inference Time(s)
Memory Usage
Methods
Diest
Linuge
ReferenceNet
UNet
512x384
1024x768
512x384
1024x768
OOTDiffusion (Xu al. 2024
13.08
155.62
509.12
547.34
4.76
36.23
6854 M
8892 M
IDM- NOO- Choi ala
2024
110.04
155.62
1340.15
1163.98
12.96
17.32
17112 M
18916 M
StableVTON Kim al.
2025
2023
155.62
173.80
545.27
12.17
36.10
9828
14176 M
CaIV
973.59
2.58
9.25
3276
5940
results and comparisons in various in-the-wild scenes, shown in Figure exceptionally well on fine patterns of garments, without altering or distorting the text and patterns on the garments. can also accurately reproduce special clothing designs, producing realistic effects such as wrinkles, lighting, and shadows.
Our method performs
4.5 QUANTITATIVE COMPARISON
Comparison of Effect. We conducted the quantitative comparison of effect with several open- source try-on methods on the VITON-HD and DressCode datasets under both paired and unpaired settings as presented in Table Our method outperformed others across the metrics. GP-NTON (Xie et al. 2023), IDM-VTON (Choi et al. 2024), and OOTDiffusiin (Xu et al. 2024) also showed good performance. GP-VTON, as warping-based method, had advantages in SSIM and LPIPS but performed weaker in KID and FID. This result suggests that warping-based methods may focus more on ensuring structural and perceptual similarity but lack realism and detailed naturalness.
Comparison of Efficiency. Table 3 and Figure 2 (b) demonstrate the quantitative comparison of efficiency, including parameters, memory usage, and extra conditions for inference. Our method contains only modules, VAE and UNet, with 899.06M parameters. Moreover, our trainable parameters are reduced by 10+ times compared to other methods. During inference, our method has a significant advantage in memory usage and does not require extra conditions such as pose or text, alleviating the burden of inference.
Table presents the inference efficiency comparison of GFLOPs, inference speed, and memory usage across different methods at 5 512x384 and 1024x768 resolutions on a single NVIDIA A100 GPU. Inference was performed with batch size of 1. For inference time, we averaged the results of 10 runs with the same input to ensure accuracy. GFL.PPs were calculated using the calflops (xiaoju ye (023) library. These comparison results demonstrate that our model can be deployed on resource-constrainet devices, such as consumer-level GPUs with less than 8 GB of VRAM, while maintaining significantly better inference speed compared to other models. However, deploying high-resolution image generation models terminal devices, such smartphones, remains area that requires further exploration.
4.6 ABLATION STUDIES
Trainable Module. We evaluated three modules for training: (1) UNet, (2) transformer blocks, and (3) self-attention. As shown in Table more training weights do not bring significant improvements in performance but increase the memory requirement and decrease the training speed. Slight advan- tages brought by additional training weights may be due to the increased trainable components, which allow the model to fit the data distribution quickly. Besides, trained self-attention
8

---

Published as a conference paper at ICLR 2025
Table Detailed comparison of model efficiency. UNetref Erezt, and Eimage represent the Refer- enceNet, text encoder, and image encoder, respectively. Compared to other diffusion- methods, CatVTON uses fewer modules, reducing total parameters by about 2x and trainable parameters by 10x+. CatVTON requires significantly less memory during inference and does not need additional conditions such as pose or text.
Params (M)
Memory Usage(G)
Conditions
Methods
VAE
UNet
UNeivef
Etest
Eumage
Total
Trainable
Pose Text
OOTDiffuuion (Xi et al.
2024
83.61
859.53
859.52
85.06
303.70
2191.42
1719.05
10.20
IDM-VTON CChol et al.
2024
83.61
2567.39
2567.39
716.38
303.70
6238.47
2871.09
26.04
StableVTON (Kim ct al.
2025
83.61
859.41
361.25
303.70
1607.97
500.73
7.87
StahleCiarment Wang el al. 2024cc
83.61
859.53
1220.77
85.06
2248.97
1253.49
11.60
MV-VTON (Wang el al.
2024a
83.61
859.53
361.25
316.32
1620.71
884.66
7.92
CaVYTON (OurS)
83.61
815.45
899.06
49.57
4.00
Table 4: Ablation results of different trainable modules. More trainable modules slightly impact performance but increase memory usage and slow training. Extra text conditions harm performance. IPS (items per second) indicates training speed.
Paired
Unpaired
Trainable Parums (M)
Training IPS
Training Memory (M)
Trainable Module
SSIM 1 FDD
KID 1 LPIPS
FID
KID
UNet
0.8692
5,2496
0.4017
0.0550
8.8131
0.9559
815.45
3,21
14289
Transformers
0.8558
5.4496
0.4434
0.0558
8.8423
1.0082
267.24
4.10
9981
Self Attention + Text
0.8517
6.5744
1.0690
0.0772
9.6998
1.6683
49.57
4.50
88055
Self Attention
0.8704
5.4252
0.4112
0.0565
9.0151
1.0914
49.57
4.75
8451
version with text conditions in the same setting, and the results show a decrease in performance, indicating that text conditions redundant in image-based try-ons, Training only the self-attention modules and removing unnecessary text conditions achieves a balance between model performance and efficiency. The IPS and memory statisties are calculated in a setting with a batch size of I to avoid the impact of other environmental factors.
Classifter-Free Guidance. To evaluate the effect of classifier-free guidance (CFG), we run infer- ences with CFG strengths of 0.0, 1.5, 2.5, 3.5, 5.0, and 7.5 while keeping all other parameters constant. Figure (b) shows that increasing CFG strength enhances image detail and fidelity. How- ever, beyond a strength of 3.5, the results developed severe color distortions and high gigh-freuueccy noise, degrading visual quality. We found that CFG strength between 2.5 and 3.5 produces the most realistic and natural results. A CFG strength of 2.5 is used for all the other experiments.
hyperparameter that controls the strength of DREAM. Specifically, À means DREAM is disabled. As shown in Fig- ure 7 (a), small À causes overly smooth im- ages, while a large À introduces excessive high- frequency noise, reducing naturalness. Table 5 shows resutls trained with different À values on the VITON-HD dataset. SSIM increases with X, while FID and LPIPS first improve and then degrade, highlighting a trade-off between re- duced distortion and perceptual quality. We
DREAM.
Table 5: Ablation results of different À in DREAM on VITON-HD dataset. λ=00 means no DREAM. Increasing À improves perceptual qual- ity (lower LPIPS, KID, and FID) but increases dis- tortion (lower SSIM) an empirical range.
Paired
Unpaired
SSIM
FID J
KID
IPIPS 4
FID |
KID
0,8740
10.4534
3.8866
0.0692
14.1045
5.2824
0.8716
8.0983
2.1977
0.0646
11.7652
3.2942
0.8704
5.4252
0.4112
0.0565
9.0151
1.0914
0.8633
5.5861
0.4005
9.0877
find that À = 10 best balances naturalness with detail fidelity in our training.
0,8614
5.5561
0.3657
0.0631
8.9114
1.0049
CONCLUSION
In this work, we present CaIVTON, a virtual try-on diffusion model with lightweight architec- ture, efficient training, and streamlined inference. CatVTON achieves a compact structure with 899.06M parameters by removing unnecessary text-related modules, reducing model complexity significantly. CatVTON proposes a parameter-efffcient training strategy to focus the most essen- tial components, specifically the self-attention layers, preserving high-quality virtual try-on perfor- mance while minimizing training costs with only 49.57M trainable parameters. During inference,

---

Published as a conference paper at ICLR 2025
CTG Strength
Condition
Condition
10
20
1.5
2.5
3.5
(a) Comparisons for 2. in DREAM
(b) Comparisons for CFG Strengths
Figure 7: Visual comparisons for different À in DREAM and different CFG strengths. When too small, results are overly smooth and lack detail; when À is too large, results have excessive high- frequency details and appear unnatural. As the CFG strength increases, the details in the generated images increase, but beyond 3.5, it leads severe color distortion and high-frequency noise.
CatVTON eliminates the need for additional information such as pose estimation, human parsing, and text-based inputs, reducing memory requirements and enhancing inference speed. Extensive experiments demonstrate that CatVTON delivers superior qualitative and quantitative results, out- performing state-of-the-art methods while maintaining a compact and efficient architecture. These findings underscore CatVTON's potential for practical applications and open new research direc- tions in virtual try-on technology.
ACKNOWLEDGMENTS
Supported by Shenzhen Science and Technology Program No. .,GUH172022099111422000000 National Natural Science Foundation Youth Basic Research Program (Doctoral Students) No.623B2100 and Nansha Key R&D Program under Grant No. 2022DD42
REFERENCES
Mikolaj Binkkwski, Danica J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans, 2021. URL
01401]
Fred L. Bookstein. Principal warps: Thin-plate splines and the decomposition of deformations.
IEEE Transactions on pattern analysis and machine intelligence, 11(6):567-585, 1989.
Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix22ixx: Learning to follow image
editing instructions, 2023. URL httppp:::///rivii/ http:////wwww..r//////211.....1. 09800
Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: Zero- shot object-level image customization. arXiv preprint arXiv:2307.09488. 2023.
Seunghwan Choi, Sunghyun Park, Minsoo Lee, and Jaegul Choo. Viton-hd: High-resolution virtual try-on via misalignment-awvee aware normalization. In Proc. the IEEE conference on computer vision and pattern recognition (CVPR), 2021.
Yisol Choi, Sangkyung Kwak, Kyungmin Lee, Hyungwon Choi, and Jinwoo Shin. Improving dif- fusion models for virtual try-on. arXiv preprint arXiv:2403.051.9, 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x words: Transformers for image recognition at scale, 2021. URL htt.tt::/saaii..oottt /abs/////////// 021.0.20.11999
Yuying Ge, Ruimao Zhang, Lingyun Wu, Xiaogang Wang, Xiaoou Tang. and Ping Luo. versatile benchmark for detection, pose estimation, segmentation and re-identification of clothing images. CVPR, 2019.
Yuying Ge, Yibing Song, Ruimao Zhang, Chongjian Ge, Wei Liu, and Ping Luo. Parser-free virtual 2021.
try-on via distilling appearance flows. arXiv preprint
10

---

Published as a conference paper at ICLR 2025
Ke Gong, Xiaodan Liang. Dongyu Zhang, Xiaohui Shen, and Liang Lin. Look into person: Self- supervised structure- sensitive learning and a new benchmark for human parsing, 2017. URL
https://arxiv www./o/gg////ooooooot.ooose.ssst 1703.05445
Junhong Gou, Siyu Sun, Jianfu Zhang, Jianlou Si, Chen Qian, and Liqing Zhang. Taming the power of diffusion models for high-quality virtual try-on with appearance flow. In Proceedings of the s1 ACM International Conference on Multimedia, 2023.
Riza Alp Güler, Natalia Neverova, and lasonas Kokkinos. Densepose: Dense human pose estimation in the wild, 2018.
Xintong Han, Zuxuan Wu, Zhe Wu, Ruichi Yu, and Larry S Davis. Viton: An image-based virtual try-on network. In CVPR, 2018.
Xintong Han, Xiaojun Hu, Weilin Huang, and Matthew Scott. Clothflow: A flow-based model for clothed person generation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 10471-10480, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition, 2015. URL httthhttt///aattttt org/ abs 1512. 03385
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance, 2022. URL BtlLhttss://arttt
org/ abs/ 2207 12598
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https:
09685
Xuan Ju, Xian Liu, Xintao Wang, Yuxuan Bian, Ying Shan, and Qiang Xu. Brushnet: A plug-and- play image inpainting model with decomposed
diffusion, 2024.
Jeongho Kim, Gyojung Gu, Minho Park, Sunghyun Park, and Jaegul Choo. Stableviton: Learning semantic correspondence with latent diffusion model for virtual try-on, 2023.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2022. URL https:
7/ arriivv.co org. abs/13/ss1111//. 6114
Peike Li, Yunqiu Xu, Yunchao Wei, and Yi Yang. Self-correction for human parsing. IEEE Transac-
fions on Pattern Analysis and Machine Intelligence, 2020. doi: 10.
Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang. Xingchao Liu, Minbin Huang, Zedong Xiao, et al. Hunyuan-dit: A powerful multi-resolltion diffusion transformer with fine-grained chinese understanding. arXiv preprint 0rX8v22405.08748 2024.
Xiaodan Liang. Si Liu, Xiaohui Shen, Jianchao Yang. Luoqi Liu, Jian Dong, Liang Lin, and Shuicheng Yan. Deep human parsing with active template regression. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(12):240-24414 December 2015. ISSN 2160- 9292. doi: 10. 1009tppmi.2055.440000 2015.240833
URL http:://k/x /ttp:////xxxx.....o. wttp:/...///////........
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. 2019.
Matiur Rahman Minar, Thai Thanh Tuan, Heejune Ahn, Paul Rosin, and Yu-Kun Lai. Cp-vton+: Clothing shape and texture preserving image-based virtual try-on. In The IEFE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. June 2020.
Davide Morelli, Matteo Fincato, Marcella Cornia, Federico Landi, Fabio Cesari, and Rita Cucchiara. Dress code: High-resolution multi multi-category virtual try-on, 2022.
Davide Morelli, Alberto Baldrati, Giuseppe Cartella, Marcella Cornia, Marco Bertini, and Rita Cuc- chiara. LaDI-VTON: Latent Diffusion Eextual-Inversion Enhanced Virtual Try-On. In Proceed- ings of the ACM International Conference Multimedia, 2023.

---

Published as a conference paper at ICLR 2025
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniee, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXth:2304.07193, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models, 2021.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. 2022.
Maximilian Seitzer. pytorch-fid: FID Score for PyTorch. htttp:://github.com pytorch- fid, August 2020. Version 0.3.0.
tzer/
Fei Shen, Hu Ye, Jun Zhang, Cong Wang, Xiao Han, and Wei Yang. Advancing pose-guided image synthesis with progressive conditional diffusion models, 2024. URL httpsss///axii..orrgggg
Abs/2310.06313
Ke Sun, Jian Cao, Qi Wang, Linrui Tian, Xindi Zhang, Lian Zhuo Bang Zhang, Liefeng Bo, Wenbo Zhou, Weiming Zhang, and Daiheng Gao. Outfitanyone: Ultra-high quality virtual try-on for any clothing and any person, 2024. WtRLhttps:///// http://awww....//////////00/0/ 240p://bb//24777 16224
Ashish Vaswani, Noam Shazeer. Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL https:
org/ org/abs/ 1706/7706.000 03762
Bochao Wang, Huabin Zheng, Xiaodan Liang, Yimin Chen, and Liang Lin. Toward characteristic- preserving image-based virtual try-on network. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 589-604, 2018.
Haoyu Wang, Zhilu Zhang, Donglin Di, Shiliang Zhang, and Wangmeng Zuo. Mv-vton: Multi-view virtual try-on with diffusion models. arXiv preprint arXiv:2404.127 /7364, 2024a.
Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, Anthony Chen, Huaxia Li, Xu Tang, and Yao Hu. Instantid: shoo identity-preserviee preserving generation in seconds, 2024b. URL httpss:/arxxv.. org/abs/2401. 07519
Rui Wang, Hailong Guo, Jiaming Liu, Huaxia Li, Haibo Zhao, Xu Tang, Yao Hu, Hao Tang, and Peipei Li. Stablegarment: Garment-centric generation via stable diffusion. arXiv preprint MrXiv:2403110783 2024c.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions image processing, 13(4):600- 612, 2004.
xiaoju ye. calflops: flops and params calculate tool for neural networks in pytorch framework, 2023. htLlhttps://githhe
flops. wwt.ffl..sss.pp.plooooooooomo
Zhenyu Xie, Zaiyu Huang, Fuwei Zhao, Haoye Dong, Michael Kampffmeyer, and Xiaodan Liang. Towards scalable unpaired virtual try-on via patch-routed spatially-adaptive gan, 2021a. URL
https: attps://arxiv. / abs 2111 1111.11544..
Zhenyu Xie, Xujie Zhang, Fuwei Zhao, Haoye Dong, Michael C. Kampffmeyer, Haonan Yan, and Xiaodan Liang. Was-vton: Warping architecture search for virtual try-on network, 2021b. URL
https::/arxiss otg/abs 2108 00386
Zhenyu Xie, Zaiyu Huang, Fuwei Zhao, Haoye Dong, Michael Kampffmeyer, Xin Dong, Feida Zhu, and Xiaodan Liang. gastt- A versatile framework for high-resolution unpaired virtual try- on, 2022. URL htt.hhttp://eete http://awwiiiorrrr//////////////
13475
12

---

Published as a conference paper at ICLR 2025
Zhenyu Xie, Zaiyu Huang, Xin Dong, Fuwei Zhao, Haoye Dong, Xijin Zhang, Feida Zhu, and Xiaodan Liang. Gp-vton: Towards general purpose virtual try-on via collaborative local-flow global-parsing learning. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23550-23559, 2023.
Yuhao Xu, Tao Gu, Weifeng Chen, and Chengcai Chen. Ootdiffusion: Outfitting fusion based latent diffusion for controllable virtual try-on. arXiv preprint arXiv:2403.0177, 2024.
xujie zhang. Xiu Li, Michael Kampffmeyer, Xin Dong, Zhenyu Xie, Feida Zhu, Haoye Dong, and Xiaodan Liang. Warpdiffusion: Efficient diffusion model for high-fidelity virtual try-on, 2023.
URL htttttts///tiiiiise wwwp/as///////oo/////.../...ee/e 2312. 03667
Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and
Fang Wen. Paint by example: Exemplar-based image editing with diffusion models. arXiv preprint arXiv:2211.13227 2022.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. 2023.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models, 2023.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as a perceptual metric. In CVPR, 2018.
Xuanpu Zhang, Dan Song, Pengxin Zhan, Qingguo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Anan Liu. Boow-vton: Boosting in- in-the-we wild virtual try-on via mask-free pseudo data training, 2024a. URL/httss:/ /tattps://arivvi org/ abs/ 2408. 06047
Xujie Zhang, Ente Lin, Xiu Li, Yuxuan Luo, Michael Kampffmeyer, Xin Dong, and Xiaodan Liang, Mmtryon: Multi-modal multi-reference control for high-quality fashion generation, 2024b. URL
https://arxxt org/ abs /0...22055..0 00448
Jinxin Zhou, Tianyu Ding, Tianyi Chen, Jiachen Jiang. Ilya Zharkov, Zhihui Zhu, and Luming
Liang Dream: Diffusion rectification and estimation-adaptive models, 2024. URL https:
00210
Luyang Zhu, Dawei Yang. Tyler Zhu, Fitsum Reda, William Chan, Chitwan Saharia, Mohammad Norouzi, and Ira Cemelmacher-Shlizemaa Tryondiffusion: A tale of two unets, 2023.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 PRELIMINARY
Latent Diffusion Models. The core idea of Latent Diffusion Models (LDMs) (Rombach et al, 202T) is to map image inputs into a lower-dimensional latent space defined by pre-trained Varia- tional Autoencoder (VAE) (Kingma Welling 2022). In this way, Diffusion Models can be trained and inferred at a reduced computational cost while retaining the capability to generate high-qualit images. The components of LDMs are primarily a denoising UNet (o t) and a VAE which con- sists of encoder & and decoder D. Given input 2, the training of LDM is carried out by minimizing the following loss function:
LLDM == R ((0..)) [lle -
co( 1)]3]
where / E [I,...T} denotes the timestep of the forward diffusion process. In the training phase, the latent representation 2, is readily derived from E with the added Gaussian noise ~ NNO, 1). Subsequently, the latent samples, drawn from the distribution p(2). are translated back into the image domain with just one traversal of D.
Diffusion Rectification and iidimaation-daaiivi Models (DREAM). DREAM (Zhou et al, 2024) is a training strategy designed to skillfully navigate the trade-off between minimizing dis- tortion and preserving high image quality in image super-resolution tasks. Specifically, during train- ing, the diffusion model is used to predict the added noise ég. This Eg is then combined with the original added noise e to obtain é, which is used to compute by:
2 + vI (E + Xεạ),
where is a parameter to adjust the strength of ea and ā = I 1-11111 with the variance scheduler {3 E (0, 1)) The training objective for DREAM can be expressed as:
LDREAM == Ex(x)) c, (O., t), [I[e - Aco) - ()
(8)
DREAM enhances training efficiency and accuracy, although it requires additional forward pass before the training prediction process, slightly slowing down the training process.
A.2 IMPLEMENTATION DETAILS
A.2.1 MASK-FREE DATASET
The construction of the mask-free dataset is based on the VITON-HD (Choi et al. (Morelli et al.) 2022), and DeepFashion (Ge et al. 2019) datasets. We the mask-based model to randomly generate pseudo-data for constructing mask-free paired data in the same garment category (uppers, lowers, and dresses). To ensure the accuracy of the masks, we employ multiple human parsing models (ATR(Liang et al. 2015). LIP(Gong et al.) (017]) and body part information from DensePose (Güler et al.,|200 [2018) cross-validate the required mask regions. Additionally, convex hull and pooling operations are applied to ensure no information leakage in areas outside the mask. This comprehensive approach guarantees the quality of the generated data required for training the mask-free model, thereby enabling it to focus on the try-on regions.
2021 DressCode
A.2.2 HARDWARE ENVIRONMENT
We conducted our experiments on Linux server with an x86 architecture. It is equipped with an Intel Xeon CPU and NVIDIA A800 GPUs, each with 80GB VRAM.
A.2.3 SOFTWARE ENVIRONMENT
Our work is implemented based on the PyTorch deep learning framework, with the version being 2.1.2. The code for the diffusion model is modified and implemented based on HuggingFace's Diffusers library.
A.2.4 CONCATENATION ALONG X/Y-AXIS
During training, we experimented with the direction of spatial concatenation (along the x-axis or y-axis). Theoretically, for convolutional neural networks and Transformers without positional em- beddings, the direction of spatial concatenation- whethhe along the x-axis or y-axis should make no difference. Our experimental results consistent with this theory; training with either x-axis or y-axis concatenation can produce normal results. Moreover, a model trained with x-axis concatena- tion also yield normal results when using y-axis concatenation during inference.
14

---

Published as a conference paper at ICLR 2025
SUSIeVTUU
StableGamen
DUEVIUN
GRVTIN
LaDEVION
MV-VTON
IDSE-YTON
WITZIN
MOME
MAT
Lec
Figure 8: More visual comparisons on the VITON-HD dataset with baseline methods. Please zoom in for more details.
A.3 MORE VISUAL COMPARISONS
A.3.1 VTION-HD DATASET
Figure 8 presents additional virtual try-on results on the VITON-HD (Choi et al.) 2021 test dataset, where our method demonstrates an advantage in preserving the details of text, patterns, and logos, and can adaptively fuse with the target person while maintaining reasonable scale. In addition, CatVTON exhibits a more natural representation of garment designs such as sleeves and collars.
A.3.2 DRESSCODE DATASET
15

---

Published as a conference paper at ICLR 2025
Puired
Unpuired
Methods
SSIM
FID 4
KID
LPIPS 4
FID 4
KID +
StableGarment Wang al.
2024c
0.8029
15.567
8.519
0.1042
17.115
8.851
MV-VTON Wang 20244
0,8083
15.442
7.501
0.1171
17.900
8.861
LaDI-VTON Morelli et 2023
0.8603
11.386
7.248
0.0733
14.648
8.754
DCI-VTON
0.8620
9.408
4.547
0.0606
12.531
5.251
OOTDitfusion
2012
0.8187
9.305
4.086
0.0876
12,408
4.689
GP-VTON Xie 2025
0.8701
8.726
3.944
0.0585
11.844
4.310
StableVITON Kim 2023
0.8543
6.439
0.942
0.0905
11.054
3.914
CatVTON (DiT)
0.9118
5.786
0.939
0.0393
10.019
1.864
Table 6: Quantitative comparison of DiT version with other methods on VITON- Choi et al.
2021 dataset. The best and second-best results are demonstrated in bold and underlined, respec- tively.
GP-VTON
OOTDifuusion
IDM-VTON
Ouns
The visual comparisons the DressCode (Morelli et al. 2022) test dataset are further dis- played in Figure 9 where our method can bet- ter recognize and match the lengths of different types of clothing and can generate more coher- ent patterns for situations such as arm occlu- sion.
A.4 TRANSFERABILITY
To extend the transferability of our proposed method, we conducted experiments using Hun- yuanDiT (Li al. (024) as the pre-trained model on the VITON-HI (Choi et al. 2021) dataset. Table 6 presents comparative re- sults of our approach within the HunyuanDiT framework on the VITON-HD dataset. Al- though DiT converges more slowly compared to UNet and has not fully fitted the data, the re- sults of our DiT-based version still outperform most existing methods.
A.5 LIMITATIONS & SOCIAL IMPACTS
While leveraging LDM (Rombach et 2021 as the backbone for generation, certain limitations. Images decoded by VAE may exhibit detail loss and color discrepancies, particularly at a 512x384 resolution. Addition- ally, the effectiveness of the try-on process is contingent upon the accuracy of the provided mask; an inaccurate mask can significantly de- grade the results. Based on Stable Diffusion v1.5, our pre-trained model was trained on large-scale datasets that include not-safe-for. work (NSFW) content. Consequently, retaining most of the original weights means our model may inherit biases from the pre-trained model, potentially generating overly explicit images of people.
model faces
Figure 9: More visual comparisons on the Dress- Code dataset with other baselines. Please zoom in for more details.
16


Reviews:
[{"review_id": "yGxtIGxFdO", "reviewer": "Reviewer_mw7R", "summary": "The study re-evaluates the necessity of additional modules in virtual try-on methods based on diffusion models and proposes CatVTON, a simple and efficient model. CatVTON transfers garments of arbitrary categories to target individuals by concatenating them as inputs, removing redundant encoding modules and steps. It features a lightweight network with just 899.06M parameters, efficient training with only 49.57M parameters through self-attention modules, and simplified inference that requires only person image and garment reference, reducing memory usage by 49%+. Experiments show that CatVTON achieves superior results and generalizes well in real-world scenarios, despite being trained on a limited public dataset.", "questions": "Please refer to \"Weaknesses.\"", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ 1. This paper proposes CatVTON, a lightweight virtual try-on diffusion model with only 899.06M parameters, that achieves high-quality results by simply concatenating garment and person images as inputs, eliminating the need for extra image encoders, ReferenceNet, and text-conditioned modules.\n\n+ 2. This paper introduces a parameter-efficient training strategy to transfer pre-trained diffusion models to virtual try-on tasks while preserving prior knowledge by training necessary modules with only 49.57M parameters\n\n+ 3.  This paper simplifies the inference process by eliminating the need for extra pre-processing of input images and leveraging the robust priors from pre-trained diffusion models to infer all necessary information, reducing memory usage by 49%+ compared to other diffusion-based baselines", "weaknesses": "- 1. The effectiveness of the try-on process relies on the accuracy of the provided masks; inaccurate masks may significantly degrade the quality of the results. When the authors conducted mask-free experiments, how did they ensure that the model paid attention to the try-on areas?\n\n- 2. Some of the data in Table 1 seem to differ from those in the original paper, why?\n\n- 3. Based on Table 3, it seems that choosing to train either the Unet or Transformers module can achieve better results, albeit with higher memory demands and slower training. How to balance this relationship?\n\n- 4. Please provide the training code to demonstrate reproducibility.\n\n- 5. The method is simple and effective, however, the entire process is overly engineered, lacking rigorous theoretical validation and support in a conference like ICLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UWiSHK6gzA", "reviewer": "Reviewer_oVwb", "summary": "CatVTON is a virtual try-on model that achieves high-quality and efficient garment transfer without relying on additional encoders or complex preprocessing. Unlike traditional virtual try-on approaches, CatVTON simplifies the architecture by using only a VAE and a single, lightweight UNet with a self-attention module, reducing the parameter count to 899.06M. Moreover, this model directly concatenates the person and garment images along spatial dimensions, requiring only 49.57M trainable parameters and cutting down inference memory usage by over 49% compared to other methods. Extensive experiments demonstrate that CatVTON achieves superior results in both controlled and in-the-wild settings.", "questions": "- Can you provide additional quantitative results on CatVTON’s inference speed and computational cost, and is the model deployable on embedded or resource-constrained devices?\n\n- Given that the mask-free model builds on a trained mask-based model, have you observed any error accumulation in this process, and if so, what techniques might mitigate such errors?\n\n- Could the lightweight model design and parameter-efficient training strategy you propose for CatVTON be applicable to other virtual try-on frameworks, including non-diffusion-based methods?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- CatVTON proposes a lightweight architecture, which highly reduces computational cost during the model training. Meanwhile, it only requires 49.57M training parameters.\n\n- The model generalises well in diverse real-world scenarios, performing effectively even with limited training data.\n\n- CatVTON is able to achieve realistic try-on results with accurate garment alignment and detail preservation.", "weaknesses": "- While the authors present CatVTON as a lightweight virtual try-on diffusion model and it only requires very few training parameters, it would be beneficial to include results on inference speed and computational cost. Additionally, demonstrating the model’s deployability on embedded devices could enhance understanding of its practical applications.\n\n- The mask-free model is derived from a trained mask-based model. It would be better to discuss potential error accumulation in this transition.\n\n- CatVTON introduces a model design and training strategy aimed at creating a lightweight model with fewer training parameters. It would be useful to consider whether this approach could be extended to other virtual try-on pipelines, particularly non-diffusion-based methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YM5ETXpidM", "reviewer": "Reviewer_F9hE", "summary": "This paper aims to efficiently adapt pre-trained diffusion models to high-quality virtual try-on. It introduces efficiency improvements in three key areas: model architecture, parameter-efficient training, and simplified inference, and analyzes the motivation and feasibility from the perspective of image representation interaction, supported by experimental validation. Extensive qualitative and quantitative experiments are conducted on VITON-HD and DressCode datasets, along with ablation studies on key components of proposed method. Compared to other virtual try-on methods, this method demonstrates significant advantages in model parameters, training efficiency, and inference costs while achieving better performance. Despite being trained on small-scale public datasets, results and comparisons in in-the-wild scenarios demonstrate the method's generalization capabilities.", "questions": "1. Can the proposed efficient training method be transferred to other more novel diffusion models, such as DiT?\n\n2. Will the generated dataset used for mask-free training be open-sourced to support future research?\n\n3. Provide more details about Mask-free training, such as dataset construction and how it differs from Mask-based model training, visual comparison of its results with Mask-based model results, etc.\n\n4. Adjust the layout of the tables and figures to make them more uniform and standardized.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Originality: This paper focuses on efficiency optimizations in diffusion models for virtual try-on. By simplifying the model architecture with a concatenation operation and identifying key modules through an analysis of image representation interactions, it achieves parameter-efficient training. Additionally, by streamlining preprocessing in inference, it achieves efficient inference without extra conditions. The approach and model architecture differ significantly from other virtual try-on models, addressing the efficiency issues of previous work. The paper offers new insights into the application of image representation interactions in generative models, demonstrating innovation.\n\n2. Quality: This paper conducts extensive comparative and ablation experiments to validate its performance. Qualitative and quantitative analyses are provided on two major open-source datasets, DressCode and VITON-HD, in comparison with recent virtual try-on methods. Quantitatively, the proposed model shows significant advantages in model size, trainable parameters, and inference memory usage, while surpassing other methods in FID, KID, SSIM, and LPIPS metrics. Visually, the model generates highly realistic details such as text logos and patterns, and performs well in in-the-wild virtual try-on scenarios.\n\n3. Clarity: The writing is well-structured, with a clear and logical flow of content. Plenty of figures, tables, and formulas are used to explain and analyze key insights and experimental details. The writing is smooth, with no obvious grammatical errors or inconsistencies.\n\n4. Significance: This work offers insights and contributions to the application of diffusion models in virtual try-on tasks. Its lightweight architecture and parameter-efficient training significantly reduce model training costs and may be transferable to other downstream tasks. The model's low memory usage during inference and simplified inference process reduce hardware requirements, making it more cost-effective to deploy.", "weaknesses": "1. The description of the proposed mask-free training method is not specific and detailed enough, more information should be provided such as the dataset details used for this part of the training and the exact process of pseudo-data construction.\n\n2. The Concatenation operation applied to garment and person features lacks a more explicit and detailed discussion, such as its advantages over encoder features like CLIP or DINO.\n\n3. The transferability of the proposed efficient virtual try-on method based on Stable Diffusion is not discussed. For example, migrating this training framework to more advanced image generation diffusion models, such as DiT, to verify the generality of this framework.\n\n4. Some inappropriate formatting issues, such as inconsistent font sizes in tables. The font in Table 1 and Table 4 is relatively small, and there is excessive whitespace in Table 4. The images in Figure 5 and Figure 6, which are used to display generated results, are arranged too tightly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MrzLWiIz3k", "reviewer": "Reviewer_iWwH", "summary": "**Post-discussion period**\n\n---\n\nI will maintain my current score of 5, as the issues I raised in my review remain unresolved:\n\n+ Discrepancies in Quantitative Results: There are significant discrepancies between the reported results in the submission and the original baseline papers. This inconsistency makes the comparison less convincing. To address this, I suggested that the authors use the original evaluation scripts from the baseline papers alongside their own settings, but it is disregarded. This would provide a fairer comparison, particularly given the considerable variance observed in metrics like LPIPS, FID, and KID.\n\n+ Architectural and Textual Inconsistencies: There is a minor but important inconsistency between the overall architecture figure and the description in the main text. Specifically, the process of input concatenation, whether in pixel-space or latent-space, is unclear. This discrepancy raises questions about whether the results are indeed identical to what was claimed.\n\nFor me, if there are unclear aspects in a submission and the authors disregard suggestions such as using original scripts for reevaluation, I cannot raise the score or recommend acceptance unless these concerns are fully addressed. The final decision I would let the effort of ACs.\n\n---\n\nThis work introduces CatVTON, a diffusion-based virtual try-on model that enhances efficiency by removing complex modules and preprocessing steps typical in other approaches. Diffusion-based virtual try-on methods usually require additional encoding, extensive training, and preprocessing, which CatVTON addresses by simplifying the architecture to a VAE and denoising UNet, reducing parameters to 899.06M. Through analysis, the authors find that self-attention modules are essential for adapting diffusion models to this task, enabling high-quality outputs with only 49.57M training parameters. CatVTON further reduces inference complexity, eliminating pre-processings like pose estimation and human parsing, which cuts memory usage by over 49%. Extensive testing demonstrates CatVTON’s strong performance and generalization, even with a limited 73K sample dataset, achieving better results than baseline methods in both controlled and real-world scenarios. Experiments are conducted on two datasets: VITON-HD and DressCode.", "questions": "Q1. What is the IPS metric? It is better to provide its definition and description of it if it is newly proposed or referred to if needed.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "+ Removing the redundant reference Unet of most existing work is a good direction as it makes the model lightweight and simplified.\n+ Results on two benchmark datasets VTON-HD and DressCode show good performance.", "weaknesses": "**Concerns**\n+ Inference speed comparison is missing since the proposed model is more lightweight, it is more beneficial to see this metric.\n\n+ Many works (AnyDoor, MimicBrush, Uni-ControlNet) show that a strong pre-trained image encoder (CLIP, DINOv2, Unet) would provide more priors and good features for better imager generation with better details, but this work shows that removing all of them, and using only VAE outputs can give even better performance. Which in my opinion, it might not be very reasonable. A valid explanation is needed to this point and also I believe that whether the code of this paper is publicly available for the community to verify its validity and reproducibility is crucial.\n\n+ The detail of all numeric numbers in Table 1, and Table 2 should be clarified. Were all of them reproduced by the author or were they quoted from somewhere else? Looking at the original papers (OOTDiffusion and others) I find that they report much higher numbers compared to the numbers shown in this submission. It raised questions about the correctness and what made the differences since they used also the same dataset.\n\n+ \"DREAM \\lambda\", just putting this terminology here is not very informative, it is quite hard to understand what is that parameter \\lambda and what is DREAM here without some more detailed explanations.\n\nFor the whole paper, the current form seems to use all \\cite{} making it very messy for all references. I think the use of \\citep{} in latex would produce a more correct presentation of citations for many parts of the paper. Using \\cite{} for cases where the citing author is subject (S), but \\citep{} for other cases when referring to the paper.\n\n[1] AnyDoor: Zero-shot Object-level Image Customization, CVPR 2024\n\n[2] MimicBrush: Zero-shot Image Editing with Reference Imitation, arXiv:2406.07547\n\n[3] Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models, EnruIPS 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)