CatVTON: Concatenation Is All You Need for Virtual Try-On with Diffusion Models

## Abstract

Abstract Virtual try-on methods based on diffusion models achieve realistic try-on effects but replicate the backbone network as a ReferenceNet or leverage additional image encoders to process condition inputs, resulting in high training and inference costs.
In this work, we rethink the necessity of ReferenceNet and image encoders and innovate the interaction between garment and person, proposing CatVTON, a simple and efficient virtual try-on diffusion model. It facilitates the seamless transfer of in-shop or worn garments of arbitrary categories to target persons by simply concatenating them in spatial dimensions as inputs. The efficiency of our model is demonstrated in three aspects: (1) Lightweight network. Only the original diffusion modules are used, without additional network modules. The text encoder and cross attentions for text injection in the backbone are removed, further reducing the parameters by 167.02M. (2) Parameter-efficient training. We identified the try-on relevant modules through experiments and achieved high-quality try-on effects by training only 49.57M parameters ( ∼ similar-to \sim ∼ 5.51% of the backbone network’s parameters). (3) Simplified inference. CatVTON eliminates all unnecessary conditions and preprocessing steps, including pose estimation, human parsing, and text input, requiring only garment reference, target person image, and mask for the virtual try-on process. Extensive experiments demonstrate that CatVTON achieves superior qualitative and quantitative results with fewer prerequisites and trainable parameters than baseline methods. Furthermore, CatVTON shows good generalization in in-the-wild scenarios despite using open-source datasets with only 73K samples.

## 1 Introduction

Virtual Try-On (VTON) is a technology that superimposes specific garments onto user photos.
It has recently garnered widespread attention due to its potential applications in the e-commerce industry.
Traditional try-on methods [, , , , , , , , , ] typically employ a two-stage process which involves warping the garment based on the pose of target person and then blending the warped garment onto the person, as shown in Fig. 3.
However, these methods have drawbacks, including producing unnatural and ill-fitting garments and having difficulties in handling complex poses.

Figure: Figure 2: Efficiency comparison with diffusion-based try-on methods. Each method is represented by two concentric circles, where the outer circle denotes the total parameters and the inner circle denotes the trainable parameters, with the area proportional to the parameter number. CatVTON achieves lower FID[] on the VITONHD[] dataset with fewer total parameters, trainable parameters, and memory usage. More details see Sec. 4.5.
Refer to caption: x2.png

Recently, inspired by the success of conditional generation method based on diffusion models[], numerous virtual try-on approaches
[, , , , , , , , ] based on diffusion models have emerged, achieving breakthroughs in high-quality try-on results. Most of these methods adopt a structure called Dual-UNet or ReferenceNet (as shown in Fig. 3b), which entirely or partially replicates the backbone UNet [] to encode garment features and utilize attention mechanisms to facilitate interaction between garment and person features in the original UNet. Despite mitigating the drawbacks of traditional methods, these approaches rely on complex networks and entail a substantial number of trainable parameters.
Some methods [, , , ] also utilize additional pre-trained image encoders such as CLIP [] and DINOv2 [] to extract garment features, further increasing the computational burden for training and inference.

To address the efficiency issues in previous works, we reconsider the necessity of ReferenceNet and extra image encoders for virtual try-on tasks.
The primary goal in virtual try-on tasks is ensuring the consistency of garment details between the try-on result and the original reference.
However, pre-trained image encoders are not designed for this purpose. For example, DINOv2[] focuses on self-supervised pre-training with extensive collections of clustered images, primarily for classification, segmentation, and detection tasks. Similarly, CLIP [] undergoes extensive pre-training using text-image pairs to align semantics across modalities.

ReferenceNet, proposed as another form of image encoder to encode garment images into multi-scale features, is introduced to complement the limitations of pre-trained image encoders.
The key idea of ReferenceNet is to replicate weights from the backbone UNet and maintain an identical network structure. It naturally shares latent spaces with the backbone UNet, significantly facilitating feature interaction.
However, it raises the question of whether it would be more logical to use the same UNet to process the person and the garment simultaneously if the shared latent spaces are essential for high-quality try-on.

Figure: Figure 3: Simple structure comparison of different kinds of try-on methods. Our approach neither relies on warped garments nor requires the heavy ReferenceNet for additional garment encoding; it only needs simple concatenation of the garment and person images as input to obtain high-quality try-on results.
Refer to caption: x3.png

Following this line of thought, we simply concatenated the garment and person images along the spatial dimension and fed them into a single UNet backbone (as shown in Fig. 3c), training it to predict the try-on results, and yielded high-quality outputs.
After verifying the feasibility of using a single UNet, we identified the modules that contribute to producing high-quality virtual try-on results. The UNet in LDM consists of alternating ResNet [] and transformer[] blocks. Theoretically, ResNet, as a convolutional network, is only responsible for feature extraction due to its limited receptive field, while Transformer blocks can learn global context.

Furthermore, looking into the transformer block, the cross-attention in the UNet interacts with texts and images, which is not required in image-based virtual try-on tasks. In contrast, the self-attention mechanism, responsible for global interaction, is the most essential module for the try-on task. As a result, we removed the text encoder and cross-attentions in the original diffusion model, making self-attention the only trainable module and achieving a lightweight network with parameter-efficient training.

After that, we explored a more straightforward and more efficient inference process. Many methods [, , ] rely on extensive pre-processing, such as pose estimation and human parsing, to guide the try-on process. Some methods use text to provide additional information, such as garment categories.
We argue that much of this information is inherently present in the original person or garment images, and pre-trained diffusion models have already acquired robust priors from extensive data, enabling them to infer this information effectively. Therefore, we streamlined the inference process by eliminating all pre-processing and conditional information, requiring only person and garment images and binary masks to complete the try-on process.

In summary, the contributions of this work include:

- •
We propose CatVTON, a lightweight virtual try-on diffusion model that simply concatenates garment and person images in spatial dimensions as the input to achieve high-quality try-on results. It eliminates the need for additional image encoders or ReferenceNet and removes unnecessary text encoders and cross attentions for more efficient architecture.
- •
We introduce a parameter-efficient training strategy to transfer a pre-trained diffusion model to the virtual try-on task while preserving prior knowledge by training only necessary modules with 49.57M parameters.
- •
We simplify the inference process by eliminating the need for any pre-processing of input person and garment images or text information. The pre-trained diffusion model with robust priors is responsible for inferring all necessary information from the input images.
- •
Extensive experiments on the VITON-HD and DressCode datasets demonstrate that our method produces high-quality virtual try-on results with consistent details, outperforming state-of-the-art baselines in qualitative and quantitative analyses., and performs well in in-the-wild scenarios.

## 2 Related Work

### 2.1 Subject-driven Image Generation

In the field of image generation, especially within text-to-image tasks, diffusion models [, , , , ] have become widely employed backbones due to their robust semantic priors accumulated through extensive training on large-scale datasets. Efforts to adapt these models for subject-driven image generation have produced several notable approaches.
Paint by Example [] replaces text conditions with image conditions, utilizing a CLIP [] image encoder to prompt image generation. This approach enables image-to-image generation but often preserves the original subjects’ features only roughly. Similarly, IP-Adapter [] injects CLIP image features into a pre-trained latent diffusion model via an adapter, allowing for plug-and-play subject-to-image generation with similar limitations in feature preservation.
DreamBooth [] refines diffusion models by introducing specific vocabulary to denote particular subjects, achieving consistent subject-driven text-to-image generation. However, it requires fine-tuning the model for each new subject. In contrast, AnyDoor [] employs DINOv2 [] and a ControlNet [] conditioned on high-frequency maps to achieve relatively accurate subject-driven image generation without the need for extensive model adjustments.
BrushNet [] introduces a pluggable dual-branch model, which utilizes pixel-level masked image features for image restoration, enhancing the precision of the generated images. MS-Diffusion [] leverages layout guidance to facilitate multi-subject zero-shot image personalization, providing a flexible and efficient approach to generating personalized images without prior subject-specific training. These methods highlight the diverse strategies employed to enhance subject-driven image generation, each addressing different aspects of the challenge with varying degrees of success and specificity.

### 2.2 Image-based Virtual Try-On

In image-based virtual try-on, the goal is to create a composite image of a person wearing a specified garment while maintaining identity and consistency.
Warping-based methods typically decompose the task into two stages: training a warping module and constructing a generation module based on the warping results. VITON [] utilizes Thin Plate Spline (TPS)[] transformation for garment warping with a coarse-to-fine pipeline. CP-VTON [] explicitly delineates the try-on strategy in two stages: warping and generation.
PF-AFN [] combines knowledge distillation with appearance flow constraints to produce high-quality virtual try-on effects.
GP-VTON [] uses local flow to warp garment components and global parsing to compose them, improving consistency in detail and overall appearance.
However, these methods often struggle with alignment issues caused by explicit flow estimation or distortions.
Diffusion-based methods leverage pre-trained diffusion models’ generative capabilities to avoid the limitations of warping.
WarpDiffusion[] bridges the warping-based and diffusion-based paradigms via an informative and local garment feature attention mechanism.
TryOnDiffusion [] utilizes two UNets for feature extraction and interaction of garment, person, and pose conditions, achieving impressive results with expensive computational costs. LaDIVTON [] maps garment features to the CLIP embeddings and conditions the latent diffusion model along with the warped input. StableVITON [] proposes a zero cross-attention block to learn the semantic correlation between garment and person features.
MMTryon[] combines multi-modal conditions and multi-reference garments into a diffusion-based framework. OOTDiffusion [] finetunes a pre-trained outfitting UNet to learn garment details in one step and incorporates them into the denoising UNet via outfitting fusion.
However, these methods often require complex network structures, numerous trainable parameters, and various conditions to assist inference. In this work, we achieve a simpler and more efficient virtual try-on diffusion model by simplifying the architecture and inference process.

## 3 Methods

Figure: Figure 4: Overview of CatVTON. Our method achieves the high-quality try-on by simply concatenating the conditional image (garment or reference person) with the target person image in the spatial dimension, ensuring they remain in the same feature space throughout the diffusion process. Only the self-attention parameters, which provide global interaction, are learnable during training. Unnecessary cross-attention for text interaction is omitted, and no additional conditions, such as pose and parsing, are required. These factors result in a lightweight network with minimal trainable parameters and simplified inference.
Refer to caption: x4.png

### 3.1 Preliminary

Latent Diffusion Models. The core idea of Latent Diffusion Models (LDMs) [] is to map image inputs into a lower-dimensional latent space defined by a pre-trained Variational Autoencoder (VAE) . In this way, Diffusion Models can be trained and inferred at a reduced computational cost while retaining the capability to generate high-quality images.
The components of LDMs are primarily a denoising UNet $E_{\theta}(\circ,t)$ and a VAE which consists of an encoder $\varepsilon$ and a decoder $\mathcal{D}$. Given an input $x$, the training of LDM is carried out by minimizing the following loss function:

$$ $L_{LDM}:=\mathbb{E}_{{\large\varepsilon}(x),\epsilon\sim\mathcal{N}(0,1),t} \left[\|\epsilon-\epsilon_{\theta}(z_{t},t)\|_{2}^{2}\right],$ (1) $$

where $t\in\{1,...,T\}$ denotes the timestep of the forward diffusion process. In the training phase, the latent representation $z_{t}$ is readily derived from $\varepsilon$ with the added Gaussian noise $\epsilon\sim\mathcal{N}(0,1)$. Subsequently, the latent samples, drawn from the distribution $p(z)$, are translated back into the image domain with just one traversal of $\mathcal{D}$.

Diffusion Rectification and Estimation-Adaptive Models (DREAM). DREAM [] is a training strategy designed to skillfully navigate the trade-off between minimizing distortion and preserving high image quality in image super-resolution tasks. Specifically, during training, the diffusion model is used to predict the added noise as $\epsilon_{\theta}$. This $\epsilon_{\theta}$ is then combined with the original added noise $\epsilon$ to obtain $\hat{\epsilon}$, which is used to compute $\hat{z_{t}}$:

$$ $\hat{z_{t}}=\sqrt{\overline{\alpha}_{t}}z_{0}+\sqrt{1-\overline{\alpha}_{t}}( \epsilon+\lambda\epsilon_{\theta}),$ (2) $$

where $\lambda$ is a parameter to adjust the strength of $\epsilon_{\theta}$ and $\overline{\alpha}_{t}=\prod^{t}_{i=1}1-\beta_{i}$ with the variance scheduler $\{\beta_{t}\in(0,1)\}_{t=1}^{T}$ . The training objective for DREAM can be expressed as:

$$ $L_{DREAM}:=\mathbb{E}{\large\varepsilon(x),\epsilon,\epsilon{\theta}\sim \mathcal{N}(0,1),t}\left[|(\epsilon+\lambda\epsilon_{\theta})-\epsilon_{\theta }(\hat{z_{t}},t)|_{2}^{2}\right].$ (3) $$

DREAM enhances training efficiency and accuracy, although it requires an additional forward pass before the training prediction process, slightly slowing down the training process.

### 3.2 CatVTON

CatVTON aims to streamline diffusion-based virtual try-on methods by eliminating redundant components, thereby simply and efficiently transferring the latent diffusion model to the try-on task.
In this section, we detail the composition and principle of our lightweight network (Sec. 3.2.1), the exploration of the parameter-efficient training strategy (Sec. 3.2.2), and the simplified inference process (Sec. 3.2.3) that removes unnecessary conditions.

#### 3.2.1 Lightweight Network

The key innovation of CatVTON is its lightweight network, which is designed to operate efficiently without compromising performance. This stems from our consideration of the image representation of garments and persons and their effective interaction.
The most comprehensive representation of an image is the image itself; any form of image encoding inherently involves compression and extraction of the information it contains.
Specifically, image encoders like DINOv2[] employ a substantial collection of clustered images for self-supervised pre-training, utilized for downstream tasks such as classification, segmentation, and detection. CLIP[] undergoes extensive pre-training with text-image pairs to align the semantics of the two modalities. However, the objectives of these approaches do not align with the precise image detail alignment required for try-on tasks.
ReferenceNet is then proposed for detail alignment in try-on, ensuring that the person and garment features are aligned during diffusion. However, this alignment can be effectively achieved by employing a single network for person and garment conditions.

Drawing from this insight, CatVTON removes the ReferenceNet or other image encoders utilized in other diffusion-based methods for garment encoding and keeps only two necessary modules (as shown in Fig. 4):

VAE. The VAE encoder is responsible for encoding input images into latent representations, thereby optimizing computational efficiency. The VAE decoder reconstructs the latent features into the original pixel space at the end. Unlike traditional methods that use different image encoders for the person and the garment, CatVTON employs a single shared encoder to process both inputs. This not only reduces the number of parameters but also ensures that the inputs are aligned with each other.

Denoising UNet. The denoising UNet synthesizes the final try-on image by combining the features in the latent space. It accepts concatenated garment and person features, along with noise and masks, as inputs and learns to integrate all information from these conditions seamlessly.
Besides, textual information is not required as an additional condition In the context of image-based virtual try-on tasks. Therefore, we excise the cross-attention modules and the text encoder to minimize redundancy further. This refinement further reduces 167.02M parameters.

Finally, we have arrived at a streamlined network architecture with a modest parameter count of 899.06M, marking a significant reduction of over 44% compared to other diffusion-based methods. This lightweight network avoids the need for complex warping algorithms or specialized network designs, streamlining the overall process.

#### 3.2.2 Parameter-Efficient Training

CatVTON explores the modules within the denoising UNet that are pivotal in facilitating the interaction between garment and person features to produce photorealistic try-on results.
Diffusion-based methods typically entail training the entirety of a U-Net or ReferenceNet to adapt pre-trained models to the virtual try-on task. However, these pre-trained models already possess a wealth of prior knowledge. An overabundance of trainable parameters exacerbates the training burden—requiring increased GPU memory and extended training duration—and may also potentially impair model performance.

We posit that a pre-trained LDM has already been thoroughly trained in its capacity to encode images into features. Therefore, when transferring it to the try-on task, it is only necessary to finetune its ability to facilitate the interaction between human and garment features.
The U-Net component of the LDM comprises alternating ResNet[] and transformer[] blocks. The ResNet, featuring a local receptive field through its convolutional architecture, can encode local features effectively. In contrast, the transformer blocks are endowed with the capacity for global interaction, with the self-attention layers being pivotal to this capability.

We conducted experiments to gradually lock the most relevant modules, as illustrated in Sec. 4.6. We set the trainable components to the entire U-Net, the transformer blocks, and the self-attention layers. The results indicated that despite a significant disparity in the number of trainable parameters (815.45M for the full U-Net, 267.24M for the transformer blocks, and 49.57M for the self-attention layers), all three models produced satisfactory virtual try-on results. Furthermore, no substantial difference was observed in visual quality and performance metrics among them.

Consequently, we adopted a parameter-efficient training strategy by finetuning only the self-attention modules with 49.57M parameters. Additionally, during training, we followed the recommended value in [], adopting a 10% conditional dropout to support classifier-free guidance and employing the DREAM strategy.
This training strategy has allowed us to reduce the number of trainable parameters significantly, optimizing computational efficiency without compromising the quality of the virtual try-on results.

#### 3.2.3 Simplified Inference

Many existing methods utilize numerous preprocessing techniques, such as pose estimation and human parsing, to guide the try-on process, and text is even employed to provide additional information, such as garment type.
However, most of the information is inherently contained within the original images, and the pre-trained model has already learned strong priors from large-scale data.

In this work, we explored a more straightforward and more efficient inference process. We simplified the inference process by eliminating the need for any preprocessing or conditional information. The whole process can be completed with only the person image, garment reference, and cloth-agnostic mask.
Specifically, given a target person image $I_{p}\in\mathbb{R}^{3\times H\times W}$ and a binary cloth-agnostic mask $M\in\mathbb{R}^{H\times W}$, a cloth-agnostic person image $I_{m}$ is obtained by:

$$ $I_{m}=I_{p}\otimes M,$ (4) $$

where $\otimes$ represents the element-wise (Hadamard) product. Then cloth-agnostic person image $I_{m}$ and the garment reference (either in-shop garment or worn person image) $I_{g}\in\mathbb{R}^{3\times H\times W}$ is encoded into the latent space by the VAE encoder $\varepsilon$:

$$ $\displaystyle X_{m}$ $\displaystyle=\varepsilon(I_{m}),$ (5) $\displaystyle X_{g}$ $\displaystyle=\varepsilon(I_{g}),$ $$

where $X_{m},X_{g}\in\mathbb{R}^{4\times\frac{H}{8}\times\frac{W}{8}}$. $M$ is also interpolated to match the size of latent space, resulting in $m\in\mathbb{R}^{\frac{H}{8}\times\frac{W}{8}}$.
Next, $X_{m}$ and $X_{g}$ are concatenated along the spatial dimension to form $X_{c}\in\mathbb{R}^{8\times\frac{H}{8}\times\frac{W}{8}}$, and the mask $m$ is concatenated with an all-zero mask of the same size to create $m_{c}\in\mathbb{R}^{8\times\frac{H}{8}\times\frac{W}{8}}$:

$$ $\displaystyle X_{c}$ $\displaystyle=X_{m}\copyright X_{g},$ (6) $\displaystyle m_{c}$ $\displaystyle=m\copyright O,$ $$

where $\copyright$ denotes the concatenation operation along the spatial dimension, $O$ represents the all-zero mask.

At the beginning of the denoising, $X_{c}$, $m_{c}$, and a random noise $z_{T}\sim\mathcal{N}(0,1)$ of the same size as $X_{c}$ are concatenated along the channel dimension and input to the denoising UNet to get predicted $z_{T-1}$, and this process is repeated for $T$ times to predict the final latent $z_{0}$. For denoising step $t$, this process can be written as:

$$ $z_{t-1}=UNet(z_{t}\odot m_{c}\odot X_{c}),$ (7) $$

where $\odot$ denotes the concatenation operation along the channel dimension, finally, $z_{0}\in\mathbb{R}^{8\times\frac{H}{8}\times\frac{W}{8}}$ is then split across the spatial dimension to extract the person part $z_{0}^{p}\in\mathbb{R}^{4\times\frac{H}{8}\times\frac{W}{8}}$, we use the VAE decoder $D$ to transform the denoised latent representation $\widetilde{X}_{0}^{p}$ back into the image space, producing the final output image ${\widetilde{I}_{p}}\in\mathbb{R}^{3\times H\times W}$.

## 4 Experiments

### 4.1 Datasets

Our experiments are conducted on three publicly available fashion datasets: VITON-HD [], DressCode [], and DeepFashion []. VITON-HD comprises 13,679 image pairs, with 11,647 pairs in the training set and 2,032 pairs in the test set. These pairs include frontal upper-body person images paired with in-shop upper garment images, all at a resolution of 1024$\times$768. DressCode consists of 48,392 training and 5,400 test image pairs, featuring frontal full-body person images and in-shop garments encompassing upper garments, lower garments, and dresses, also at 1024$\times$768 resolution. The in-shop clothes retrieval benchmark of DeepFashion dataset includes 52,712 high-resolution person images. From these, we selected 13,098 person image pairs for garment transfer tasks, with 1,896 pairs used for testing and the remainder for training, and then adjust these images to 1024$\times$768. DressCode and DeepFashion datasets underwent clothing-agnostic mask processing using body parsing results from DensePose [] and SCHP [] (LIP [] and ATR [] version), while VITON-HD employed preprocessed masks.

Figure: Figure 5: Qualitative comparison on the VITONHD[] dataset. CatVTON demonstrates a distinct advantage in handling complex patterns and text. Please zoom in for more details.
Refer to caption: x5.png

### 4.2 Implementation Details

We train our models based on the inpainting version’s pre-trained StableDiffusion v1.5 [].
Moreover, for a fair comparison, we train two models for the VITONHD[] and DressCode[] datasets separately and evaluated them on respective test sets as previous methods did.
All the models are trained with identical super-parameters: the AdamW [] optimizer is employed with a batch size of 128 and a constant learning rate of $1e-5$ for $16,000$ steps training under 512$\times$384 resolution, and DREAM [] is used in our training with $p=10$.
All experiments are conducted on 8 NVIDIA A800 GPUs, and each model takes approximately 10 hours for training.
Additionally, a multi-task model that supports transfer in-shop and worn garments in resolution of 1024$\times$768 is trained across the three datasets mentioned in Sec. 4.1 ($\sim$ 73K pairs) for 48,000 steps under identical setup but a batch size of 32. Multi-task training exhibited slower convergence than single-task training due to more complex scenarios.
Our visualizations and qualitative analysis results were obtained using the multi-task model, while quantitative analysis results were derived from single-task models trained on each dataset.

### 4.3 Metrics

For paired try-on settings with ground truth in test datasets, we employ four widely used metrics to evaluate the similarity between synthesized images and authentic images: Structural Similarity Index (SSIM) [], Learned Perceptual Image Patch Similarity (LPIPS) [], Frechet Inception Distance (FID) [], and Kernel Inception Distance (KID) []. For unpaired settings, we use FID and KID to measure the distribution of the synthesized and real samples.

### 4.4 Qualitative Comparison

We compared the visual quality on two datasets: VITONHD[] and DressCode[]. Fig. 5 presents the try-on results of garments with complex patterns from the VITONHD dataset, comparing different methods to highlight the differences in fine-grained consistency. While other methods often exhibit artifacts, loss of detail, and blurry text logos when dealing with complex textures, CatVTON demonstrates its superiority by effectively handling texture positioning and occlusions and producing more photo-realistic results.

Fig. 6 illustrates the comparison for different garment types (including upper, lower, and dress) on full-body person images from the DressCode dataset.
For uppers, akin to the results in Fig. 5, our approach can generate results more consistent with the garment textures, devoid of artifacts. Regarding lowers and dresses, our method can more accurately recognize the length and texture of the garments, generating reasonable results under occlusions and performing better in rendering semi-transparent materials.

Figure: Figure 6: Qualitative comparison on the DressCode[] dataset. CatVTON demonstrates a distinct advantage in handling complex patterns and text. Please zoom in for more details.
Refer to caption: x6.png

Furthermore, we extended the evaluation to in-the-wild scenarios to test the robustness and applicability in real-world conditions. As shown in Fig. 7, CatVTON accurately recognizes and integrates the shape of complex garments, such as off-shoulder designs, with the person. It can generate interlaced parts for complex poses, such as sitting. Additionally, it effectively completes and integrates the background with the garment in complex in-the-wild scenarios.

Figure: Figure 7: Qualitative comparison in in-the-wild scenarios. OutfitAnyone only supports inference on its provided personal images. Our method combines background, person, and garment more naturally in complex scenarios. Please zoom in for more details.
Refer to caption: x7.png

### 4.5 Quantitative Comparison

**Table 1: Quantitative comparison with other methods. We compare the metrics under both paired and unpaired settings on the VITONHD[] and DressCode[] datasets. The best and second-best results are demonstrated in bold and underlined, respectively.**
| Methods | VITONHD | DressCode |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Paired | Unpaired | Paired | Unpaired |  |  |  |  |  |  |  |  |  |
| SSIM $\uparrow$ | FID $\downarrow$ | KID $\downarrow$ | LPIPS $\downarrow$ | FID $\downarrow$ | KID $\downarrow$ | SSIM $\uparrow$ | FID $\downarrow$ | KID $\downarrow$ | LPIPS $\downarrow$ | FID $\downarrow$ | KID $\downarrow$ |  |
| DCI-VTON[] | 0.8620 | 9.408 | 4.547 | 0.0606 | 12.531 | 5.251 | - | - | - | - | - | - |
| StableVTON[] | 0.8543 | 6.439 | 0.942 | 0.0905 | 11.054 | 3.914 | - | - | - | - | - | - |
| StableGarment[] | 0.8029 | 15.567 | 8.519 | 0.1042 | 17.115 | 8.851 | - | - | - | - | - | - |
| MV-VTON[] | 0.8083 | 15.442 | 7.501 | 0.1171 | 17.900 | 8.861 | - | - | - | - | - | - |
| GP-VTON[] | 0.8701 | 8.726 | 3.944 | 0.0585 | 11.844 | 4.310 | 0.7711 | 9.927 | 4.610 | 0.1801 | 12.791 | 6.627 |
| LaDI-VTON[] | 0.8603 | 11.386 | 7.248 | 0.0733 | 14.648 | 8.754 | 0.7656 | 9.555 | 4.683 | 0.2366 | 10.676 | 5.787 |
| IDM-VTON[] | 0.8499 | 5.762 | 0.732 | 0.0603 | 9.842 | 1.123 | 0.8797 | 6.821 | 2.924 | 0.0563 | 9.546 | 4.320 |
| OOTDiffusion[] | 0.8187 | 9.305 | 4.086 | 0.0876 | 12.408 | 4.689 | 0.8854 | 4.610 | 0.955 | 0.0533 | 12.567 | 6.627 |
| CatVTON(Ours) | 0.8704 | 5.425 | 0.411 | 0.0565 | 9.015 | 1.091 | 0.8922 | 3.992 | 0.818 | 0.0455 | 6.137 | 1.403 |

Comparison of Effect. We conducted a quantitative evaluation and comparison of effect with several state-of-the-art open-source virtual try-on methods [, , , , , , , ] on the VITON-HD[] and DressCode[] datasets. The comparison was performed under both paired and unpaired settings to measure the similarity between the synthesized results and ground truth and the generalization performance of the models. The results are presented in Tab. 1.
Our method outperformed all others across all metrics, demonstrating the effectiveness of our model architecture and training strategy in the virtual try-on task.
Besides, GP-VTON[], IDM-VTON[], and OOTDiffusion[] also showed good performance. GP-VTON, as a warping-based method, had advantages in SSIM[] and LPIPS[] but performed weaker in KID[] and FID[]. This result suggests that warping-based methods may focus more on ensuring structural and perceptual similarity but lack realism and detailed naturalness.

**Table 2: Detailed comparison of model efficiency. ${\textit{UNet}}_{ref}$, $E_{text}$, and $E_{image}$ represent the ReferenceNet, text encoder, and image encoder, respectively. Compared to other diffusion-based methods, CatVTON uses fewer modules, reducing total parameters by about $2\times$ and trainable parameters by $10\times$+. CatVTON requires significantly less memory during inference and does not need additional conditions such as pose or text.**
| Methods | Params (M) | Memory | Conditions |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| VAE | UNet | ${\textit{UNet}}_{ref}$ | $E_{text}$ | $E_{image}$ | Total | Trainable | Usage(G) | Pose | Text |  |
| OOTDiffusion[] | 83.61 | 859.53 | 859.52 | 85.06 | 303.70 | 2191.42 | 1719.05 | 10.20 | - | ✓ |
| IDM-VTON[] | 83.61 | 2567.39 | 2567.39 | 716.38 | 303.70 | 6238.47 | 2871.09 | 26.04 | ✓ | ✓ |
| StableVTON[] | 83.61 | 859.41 | 361.25 | - | 303.70 | 1607.97 | 500.73 | 7.87 | ✓ | - |
| StableGarment[] | 83.61 | 859.53 | 1220.77 | 85.06 | - | 2248.97 | 1253.49 | 11.60 | ✓ | ✓ |
| MV-VTON[] | 83.61 | 859.53 | 361.25 | - | 316.32 | 1620.71 | 884.66 | 7.92 | ✓ | - |
| CatVTON(Ours) | 83.61 | 815.45 | - | - | - | 899.06 | 49.57 | 4.00 | - | - |

Comparison of Efficiency. Tab. 2 and Fig. 2 demonstrate the comparison of our method with other diffusion-based methods in terms of model parameters, memory usage, and additional conditions for inference. Our method only requires two modules, VAE and UNet, without needing an additional ReferenceNet or text and image encoders, significantly reducing the total number of model parameters (by approximately 44%). Moreover, based on our exploration of task-specific modules, the trainable parameters in CatVTON are reduced by 10+ times compared to other methods. During inference, our method has a significant advantage in memory usage (the memory required to load all modules) compared to other methods, and it does not require additional conditions such as pose or text information, greatly alleviating the burden of inference.

### 4.6 Ablation Studies

We conducted ablation experiments to investigate the effects of different 1) trainable modules, 2) $\lambda$ in DREAM, and 3) the strength of classifier-free guidance on try-on results. The different versions of models for comparison were trained on the VITONHD[] dataset, following the setup mentioned in Sec. 4.2.

Trainable Module. We evaluated three trainable modules: 1) UNet, 2) transformer blocks, and 3) self-attention, with the number of trainable parameters decreasing in that order. Tab. 3 shows the evaluation results on the VITONHD[] dataset. We observed that increasing the number of trainable parameters slightly improved most metrics, suggesting that more parameters for training can enhance model performance. However, as the number of trainable parameters increases, the model fits the dataset more quickly, which may be accompanied by overfitting and the loss of prior knowledge from the pre-trained model. Despite the slight improvements, the other two trainable parameters are 5$\times$+ and 16$\times$+ compared to self-attention.

**Table 3: Ablation results of different trainable module on VITONHD[] dataset. When expanding the trainable modules to the Transformer Block and UNet, the metrics showed slight improvements, but the trainable parameters increased by 5$\times$+ and 16$\times$+, respectively.**
| Trainable Module | Paired | Unpaired | Trainable Params (M) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| SSIM $\uparrow$ | FID $\downarrow$ | KID $\downarrow$ | LPIPS $\downarrow$ | FID $\downarrow$ | KID $\downarrow$ |  |  |
| UNet | 0.8692 | 5.2496 | 0.4017 | 0.0550 | 8.8131 | 0.9559 | 815.45 |
| Transformers | 0.8558 | 5.4496 | 0.4434 | 0.0558 | 8.8423 | 1.0082 | 267.24 |
| Self Attention | 0.8704 | 5.4252 | 0.4112 | 0.0565 | 9.0151 | 1.0914 | 49.57 |

DREAM. As shown in Fig. 8, when $\lambda$ is relatively small, the generated results are overly smooth and lack high-frequency details. When $\lambda$ is too large, the high-frequency details are excessive, making the results less natural. DREAM[] uses $\lambda$=$1$ as an empirical value, but $\lambda$=$10$ achieves a better trade-off between naturalness and detail fidelity for our training. We also compared the quantitative analysis results of models trained with different values of $\lambda$ on the VITONHD[] dataset. We found the same phenomenon depicted in DREAM[]: SSIM is positively related to $\lambda$, but FID and LPIPS first improved and then deteriorated as $\lambda$ increased. It indicates a reduction in distortion at the cost of perceptual quality.

Figure: Figure 8: Visual comparison for different $\lambda$ in DREAM[]. When $\lambda$ is too small, results are overly smooth and lack detail; when $\lambda$ is too large, results have excessive high-frequency details and appear unnatural.
Refer to caption: x8.png

**Table 4: Ablation results of different $\lambda$ in DREAM[] on VITONHD dataset []. $\lambda$=$\infty$ means not using DREAM. Increasing $\lambda$ improves perceptual quality (lower LPIPS, KID, and FID) but also increases distortion (lower SSIM) in an empirical range.**
| DREAM $\lambda$ | Paired | Unpaired |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| SSIM $\uparrow$ | FID $\downarrow$ | KID $\downarrow$ | LPIPS $\downarrow$ | FID $\downarrow$ | KID $\downarrow$ |  |
| 0 | 0.8740 | 10.4534 | 3.8866 | 0.0692 | 14.1045 | 5.2824 |
| 1 | 0.8716 | 8.0983 | 2.1977 | 0.0646 | 11.7652 | 3.2942 |
| 10 | 0.8704 | 5.4252 | 0.4112 | 0.0565 | 9.0151 | 1.0914 |
| 20 | 0.8633 | 5.5861 | 0.4005 | 0.0620 | 9.0877 | 1.0416 |
| $\infty$ | 0.8614 | 5.5561 | 0.3657 | 0.0631 | 8.9114 | 1.0049 |

Classifier-Free Guidance. To evaluate the effect of classifier-free guidance (CFG) on generated try-on images, we run inferences with CFG strengths of 0.0, 1.5, 2.5, 3.5, 5.0, and 7.5, keeping all other parameters constant. Fig. 9 shows that increasing CFG strength enhances image detail and fidelity. However, beyond a strength of 3.5, the images developed severe color distortions and high-frequency noise, degrading visual quality. We found that a CFG strength of 2.5 to 3.5 produces the most realistic and natural results. For our experiments, we consistently used a CFG strength of 2.5.

Figure: Figure 9: Visual comparison for different CFG[] strength. As the CFG strength increases, the details in the generated images increase, but beyond 3.5, it leads to severe color distortion and high-frequency noise.
Refer to caption: x9.png

### 4.7 Limitations & Social Impacts

While leveraging LDM [] as the backbone for generation, our model faces certain limitations. Images decoded by VAE may exhibit detail loss and color discrepancies, particularly at a 512×384 resolution. Additionally, the effectiveness of the try-on process is contingent upon the accuracy of the provided mask; an inaccurate mask can significantly degrade the results.
Based on Stable Diffusion v1.5, our pre-trained model was trained on large-scale datasets that include not-safe-for-work (NSFW) content. Consequently, retaining most of the original weights means our model may inherit biases from the pre-trained model, potentially generating overly explicit images of people.

## 5 Conclusion

In this work, we introduce CatVTON, a lightweight and efficient virtual try-on diffusion model boasting just 49.57M trainable parameters. CatVTON seamlessly transfers garments from in-shop or person images onto a target person by spatially concatenating garment and person images. This innovative approach eliminates the need for ReferenceNet and additional image encoders, cutting model parameters and memory usage by over 40% compared to other diffusion-based methods. Moreover, CatVTON streamlines the virtual try-on process, requiring no extra steps like pose estimation, human parsing, or text prompts. Extensive experiments reveal that CatVTON delivers superior qualitative and quantitative results and outperforms state-of-the-art methods while maintaining a compact and efficient architecture. These findings underscore the potential of CatVTON for practical applications in virtual try-on technology, making it a promising tool for the fashion industry.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 6
presentation: 3
questions: Please refer to "Weaknesses."
soundness: 3
strengths: + 1. This paper proposes CatVTON, a lightweight virtual try-on diffusion model with only 899.06M parameters, that achieves high-quality results by simply concatenating garment and person images as inputs, eliminating the need for extra image encoders, ReferenceNet, and text-conditioned modules.

+ 2. This paper introduces a parameter-efficient training strategy to transfer pre-trained diffusion models to virtual try-on tasks while preserving prior knowledge by training necessary modules with only 49.57M parameters

+ 3.  This paper simplifies the inference process by eliminating the need for extra pre-processing of input images and leveraging the robust priors from pre-trained diffusion models to infer all necessary information, reducing memory usage by 49%+ compared to other diffusion-based baselines
summary: The study re-evaluates the necessity of additional modules in virtual try-on methods based on diffusion models and proposes CatVTON, a simple and efficient model. CatVTON transfers garments of arbitrary categories to target individuals by concatenating them as inputs, removing redundant encoding modules and steps. It features a lightweight network with just 899.06M parameters, efficient training with only 49.57M parameters through self-attention modules, and simplified inference that requires only person image and garment reference, reducing memory usage by 49%+. Experiments show that CatVTON achieves superior results and generalizes well in real-world scenarios, despite being trained on a limited public dataset.
weaknesses: - 1. The effectiveness of the try-on process relies on the accuracy of the provided masks; inaccurate masks may significantly degrade the quality of the results. When the authors conducted mask-free experiments, how did they ensure that the model paid attention to the try-on areas?

- 2. Some of the data in Table 1 seem to differ from those in the original paper, why?

- 3. Based on Table 3, it seems that choosing to train either the Unet or Transformers module can achieve better results, albeit with higher memory demands and slower training. How to balance this relationship?

- 4. Please provide the training code to demonstrate reproducibility.

- 5. The method is simple and effective, however, the entire process is overly engineered, lacking rigorous theoretical validation and support in a conference like ICLR.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: - Can you provide additional quantitative results on CatVTON’s inference speed and computational cost, and is the model deployable on embedded or resource-constrained devices?

- Given that the mask-free model builds on a trained mask-based model, have you observed any error accumulation in this process, and if so, what techniques might mitigate such errors?

- Could the lightweight model design and parameter-efficient training strategy you propose for CatVTON be applicable to other virtual try-on frameworks, including non-diffusion-based methods?
soundness: 3
strengths: - CatVTON proposes a lightweight architecture, which highly reduces computational cost during the model training. Meanwhile, it only requires 49.57M training parameters.

- The model generalises well in diverse real-world scenarios, performing effectively even with limited training data.

- CatVTON is able to achieve realistic try-on results with accurate garment alignment and detail preservation.
summary: CatVTON is a virtual try-on model that achieves high-quality and efficient garment transfer without relying on additional encoders or complex preprocessing. Unlike traditional virtual try-on approaches, CatVTON simplifies the architecture by using only a VAE and a single, lightweight UNet with a self-attention module, reducing the parameter count to 899.06M. Moreover, this model directly concatenates the person and garment images along spatial dimensions, requiring only 49.57M trainable parameters and cutting down inference memory usage by over 49% compared to other methods. Extensive experiments demonstrate that CatVTON achieves superior results in both controlled and in-the-wild settings.
weaknesses: - While the authors present CatVTON as a lightweight virtual try-on diffusion model and it only requires very few training parameters, it would be beneficial to include results on inference speed and computational cost. Additionally, demonstrating the model’s deployability on embedded devices could enhance understanding of its practical applications.

- The mask-free model is derived from a trained mask-based model. It would be better to discuss potential error accumulation in this transition.

- CatVTON introduces a model design and training strategy aimed at creating a lightweight model with fewer training parameters. It would be useful to consider whether this approach could be extended to other virtual try-on pipelines, particularly non-diffusion-based methods.

---

confidence: 5
contribution: 4
initial_rating: 8
presentation: 4
questions: 1. Can the proposed efficient training method be transferred to other more novel diffusion models, such as DiT?

2. Will the generated dataset used for mask-free training be open-sourced to support future research?

3. Provide more details about Mask-free training, such as dataset construction and how it differs from Mask-based model training, visual comparison of its results with Mask-based model results, etc.

4. Adjust the layout of the tables and figures to make them more uniform and standardized.
soundness: 4
strengths: 1. Originality: This paper focuses on efficiency optimizations in diffusion models for virtual try-on. By simplifying the model architecture with a concatenation operation and identifying key modules through an analysis of image representation interactions, it achieves parameter-efficient training. Additionally, by streamlining preprocessing in inference, it achieves efficient inference without extra conditions. The approach and model architecture differ significantly from other virtual try-on models, addressing the efficiency issues of previous work. The paper offers new insights into the application of image representation interactions in generative models, demonstrating innovation.

2. Quality: This paper conducts extensive comparative and ablation experiments to validate its performance. Qualitative and quantitative analyses are provided on two major open-source datasets, DressCode and VITON-HD, in comparison with recent virtual try-on methods. Quantitatively, the proposed model shows significant advantages in model size, trainable parameters, and inference memory usage, while surpassing other methods in FID, KID, SSIM, and LPIPS metrics. Visually, the model generates highly realistic details such as text logos and patterns, and performs well in in-the-wild virtual try-on scenarios.

3. Clarity: The writing is well-structured, with a clear and logical flow of content. Plenty of figures, tables, and formulas are used to explain and analyze key insights and experimental details. The writing is smooth, with no obvious grammatical errors or inconsistencies.

4. Significance: This work offers insights and contributions to the application of diffusion models in virtual try-on tasks. Its lightweight architecture and parameter-efficient training significantly reduce model training costs and may be transferable to other downstream tasks. The model's low memory usage during inference and simplified inference process reduce hardware requirements, making it more cost-effective to deploy.
summary: This paper aims to efficiently adapt pre-trained diffusion models to high-quality virtual try-on. It introduces efficiency improvements in three key areas: model architecture, parameter-efficient training, and simplified inference, and analyzes the motivation and feasibility from the perspective of image representation interaction, supported by experimental validation. Extensive qualitative and quantitative experiments are conducted on VITON-HD and DressCode datasets, along with ablation studies on key components of proposed method. Compared to other virtual try-on methods, this method demonstrates significant advantages in model parameters, training efficiency, and inference costs while achieving better performance. Despite being trained on small-scale public datasets, results and comparisons in in-the-wild scenarios demonstrate the method's generalization capabilities.
weaknesses: 1. The description of the proposed mask-free training method is not specific and detailed enough, more information should be provided such as the dataset details used for this part of the training and the exact process of pseudo-data construction.

2. The Concatenation operation applied to garment and person features lacks a more explicit and detailed discussion, such as its advantages over encoder features like CLIP or DINO.

3. The transferability of the proposed efficient virtual try-on method based on Stable Diffusion is not discussed. For example, migrating this training framework to more advanced image generation diffusion models, such as DiT, to verify the generality of this framework.

4. Some inappropriate formatting issues, such as inconsistent font sizes in tables. The font in Table 1 and Table 4 is relatively small, and there is excessive whitespace in Table 4. The images in Figure 5 and Figure 6, which are used to display generated results, are arranged too tightly.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: Q1. What is the IPS metric? It is better to provide its definition and description of it if it is newly proposed or referred to if needed.
soundness: 2
strengths: + Removing the redundant reference Unet of most existing work is a good direction as it makes the model lightweight and simplified.
+ Results on two benchmark datasets VTON-HD and DressCode show good performance.
summary: **Post-discussion period**

---

I will maintain my current score of 5, as the issues I raised in my review remain unresolved:

+ Discrepancies in Quantitative Results: There are significant discrepancies between the reported results in the submission and the original baseline papers. This inconsistency makes the comparison less convincing. To address this, I suggested that the authors use the original evaluation scripts from the baseline papers alongside their own settings, but it is disregarded. This would provide a fairer comparison, particularly given the considerable variance observed in metrics like LPIPS, FID, and KID.

+ Architectural and Textual Inconsistencies: There is a minor but important inconsistency between the overall architecture figure and the description in the main text. Specifically, the process of input concatenation, whether in pixel-space or latent-space, is unclear. This discrepancy raises questions about whether the results are indeed identical to what was claimed.

For me, if there are unclear aspects in a submission and the authors disregard suggestions such as using original scripts for reevaluation, I cannot raise the score or recommend acceptance unless these concerns are fully addressed. The final decision I would let the effort of ACs.

---

This work introduces CatVTON, a diffusion-based virtual try-on model that enhances efficiency by removing complex modules and preprocessing steps typical in other approaches. Diffusion-based virtual try-on methods usually require additional encoding, extensive training, and preprocessing, which CatVTON addresses by simplifying the architecture to a VAE and denoising UNet, reducing parameters to 899.06M. Through analysis, the authors find that self-attention modules are essential for adapting diffusion models to this task, enabling high-quality outputs with only 49.57M training parameters. CatVTON further reduces inference complexity, eliminating pre-processings like pose estimation and human parsing, which cuts memory usage by over 49%. Extensive testing demonstrates CatVTON’s strong performance and generalization, even with a limited 73K sample dataset, achieving better results than baseline methods in both controlled and real-world scenarios. Experiments are conducted on two datasets: VITON-HD and DressCode.
weaknesses: **Concerns**
+ Inference speed comparison is missing since the proposed model is more lightweight, it is more beneficial to see this metric.

+ Many works (AnyDoor, MimicBrush, Uni-ControlNet) show that a strong pre-trained image encoder (CLIP, DINOv2, Unet) would provide more priors and good features for better imager generation with better details, but this work shows that removing all of them, and using only VAE outputs can give even better performance. Which in my opinion, it might not be very reasonable. A valid explanation is needed to this point and also I believe that whether the code of this paper is publicly available for the community to verify its validity and reproducibility is crucial.

+ The detail of all numeric numbers in Table 1, and Table 2 should be clarified. Were all of them reproduced by the author or were they quoted from somewhere else? Looking at the original papers (OOTDiffusion and others) I find that they report much higher numbers compared to the numbers shown in this submission. It raised questions about the correctness and what made the differences since they used also the same dataset.

+ "DREAM \lambda", just putting this terminology here is not very informative, it is quite hard to understand what is that parameter \lambda and what is DREAM here without some more detailed explanations.

For the whole paper, the current form seems to use all \cite{} making it very messy for all references. I think the use of \citep{} in latex would produce a more correct presentation of citations for many parts of the paper. Using \cite{} for cases where the citing author is subject (S), but \citep{} for other cases when referring to the paper.

[1] AnyDoor: Zero-shot Object-level Image Customization, CVPR 2024

[2] MimicBrush: Zero-shot Image Editing with Reference Imitation, arXiv:2406.07547

[3] Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models, EnruIPS 2023

Meta-review: In this paper, the authors propose CatVTON, a simple and efficient virtual try-on diffusion model using a VAE and denoising UNet with spatial concatenation, reducing memory usage and training parameters. The paper demonstrated competitive performance on standard benchmarks and in real-world scenarios, with strong generalization and efficient inference. However, the reviewers raised concerns about reproducibility due to discrepancies in evaluation metrics compared to prior works, insufficient explanation of the concatenation's advantages, and a lack of detailed theoretical validation. 

After the author-reviewer discussion period, most of the reviewers confirmed their concerns had been addressed. However, Reviewer iWwH still has two important remaining concerns:
- Discrepancies in quantitative results: there are significant discrepancies between the reported results in the submission and the original baseline papers. 
- Architectural and textual inconsistencies: there is a minor but important inconsistency between the overall architecture figure and the description in the main text.

Considering most of the reviewers are positive towards this submission, and Reviewer iWwH is not strongly against accepting it, I recommend acceptance. However, the discussions between the reviewers and authors, especially the concerns from Reviewer iWwH, should be included in the final version.

Decision: Accept (Poster)