MMTryon: Multi-Modal Multi-Reference Control for High-Quality Fashion Generation

## Abstract

Abstract This paper introduces MMTryon, a multi-modal multi-reference VIrtual Try-ON (VITON) framework, which can generate high-quality compositional try-on results by taking a text instruction and multiple garment images as inputs. Our MMTryon addresses three problems overlooked in prior literature: 1) Support of multiple try-on items. Existing methods are commonly designed for single-item try-on tasks (e.g., upper/lower garments, dresses).
2) Specification of dressing style . Existing methods are unable to customize dressing styles based on instructions (e.g., zipped/unzipped, tuck-in/tuck-out, etc.)
3) Segmentation Dependency . They further heavily rely on category-specific segmentation models to identify the replacement regions, with segmentation errors directly leading to significant artifacts in the try-on results. To address the first two issues, our MMTryon introduces a novel multi-modality and multi-reference attention mechanism to combine the garment information from reference images and dressing-style information from text instructions. Besides, to remove the segmentation dependency, MMTryon uses a parsing-free garment encoder and leverages a novel scalable data generation pipeline to convert existing VITON datasets to a form that allows MMTryon to be trained without requiring any explicit segmentation.
Extensive experiments on high-resolution benchmarks and in-the-wild test sets demonstrate MMTryon’s superiority over existing SOTA methods both qualitatively and quantitatively.
MMTryon’s impressive performance on multi-item and style-controllable virtual try-on scenarios and its ability to try on any outfit in a large variety of scenarios from any source image, opens up a new avenue for future investigation in the fashion community. Project page is this link

## 1 Introduction

VIrtual Try-ON (VITON) technology aims to generate photo-realistic images featuring individuals adorned in specific garments according to input images of both the garments and the person. Due to its promising potential to revolutionize the online shopping experience, VITON has garnered considerable attention both in the academic and industry communities.
Despite the substantial progress achieved in academic benchmarks [, ], previous methods [, , , , , , ] still fall short of meeting the requirements of real-world applications, in particular for: 1) Multiple-garment Compositional Try-On: Users should be able to freely choose and combine a diverse range of tops, bottoms, and accessories from in-the-wild images, rather than being confined to the single-garment try-on scenario. 2) Flexible Try-On Styles: Users should have the flexibility to select their preferred try-on style, allowing them to specify how a garment is worn, such as deciding whether a top should be tucked into the trousers or draped over them.

Due to their focus on single garments, these methods encounter challenges when multiple garments and accessories need to be changed simultaneously. This limitation arises because collecting training data for single-garment try-on is relatively easy, whereas gathering data for multi-garment try-on is significantly more difficult. Additionally, these methods struggle to model the subtle differences in how each piece of clothing is styled during a compositional try-on, such as the overlapping relationship between tops and bottoms. Furthermore, given a flat in-store image of a coat with a zipper, existing methods are limited to replicating the garment in its original state onto the target, without the ability to adjust for open or closed styles.

Additionally, existing methods lack the capability to precisely identify try-on areas, often resorting to trained segmentation models [, ] for explicit try-on area localization. This reliance on segmentation models makes these approaches highly dependent on the segmentation model’s precision, which can be unsatisfactory in complex scenarios, introducing additional noise during training.
Additionally, conditioning the generation on the segmented garment results in a loss of information of how the garment is worn, leading to highly unrealistic try-on outcomes, particularly for uniquely designed garments. Moreover, the dependence on segmentation models severely restricts the flexibility of the try-on process, as these models are typically confined to a predefined set of garment classes.

Figure: Figure 1: MMTryon can follow complex instructions to generate High-quality try-on results.
Refer to caption: x1.png

To address these challenges, we propose MMTryon, a more flexible try-on model capable of achieving multi-modal compositional dressing without the need for pre-segmenting the try-on areas. In our approach, a general try-on scenario is decoupled into text and image information, where the text information encompasses high-level garment descriptions and try-on style instructions, while the image information contains fine-grained texture details.
MMTryon is based on a conditional diffusion model that, to ensure efficient incorporation of both the image and text information in challenging scenarios, leverages our novel multi-reference image attention and multi-modal instruction attention modules. Specifically, to exploit the image information, we pre-trained a robust garment encoder on a large dataset to project image features into the multi-reference image attention module. This module then can utilize text as queries to extract the necessary features with the aim of eliminating our dependency on segmentation. For the text information, we develop a sophisticated multi-modal prompt and employ a pre-trained text encoder to project the text information into the specifically designed Multi-Modal Instruction Attention module. Finally, we train our model on an synthetically enhanced dataset, facilitated by our proposed data pipeline, using the pre-trained encoders and Stable Diffusion-initialized weights. Despite the difficulty of obtaining ideal multi-garment-to-single-model pairs for compositional try-on, our pre-trained garment encoder ensures excellent results even with limited datasets. Additionally, by overcoming the segmentation accuracy limitations in existing VITON models, our model can more effectively handle the nuances of different clothing items and styles (see Fig. 1).

Overall, our contributions can be summarized as follows: First, we propose MMTryon which, to our knowledge, is the first model to support the multi-modal compositional try-on task. MMTryon enables the combination of one or multiple garments for try-on, while also allowing the manipulation of try-on effects through textual commands. We further introduce a scalable data generation pipeline and a specially designed garment encoder. This eliminates the need for any prior segmentation networks during both the training and inference phases to get garments, relying solely on text, images of individual garment items, or model images to identify the areas of interest. Finally, extensive experiments conducted on public try-on benchmarks and in-the-wild test sets are performed, demonstrating that our approach surpasses existing state-of-the-art methods such as Outfit Anyone(^1^11Outfit Anyone is available at [https://github.com/HumanAIGC/OutfitAnyone](https://github.com/HumanAIGC/OutfitAnyone)). Notably, our method delivers the most flexible and interactive dressing experience available to date.

## 2 Related work

GAN-Based Virtual Try-on.
Traditional virtual try-on methods [, , , , , , , , , , , , , , , ] usually adopt a two-stage pipeline with Generative Adversarial Networks (GANs). The first stage employs an explicit warping module to deform in-shop clothing to the target shape, while the second stage uses a GAN-based generator to fuse the deformed clothing onto the target person. For theses methods, the synthetic quality largely depends on the quality of the deformation in the first stage, prompting current methods to emphasize enhancing the non-rigid deformation capabilities of the warping module. Meanwhile, some GAN-based approaches [, ] have explored the compositional try-on task, utilizing a cyclical generation mode to achieve sequential try-on. However, when the outcome of compositional dressing is entirely determined by the try-on order, the lack of understanding of the semantic information of the garments often leads to unrealistic results.
Furthermore, most GAN-based solutions mentioned above directly utilize UNet-based generators for try-on synthesis, with little exploration into how to enhance the generator’s capabilities. This results in poor resolution and visual quality of the try-on outcomes, making it difficult to integrate them with diverse linguistic instructions.

Figure: Figure 2: Overview of the proposed MMTryon framework. The instruction prompt and garment images are combined to obtain a multi-modal interleaved instruct embedding, replacing the original textual condition. Each garment image together with the corresponding text span are further processed by the garment encoder to obtain reference features, which, along with target features, undergo multi-reference attention to ensure detailed texture transfer.
Refer to caption: x2.png

Diffusion-Based Virtual Try-on.
Compared to GAN-based models, diffusion models have made significant strides in high-fidelity conditional image generation [, , ]. Image-based virtual try-on is essentially a specialized form of the image editing/repair task conditioned on a given garment image. Therefore, a straightforward adaptation is to extend text-to-image diffusion models to accept images as conditions. While methods such as  [, , ] have demonstrated their capabilities in virtual try-on, they fall short in preserving the texture details of the try-on results due to inadequate extraction of fine-grained features by the image encoder.
Recent methods [, ] aim to integrate traditional GAN-based approaches with diffusion models. They employ explicit warping modules to create deformed garments and use diffusion models to blend these with reference person images. Beyond simply incorporating explicit warping modules, the newly introduced TryonDiffusion [], conversely, implements an implicit warping mechanism for clothing deformation. This addresses the issues of texture misalignment mentioned earlier and achieves promising try-on synthesis. However, TryonDiffusion relies on precise parsing to achieve various conditions, making it unable to handle more complex try-on scenarios.
Additionally, none of the existing approaches have effectively leveraged the capabilities of current large multi-modal models and have discarded text as an input, lacking the ability of free-form instruction following. The try-on effect in these approaches is solely inferred from the garment image.
Our approach, through the specialized design of a parsing free garment encoder and the multi-modal multi-reference attention mechanism, achieves a more versatile try-on experience. This not only addresses the limitations seen in both GAN and diffusion model-based methods but also utilizes the strengths of large multi-modal models, allowing for more detailed and flexible adaptation to various clothing styles and try-on scenarios.

## 3 Method

As shown in Fig. 2, MMTryon has a minimal input setup during inference, only requiring a source person image, multiple reference images, and a text instruction describing how the items from the references are fitted onto the desired person. To achieve multi-modal multi-reference controllable try-on, two key elements are central: the training data and the model design.
We start by presenting the design of our model to support this inference setup before describing how the data is gathered to support model training.

### 3.1 Overview

Given a target person image $I$, multiple reference images containing try-on items $\{G_{i}\}$ and a text instruction $T$ describing how the items from those reference images are dressed, MMTryon aims to seamlessly follow $T$ to synthesize photo-realistic try-on results $I^{\prime}$. We model this task as a conditional diffusion model, which takes $I$, $\{G_{i}\}$ and $T$ as conditions and gradually estimates a denoised version of $I^{\prime}$ from pure noise. Our diffusion model is built upon the public available Stable Diffusion text-to-image model [].

Stable Diffusion is a text conditioned latent diffusion model [].
For an VAE [] encoded image latent feature $z_{0}$, the forward diffusion process is performed by adding noise according to a predefined noise scheduler $\alpha_{t}$[]:

$$ $q(z_{t}|z_{0})=\mathcal{N}(z_{t};\sqrt{\alpha_{t}}z_{0},(1-\alpha_{t})I).$ (1) $$

To reverse the diffusion process, a noise estimator $\epsilon_{\theta}(\cdot)$ parameterized by an UNet is learned to predict the forward added noise $\epsilon$ with the objective function,

$$ $\mathcal{L}_{\text{dm}}=\mathbb{E}_{(z_{0},c)\sim D}\mathbb{E}_{\epsilon\sim \mathcal{N}(0,1),t}\left[||\epsilon-\epsilon_{\theta}(z_{t},t,c)||^{2}\right],$ (2) $$

where $c$ is the text condition associated with image latent $z$, and $D$ is the training set.

In MMTryon, our condition is not merely a text condition but a multimodal condition that includes texture information as well as a multimodal prompt that integrates global image information and instructions. Modulating this condition into the diffusion model is a challenging problem. Prior works [, ] have relied on transforming pixel features into text-aligned features using a CLIP encoder and injecting them into the cross-attention layers. However, this approach often results in a loss of attention to detail. In this work, we instead draw inspiration from [, , ], who show that the cross-attention layer learns correspondences between the image features and the text embedding tokens, generating an overall image semantic structure, while the self-attention layers are more related to detailed texture generation. We thus focus on enhancing the cross-attention layers conditioned on the original text, integrating it with our multi-modal instruction to improve generation of the final try-on results. Additionally we train a garment encoder to align pixel features with the stable diffusion features, enhancing the self-attention layers and seamlessly transfer detailed image features from the garment images to the desired human images.

### 3.2 Pretrain Garment Encoder

As mentioned before, one of the main challenges of multi-reference try-on is the lack of training data, i.e., most of the garment person pairs consist of only a single garment. We draw inspiration from the training scheme of Large Multi-modal Modal(e.g., []), and split the training into pretraining and supervised fine-tuning stages. During pretraining, our goal is to align features from multi-sources to the generator, here, the StableDiffusion denoising UNet. Thus we fixed the denoising UNet, and pretrain the garment encoder in this stage.

To achieve texture-consistent try-on results without relying on a prior segmentation model, this encoder should extract fine-grained image features and respond effectively to text queries. We initialized the garment encoder with the UNet encoder from the diffusion model, which takes both garment image and text query as input, we use the features after the cross-attention layer of each transformer blocks as the output garment feature. This feature is then injected into the diffusion UNet through multi-reference attention. By supervising with diffusion loss $\mathcal{L}_{\text{dm}}$, we are able to obtain features that can preserve texture consistency.

Meanwhile, to improve the accuracy of text queries, we enforce the text-queried features to have a similar response to a prior segmentation model SAM [] (see Fig.  3). Here we also introduce a text query loss, where features outside the region of the mask are passed through a sigmoid, squared, and averaged to encourage minimization of activations in areas unrelated to the text.
The final loss is:

$$ $\mathcal{L}_{\text{enc}}=\mathcal{L}_{\text{dm}}+\mathcal{L}_{\text{query}}$ (3) $$

where the text query loss $\mathcal{L}_{\text{query}}$ is:

$$ $\mathcal{L}_{\text{query}}=\frac{1}{N}\sum\sigma(F\odot(1-M))^{2}$ (4) $$

Here, $\odot$ denotes element-wise multiplication and $\sigma$ denotes the sigmoid function. $F$ represents the garment feature and $M$ denotes the prior mask corresponding to the target region’s text. $N$ denotes the number of elements in $F$ and $M$. This way, we can obtain a garment encoder that can extract only the necessary feature corresponding to the text query which also preserves low-level pixel consistency through a combination of discriminative and generative supervision.

Figure: Figure 3: Overview of the proposed pretrained garment encoder.Our garment encoder utilizes a prior mask derived from grouding dino and SAM to improve text query accuracy through cross-attention between the target text and the input features. The garment encoder is supervised by the diffusion reconstruction loss and our text query loss.
Refer to caption: x3.png

### 3.3 MMTryon

Here, we provide a detailed description of each component within the main UNet. In particular, we employ two types of attention mechanisms to enable multi-modal multi-reference try-on. Having established the powerful garment encoder in the previous section, we now eliminate the dependency on prior masks by utilizing a synthetic dataset to finetune the entire model and the garment encoder jointly. This dataset is derived from the original data through an enhanced data generation pipeline, which is detailed in Sec. 3.3.1.

#### 3.3.1 Scalable Data Generation Pipeline

To achieve multi-modal multi-reference try-on, a straightforward way is to construct training data in the same format. However, existing try-on datasets in this format are not readily available. There are generally two types of datasets, in-shop garment to person pairs or person to person pairs, where the former one only contains a single reference and is usually limited in garment types.
Leveraging the person-to-person pairs instead, a possible way to automatically construct a training dataset for the multi-modal multi-reference try-on task is to use segmentation models to crop various reference garments to construct multi-reference pairs. However, the drawback of this approach is that the style of the garments is not controllable and that the model will still be dependent on off-the-shelf segmentation models during inference.

In this work, we instead propose to leverage the increasingly powerful capabilities of large models to develop a flexible scheme for dataset construction. Starting from existing person-to-person datasets, given a pair of person images $I_{a},I_{b}$, we use SOTA large multi-modal model GPT-4V [] and CogVLM [] to caption the image in a desired format describing how the person is dressed. We then extract the main garment subjects and style (garment categories) from the image description denoted as $\langle obj_{1},obj_{2},\ldots,obj_{n}\rangle$. Subsequently, we use a SOTA open-vocabulary detection model [] to generate the bounding boxes corresponding to each garment subjects within the image. The SAM [] model is then used to generate precise mask annotations. After this caption-grounding-segmentation procedures, we now established a correspondence between a target person image $I_{a}$ with description, and all its segmented garments from $I_{b}$. We find that this process performs well for the vast majority of clothing categories. Note that we need to establish correspondences between two different poses, otherwise this try-on task will degenerate to a copy-paste problem.

Figure: Figure 4: The data generation pipeline of MMTryon. We use a large multi-modal model to describe the target person image, followed by open-vocabulary grounding and segmentation models to extract correspondences between a person image and several garment subjects. For each subject, we utilize SDXL inpainting to obtain the enhanced dataset, which serves as our training data.
Refer to caption: x4.png

To enable parsing-free inference, starting from those garment images, we then use the inpainting ability of the Stable Diffusion model (we use SDXL [] here) with a given prompt to randomly generate an image where a person is wearing such a garment. We use random seeds and random prompts to generate multiple samples, and discard samples with high similarity to prevent information leakage. This way, by iterating through all garment subjects, we are able to construct training pairs, where the target image is described using text, and the reference garments are from different images. The overall process is illustrated in Fig. 4.

#### 3.3.2 Multi-Modal Instruction Attention

In our task, it is not possible to ask the user to describe the desired garments from the reference images precisely using only text, as high-level descriptions such as garment type exhibit ambiguity and can not guarantee dressing fidelity. To mitigate this issue, we form a multi-modal instruction using an interleaved prompt template:

$$ $\text{A person wearing }\text{<garment> <style> }[Ref\#1]\\ \text{ ,..., and <garment> <style>}[Ref\#N]$ $$

For example, A person wearing a top, tucked in, [Ref#1], pants [Ref#2] and shoes [Ref#3]. Here [Ref#N] is a special placeholder token prepared for the corresponding reference image. This text prompt is first encoded using CLIP text encoders into text embedding $\mathbf{T}\in\mathbb{R}^{L\times D}$, where $L$ is the length of the tokens and $D$ is the embedding dimension. The embeddings corresponding to the placeholder tokens are then fused with the CLIP image embedding of the related garment images (see Fig. 2). To balance the embedding length and image feature granularity, and also better align the image feature with the text embedding, we do not only use the [CLS] token as in []
, but instead use the latent features from the penultimate layers of the CLIP ViT encoder. These latent features are further downsampled using a perceiver-resampler [].
This multi-modal embedding then serves as the replacement of the original text embedding of the stable diffusion model and not only aids in the generation of dressing styles but also enhances the stability of the try-on, ensuring the correct mapping of the garment.

#### 3.3.3 Multi-Reference Texture Attention

The Multi-Modal Instruction Attention cannot fully preserve the details of the garment due to the use of the CLIP ViT image encoder. To guarantee texture consistency,
we therefore first prompt the garment encoder with the textual label. The textual label $P_{i}$ is derived from the instruction prompt $T$ and used to extract features for the particular subject from image $G_{i}$. After obtaining the features, in each self-attention layer of the diffusion UNet, we extract
a set of garment features corresponding to that layer, denoted as $\{F_{G_{1}},F_{G_{2}},\ldots,F_{G_{n}}\}$. We denote the original feature map from the diffusion UNet as $F_{\text{target}}$. To perform the attention operation, the query $Q$ is derived from $F_{target}$, while the $K,V$ pairs are obtained from the concatenated features of $\{F_{\text{target}},F_{G_{1}},F_{G_{2}},\ldots,F_{G_{n}}\}$. This multi-reference warping attention allows us to transfer the features of different reference images onto the target image, thus effectively preserving the texture features of each object without mutual interference.
The process is illustrated in Fig. 2.

Figure: Figure 5: Qualitative comparisons on VITON-HD in the single try-on task. Compared with other methods, our method MMTryon produces more realistic and texture-consistent images.
Refer to caption: x5.png

## 4 Experiment

### 4.1 Datasets

We conduct extensive experiments on two public high-resolution Virtual Try-ON (VITON) benchmarks, namely VITON-HD [] and DressCode [], as well as on an additional large-scale proprietary e-commerce dataset. Experiments are conducted under a resolution of $1024\times 768$. Specifically, VITON-HD comprises 13,679 image pairs of front-view upper-body women and upper-body in-store garment images, which are further divided into 11,647 training pairs and 2,032 testing pairs. DressCode includes 48,392 training pairs and 5,400 testing pairs of front-view full-body person and in-store garment images, consisting of three subsets with different category pairs (i.e., upper, lower, dresses).
Our proprietary e-commerce dataset contains 203,239 training pairs and 5,219 testing pairs of front-view full-body person and in-store garment images, and 450,931 training pairs and 9,198 testing pairs of multiple front-view images of the same full body.

### 4.2 Implementation Details

During the data augmentation process, we enhanced the data across the three datasets, running the entire enhancement workflow on 16 A800 GPUs for 5 days. The augmented dataset comprises approximately 2 million images. In our MMTryon implementation, we utilize the SD 1.5 pretrained model [] to initialize the weights of our main UNet and the weights of our pretrained garment encoder. For the encoding of both text and global image information, we use clip-large-14 [].The garment encoder was pretrained on 8 A100 GPUs for 3 days with a total batch size of 8. MMTryon was trained on 8 A100 GPUs for 1 days with a total batch size of 8.

### 4.3 Baselines

In our study, we initially compare our approach on traditional try-on tasks with previous baseline methods. We selected four advanced GAN-based methods, namely PF-AFN [], FS-VTON [], SDAFN [], GP-VTON [] and the latest diffusion-based methods, DCI-VTON [] and StableVTON []. We directly utilize their released pretrained models for this comparison. Unfortunately, we are not able to conduct an extensive comparison with
TryonDiffusion [] as it is not publicly available at the time of submission.
Secondly, for the compositional and multi-modal try-on tasks, we compare our work with paint by example [], Stable diffusion  [], and DALLE3 [].
For all baselines, we strictly adhere to the official instructions for running their training and testing scripts. Additionally, to further demonstrate our models zero-shot capabilities, we further perform a comparisons with the state-of-the-art community model, Outfit Anyone.

### 4.4 Qualitative Results

Figure: Figure 6: Qualitative comparisons in the wild. Compared with OutfitAnyone, our method MMTryon produces more realistic and stable images. Note, as Outfit Anyone is only available through their user interface, the comparisons here are limited to their provided model images.
Refer to caption: x6.png

Figure: Figure 7: Qualitative comparisons with Paint-by-Example [], Midjourney and DALLE3 [] in the Multi-Modal Multi-Reference task. Compared with other methods, our method can achieve the best results both in terms of faithfulness and realism.
Refer to caption: x7.png

Comparison on single garment try-on.
Fig. 5 and Fig. 6 provides qualitative comparison between MMTryon and the state-of-the-art baselines in the single garment try-on task on the VITON-HD dataset []. The results demonstrate the superiority of our MMTryon over the baselines. Firstly, our method employs a more powerful garment encoder, which results in better texture consistency compared to the baseline. Secondly, as our approach utilizes more textual information, it achieves greater stability in try-on styles than the baseline, showcasing the effectiveness of our method.

Comparison on multi-modal and compositional try-on. Fig. 7 provides a qualitative comparison between MMTryon and the state-of-the-art baselines for the compositional try-on and multi-modal try-on tasks in the wild. The results demonstrate the superiority of our approach relative to the baselines. Firstly, for compositional try-on, our method exhibits better texture consistency and more realistic try-on effects compared to the baselines.
Secondly, for multi-modal try-on, our approach demonstrates more refined performance in terms of texture consistency.
At the same time, baselines often overlook the textual descriptions of dressing styles, whereas our method demonstrates stronger text-image consistency, effectively reflecting the intentions expressed in the text. This demonstrates that our approach achieves state-of-the-art results for the compositional and multi-modal try-on tasks.

**Table 1: Quantitative comparisons on the single garment tryon.**
| Method |  | SSIM $\uparrow$ | FID $\downarrow$ | LPIPS $\downarrow$ | KID $\downarrow$ | Human evaluation $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| PF-AFN [] |  | 0.885 | 9.616 | 0.087 | 3.85 | 0.03 |
| FS-VTON [] |  | 0.881 | 9.735 | 0.091 | 3.69 | 0.04 |
| SDAFN [] |  | 0.881 | 9.497 | 0.092 | 2.73 | 0.03 |
| GP-VTON [] |  | 0.893 | 9.405 | 0.079 | 0.88 | 0.18 |
| DCI-VTON [] |  | 0.868 | 9.166 | 0.096 | 1.10 | 0.06 |
| StableVTON [] |  | 0.866 | 8.992 | 0.079 | 1.03 | 0.20 |
| MMTryon |  | 0.912 | 8.702 | 0.069 | 0.58 | 0.40 |

Figure: Figure 8: Qualitative results of the ablation study. MMTryon w/o PR indicates MMTryon where only the second stage is trained, MMTryon w/o TQL indicates MMTryon without the text query loss in the first stage, MMTryon w/o MRA indicates MMTryon without the multi-reference attention.
Refer to caption: x8.png

### 4.5 Quantitative Results

Metrics.
For traditional single-garment try-on tasks, we utilize four widely-used metrics, namely the Structural Similarity Index (SSIM) [], Perceptual Distance (LPIPS) [], Kernel Inception Distance  [], and Fréchet Inception Distance (FID) [], to evaluate the similarity between synthesized and real images.
Additionally, we conducted a Human Evaluation (HE) study, inviting 100 reviewers to assess the synthesis quality and text-image consistency across single-garment try-on scenarios

Evaluation.
As reported in Tab. 1, our MMTryon consistently outperforms the baseline methods across all metrics, proving that MMTryon can generate try-on results with better visual quality in single garment try-on. Furthermore, in our human evaluation, we compare the generated results of the baselines to our model. Higher human evaluation scores indicate that a larger proportion of participants prefer the outcomes of the given method. The human evaluators prefer the results generated by MMTryon in most cases, indicating superior texture consistency and text-image consistency. Please note that we only present qualitative results for multi-modal compositional try-on as there is no publicly available dataset for compositional try-on.

### 4.6 Ablation study

**Table 2: Ablation results demonstrate the benefit of the pretrained garment encoder (PR), text query loss (TQL) and multi-reference attention (MRA).**
| Method | FID $\downarrow$ | KID $\downarrow$ | CLIP Score $\uparrow$ |
| --- | --- | --- | --- |
| w/o PR | 10.23 | 0.68 | 0.22 |
| w/o TQL | 9.25 | 0.65 | 0.24 |
| w/o MRA | 12.89 | 0.87 | 0.18 |
| MMTryon | 8.90 | 0.58 | 0.27 |

To validate the effectiveness of the pretrained garment encoder, we conduct an ablation study by comparing MMTryon with two ablated versions. One that directly trains the entire model without pretrained garment encoder (denoted as w/o PR) and one that does not include the text query loss (denoted as w/o TQL). The results, as shown in Fig. 2, indicate that omitting the pretrained garment encoder or text query loss will lead to a decrease in garment consistency and overall generation quality. The quantitative results, as presented in Fig. 8, demonstrate that MMTryon with pretrained garment encoder and supervised by text query loss significantly outperforms the other two versions as the garment consistency score and overall generation quality score are both higher.
Next, to further verify the effectiveness of the multi-reference attention, we conduct another ablation study. In this study, we compare MMTryon with a version that only utilizes the CLIP features of the clothing, denoted as w/o MRA, where we replaced the multi-reference attention with the original self-attention module. We again compare the main metric scores and observe a decline in both image quality and texture consistency when multi-reference attention is removed. Meanwhile, Fig. 8(b) indicates that rich clothing features are key to maintaining texture consistency.

## 5 Conclusion

In this work, we introduce MMTryon, a novel and powerful try-on model capable of freely generating high-fidelity VITON results with realistic try-on effects based on text and multiple garments. By using a pretrained encoder, MMTryon not only avoids the need for segmentation but also achieves compositional try-on even with limited data. To support multi-modal and multi-reference dressing modes, MMTryon introduces multi-modal instruction attention and multi-reference attention modules. Experiments conducted on high-resolution VITON benchmarks and in-the-wild test sets demonstrate MMTryon’s superior efficacy compared to existing methods.

## Appendix A Supplemental material

The outline of this supplementary material is as follows:

- •
Additional details of the training dataset ;
- •
Human Evaluation Details of our MMTryon;
- •
Societal impact ;
- •
Limitation and future work
- •
Additional visual results of our MMTryon.

### A.1 The Training dataset Details

For the training data, during the labeling process, we utilized 7 prompts to guide SDXL for inpainting, where the prompts mainly include different categories of clothing, such as adding a pair of trousers, a skirt, etc. We filtered out some inpainted data with poor image quality through human detection. Ultimately, we retained 3-5 of the most suitable and realistic images as our training data. Figure 9 illustrates visualizations of a random subset of the training data.To further make use of the garment-to-person pairs, we first train a single reference try-on network using a similar model design as described above, and transfer those images to person-to-person pairs, generating synthetic garment-person-person pairs. We now have constructed a multi-reference multi-modal instruction dataset that includes synthetic multi-reference data and single-reference garment-person pairs

### A.2 Human Evaluation Details

For the human evaluation, we designed questionnaire for the single garment try-on 100 volunteer were invited to complete a questionnaire comprising 50 assignments for each task. Specifically, given a person image and a garment image, volunteers were asked to select the most realistic and accurate try-on result out of seven options generated by our MMTryon and baseline methods (i.e., PF-AFN [], FS-VTON [], SDAFN [], GP-VTON [], DCI-VTON [], and Stable-VTON []). The order of the generated results in each assignment was randomly shuffled.

### A.3 Societal impact.

As most generative approaches, MMTryon can be used for malicious purposes by generating images that infringe upon copyrights and/or privacy. Given these considerations, responsible use of the model is advocated.

### A.4 Limitation and future work.

While our method demonstrates strong performance, it still has certain limitations. In the data generation process, our method is influenced by the limitations of pretrained models, making it challenging to produce data that meets the requirements for very fine parts, such as cuffs and collars. This restricts our ability to generate detailed components. Moving forward, we may focus on fine-tuning large models to construct a more freely detailed and fine-grained dataset, aiming to enhance the upper limit of our model.

### A.5 Additional Results

Fig  10 demonstrates a demo where up to six different categories of clothing are sequentially changed.
Fig  11 provides more comparison results on the Multi-Modal Multi-Reference scenarios.
Additionally, Fig. 12 and Fig.  13 and Fig.  14 displays additional results of MMTryon.

Figure: Figure 9: Visualization of our training dataset.
Refer to caption: x9.png

Figure: Figure 10: More results of MMTryon. Our method demonstrates high quality result in a large variety of scenarios and precise control over dressing styles through instruction-based manipulation.
Refer to caption: x10.png

Figure: Figure 11: More comparision result of MMTryon.
Refer to caption: x11.png

Figure: Figure 12: More results of our MMTryon. Please zoom in for more details.
Refer to caption: x12.png

Figure: Figure 13: More results of our MMTryon. Please zoom in for more details.
Refer to caption: x13.png

Figure: Figure 14: More results of our MMTryon. Please zoom in for more details.
Refer to caption: x14.png

## Peer Reviews

confidence: 5
contribution: 4
initial_rating: 8
presentation: 4
questions: 1.As mentioned in the weaknesses, please explain why the pants in the last rows of the first image changed unexpectedly.

2.the method is good，Are there any plans to release the code and model to support the  research community?

3.What is the inference time and required GPU memory for MMTryon in different task?
soundness: 4
strengths: 1. The paper introduces innovative multi-modal and multi-reference attention mechanisms, allowing for multi-garment try-ons with customized, fine-grained text guidance.

 2. Leveraging a pretrained garment encoder, MMTryon can capture detailed features of garments specified by text, improving model precision and contributing to advancements in the field.

 3. Extensive qualitative and quantitative evaluations demonstrate MMTryon’s superior performance compared to state-of-the-art methods, with high-quality visual results showcasing realistic try-on effects and robust editing capabilities.

 4. The proposed data generation pipeline creates a comprehensive paired multi-garment dataset without segmentation, broadening the scope of try-on tasks and supporting complex garment compositions.The parsing-free approach in this article presents a promising direction, contributing to the advancement of try-on technology toward more practical applications.

 5.This article is well-written
summary: The paper introduces MMTryon, a model that integrates a multi-modality and multi-reference attention mechanism to enhance virtual try-on tasks by combining garment details from reference images with dressing styles from text guidance. The model is designed for multi-garment try-ons and does not require segmentation networks, thanks to a new parser-free garment encoder and a scalable data generation pipeline. MMTryon achieves superior performance in single and multi-garment try-on tasks, addressing limitations in previous models such as single-garment constraints, fixed dressing styles, and segmentation dependencies.
weaknesses: 1.some descriptions could be clearer to avoid reader confusion. For example, in Fig. 3, labeling the UNet as “MMTryon UNet” rather than specifying it as a pretrained SD UNet might mislead readers at this training stage. Similarly, terms like “w/o TQL” and “w/o MRA” in Fig. 8 are somewhat unclear;  

2.the inference process requires more detailed explanation, especially given the model’s support for multiple tasks. Currently, some detail of inference lack explaination, which may lead to unintended changes in specific garments, such as unexpected alterations in lower garments shown in the first image.

---

confidence: 5
contribution: 4
initial_rating: 8
presentation: 4
questions: 1. Could you provide more information about the e-commerce dataset used in the experiments?
2. Do you have any plans to open-source the dataset and the data processing pipeline? Making these resources publicly available could greatly benefit research in this area and facilitate further advancements in virtual try-on methods.
soundness: 4
strengths: 1. The paper introduces a unique attention mechanism that enables multi-item try-on and customizable styling, addressing limitations in previous models.
2. The paper adopts a novel approach by eliminating segmentation dependency using a parsing-free garment encoder and scalable data pipeline. This method presents a practical solution to common issues in virtual try-on tasks, improving model efficiency and minimizing artifacts.
3. The paper is well-organized, and there are a lot of ablation study in main paper and appendix to demonstrate impact of each component in the proposed approach.
summary: This paper introduces MMTryon, a multi-modal, multi-reference Virtual Try-On (VITON) framework capable of generating high-quality, compositional try-on results from text instructions and multiple garment images. MMTryon addresses three main limitations in existing methods: supporting multiple garments, specifying dressing styles, and reducing dependency on segmentation models. It achieves this through a novel multi-modality attention mechanism that combines garment details from images with style cues from text, and by using a parsing-free garment encoder alongside a scalable data pipeline to eliminate the need for segmentation. Extensive experiments demonstrate MMTryon’s superior performance in multi-item and style-controllable try-on tasks, offering new potential for applications in the fashion field.
weaknesses: 1. The Related Work section lacks recent studies on control and try-on methods,
2. More detail information about dataset should be provided.
3. in some case, the fine texture on the garment still cannot be recovered completely,

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. The paper makes overstated claims, such as line 96 where it states, "is the first model to support the multi-modal compositional try-on task." However, multi-modal compositional try-on is already supported by M&M VTO [1], AnyFit [2], and VITON-DiT [3]. I don't think the relevant work is sufficient. And I recommend the authors explicitly compare these previous works in terms of their multi-modal capabilities to clarify if the proposed approach indeed offers novel advantages or is simply an extension of existing techniques.

2. The paper lacks clear motivation and has weak contributions. For example, Question 1 has already been explored, but the authors do not clearly articulate its relevance or contribution here. Additionally, texture consistency—an essential issue in VTON—seems neglected. I suggest the authors further clarify the connection between their research questions and contributions and add a discussion on texture consistency, as this factor is critical to the realism of virtual try-on. This improvement would enhance the clarity and relevance of the study.

3. The novelty is limited. The garment encoder appears to use cross-attention, which raises the question of whether there is anything unique about it. I recommend the authors explain how their garment encoder differs from or improves upon standard cross-attention mechanisms, especially if there are any specific design or functional enhancements, to demonstrate genuine innovation.

4. Serious concerns arise from fusing text and image features directly without decoupling, which may lead to unintended consequences, yet the paper does not address this. I suggest the authors provide a more thorough justification for this fusion strategy, as bypassing decoupling could compromise the model’s interpretability or adaptability. Discussing the rationale behind this choice and evaluating potential pitfalls would strengthen the explanation.

5. The paper lacks thorough comparison methods. In particular, quantitative results for single-garment cases are not provided for methods like Outfit Anyone, Kolors-Virtual-Try-On [4], Wear-Any-Way [5], and MV-VTON [6]. Additionally, Figure 6 does not display clear advantages. The omission of quantitative comparisons with key methods weakens the study’s validity. I suggest including comparative performance metrics with these established models in Table 1 and providing more insightful analysis of Figure 6 to support the claimed improvements. Besides,  Under multi-modal conditions, Table 7 suggests that background is not controllable. However, Imagedressing [7] has achieved background control, yet it is not referenced or discussed. If background control is indeed relevant, I suggest that the authors provide a direct comparison with Imagedressing or explicitly state if background control is beyond the scope of their study to give readers a clearer understanding of the method’s applicability.

6. Texture details (such as Pants, logo changed) appear poorly rendered across multiple cases, including those in the appendix. I recommend identifying specific areas with poor texture rendering and offering suggestions to improve texture detail quality, as this would enhance the realism and appeal of the model’s outputs.

7. The implementation details concerning SDXL on line 295 and SD 1.5 on line 352 are confusing. I suggest that the authors clarify why they employ different Stable Diffusion versions in various pipeline segments and explain the implications for model outputs to improve method transparency.

8. Adding results under similar conditions, such as challenging cases with letters, numbers, and diverse textures, is advised. The paper would benefit from showcasing the model’s performance on complex garments, such as those with letters or intricate textures, as these are realistic challenges in virtual try-on systems.


Other details:
1. In Table 2,  '0.7' should be '0.70'
2. Formulas 1 and 2 have commas at the end, while formulas 3 and 4 are missing.
---
[1] Zhu L, Li Y, Liu N, et al. M&M VTO: Multi-Garment Virtual Try-On and Editing[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 1346-1356.

[2] Li Y, Zhou H, Shang W, et al. AnyFit: Controllable Virtual Try-on for Any Combination of Attire Across Any Scenario[J]. arXiv preprint arXiv:2405.18172, 2024.

[3] Zheng J, Zhao F, Xu Y, et al. VITON-DiT: Learning In-the-Wild Video Try-On from Human Dance Videos via Diffusion Transformers[J]. arXiv preprint arXiv:2405.18326, 2024.

[4] https://huggingface.co/spaces/Kwai-Kolors/Kolors-Virtual-Try-On.

[5] Chen M, Chen X, Zhai Z, et al. Wear-any-way: Manipulable virtual try-on via sparse correspondence alignment[J]. arXiv preprint arXiv:2403.12965, 2024.

[6] Wang H, Zhang Z, Di D, et al. MV-VTON: Multi-View Virtual Try-On with Diffusion Models[J]. arXiv preprint arXiv:2404.17364, 2024.

[7] Shen F, Jiang X, He X, et al. Imagdressing-v1: Customizable virtual dressing[J]. arXiv preprint arXiv:2407.12705, 2024.
soundness: 2
strengths: 1. MMTryon is able to combine multiple garments for fitting, while also allowing the fitting effect to be manipulated by text commands. 
2. A scalable data generation pipeline and a specially designed clothing encoder are introduced. 
3. Eliminates the need for any prior segmentation network during the training and reasoning phases.
summary: This paper combines multi-modal implementation of VITON. 
Three problems are proposed and solved by the designed data set and multi-model instruction attention. 
The experimental results show that the data in the wild well.
weaknesses: 1. The definition of the task is not very clear, and similar work may already exist, leading to unclear motivation. Related work needs to be strengthened.
2. multi-modal multi-reference attention is just a simple extension of reference attention [1], and it is not clear how to decouple and align the details.
3. The quantitative and qualitative results are not complete, especially the texture details are not so good.
4. The data set is the highlight of this paper, but it doesn't seem to claim to be open data, so I don't think it's a contribution either.
---
[1] Hu L. Animate anyone: Consistent and controllable image-to-video synthesis for character animation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 8153-8163.

Meta-review: The paper receives mixed scores from the reviewers. While the reviewers appreciate the innovative design and extensive experiments, they also raised some concerns including claims overstated, the contributions incremental, and the texture quality limited.

In particular, M&M VTO already demonstrated several of the paper's claims (e.g. multi-modal compositional try-on) and indeed significantly weakens the paper's contributions. While M&M VTO admits it is not designed for this task, it is hard to deny the work can achieve such capability. The authors are encouraged to conduct more complete literature survey prior to the submission.

Moreover, the texture quality (e.g. text) is indeed limited in many cases (e.g. in Fig. 15). It appears that examples shown in M&M VTO (e.g. Fig. 4 in their paper) has better text quality. For qualitative comparisons to existing methods (Fig. 5 and 6), it seems a bit unclear if the proposed model is more correct than other methods.

Based on the above observations, the authors are encouraged to perform more detailed analysis and comparisons to existing works in the revised version.

Decision: Reject