GMAIL: Generative Modality Alignment for generated Image Learning
Authors: Shentong Mo
Sukmin Yun
Venue: ICML 2025 spotlightposter (2025)
Keywords: diffusion models
generated visual learning
vision-language models
Primary area: general_machine_learning->representation_learning
TL;DR: In this paper, we propose a novel framework for discriminative use of generated images that explicitly treats generated images as a separate modality from real images.

Abstract: Generative models have made it possible to synthesize highly realistic images, potentially providing an abundant data source for training machine learning models. Despite the advantages of these synthesizable data sources, the indiscriminate use of generated images as real images for training can even cause mode collapse due to modality discrepancies between real and synthetic domains. In this paper, we propose a novel framework for discriminative use of generated images, coined \textit{GMAIL}, that explicitly treats generated images as a separate modality from real images. Instead of indiscriminately replacing real images with generated ones in the pixel space, our approach bridges the two distinct modalities in the same latent space through a multi-modal learning approach. To be specific, we first fine-tune a model exclusively on generated images using a cross-modality alignment loss and then employ this aligned model to further train various vision-language models with generated images. By aligning the two modalities, our approach effectively leverages the benefits of recent advances in generative models, thereby boosting the effectiveness of generated image learning across a range of vision-language tasks. Our framework can be easily incorporated with various vision-language models, and we demonstrate its efficacy throughout extensive experiments. For example, our framework significantly improves performance on image captioning, zero-shot image retrieval, zero-shot image classification, and long caption retrieval tasks. It also shows positive generated data scaling trends and notable enhancements in the captioning performance of the large multimodal model, LLaVA.

GMAIL: Generative Modality Alignment for generated Image Learning
Shentong Mo Sukmin Yun
Abstract
1. Introduction
Generative models, such as GANs (Goodfellow et al., 2020; Chen al., 2016) and diffusson models (Song al., 2021a; Dhariwal & Nichol, 2021; Rombach al., 2022), have rev- olutionized the field of computer vision by enabling the synthesis of highly realistic images. These generated im- ages offer a rich and scalable source of data, which can significantly augment training datasets, enhance data diver- sity, and reduce the dependency on costly real-world data collection. However, despite their potential, incorporating generated images directly into training pipelines poses sub- stantial challenges due to inherent modality discrepancies between generated and real images. This misalignment of- ten leads to phenomenon known as mode collapse (LeCun, 2022), where the model's performance severely deteriorates due to an over-reliance on generated content that fails to generalize well to real-world scenarios. To address this, essential to solve the generated-to-real (Gen-Real) modality discrepancy problem first.
Generative models have made it possible to syn- thesize highly realistic images, potentially provid- ing an abundant data source for training machine learning models. Despite the advantages of these synthesizable data sources, the indiscriminate use of generated images as real images for training can even cause mode collapse due to modality discrepancies between real and synthetic domains. In this paper, we propose a novel framework for discriminative use of generated images, coined GMAIL, that explicitly treats generated images as separate modality from real images. Instead of indiscriminately replacing real images with gener- ated ones in the pixel space, our approach bridges the two distinct modalities in the same latent space through a multi-modal learning approach. To be specific, we first fine-tune a model exclusively on generated images using a cross- modslityy align- ment loss and then employ this aligned model further train various Ession-language models with generated images. By aligning the two modalities, our approach effectively leverages the benefits of recent advances in generative models, thereby boosting the effectiveness of generated image learning across a range of vision-language tasks. Our framework be easily incorporated with various vision-language models, and we demon- strate its efficacy throughout extensive experi- ments. For example, our framework significantly improves performance on image captioning, zero- shot image retrieval, zero-shot image classifica- tion, and long caption retrieval tasks. also shows positive generated data scaling trends and notable enhancements in the captioning performance of the large multimodal model, LLaVA.
Existing approaches (Tian et al., 2024) typically integrate generated images into the training process without ade- quately addressing the modality gap between generated and real images. The resulting models are prone to overfitting the peculiarities of synthetic data, which negatively impacts performance across various downstream tasks, particularly when the model encounters real-world data. The primary source of this collapse lies in the failure to recognize that generated images, despite their realism, represent distinct data modality that deviates from real images in subtle but significant ways. Addressing this modality gap is crucial to harnessing the full potential of generated data while main- taining robust performance on real-world tasks.
The challenge of using generated images stems from the fundamental differences between generated and real-world data distributions. Even when generated images appear vi- sually convincing, they often contain subtle artifacts, biases, or domain-specific noise introduced during the generation process. These discrepancies are not just visual but can also affect higher-level semantic representations. resulting in a misalignment in the feature space that can propagate through the training pipeline. Furthermore, generative mod- els may inadvertently capture and amplify biases present in their training data, leading to synthetic images that devi-
'Department of Machine Learning, CMU, USA Departmeet of Machine Learning. MBZUAI, UAE Depparmennt of Artificial Intelligence, Hanyang University ERICA, South Korea. Corre- spondence to: Sukmin Yun <sukminyun@ hachonyyuuuhhaannn....mm an..anng.a..r>...
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

GMAIL: Generative Modality Alignment for generated Image Learning
ate in unexpected ways from real-world distributions. This modality gap poses significant challenges for downstream tasks, where models trained misaligned data struggle with overfitting to generated features, reduced robustness, and degraded performance when applied to real images. Bridging this gap is critical to leveraging the strengths of generative models while avoiding pitfalls that compromise model reliability.
through extensive experiments on diverse of vision-language benchmarks, including image caption- ing. zero-shot image retrieval, and zero-shot image classification, and further validate its compatibility with the recent large multimodal model, LLaVA.
We explore the generated data scaling trend of our framework using large-scale generated datasets, demonstrating that our approach consistently improves as the volume of training data increases.
To tackle this challenge, we introduce novel framework for Generative Modality Alignment for generated Image Learning, namely GMAIL, that explicitly treats generated images as a separate modality from real images. Unlike conventional methods that mix generated and real data in- discriminately, our approach bridges the two distinct modal- ities in the latent space by embedding generated images alongside real images having the same descriptions. Specifi- cally, we fine-tune a model exclusively on generated images using a cross-modality alignment loss while keeping the pre- trained model for real images unchanged. This allows for explicit and adaptive alignment between the two modalities, enabling us to utilize the aligned model for training various vision-language models (Radford et al., 2021; Liu et al., 2023; Zhang al., 2024) with highly realistic generated images. Thereby, we fully exploit the advantages of recent advances in generative models (Rombach et al., 2022), en- hancing the performance of generated image training across various vision- Assion-languag tasks.
2. Related Work
Diffusion Models. Diffusion models (Ho et al., 2020; Song et al., 2021b;a) have emerged as powerful class of gen- erative models, capable of producing high-quality images that closely mimic the distribution of real-world images, Prominent examples include Stable-Diffusion (Rombach et al., 2022), DreamBooth (Ruiz et al., 2023; 2024), and the DALL-E series (Ramesh et al., 2021; 2022; Betker et al., 2023), which have demonstrated remarkable success in gen- erating diverse and complex images from textual descrip- tions. These models leverage advanced diffusion processes to iteratively refine images from noise, capturing intricate details and generating visually convincing outputs that can closely resemble real-world imagery. Our work utilizes the power of diffusion models to generate images, offering an innovative and cost-effective source of training data derived from textual descriptions. By aligning these generated im- ages with real image modalities through our GMAIL frame- work, we bridge the gap between synthetic image generation and practical machine learning applications, addressing the challenges of modality discrepancies. This application of diffusion models represents a novel contribution to the field, as it not only enhances training efficiency but also expands the use of generative models beyond mere content creation, embedding them directly into the model training process to improve real-world performance.
Through the extensive experiments across a wide range of vision-language tasks, we demonstrate the effective- ness of our framework by incorporating it with various vision-language models such as LLaVA (Liu et al., 2023). For example, our approach enhances image captioning on COCO (Lin et al., 2014), zero-shot image retrieval on COCO (Lin et al., 2014) and Flickr30k (Young et al., 2014), zero-shot image classification across eight widely used datasets, and long caption retrieval on ShareGPT4V (Chen observe positive generated
et al., 2024). Furthermore, data scaling trends in our framework across diverse datasets such as COCO (Lin et al., 2014), CC3M (Sharma et al., 2018), and CC12M (Changpinyo et al., 2021). highlighting the scalability of our method. Notably, our approach also improves the captioning performance of the recent large multimodal model, LLaVA (Liu et al., 2023), demonstrating its broad compatibility.
Generated Image Learning. Generated image learning has gained traction as researchers explore the potential of syn- thetic data augment traditional training paradigms. Syn- CLR (Tian et al., 2024) proposed a self-supervised frame- work that employs synthetic data to pre-train visual repre- sentations, showing that models trained on generated data achieve competitive results compared to those trained on real data. However, a critical challenge in this domain is the issue of mode collapse, where the over-reliance on synthetic data without proper alignment leads performance degra- dation when models are applied to real-world tasks. Recent work (Shumailov et al., 2024) highlights the inherent risks of training models on recursively generated data, emphasiz- ing that models can inherit and amplify errors in synthetic data, ultimately compromising their ability to generalize. Our research directly addresses these challenges by propos-
Our main contributions are summarized as:
We introduce a novel framework for discriminative use of generated images, explicitly treating them as distinct modality and aligning them with real images within the same latent space. It enables researchers to exploit highly realistic generated images effectively.
We demonstrate the effectiveness of our framework
2

---

GMAIL: Generative Modality Alignment for generated Image Learning
ing a novel strategy that treats generated images distinct modality and aligns them with real images in the same latent space. This approach not only mitigates the risk of collapse but also enhances the robustness by embedding generated images within the same latent space as real images.
models trained on both generated and real data.
3.1. Preliminaries
In this subsection, we introduce the problem setup and nota- tions, followed by an overview of the contrastive language- image pre-training methodology that forms the foundation of our approach.
Meanwhile, there also exist attempts (Zhang et al., 2021; Ye et al., 2024; Wu et al.. 2023) at generating high-quality labeled datasets. While prior works have primarily focused on improving image segmentation tasks by generating mask labels, we instead aim to bridge modality gaps when us- ing synthetic data to further train lanion- models, mitigating over-reliance on the synthetic domain.
Problem Setup and Notations. Let D. = { V-)) rep- resent a dataset of real images with corresponding labels or annotations, and D, = {(1g bo)} denote a dataset of generated images synthesized by generative models, such as GANs or diffusion models. Our objective is to train a model f()) that performs well across broad set of downstream tasks, utilizing both D, and Dg, while mitigating the risk of mode collapse caused by the inherent modality gap between D,, and Dz. To formally define the alignment process, we introduce two models: a base model fr. pre-trained on real images, and a fine-tuned model fg. trained specifically on generated images. The primary goal of our framework is to align fg with fr. ensuring that the feature representations of generated images are semantically consistent with those of real images. This alignment facilitates a unified understand- ing of both modalities, allowing the model to generalize across real data during inference.
Vision- Vision-Language Models. Vision-language models, such as CLIP (Radford et al., 2021). have revolutionized cross- modal understanding by learning joint representations of im- ages and text through contrastive learning objectives. While these models excel at leveraging large-scale real-world data, they often struggle when trained on generated images due to the modality gap. To overcome this, recent methods have explored various aligmment techniques to improve cross- modal performance. For example, Long-CLII (Zhang et al., 2024) extended CLIP by integrating longer captions, improv- ing its ability to handle more descriptive texts. Similarly, LLaVA (Liu et al., 2023) has demonstrated the potential for vision-languesd models to handle multimodal tasks like vi- sual question answering and captioning by leveraging large- scale data. Our work builds on these foundational efforts by introducing an explicit Gen-Real alignment framework that enhances the adaptability of Dision-languae models when using generated data. By embedding generated im- ages within the same latent space as real images and training the alignment, our approach directly addresses the modal- ity discrepancies that limit model performance, offering a scalable solution that significantly boosts cross-modal learn- ing across diverse vision- languag tasks, including image captioning, zero-shot retrieval, and classification.
Contrastive Language-Image Pre-training. Our frame- work builds on the foundation of Contrastive Language- Image Pre- Tre-Training (CLIP) (Radford et al., 2021), which learns joint embeddings for images and textual descriptions. CLIP leverages contrastive loss that brings the embeddings of paired images and texts closer, while pushing apart the embeddings of unpaired ones, fostering cross- mooal align- ment. However, traditional CLIP training does not explicitly address the discrepancy between generated and real images, often leading to performance degradation when integrating generated data directly. To extend CLIP to handle generated images as a distinct modality, we propose a modified train- ing objective that incorporates contrastive learning not only between real images and text but also between generated images and text. This treats generated and real images in- dependently, preserving the unique characteristics of each modality during training.
3. Method
In this section, we describe our proposed Generative Modality Alignment for generated Image Learning (GMAIL) framework, which tackles the challenge of training on generated images while ensuring robust performance dur- ing inference on real-world data, as illustrated in Figure Our approach introduces two key components: (1) Gen- CLIP flow on training and inference that handles generated and real images as separate modalities, and (2) an explicit alignment strategy with vision-language models to facili- tate better integration with large language models (LLMs) such as CLIPCap (Mokady et al., 2021), LLaVA (Liu et al., 2023), and Llama3 (Meta, 2024). In this part, we detail the problem setup, the key components of our framework, and the alignment strategy used to enhance the performance of
3.2. Gen-CLIP Flow: Training on Generated Images
The first key component of our method is the Gen-CLIP flow, which focuses on training the model on generated images while treating them as a distinct modality. Unlike traditional approaches that mix generated and real images indiscriminately, we handle generated images separately to prevent the model from overfitting to the peculiarities of synthetic data. In the Gen-CLIP flow, we fine-tune a pre-trained CLIP model (i.e., image encoder of f.) using generated images, paired with the same textual descriptions
3

---

GMAIL: Generative Modality Alignment for generated Image Learning
CLIP Flow:
Inference
Real Imuaee Encoder
Projection For Real
CLIPCop (GPTT
roal images
Inuge-text nligmment
plate of fruit the table."
Text Encoder
Gen-real aligument
LLM
LLaVA
Image-text aligument
Training generated images
Generated Image Encoder
Projection Generated
LLaMA-3
O LLAMA
Figure 1. Illustration of the proposed framework for vision-language tuning with Gen-Real alignment from diffusion models. We propose a method that explicitly aligns CLIP trained on real images with another CLIP model trained on generated images and then leverages the aligned CLIP to train state-of-the art vision-language models on generated images (i.e, Cen-CLIP Flow). inference with real images, the original CLIP is used to process real images, thereby avoiding discrepancies between real and generated modalities.
used for real images. During fine-tuning, we employ cross-modality alignment loss to minimize the feature space discrepancy between generated and real images (see Eq. 1). This contrastive alignment loss encourages the model ʃg to learn representations that place generated and real images with the same descriptions close each other in the latent space, while maintaining their distinct speccfc characteristics. To maintain computational efficiency and prevent catastrophic forgetting of real image representations, we apply Low-Ratn Rank Adaptation (LoRA) (Hu et al., 2022) during fine-tuning. LoRA introduces lightweight, efficient updates the model, ensuring that the alignment process does degrade the model's ability to generalize across different data modalities.
3.3. Alignment with Vision-Language Models
Our alignment strategy is designed to enhance the integra- tion of generated data into vision-language models, par- ticularly large language models (LLMs) such as CLIP- Cap (Mokady et al., 2021), LLaVA (Liu al., 2023), and Llama3 (Meta, 2024). This extension of GMAIL ensures that generated images can be utilized effectively within these models for tasks such as image captioning, retrieval, and long-form question answering.
The key to our framework is the cross-modality alignment loss, which ensures that generated images are embedded within the same latent space as real images, while maintaining their distinct characteristics. The alignment loss is formulated as:
Gen-Real Alignment.
CLIP Flow: Inference on Real Images. In the inference phase with real images, the model So fine-tuned on gen- crated images in the Gen-CLIP flow can be deployed to process real images through the original image encoder of fr without further fine-tuning mtninn. By keeping the pre-tran mreitrained CLIP model for real images unchanged during the generated image training process, sentations from the generated data remain aligned with real data. The CL/P flow leverages these aligned representations for inference on real images, allowing the model to gen- eralize well to real-world data without suffering from the typical mode collapse associated with over-reliance on gen- erated content. This dual-model structure allows the model to benefit from the complementary strengths of both real and generated images, ensuring that it performs robustly during real-world deployment while still benefiting from the scalability of generated training data. Note that the encoder fine-tuned with the LoRA and the projection for real used on real images during inference time.
Lalign
[B
(Eg.z.)EB
exp(sim([g(t,))) (A.))) log Σn EB exp(sim(ff(( (4)))))
(1)
ensure that the learned repre-
where 2g and x, represent generated and real images, f, and fr are their corresponding image representations, sim(..) denotes cosine similarity between embeddings, and 7 is a temperature parameter. This loss encourages generated images to be aligned with their real counterparts, facilitating effective transfer of knowledge across both modalities.
CLIPCap (Mokady et al., 2021) combines CLIP's 's image embeddings with mansformer-basee language model to generate captions from images. By aligning generated im- ages with real image embeddings, we ensure that CLIPCap can generate high-quality captions from both real and gener- ated data. Fine-tuning CLIPCap with our alignment frame-

---

GMAIL: Generative Modality Alignment for generated Image Learning
work allows the model to handle both modalities effectively, resulting in enhanced performance image captioning.
We measure both image-to-text and text-to-image retrieval capabilities using Recall 1, Recall and Recall @ 10. These metrics assess the model's ability to correctly retrieve relevant items based on the provided query, highlighting its cross-modal understanding. Zero-Shot Image Classifica- tion: The classification performance on unseen categories is evaluated using top-1 accuracy, reflecting the model's generalization ability to new classes without prior training those specific categories.
LLaVA (Liu et al., 2023) & Llama3 (Meta, 2024) are advanced multimodal models designed to perform vision- language tasks. To align generated images with these mod- els, we first fine-tune the vision-language models using our GMAIL strategy to ensure that representations from gen- erated data are aligned with real data. The aligned vision representations are then integrated with the LLMs, allowing the models to handle complex vision-language tasks such as long captioning and retrieval more effectively. This align- ment enhances the robustness and flexibility of LLaVA and Llama3 in both real and generated images.
Implementation. For image captioning, we adhere to the implementation strategy of ClipCap (Mokady et al., 2021), which combines CLIP with a text generation model to pro- duce descriptive captions for images. ClipCap uses CLIP's image embeddings as input to a transformer-basee caption- ing model, enabling the generation of semantically accurate and contextually rich captions for both real and generated images. For zero-shot evaluation on both retrieval and im- age classification tasks, we follow the setup detailed in the original CLIP (Radford et al., 2021) paper. This setup em- phasizes the model's ability to generalize across unseen data by using natural language prompts to guide image clas- sification and retrieval, leveraging the contrastive training between images and textual descriptions without explicit fine-tuning on target datasets. We adopt Stable Diffusion v2 (Rombach et al., 2022) to generate synthetic images us- ing captions from the COCO (Lin et al., 2014) train2014 set. Stable Diffusion provides high-quality image synthe- sis, enabling us to produce generated images that are both visually realistic and semantically aligned with the training captions, serving as the generated modality in our align- ment framework. During fine-tuning, we use rank of 4 in Low-Rank Adaptation (LoRA) to adjust the model pa- rameters specifically for generated images, ensuring that the adaptation remains efficient and computationally manage- able. For optimization, we use the Adam W optimizer with learning rate of 1 x 10-4 and weight decay of 0.01. We employ a cosine annealing schedule with warm restarts to dynamically adjust the learning rate, enhancing convergence stability across training phases. Batch normalization and gradient clipping are applied to prevent exploding gradients and ensure smooth training dynamics.
Our framework designed to scale effectively with larger datasets, as evidenced by the performance improvements observed on large-scale datasets such CC3M (Sharma et al., 2018) and CC12M (Changpinyo et al., 2021). The alignment strategy ensures that as the volume of generated training data increases, the model continues to generalize effectively to real-world data. This scalability demonstrates the potential of GMAIL as a cost-effective solution for train- ing robust vision-language models using synthetic data.
4. Experiments
In this section, we provide the experimental setup, evalua- tion metrics, and comparative analysis conducted to validate the effectiveness of our method. Through rigorous experi- mentation on a diverse set of datasets, we assess our model on image captioning, zero-shot image retrieval, and zero- shot image classification tasks, comparing it against existing benchmarks to highlight our contributions.
4.1. Experimental Setup
Datasets. Our experiments leverage a comprehensive collec- tion of datasets to evaluate the versatility and effectiveness of our proposed Gen-Real alignment framework. We fo- cus on a diverse set of tasks, including image captioning, zero-shot image retrieval, and zero-shot image classification, ensuring broad coverage across various domains. Please refer Appendix Section A for the detailed dataset settings.
4.2. Comparison to prior work
Evaluation Metries. To comprehensively evaluate our framework, we employ task-specific metrics tailored to im- age captioning, zero-shot image retrieval, and zero-shot image classification: Image Captioning: Performance is as- sessed using standard metrics such as BLEU@4 (B@4) (Pa- pineni al., 2002), METEOR (Denkowski & Lavie, 2014), CIDEr (Vedantam et al., 2014), SPICE (Anderson et al., 2016), ROUGE-L (Lin & Och, 2004), and Word Mover's 's Distance (WMD) (Kusner et al., 2015). These metrics evalu- ate the quality and semantic accuracy of generated captions compared to ground truth. Zero-Shot Image Retrieval:
Image Captioning. We compare our model's performance on the COCO dataset against prior commonly-used base- lines, including ClipCap (Mokady et al., 2021), LLaVA (Liu et al., 2023), and LLAMA-3 (Meta, 2024). The results, detailed in Table 1, demonstrate significant improvements across all evaluated metrics, underscoring the efficacy of our GMAIL approach when combined with synthetic images and LoRA optimization. For ClipCap, the proposed Clip- Cap + GMAIL configuration achieves 38.12 B @4, 31.67 METEOR, 119.53 CIDEr, 23.75 SPICE, 56.27 ROUGE-L,
5

---

GMAIL: Generative Modality Alignment for generated Image Learning
Table Image captioning, We perform tue-tuning on pre-trained ClipCap, IFCap, LLaVA, and 3LaMA-3 for image captioning on COCO. We report the standard metrics to evaluate the quality of generated captions. The best results are indicated in bold.
Method
B@4 (*)
METEOR(1)
CIDEr (1)
SPICE (1)
ROUGE-L (1)
WMD (1)
ClipCap (Mokady et al., 2021)
32.15
27.10
108.35
20.12
ClipCap + GMAII (ours)
38.12
31.67
119.53
23.75
56.27
62.16
IFCap (Lee et al., 2024)
33.25
28.60
115.27
21.58
51.35
56.72
IFCap GAMIL (ours)
39.32
32.07
127.86
23.98
59.83
63.51
LLaVA (Liu et al., 2023)
39.67
32.38
134.29
24.17
61.36
65.78
LLaVA + GMAIL (ours)
43.26
34.89
146.38
27.23
65.25
71.39
Llama3 (Meta, 2024)
47.36
35.21
158.13
28.35
68.32
75.13
Llama3 GMAIL (ours)
50.21
38.59
168.53
32.58
73.29
80.25
Table 2. Zero-shot image retrieval on COCO. We perform zero-shot retrieval pre-trained CLIP models for image retrieval on the
COCO benchmark. We report the image-to-text and lext-to-image Recall 1,5,10 metrics to evaluate the quality retrieved images.
Image-to Text
Text-to-Iman Image
Method
R@ 1 (f)
R@5 (1)
R@ R@10 (1)
R@ (1)
R@5 (*)
R@ 10 (1)
CLIP (Radforr et al., 2021)
51.8
76.8
84.3
32.7
57.7
68.2
CLIP + GMAIL (ours)
56.8
80.1
87.2
37.5
62.7
73.2
Long-CLIP (Zhang et al., 2024)
57.2
80.8
87.8
40.4
65.9
75.7
Long-CLIP- + GMAIL (ours)
62.3
84.1
91.2
45.6
69.8
79.5
and 62.16 WMD, significantly outperforming the baseline ClipCap and the ClipCap+ LoRA setup. Specifically, our GMAIL, approach boosts the original ClipCap (Mokady et al., 2021) by 5.97 @@4, 4.57 METEOR, 11.18 CIDEr, and 3.63 SPICE. These results highlight the advantages of aligning generated and real images within a unified seman- tic space, allowing for enhanced image captioning perfor- mance. Similarly, when applied to LLAMA-3, our LLAMA- 3 + GMAIL model reaches 50.21 B @4, 38.59 METEOR, 168.53 CIDEr, 32.58 SPICE, 73.29 ROUGE-L, and 80.25 WMD, demonstrating notable improvements over both the baseline and the LoRA fine-tuning strategy. Compared to LLAMA-3 alone, GMAIL achieves gains of 2.85 B @ 2.46 METEOR, 10.35 CIDEr, and 4.30 SPICE, establish- ing our approach as robust technique for enhancing mod- els through Gen-Real alignment. The substantial gains ob- served across both model architectures confirm the effec- tiveness of our GMAIL framework. By tne-tuning with generated images while maintaining alignment with real image modalities, our method effectively bridges the modal- ity gap, resulting in better understanding and generation of descriptive captions aligned with
on real images by 5.0 R@ 1, 3.3 R@5, and 2.9 R@ 10. For text-to-image retrieval, CLIP GMAIL scores 37.5 R@
62.7 R@5, and 73.2 R @ 10, demonstrating gains of 4.8 R@ 1, 5.0 R and 5.0 R@ 10 compared to the baseline CLIP. These improvements validate the effectiveness of our alignment strategy in bridging the gap between generated and real image modalities, enhancing zero-shot retrieval capabilities. Similarly, when applied to the Long-CLIP ar- chitecture (Zhang et al., 2024), our Long-CLIP + GMAIL configuration further boosts performance, achieving 62.3 @@I 1, 84.1 R @5, and 91.2 R @@I 10 on imae-ttotexxt retrieval, and 45.6 @ 1, 69.8 &@S, and 79.5 R @ 10 on text-to-image retrieval. This demonstrates that GMAIL consistently en- hances model performance across different backbone archi- tectures by facilitating better aligmment of generated images with real-world data. On the Flickr30k dataset, our CLII GMAIL model achieves 47.1 R @@) 71.2 R @ and 79.6 R 10 for image- textet retrieval, outperforming CLIP by 3.0 R@ 1, 3.2 R@5, and 2.6 R @@ 10. In text-to-image re- trieval, the model scores 39.3 R @@@1 1, 61.5 R @5,, and 71.8 R@ 10, with respective gains of 14.6 R @ 1, 16.0 R@5, and 17.2 R @@ 10 over CLIP. These results validate the robustness of our approach in learning meaningful representations from generated images for zero-shot retrieval real images.
data.
Zero-shot Image Retrieval. The comparative results in Tables 2 and 3 highlight our model's superior recall rates, showcasing its robustness in understanding and associating visual and textual data. Our method is evaluated on two benchmarks: COCO and Flickr30k, using both image-to- text and text-to-image retrieval tasks, demonstrating signif- icant improvements over prior baselines. On the COCO dataset, our approach, CLIP+ GMAIL, achieves 56.8 R@ 80.1 @@5, and 87.2 R@ @ 10 for image-to- retrieval, out- performing the original CLIP (Radford et al., 2021) trained
Zero-shot Image Classification. We evaluate the zero-shot classification performance of our model across eight di- verse benchmarks, including DTD, Stanford Cars, SUN397, Food 101, Aircraft, Oxford Pets, Caltech 101, and ImageNet IK. As shown in Table our model consistently achieves top-1 accuracy surpassing previous approaches, validating the advantage of leveraging generated images through our framework for enhancing zero-shot learning capabilities.
6

---

GMAIL: Generative Modality Alignment for generated Image Learning
Table Zero-shot image retrieval on Flickr30k. We perform zero-shot retrieval on pre-trained CLIP models for image retrieval on the Flickr30k benchmark. We report the image-to-text and text- text-to-image Recall @ 1,5.10 metrics to evaluate the quality of retrieved images.
Image-to- to- Text
-exa-to-Image
Method
@ I (1)
R@5 (1)
R@ (1)
R@I 1 (1)
R@5 (*)
R@ 10 (1)
CLIP (Radford et al., 2021)
44.1
68.2
77.0
24.7
45.1
54.6
CLIP + GMAII. (ours)
47.1
71.2
79.6
30.2
50.3
60.5
Long-Cli Long-CLIP (Zhang al., 2024)
47.2
71.5
80.0
33.1
55.6
64.9
Long-CLIP CLIP + GMAIL (ours)
51.6
75.3
83.6
39.3
61.5
71.8
Table 4. Zero-shot image classification. We perform a zero-shot evaluation pre-trained CLIP models for image classification eight henchmarks. We report the top-1 accuracy to evaluate the quality of learned representations. The best results are indicated in bold.
Method
DTD
Stanford Cars
SUN397
Food 101
Aircraft
Oxford Pets
Caltech 101
ImageNet
CLIP (Radford et al., 2021)
55.20
77.53
69.31
93.08
32.88
93.33
93.24
75.54
CLIP + GMAIL (ours)
65.26
81.32
75.53
95.21
37.85
95.23
95.57
77.68
SynCLR (Tian et al., 2024)
79.90
93.80
76.20
91.60
81.70
93.60
95.30
85.80 (ft)
SynCLR + GMAII. (ours)
83.67
96.56
81.25
96.38
86.75
95.70
98.35
87.95 (ft)
Table 5. Long caption retrieval on ShareGPT4V. We report the image-to to-extt-ttttttaa and text-to-image Recall 1 to evaluate the quality of retrieved images. The best results are indicated in bold.
our CLIP GMAIL configuration achieves 85.2 for image- to-text and 86.7 for text-to-image retrieval, outperforming both the original CLIP (Radford et al., 2021) and the CLIP
Method
Image-to-Text
Text-to-Image
+ LoRA variants. This result highlights the effectiveness of our alignment strategy in bridging the semantic gap be- tween generated and real images, particularly when handling complex, long-caption scenarios. When applied to the Long- CLIP architecture (Zhang et al., 2024), our Long-CLII + GMAIL configuration reaches 97.2 for image-to-text and 96.1 for text-to- eott-to-image retrieval, marking the highest perfor- mance among all tested configurations. These gains of 2.6 and 1.6 over Long-CIL CLIP LoRA confirm that our approach not only strengthens the alignment between modalities but also substantially improves the retrieval of images and cap- tions involving extended and intricate descriptions. Overall, these results confirm the robustness and scalability of our framework in managing complex captioning tasks.
CLIP (Radford et al., 2021)
78.2
79.6
CLIP + GMAIL (ours)
85.2
86.7
Long-CLIP (Zhang al., 2024)
94.6
93.3
Long-CLIP- GMAIL (ours)
97.2
96.1
Our CLIP + GMAIL approach achieves top-1 accuracy of 65.26 on the DTD benchmark, outperforming the original CLIP (Radford et al., 2021) by 10.06 points, demonstrating the significant benefit of aligning generated images with real data. On the Stanford Cars dataset, our model reaches 81.32, showing robust performance gains, particularly in fine-grained classification tasks. For the challenging FGVC Aircraft benchmark, our method scores 37.85, marking substantial improvement of 4.97 over the baseline CLIP, highlighting our model's 's capacity to handle complex visual distinctions. Additionally, our model performs exception- ally well on other benchmarks, achieving 75.53 on SUN397, 95.21 on Food 101, 95.23 on Oxford Pets, 95.57 on Cal- tech 101, and 77.68 on ImageNet 1K. These results consis- tently outperform both the standard CLIP and the CLIP LoRA setup, confirming the effectiveness of our Gen-Real alignment strategy in broadening the model's generalization capabilities across various domains.
4.3. Experimental analysis
In this section, we performed ablation studies to demonstrate the benefit of Gen-Real alignment. We also conducted ex- tensive experiments to explore the scaling trend on different training data sizes.
Gen-Real Alignment. To quantify the impact of Gen-Real alignment fine-tuning on our model's performance, we con- ducted ablation studies comparing models with and without alignment optimization. The results, presented in Table 6, demonstrate significant improvements across all metrics when alignment tuning is applied, validating the effective- ness of our proposed approach. In the context of image captioning tasks, models fine-tuned with Gen-Real align- ment consistently outperform their counterparts that lack this optimization step. Specifically, adding Gen-Real align- ment to the vanilla bascline using synthetic images to fine- tune all parameters led to substantial increases across all
Long Caption Retrieval. We evaluate our model's 's capabil- ity to handle long captions using the ShareGPT4V (Chen et al., 2024) benchmark, as reported in Table 5. The evalu- ation focuses on image-to-text and text-to-inng -o--gee-iageeea retrieval tasks, with Recall @1 I used to assess the quality of retrieved results. Our model demonstrates an enhanced ability to com- prehend and generate relevant responses to extended textual inputs, affirming its utility in applications that require de- tailed and deseriptive outputs. For the CLIP-baseed models,
7

---

GMAIL: Generative Modality Alignment for generated Image Learning
Table Ablation study on Gen-Real Alignment. We perform ablation studies on image captioning from pre-trained CLIP on generated images. The best results are indicated in bold.
Alignment @4 (1)
METEOR(1)
CIDEr (1)
SPICE (t)
ROUGE-L (1)
WMD (1)
36.15
30.32
115.35
22,95
55.12
61.08
38.12
31.67
119.53
23.75
56.27
62.16
Table 7. Scaling trend of Gen-Real alignment on zero-shot image retrieval on Flickr30k. We perform zero-shot retrieval on models trained from COCO, CC3M, and CC12M Flickr30k. We report the Recall 1,5,10 metrics to evaluate the quality of retrieved images.
Image-to-Text
Texl-to-Image R@5 (t)
Train Data
R@I (?)
R@5 (1)
R@ 1010 (t)
R@ I (f)
R 10 (1)
COCO
47.1
71.2
79.6
30.2
50.3
60.5
CC3M
48.6
73,6
82.2
32,6
52.6
62.3
CC12M
50.9
75.3
84.6
34.9
54.7
64.8
Table 8. Comparison with SigLIP on COCO captioning. Our GMAIL significantly improves SigLIP by effectively addressing the synthetic-real discrepancy. The best results are bold.
Method
B #@4 (1)
CIDEr (t)
SigLIP
37.51
117,82
SigLIP GMAIL (ours)
42.35
125.68
The results reveal a clear scaling trend, where increasing the volume of training data from COCO to CC3M and then to CCI2M consistently enhances the model's performance on both image-to-text and text-to-image retrieval tasks. Specif- ically, model trained est scores with 50.9 R @@ 1, 75.3 R @5,, and 84.6 R @@ 10 for image-to-text retrieval, and 34.9 R @ 1, 54.7 R@5, and 64.8 R@ 10 for text-to-image retrieval, outperforming the models trained on the smaller COCO and CC3M datasets. These improvements demonstrate that our Gen-Rea align- ment framework benefits significantly from larger and more diverse training datasets of generated images, effectively capturing richer semantic representations and enhancing re- trieval capabilities. The results underscore the effectiveness of our method in leveraging the scaling trend of generated data, showing that as the scale of synthetic images increases, our model continues to learn and generalize better across zero-shot retrieval tasks.
CC12M achieves the high-
Figure 2. Visualizations of real (Column 1) and generated im- ages (Columns 2-6) using the same caption. Those generated images generally capture high-level semantics in real images.
evaluated metrics: 3.56 in B@4, 1.13 in METEOR, 4.18 in CIDEr, 0.8 in SPICE, 1.15 in ROUGE-L, and 1.09 in WMD. These improvements highlight the critical role of alignment fine-tuning in bridging the modality gap between generated and real images, which enables the model to better cap- ture and replicate the semantic richness found in real-world data. The results underscore the effectiveness Gen-Real alignment in optimizing model performance, particularly in adapting the nuances of generated images and their associated textual descriptions. By embedding generated images within the same latent space as real images, our approach enhances the model's ability to understand and process complex visual-language relationships, ultimately leading to superior performance in downstream tasks.
Visualization of Generated Images. To further understand the quality and semantic aligmment of the generated images used in our training process, we provide visualizations of a subset of generated images alongside their corresponding real counterparts, as shown Figure 2. These images were generated using Stable Diffusion (Rombach et al., 2022). and are designed to closely match the real-world data in terms of visual realism and content. Through these visual- izations, we observe that while generated images generally capture high-level features and structures present in real images, they may still exhibit subtle artifacts or variations that could contribute to the modality gap. Despite these dif- ferences, our Gen-Real Alignment framework successfully
Scaling trend of Gen-Real alignment. To further eval- uate the scalability of our proposed Gen-Real alignment, explore its performance across varying scales of train- ing data. Specifically, we apply our training framework synthetic images derived from COCO (Lin et al., 2014), CC3M (Sharma et al., 2018), and CC12M (Changpinyo et al., 2021). The comparison results on zero-shot image retrieval on the Flickr30k benchmark are reported in Table
8

---

GMAIL: Generative Modality Alignment for generated Image Learning
Table Comparison with models trained on real images. We perform experiments on image captioning from pre-traines trained CLIP on generated and real images. The best results are indicated in bold.
Dual Projection
Alignment
Fine-tuning Data
B@4 (1)
CIDEr (1)
SPICE (1)
X
X
32.15
108.35
20.12
X
X
Synthetic
36.15
115.35
22.95
/
J
Synthetic
38.12
119.53
23.75
X
X
Real
38.24
119.78
23.86
Real
38.37
119.95
23.98
Table 10. Visual question answering on SeienceQA. We report the average accuracy on questions with the image context from the SciencoQA benchmark. The best results bold.
Table II. Ablation study LoRA rank and full fine-tuning. We perform experiments on image captioning from pre-trained CLIP on generated images. The best results are indicated in bold.
Method
Accuracy (%)
Method
B@4 (1)
CIDEr (t)
SPICE (t)
LLaVA
85.2
LoRA (rank = 2)
36.85
117.62
23.10
LLaVA GMAIL (ours)
87.6
LoRA (rank 4)
38.12
119.53
23.75
LLaMA-3
88.5
LoRA (rank 6)
37.96
119.12
23.60
LLaMA-3 + GMAIL (ours)
91.2
Full fine-tuning
37.50
118.95
23.50
These results highlight GMAIL's ability to enhance VLM's generalization across multimodal tasks.
bridges this gap, evidenced by the alignment of seman- tic features between the generated and real images in the learned latent space. The visualizations not only illustrate the potential of generated data as cost-effective supple- ment to real-world data but also highlight the importance of explicit alignment strategies to mitigate discrepancies between generated and real data distributions.
A Ablation on LoRA. LoRA (Hu et al., 2022) allows efficient adaptation to synthetic data while preserving the knowledge from pre-training on large-scale real data. This avoids the need for full fine-tuning, which can overwrite important pre- trained weights, especially when synthetic data is noisy or biased. The ablation results are reported in Table 11. As can be seen, LoRA with rank 4 achieves the best performance, balancing computational efficiency and alignment quality. Meanwhile, LoRA updates 35% fewer parameters compared to full fine-tuning while achieving better performance.
Comparison with SigLIP. To strengthen the effectiveness of our work, we compare GMAIL with SigL.P (Zhai et al., 2023) on COCO captioning. The results are shown in Ta- ble 8. SigLIP (Zhai et al., 2023) adopts sigmoid loss for better image-text pre-training, focusing solely on real im- ages. In contrast, our GMAIL, aligns real and generated images as distinct modalities, addressing the challenges of integrating synthetic data into further training. GMAIL is particularly relevant in scenarios requiring synthetic data, such as handling expensive attribute annotations or gener- ating diverse samples. These results demonstrate that our GMAIL complements SigLIP by effectively addressing the synthetic-real discrepancy, allowing for enhanced general- ization and performance improvements.
5. Conclusion
In this work, we present G.AII, novel framework for generative-to-reet alignment that addresses the modality gap between generated and real images, a key challenge that often leads to mode collapse when integrating generated data into training pipelines. Our approach explicitly treats generated images a separate modality and employs training scheme that aligns these images within the same latent space as real images. By fine-tuning models on gen- erated images, while maintaining pre- pre-trained model for real images, our framework facilitates explicit alignment be- tween the two modalities, leading to significant performance improvements across various vision-language tasks. Exten- sive experiments demonstrate the efficacy of our method on wide range of benchmarks, including image captioning, zero-shot image retrieval, and zero-shot image classifica- tion. Our results consistently show that GMAIL enhances the model's ability to generalize and perform across tasks, particularly when trained on large-scale datasets. The seal- ing trend observed with larger generated datasets such as CC12M further highlights the robustness and adaptability of our approach.
Training on Real Images. To illustrate the impact of real images on GMAIL, we compare performances COCO captioning using real-only (first and fourth rows). mixed real-generated data (second row), and GMAIL aligmment strategies (third and fifth rows). The results are shown in Table 9. These results indicate that GMAIL's alignment strategy not only bridges the synthetic-real gap but also could improve models trained exclusively real data.
ScienceQA Results. We also evaluated GMAIL's perfor- mance on ScienceQA (Lu et al., 2022) when integrated with LLaVA (Liu et al., 2023) and LLaMA-3 (Meta, 2024). We calculated the average accuracy of questions with the image context. The comparison results are reported in Table 10.

---

GMAIL: Generative Modality Alignment for generated Image Learning
Acknowledgements
Chen, L., Li, J., Dong, X., Zhang, P., He, C., Wang. J., Zhao, F., and Lin, D. Sharegpt4v: Improving large multi-modal models with better captions. In European Conference on Computer Vision, pp. 370-387. Springer, 2024.
This work was supported by the research fund of Hanyang Lniversity(HY-2022-2200 and the National Supercomput- ing Center with supercomputing resources including techni- cal suppor(KSC-222-CRE..000
Chen, X., Duan, Y., Houthooft, R., Schulman, Sutskever, I., and Abbeel, Infogan: Interpretable representation learning by information maximizing generative adver- sarial nets. Advances in neural information processing systems, 29, 2016.
Impact Statement
Our work leverages generative models, such as stable dif- fusion models (Rombach et al., 2022), to create generated images that can supplement real-world datasets in training machine learning models. While this approach offers signifi- cant benefits in terms of reducing the need for expensive and time-consuming real-woo world data collection, we recognize the potential ethical risks associated with generated data. Generated images may inadvertently reflect biases present in the data used to train the generative models, potentially per- petuating harmful stereotypes or inaccuracies. To mitigate this, we emphasize the importance of careful curation of training datasets and encourage the community to develop strategies for auditing and debiasing generative models. Ad- ditionally, the alignment of generated data with real-world data must be handled with caution, as over-reliance on gen- crated content can obscure important real-world variations.
Cimpoi, M.. Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In Proceed- ings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei- Fei, L. ImageNet: A Large-Scale Hierarchical Image Database. In Proceedings of IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 248-255, 2009.
Denkowski, M. and Lavie, A. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the ninth workshop statistical machine translation, pp. 376-380, 2014.
References
Dhariwal, P. and Nichol, Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 118780-8794, 2021.
Anderson, P., Fernando, B., Johnson, M., and Gould, S. Spice: Semantic propositional image caption evaluation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 382-398, 2016
Fei-Fei, L., Fergus, R., and Perona, P. Learning generative visual models from few training examples: An incremen- tal bayesian approach tested on 101 object categories. In 2004 Conference on Computer Vision and Pattern Recog- nition Workshop, pp. 178-178, 2004.
Assran, M., Duval, Q., Misra, I., Bojanowski, P., Vincent, P., Rabbat, M., LeCun, Y., and Ballas, N. Self-supervised learning from images with a joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 15519- 15629, 2023.
Goodfellow, I., Abaget-Abadic, J., Mirza, M., Xu, B., Farle- D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial networks. Communications of the ACM, 63911):19--144, 2020.
Betker, J., Goh, G., Jing, L., Brooks, T., Wang, J., Li, L., Ouyang, L., Zhuang, J., Lee, J., Guo, Y., Manassra, W., Dhariwal, P., Chu, C., Jiao, Y., and Ramesh, A. Improving image generation with better captions. OpenAl Technical Report, 2023.
Ho, J., Jain, A., and Abbeel, Denoising diffusion proba- bilistic models. In Proceedings of Advances In Neural In- formation Processing Systems (NeurIPS), pp. 6840-6851, 2020.
Bossard, L., Guillaumin, M., and Van Gool, L. Food-101 mining discriminative components with random forests. In European Conference on Computer Vision, 2014.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Changpinyo, S., Sharma, P., Ding, N., and Soricut, R. Con- ceptual 12M: Pushing web-scale image-text pre-training to recognize loig visual concepts. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Krause, J., Stark, M., Deng, J., and Fei-Fei, L. 3d object rep- resentations for fine -naiiind categorization. In 2013 IEEE International Conference on Computer Vision Workshops, pp. 554,561,72 2013.
10

---

GMAIL: Generative Modality Alignment for generated Image Learning
Kusner, M. J., Sun, Y., Kolkin, N. I., and Weinberger, K. Q- From word embeddings to document distances. In Pro- ceedings of International Conference Machine Learn- ing (ICML), 2015.
Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar,
C. V. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition, 2012.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-888 8763. PmLR, 2021.
Labs, B. black- peazest-labs/feee
Flux.
https://githis com/ 2024.
LeCun, Y. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62, 2022.
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Rad- ford, A., Chen, M., and Sutskever, I. Zero-shot text-to- image generation. In International conference on ma- chine learning, pp. 8821-8831, Pmlr, 2021.
Lee, S., Kim, S.-W., Kim, I., and Kim, D.-J. Ifcap: Image- like retrieval and frequency-basee based entity filtering for zero- shot captioning. arXiv preprini arXiv:2409. 18046, 2024.
Lin, C.-Y. and Och, F. J. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Annual Meeting of the Association for Computational Linguistics, 2004.
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.0612. 2022.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., and Zitnick, C. L. Microsoft coco: Com- mon objects in context. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 740-755, 2014.
Ommer, B. High-resolution image synthesis with la- tent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, June 2022.
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF conference on computer vision and paltern recognition, pp. 22500-22510, 2023.
Liu, H., Li, C., Wu,, Q. and Lee, J. Visual instruction tun- ing. In Proceedings of Advances In Neural Information Processing Systems (NeurIPS), 2023.
Lu, P., Mishra, S., Xia, T., Qiu, L.,, Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. The 36th Conference on Neural Information Processing Systems (NeurIPS). 2022.
Ruiz, N., Li, Y., Jampani, V., Wei, W., Hou, T., Pritch, Y., Wadhwa, N., Rubinstein. M., and Aberman, K. Hyper- dreambooth: Hypernetworks for fast personalization of text-to-image models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 6527-6536, 2024.
Maji, S., Rahtu, E., Kannala, J., Blaschko, M., and Vedaldi, A. Fine-graide Fine-grained visual classification of aircraft. arXiv preprint 1306vv1306.5// 2013.
Sharma, P., Ding, N., Goodman, S., and Soricut, R. Con- ceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Gurevych, I. and Miyao, Y. (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 2556-2565, 2018.
Meta, A. Introducing meta llaaa 3: The most capable openly available Ilm to date. Meta AI, 2024.
Mishra, S., Panda, R., Phoo, C. P., Chen, C.-F. R., Karlinsky, L., Saenko, K., Saligrama, V., and Feris, R. S. Task2sim: Towards effective pre-tra pre-training and transfer from synthetic data. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9194 -204, 2022.
Shumailov, I., Shumaylov, 7.,, Zhao, Y., Papernot, N., Ander- son, R., and Gal, Y. Ai models collapse when trained on recursively generated data. Nature, 631:755-759, 2024.
Mokady, R., Hertz, A., and Bermano, A. H. Clip- cap: Clip prefix for image captioning. arXiv preprint arXiv:2111.09734, 2021.
Song. J., Meng, C., and Ermon, Denoising diffusion im- plicit models. arXiv preprint arXiv:2010.02502. 2021a.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In Proceedings of International Conference on Learning Represenaatonss (ICLR), 2021b.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-. Bleu: a method for automatic evaluation of machine translation. In Annual Meeting of the Association for Computational Linguistics, 2002.

---

GMAIL: Generative Modality Alignment for generated Image Learning
Tian, Y., Fan, L., Chen, K., Katabi, D., Krishnan, D., and Isola, P. Learning vision from models rivals learning vision from data. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 15887-15898, 2024.
Zhang, B., Zhang, P., Dong, X., Zang, Y., and Wang. J. Long-clip: Unlocking the long-text capability of clip. In European Conference Computer Vision, pp. 310-325. Springer, 2024.
Zhang, Y., Ling, H., Gao, J., Yin, K., Lafleche, J.-F, Bar- riuso, A., Torralba, A., and Fidler, S. Datasetgan: Effi- cient labeled data factory with minimal human effort. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 10145-10155, 2021.
van der Maaten, L. and Hinton, G. Visualizing data using t-sne. Journal of Machine Learning Research, 9(86): 2579-2605, 2008.
Vedantam, R., Zitnick, L., and Parikh, D. Cider: Consensus-bard based image description evaluation. 2015 IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 4566-4575,
Wu, W., Zhao, Y., Shou, M. Z.., Zhou, H., and Shen, C. Dif- fumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1206-1217, 2023.
Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE Computer Society Conference Computer Vision and Pattern Recognition, pp. 3485- 3492, 2010.
Xiao, J., Ehinger, K. A., Hays, J., Torralba, A., and Oliva, A. Sun database: Exploring large collection of scene categories. International Journal of Computer Vision, 119:3-22, 2014.
Ye, H., Kuen, J., Liu, Q. Lin, Z., Price, B., and Xu, D. Seggen: Supercharging segmentation models with text2mask and mask2img synthesis. European Confer- ence on Computer Vision, pp. 352-370. Springer, 2024.
Young, P., Lai, A., Hodosh, M., and Hockenmaier, J. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Lin- guistics, 2:67-78, 2014.
Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., Wei, C., Yu, B., Yuan, R., Sun, R., Yin, M., Zheng, B., Yang, Z., Liu, Y., Huang. W. Sun, H., Su, Y., and Chen, W. Mmmu: A massive multi-dis multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pp. 9556-9567, June 2024.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training. Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision (ICCV), pp. 11975-1 1196,, 2023.
12

---

GMAIL: Generative Modality Alignment for generated Image Learning
Appendix
In this appendix, we provide the following material:
addition implementation and datasets details in Section A,
algorithm for our GMAIL in Section B,
more discussions on Gen-Real alignment in Section C,
more experimental analyses in Section D,
qualitative visualization results in Section E.
A. Implementation & Dataset Details
In this section, we provide additional implementation details to ensure the reproducibility of our experiments, along with a comprehensive description of the datasets used.
Implementation. The base model used in our framework is the CLIP model (Radford et al., 2021), pre-trained on real images and paired with their textual descriptions. We fine-tune the pre-trained CLIP model on generated images using the LoRA (Hu et al., 2022) method to introduce low-rank updates, ensuring that the training remains computationally efficient. For contrastive learning, we set the temperature parameter
0.07 and optimize using the Adam optimizer with learning rate of 1 x 10 and a batch size of 256. The synthetic training data were generated using Stable Diffusion v2 on NVIDIA A A100-80GB GPUs. The number of generated images is consistent with the number of text-image pairs in the original training set: 560k for COCO, 3.3 million for CC3M, and 12 million for CC12M. Each image was generated with 50 inference steps, balancing quality and computational efficiency. The total generation time is 5 GPU days for COCO, 30 GPU days for CC3M, and 109 GPU days for CC12M. Parallelized generation was employed for larger datasets like CC12M. Fine-tuning for "Proj for Real" and "Proj for Gen" was performed for 50,000 steps.
To evaluate the versatility and effectiveness of our Gen-Real Alignment framework, we employ a comprehensive suite of datasets across a variety of tasks, including image captioning, zero-shot image retrieval, and zero-shot image classification. This ensures a broad assessment of our model's performance across multiple domains and challenges.
Datasets.
COCO (Lin et al., 2014): The COCO dataset is used for both image captioning and zero-shot image retrieval tasks. It offers a large and diverse collection of real-world images paired with detailed textual descriptions, serving as a benchmark for evaluating the alignment generated and real image modalities.
Zero-Shot Image Classification: To evaluate the generalization capabilities of our model, we utilize eight well- kell-nnown benchmarks, following the setup of the original CL.IP (Radford et al., 2021):
DTD (Cimpoi et al., 2014): Tests the model's ability to classify textures across various images.
Stanford Cars (Krause et al., 2013): A dataset focusing on fine-grained classification of car models, used to assess the model's capacity distinguish between visually similar objects.
-UN3997 (Xiao et al., 2010; 2014): A large-scale scene classification dataset used to evaluate scene understanding,
Food 101 (Bossard et al., 2014): A benchmark used to assess the model's ability to classify food items from various cuisines.
Aircraft (Maji et al., 2013): Used for fine-grained classification of aircraft models, testing the model's accuracy in distinguishing similar objects.
Oxford Pets (Parkhi et al., 2012): A dataset focused on the classification of various pet breeds, including both dogs and cats.
Caltech 101 (Fei-Fei et al., 2004): A widely used object recognition dataset covering variety of general categories.
- -ImageNet 1K (Deng et al., 2009): A benchmark for large-scale object classification, testing the model's ability to handle diverse image categories.
13

---

GMAIL: Generative Modality Alignment for generated Image Learning
Algorithm 1 GMAIL Algorithm: Training and Inference Generated and Real Images
Require: Datasets of real images D, = {x+, Mr) generated images D, - ( Nalh and image pairs D { (z)) pre-trained CLIP models fr and g. learning rate 7, batch size [B], temperature т, LoRA parameters.
Ensure: Fine-tuned model fg for generated images, aligned with f+ real images.
1: Initialize: Load the pre-trained CLIP models fr and fg trained on real and generated images, respectively.
2: Gen-CLIP Flow: Aligning Generated and Real Images.
3: for each mini-batch B {zg. 2+)} from D do
4: Extract generated image features fg (xyy for each Tg B using fg.
5: Extract real image features (x,) for each &, E B using fr.
6: Compute tross-modaliyg alignment loss Lotign:
log exp(sin([g(gg) [xxx))//T 2 EB ex exp(sim (z)gg) (+(4))//T)
Latign
B (ag.te JEB
7: Apply LoRA updates to minimize Catign.
8: Update model parameters fg 5g nV fatign.
9: end for
10: Alignment with Vision-Language Models for downstream tasks.
11: for each LLM (c.g., from CLIPCap, LLaVA, LLaMA3) do
12: Fine-tune the LLM using the aligned generated and real image embeddings from D, and D,, respectively.
13: end for
CLIP Flow: VLM Inference on Real Images.
for each real image T, for inference do
16: Extract real image features fx, using fr-
17: Use (x,, instead (x,) for inference the VLM model on real images.
18:
end for
19:
Return: VLM models with fg trained on generated images, aligned with the real-image model fy.
CC3M (Sharma et al., 2018) and CC12M (Changpinyo et al., 2021): These large-scale datasets provide millions of image-caption pairs, allowing us to explore the scalability of our Gen-Real alignment framework. We evaluate our model's performance when trained on both real and generated data from these expansive datasets.
ShareGPT4V: To evaluate long caption retrieval,
the ShareGPT4V dataset, which includes complex and descriptive captions associated with both generated and real images. This dataset emphasizes the importance of strong cross-modal alignment for retrieving long, detailed captions.
Evaluation Metrics. To comprehensively evaluate our framework, we employ task-specific metrics tailored to image captioning, zero-shot image retrieval, and zero-shot image classification:
Image Captioning: Performance is assessed using standard metrics such as BLEU@4 (B@4) (Papineni et al., 2002). METEOR (Denkowski & Lavie, 2014), CIDEr (Vedantam et al., 2014), SPICE (Anderson et al., 2016), ROUGE-L (Lin & Och, 2004), and Word Mover's Distance (WMD) (Kusner al., 2015). These metrics evaluate the quality and semantic accuracy of generated captions compared to the ground truth.
Zero-Shot Image Retrieval: measure both image-to-text and text-to-image retrieval capabilities using Recall @ 1, @5, and Recall@ 10. These metrics assess the model's ability to correctly retrieve relevant items based on the provided query, highlighting its cross-modal understanding.
Zero-Shot Image Classificatiin: Classification performance unseen categories is evaluated using top-1 accuracy, which reflects the model's generalization ability to classify new classes without prior training on those specific categories.
This experimental setup allows us thoroughly validate our Gen-Real alignment framework across a wide range of
14

---

GMAIL: Generative Modality Alignment for generated Image Learning
Table 12. Computational costs comparisons on COCO training. Our GMAIL introduces a slight increase in memory usage but remains
more efficient on the convergence training time and steps than the baseline of indiscriminate mixing (gentreal) without alignment.
Synthetic Data
Training
Training
Memory
FLOPs
Dual Projection
Alignment
Time (hrs)
Steps
Usage (GB)
(G)
X
X
8
50k
24
70.2
X
/
12
70k
26
85,5
/
10
60k
28
85.5
Table 13. Comparison with the same training steps. We compare CLIP trained on real images, fine-tuned CLIP on generated images
without alignment, and ours under COCO training with the same training steps. The best results are indicated in bold.
Dual Projection
Alignment
Synthetic Data
@@4 (t)
CIDEr (1)
SPICE (f)
x
x
32.15
108.35
20.12
X
/
35.76
113.42
22.63
/
37.92
117.6
23.42
tasks, demonstrating its effectiveness in addressing the modality gap between generated and real images and enhancing performance across diverse vision-language applications.
B. GMAIL Algorithm
In this section, we outline the algorithm that implements the Generative Modality Alignment for generated Image Learning (GMAIL.) framework, incorporating the Gen-CIII flow for training on generated images and the CLIP flow for inference on real images. This algorithm also details the cross-modality alignment loss and how we ensure alignment with vision- vision-language models (VLMs) such as CLIPCap (Mokady et al., 2021), LLaVA (Liu al., 2023), and LLaMA-3 (Meta, 2024).
Algorithm summarizes the training and inference process for the GMAII. framework, detailing how the model is trained on generated images using the Gen-CLIP flow, and subsequently applied to real images during inference. The algorithm also explains how to integrate aligned generated and real data with lssiun-l models such as CLIPCap, LLaVA, and LLaMA-3 for downstream tasks.
C. More Discussions on Gen-Real Alignment
In this section, provide comprehensive discussion of Gen-Real Alignment. Given training samples having the same text: real image R, synthetic S, and text T, let us denote our dual encoders as f, h for real-encoder, syn-encoder, and text-encoder, respectively.
Single vs. Dual Modality, In single-modality scenario (i.e., a single encoder setup where f = g). given training would reduce distance (ff(R), h(T)) and D(f(S), h(T)), and then (f((.....))) f(S) would be reduced together. However, due to the nature of synthetic images, there could exist a gap between R and S, such unnatural artifacts, assuming S contains spurious features. Therefore, under such approaches put real and generated images into the same embedding space, generated artifacts may dominate, causing poor generalization and overfitting to synthetic patterns. Moreover, if the encoder ignores such different inputs R and S, and produces representations that remain constant and equal, it can lead to "mode collapse" (LeCun, 2022; Assran et al., 2023), where the model overfits generated patterns, degrading performance on real data. On this line, we consider a dual-modality scenario, (i.e., dual encoder setup where f * g) to prevent such a problem ((S)). Here, we instead minimize D(J ((T)) and ((g(S), ((T)), so allowing small (f)) hS)), not (R), f()). Specifically, the expected role of f is to ignore a synthetic complement of S and produce representations that remain intersection of 5 and R (having the same T). Such separate mappings of and y would allow learning focused on shared characteristics between the real and generated modalities. Thereby treating generated images distinct modality, GMAIL. could prevent "mode collapse", enabling the effective use of synthetic data to augment real datasets without poor generalization and overfitting to synthetic patterns.
caused by reducing a distance
Cross-Modality Alignment Loss. Furthermore, the proposed cross- coos--odaaltty aligmment loss aims to directly reduce a distance (R) h(S)) allowing effective and faster training to convergence. As shown in Table 12, the proposed loss
15

---

GMAIL: Generative Modality Alignment for generated Image Learning
Table 14. Quantitative similarity metrics comparisons on COCO, We computed the cosine similarity between paired real and generated image embeddings without and with alignment on COCO dataset.
Alignment
Cosine Similarity (1)
X
0.52
0.89
Table 15. Multi-modal reasoning and visual-language alignment on MMMU. We evaluated GMAIL's performance on MMMU benchmark when integrated with LLaVA. We report the average accuracy questions with the image context. The best results are bold.
Method
Accuracy (%)
LLaVA
44.7
LLaVA GMAIL (ours)
48.3
reduced training time and steps to convergence. Throughout our extensive experiments, for a given R and S having the same T, we have demonstrated the effect of minimizing distance (IJ h(S)) which learns shared semantics between real and generated images while ignoring generated artifacts of may raise poor generalization on real images.
Empirical Validation of Alignment Loss. Nevertheless, further conducted ablation study the effect of the cross-modality alignment loss (i.e., the effects of directly reducing (RJ h(8)) under the dual encoder setup on COCO captioning. The results in Table confirm that the alignment loss significantly bridges the modality gap, resulting in consistent performance improvements.
D. More Experimental Analysis
Computational Costs. We performed additional experiments to compare the computational costs. Table 12 shows the results, including explicit details on the contributions of the cross-modality alignment loss and dual-model setup. The additional costs for GMAIL stem from the cross-modality alignment loss, which facilitates aligning the features of generated and real images in shared latent space, and the dual-projection setup, which processes the two modalities separately, Compared to CLIP without the dual projection, our GMAIL introduces a slight increase in memory usage but remains more efficient on the convergence training time and steps than the baseline of indiscriminate mixing on generative and real data without the alignment.
Same Training Steps. Here, have conducted additional experiments with equal training steps for CLIPCap (Mokady al., 2021) across all compared methods and included the results in the Table 13. Even with identical training steps, GMAIL consistently outperforms both baselines, confirming that gains stem from our alignment design, not from additional training.
MMMU Results. We also evaluated GMAIL's performance on MMMU (Yue et al., 2024), which is a benchmark that requires multi-modal reasoning and disual-language alignment. Table 15 shows that GMAIL can improve visual grounding and alignment with LLMs, demonstrating generalization to reasoning- based VLM tasks.
Comparison with Tasks2Sim. Here, we have added experiments comparing Tasks2Sim (Mishra et al., 2022) in Table 16. Specifically, GMAIL outperforms prior mix-only methods due to explicit modality disentanglement and alignment, rather than blending synthetic and real data blindly.
Other generative models. We have conducted experiments using FLUX (Labs, 2024), which introduces a more powerful and differently parameterized generation pipeline compared to Stable Diffusion v2. These additional results in Table 17 allow us to test GMAIL's robustness to shifts in generator-specife artifacts. The performance improvements remain consistent with FLUX, indicating robust alignment across varying artifact styles and photorealism levels.
Qualitative Embeddings Visualization. To further validate the alignment between real and generated data, we conducted t-SNE (van der Maaten & Hinton, 2008) visualizations and cosine similarity analyses of the embeddings without and with alignment. Figure 3 shows the t-SNE plots of real and generated embeddings from 1000 samples in the COCO dataset. Without alignment, real and synthetic embeddings form two distinct clusters, reflecting the modality gap. With alignment proposed in our GMAIL, the gap between real and synthetic embeddings is significantly reduced, with both modalities aligning closely.
16

---

GMAIL: Generative Modality Alignment for generated Image Learning
Table 16. Comparison to Task2Sim, We report comparison to Task2Sim (Mishra et al., 2022), which introduced a mixed training approach (joint usage of real and synthetic data). The best results indicated bold.
Method
B@4 (1)
CIDEr (1)
SPICE (1)
Task2Sim
36.65
115.72
23.02
GMAIL (ours)
38.12
119.53
23.75
Table 17. Image generation FLUX. We replace a generative model from Stable Diffusion v2 FLUX and perform experiments on the image captioning task. The best results are indicated in bold.
Method
B@4 (1)
CIDEr (1)
SPICE (1)
FLUX (without alignment) FLUX + GMAIL (ours)
37.20
117.82
23.40
39.54
122.36
24.15
Quantitative Similarity Metries. We also quantified the alignment score of the above 1000 samples in the COCO dataset using cosine similarity between paired real and generated embeddings. The results are shown in Table 14. These results demonstrate that the alignment loss effectively bridges the Gen-Real gap, ensuring better feature consistency across modalities. Moreover, without alignment, we observed that the cosine similarity between embeddings of real image-text pairs and generated image-text pairs was only 0.44 and 0.42, respectively. This empirically demonstrates that the misalignment between real and generated images is comparable to the gap between images and text embeddings of CLIP models.
E. Qualitative Visualizations
In this section, provide qualitative visualizations of the generated images used in our experiments. Figures 4, 5, 6, 7, 8 and 9 show examples of images generated by Stable Diffusion (Rombach et al., 2022), alongside their corresponding real-world counterparts from the COCO dataset (Lin et al., 2014). Our visualizations demonstrate that the generated images closely resemble real images, capturing key semantic details and structural elements. However, subtle differences in texture or object placement are occasionally present. These artifacts highlight the importance of our Gen-Real Alignment (GMAIL) framework, which ensures that these differences do not lead mode collapse by aligning the feature representations of generated and real images in the latent space. These visualizations further validate the effectiveness of our alignment strategy, ensuring that both generated and real data contribute equally to the model's understanding during inference.
17

---

GMAIL: Generative Modality Alignment for generated Image Learning
w/o alignment
w alignment
synthetic
synthetic real
real
Figure 3. Qualitative Visualizations of embeddings of real and synthetic images without (Left) and with (Right) alignment. Blue red dots denote the embeddings for real and synthetic images, respectively. Our GMAIL with alignment significantly reduced the gap between real and synthetic images, with both modalities aligning closely in the latent space.
IN PARA
3rd POPPED 30
10
Figure 4. Visualizations of real (Column 1) and generated images (Columns 2-6) using the caption. Those generated images generally capture high-level semantics in real images.
18

---

GMAIL: Generative Modality Alignment for generated Image Learning
Sunsbury'
Figure 5. Visualizations of real (Column 1) and generated images (Columns 2-6) using the caption. Those generated images generally capture high-level semantics in real images.
19

---

GMAIL: Generative Modality Alignment for generated Image Learning
Figure Visualizations of real (Column 1) and generated images (Columns 2-6) using the same caption. Those generated images generally capture high-level semantics in real images.
20

---

GMAIL: Generative Modality Alignment for generated Image Learning
Listrine
Figure 7. Visualizations of real (Column 1) and generated images (Columns 2-6) using the same caption. Those generated images generally capture high-leve semantics in real images.
21

---

GMAIL: Generative Modality Alignment for generated Image Learning
Figure Visualizations of real (Column 1) and generated images (Columns 2-6) using the same caption. Those generated images generally capture high-level semantics in real images.
22

---

GMAIL: Generative Modality Alignment for generated Image Learning
8
000
Figure 9. Visualizations of real (Column 1) and gemerated images (Columns 2-6) using the same caption. Those generated images generally capture high-level semantics in real images.
23


Reviews:
[{"review_id": "3evHL4uPlM", "reviewer": "Reviewer_4rMz", "summary": "Authors propose a framework to train multimodal LLM on generated images. This work recognizes that generated images offer a different distribution than real images, and then should, in those settings, be recognized as a different modality altogether to be useful for training. Authors adopts the code base of the open source multimodal LLM LLaVA and perform multiple experiments and evaluation, demonstrating than ingesting generated images as a separate modality improves performance on multiple modality tasks, including image captioning, zero-shot image retrieval, zero-shot classification and caption retrieval.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's feedback and the questions raised. Below, we address and resolve each of these questions in our responses.\n\n> Model Collapse \n\nWe agree. To clarify the severity and nature of the issue, we added a direct comparison between models trained on: Real images only (baseline), Generated images without alignment, Generated images with our GMAIL alignment. Training directly on generated images without alignment reduces generalization to real-world test data. GMAIL mitigates this through modality-aware learning.\n\n| Training   Modality | Align | B@4 ↑ | CIDEr ↑ | SPICE ↑ |\n|:----------------:|:---------------:|:------------------:|:---------------:|:------------------:|\n| Real only           | ✗           | 32.15 | 108.35  | 20.12   |\n| Generated only     | ✗           | 36.15 | 115.35  | 22.95   |\n| Generated + GMAIL   | ✔           | **38.12** | **119.53**  | **23.75**   |\n\nWithout alignment, performance improves on synthetic data, but fails to transfer well to real data. GMAIL avoids overfitting to synthetic artifacts by learning aligned representations.\n\n\n\n> More Analysis\n\nThank you for the suggestion. We now refer the term “model collapse” to better reflect what we observe: a divergence from real-image generalization, rather than complete failure. We also show latent space shifts. Without GMAIL, embeddings for the same image-caption pair differ significantly across real and synthetic inputs.\n| Alignment   | Avg. Cosine Similarity ↑ |\n|:----------------:|:---------------:|\n| ✗ (no GMAIL)     | 0.25                     |\n| ✔ (GMAIL)        | 0.89                     |\n\n> Embedding Space Collapse Without Alignment\nTo complement the cosine similarity, we include a t-SNE visualization (Figure 3, Appendix), which shows that real vs. generated images cluster apart without GMAIL, but overlap with GMAIL, supporting the hypothesis of latent space misalignment and collapse risk.\n\n> Scalability and Training Stability with GMAIL\nGMAIL introduces modest memory overhead but significantly improves convergence and stability, especially in large-scale training. We empirically found that GMAIL improves convergence speed and representation robustness with fewer training steps.\n| Method          | Synthetic Data | Steps ↓|\n|:----------------:|:---------------:|:---------------:|\n| CLIP (gen only) | ✔             | 70k     |\n| GMAIL (ours)    | ✔              | 50k    |\n\n> Effectiveness on Downstream Tasks\nWe included experiments on: Alignment improvements to SigLIP models (Table 9) and Visual QA on ScienceQA (Table 10) in the appendix. We also include the results on MMMU benchmark, which is a benchmark that requires multimodal reasoning and visual-language alignment, in the Table below. GMAIL improves visual grounding and alignment with LLMs, demonstrating generalization to reasoning-based VLM tasks. \n\n| Model         | Test Overall (%) |\n|:----------------:|:---------------:|\n| LLaVA         | 44.7             |\n| LLaVA + GMAIL (ours) | **48.3**             |\n\n> Naming \n\nThis acronym was chosen for memorability, but we understand the concern. We’ve added a footnote clarifying no relation to Google.\n\n\n> Font and Typos\n\nWe fixed the typo in Line 84, and Figure 1 now uses CMU Serif, is fully vectorized (PDF), and screen-reader accessible.", "claims_and_evidence": "Authors claim that considering generated images as a separate modality than non-generated images brings multiple benefits to train multimodal LLM. The alternative, -- considering both generated and real images as a same modality, exposes at risk of misalignment between the image space and the decoder text space and model collapse in general, as generated images are differing in nuanced but significantly ways. To support that claim authors conducted multiple experiments and evaluations.", "methods_and_evaluation_criteria": "- Generated images are generated with diffusion models, and then used to train a CLIP based encoder. Authors introduces a cross-modality loss to ensure that generated images share the same representation space with the real image, while keeping their respective characteristics. \n- Authors use a shared caption as \"anchor\" between the two spaces, ensuring that generated and real images linked by that anchor are close, ie. have a high cosine similarity, in the share space. \n- The rest of the work is fairly standard, with the use of captioning evaluation metrics (BLEU, ROUGE, etc.) on common datasets (CoCo, etc.) and standard evaluation practices for image/caption retrieval.", "theoretical_claims": "- See 'Claims and Evidence'", "experimental_designs_or_analyses": "- The experimental settings are correct, and the evaluation is done thoroughly. One could have appreciated however to see a clear evidence of the risk of 'model collapse' when trained on generated image as used as justification for this work. Instead, as shown in Table 6, it seems models trained on generated images without considering them as an external modality are more performant, but not conversely, models that don't follow that strategy seem to only perform slightly worse, -- i.e there is not an evidence of such \"model collapse\".", "relation_to_broader_scientific_literature": "- Authors articulate their contributions with other works in the literature.", "essential_references_not_discussed": "-", "other_strengths_and_weaknesses": "- The overall paper is well written, and quite clear. Authors designed their experiments to respond to the claim made in the abstract/introduction, and show indeed improvements with their method. One could have hoped a better analysis of the 'model collapse' when trained on generated image without their method used as justification.", "other_comments_or_suggestions": "- L084: \"tje\" -> \"the\" \n- Figure 1: Use a serif font like the body of text, make it vectorized (\\include{yourfigure.pdf}) so this works with accessibility readers. \n- Minor comment: The use of 'GMAIL' for the title of this work is a bit confusing, as it obviously sounds like the famous Google's mail service. Is that intended? One could have preferred Gen-Real alignment, or something that overlap less with other companies/services.", "questions_for_authors": "Dear authors, thank you for your work:\n - Can you give more details about the model collapse when training on generated images without considering as modality loss? Is that something you have witnessed first hand? Can you share results on that point please?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W1Sn1tG3jP", "reviewer": "Reviewer_EHmi", "summary": "This paper proposes a method to fine-tune a CLIP image encoder on synthetic image samples so that it can be used in training vision-language models with generated data. The \"Gen-CLIP Flow\" is learning two CLIP image encoders, one for real images and the other for synthetic images. The \"Alignment with Vision-Language Models\" uses the learned CLIP image encoders above to extract features from real and synthetic images separately, which are later fed into the Vision-Langauge models. The proposed training method shows performance improvement on zero-shot image retrieval on COCO, Flickr30k, zero-shot image classification, long caption retrieval", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's feedback and the questions raised. Below, we address and resolve each of these questions in our responses.\n\n> Training Steps\n\nThank you for this observation. We have conducted additional experiments with equal training steps across all compared methods and included the results in the Table below. Even with identical training steps, GMAIL consistently outperforms both baselines, confirming that gains stem from our alignment design, not from additional training.\n\n| Method            | B@4 ↑ | CIDEr ↑ | SPICE ↑ |\n|:----------------:|:---------------:|:----------------:|:---------------:|\n| CLIP (real only)  | 32.15 | 108.35  | 20.12   |\n| CLIP (gen only)   | 35.76 | 113.42  | 22.63   |\n| GMAIL (ours) | **37.92** | **117.6**   | **23.42**   |\n\n\n\n> Experiments on MMMU\n\nThank you! We included results on MMMU, which is a benchmark that requires multimodal reasoning and visual-language alignment. GMAIL improves visual grounding and alignment with LLMs, demonstrating generalization to reasoning-based VLM tasks. \n\n| Model         | Test Overall (%) |\n|:----------------:|:---------------:|\n| LLaVA         | 44.7             |\n| LLaVA + GMAIL | **48.3**             |\n\n\n\n> Method Description \n\nWe apologize for the confusion. We’ve made the following changes to improve clarity: Figure 1 is now redesigned and clarified with modality-specific flows and alignment stages. We added formal notation in Section 3.1 (Preliminaries) and Section 3.2 (Gen-CLIP Flow). We also introduced the key equation for alignment and clarified dual-encoder roles.\n\n> Algorithm 1\n\nWe’ve rewritten Algorithm 1 in Appendix B with clearer, formal steps and precise mathematical notation. We replaced vague language like “aligned representation​” with formal notation, and split training into two distinct phases: Gen-CLIP Flow with LoRA adaptation and Inference/Transfer to Vision-Language Models (CLIPCap, LLaVA, etc.)\n\n> Sentence in Line 188-190\n\nWe use a dual projection structure. During inference, real images are encoded via the original encoder $f_r$ (not $f_g$). This avoids degradation from overfitting to synthetic artifacts in $f_g$. The aligned model still benefits from synthetic training via shared projection space.\n\n> References to Data Synthesis for Vision Tasks\n\nThank you for the suggestion! We’ve now added and discussed all three works in Related Work:\n\n- DatasetGAN (CVPR’21): Efficient labeled data generation via GANs and semantic segmentation.\n\n- SegGen (ECCV’24): Mask2Img diffusion pipelines for boosting segmentation.\n\n- DiffuMask (ICCV’23): Pixel-level annotations via text-to-mask-to-image synthesis.\n\nWe emphasize that GMAIL is complementary to these efforts:\n\n- They focus on generating high-quality labeled datasets.\n- We focus on bridging modality gaps when using synthetic data to train vision-language models.\n\n> Typos\n\nWe carefully proofread the entire paper and corrected all identified typos.", "claims_and_evidence": "Although it is good that we see improvement on many visual-language tasks, the effectiveness of the synthetic data training is not well grounded as the experimental settings are not reasonable. As shown in Table 8, the authors use more training steps for synthetic training than baseline, which means the model received more training signals than synthetic training. I suggest the authors use the same training step for all experiments for a fair comparison.", "methods_and_evaluation_criteria": "- Experiments on VLM benchmarks (e.g. MMMU) are missing, which will be more convincing for illustrating the influence of the visual-language alignment training.", "theoretical_claims": "No theory is involved.", "experimental_designs_or_analyses": "The experimental settings are not reasonable. \n\n- As shown in Table 8, the authors use more training steps for synthetic training than baseline, which means the model received more training signals than synthetic training. I suggest the authors use the same training step for all experiments for a fair comparison.", "relation_to_broader_scientific_literature": "This paper is related to many works using data synthesis methods to boost model performance, especially in the field of computer vision and vision-langauge alignment.", "essential_references_not_discussed": "The authors should discuss many more data synthesis methods for visual understanding, for example, DatasetGAN [1], SegGen [2], Diffumask [3].\n\n\n[1] DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort. CVPR 2021\n[2] SegGen: Supercharging Segmentation Models with Text2Mask and Mask2Img Synthesis. ECCV 2024\n[3] DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Models. ICCV 2023", "other_strengths_and_weaknesses": "Other Strength:\n- The idea of utilizing synthesis to help understanding is elegant.\n\nOther Weakness:\n- The writing of this paper needs significant improvement. I find myself confused while reading the method part although the method is relatively simple. For example, the description of Gen-CLIP Flow is not clear enough. I suggest the author add necessary formulate or diagram to help to understand. Also, there are many typos.\n\n- The Algorithm 1 is also confusing. Again, I suggest more formal and accurate statements in the paper instead of using words like \"the aligned representation from f_g\".", "other_comments_or_suggestions": "- In lines 188-190, \"the model fine-tuned on generated images in the Gen-CLIP flow is deployed to process real images without further fine-tuning.\" Could the author explain this statement more clearly? I suggest using the necessary formula and notation.", "questions_for_authors": "Please check the comments above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZaVAk3iQv5", "reviewer": "Reviewer_1EhT", "summary": "This paper introduces GMAIL, a novel framework designed to bridge the modality gap between generated and real images, a common issue that can cause mode collapse in training pipelines. The approach treats generated images as a distinct modality and aligns them with real images in the same latent space. By fine-tuning models on generated images while preserving a pre-trained model for real images, GMAIL achieves explicit alignment between the two modalities. This method results in significant performance gains across various vision-language tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's feedback and the questions raised. Below, we address and resolve each of these questions in our responses.\n\n> Clarification\n\nThank you for catching this ambiguity. We've clarified both the notation and purpose in the Table below.\n\n| Training   Data  | Alignment | B@4 ↑ | CIDEr ↑ | SPICE ↑ |\n|:----------------:|:---------------:|:----------------:|:---------------:|:----------------:|\n| Real only        | ✗         | 32.15 | 108.35  | 20.12   |\n| Mix   | ✗         | 36.15 | 115.35  | 22.95   |\n| GMAIL (ours)     | ✔         | 38.12 | 119.53  | 23.75   |\n\n- Real only: Standard CLIP fine-tuned solely on real images (baseline).\n- Mix: Real + generated data combined, no alignment applied.\n- GMAIL: Separate encoders, aligned via our method.\n\nWhy does “real-only” improve? Because fine-tuning ClipCap on real data improves over frozen pre-trained CLIP. However, it underperforms compared to models using generated data, especially GMAIL, which preserves real performance while benefiting from synthetic expansion.\n\n\n> Comparison to Tasks2Sim\n\nGreat point. We have added experiments comparing Tasks2Sim in the Table below. GMAIL outperforms prior mix-only methods due to explicit modality disentanglement and alignment, rather than blending synthetic and real data blindly.\n\n| Method       | B@4 ↑ | CIDEr ↑ | SPICE ↑ |\n|:----------------:|:---------------:|:----------------:|:---------------:|\n| Tasks2Sim       | 36.65 | 115.72  | 23.02   |\n| GMAIL (ours)    | **38.12** | **119.53**  | **23.75**   |\n\n> Embedding Comparisons\n\nThank you for the suggestion! We calculated the average Cosine Similarity Between Modalities including Real, Gen, and Text in the Table below. Real vs. Gen gap is smaller than image-text, but still substantial, justifying separate modality treatment. Although the cosine similarity scores for Real vs. Text (0.44) and Gen vs. Text (0.42) appear numerically close, both are substantially higher than the similarity between Real vs. Gen (0.25). This discrepancy shows a significant gap between real and generated images, indicating that each type of image is more semantically aligned with textual descriptions than they are with each other.\n\n| Pair          | Cosine Similarity ↑ |\n|:----------------:|:---------------:|\n| Real vs. Gen  | 0.25                |\n| Real vs. Text | 0.44                |\n| Gen vs. Text  | 0.42                |\n\n\n> Citation of LeCun (2022)\n\nThank you for pointing this out. We originally referred to the broader concept of representation collapse (often called \"mode collapse\") in the context of over-optimization, as discussed in LeCun's vision report starting from Section 4.3 (\"Training Energy-Based Models,\" p. 20 onwards). Specifically, LeCun describes how certain training methods risk collapsing learned representations, thereby reducing their generalization capacity.\n\n> Typo\n\nWe fixed it.", "claims_and_evidence": "Yes. The claims made in the submission are fully supported.", "methods_and_evaluation_criteria": "Overall, the proposed methods and evaluation criteria are very reasonable.", "theoretical_claims": "The theoretical claims are correct.", "experimental_designs_or_analyses": "There are a few questions I still have regarding the experiments:\n\n1. Training on real images in the supplementary materials – The section mentions a comparison of COCO captioning performance using three strategies: real-only, mixed real-generated data, and GMAIL alignment. However, the corresponding table does not clearly indicate which settings correspond to real-only and mixed real-generated data. This is especially confusing for the only real data fine-tuning setting, as the entire paper discusses aligning real and generated data. If only real data is used, what exactly is being aligned? Moreover, why does this setting still show performance improvement? \nBesides, given that prior works (just for example, Tasks2Sim, From Fake to Real) have already demonstrated the effectiveness of mixed training, this paper should further experiment and discuss its impact on mix-pretrained models.\n\n2. Embedding visualization – Typically, I would not consider real and generated images as different modalities. The visualization only shows the existence of a gap between them but does not prove that the gap is as significant as that between images and text. To substantiate the claim that generated data should be treated as a separate modality, the comparison should include embeddings from traditionally different modalities, such as images and text.", "relation_to_broader_scientific_literature": "There is no relationship to broader scientific literature.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Overall, although mixing generated data and real data for training may not score highly in terms of novelty, this paper presents a well-structured and clearly articulated approach when specifically focusing on fine-tuning pre-trained models with generated images. The claims are clear, the methodology is well-explained, and the experiments are thorough. Additionally, the supplementary material provides ample and effective supporting experiments to further validate the proposed method.\n\nHowever, the experimental section has certain weaknesses. Please refer to the corresponding section for details.", "other_comments_or_suggestions": "Typo：line 84：tje should be the\n\nWhen mention \"mode collapse\", the author cited \"A path towards autonomous machine intelligence version 0.9. 2\", where I cannot find this phrase.", "questions_for_authors": "No further questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QJpsPwNLxM", "reviewer": "Reviewer_NFLh", "summary": "The paper introduces GMAIL (Generative Modality Alignment for generated Image Learning). It is a framework for incorporating generated images into training pipelines while explicitly addressing the modality gap between real and generated images. Instead of treating synthetic and real images interchangeably, GMAIL fine-tunes a separate model for generated images and aligns it with a pre-trained model for real images. This alignment occurs in a shared latent space, enabling vision-language models (VLMs) like CLIP, LLaVA, and LLaMA3 to effectively utilize generated images for tasks such as image captioning, zero-shot image retrieval, and zero-shot image classification.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's feedback and the questions raised. Below, we address and resolve each of these questions in our responses.\n\n> Domain Adaptation Literature\n\nOur approach is conceptually related to domain adaptation techniques [a,b], where the goal is to align feature spaces between source and target domains. However, GMAIL focuses on a unique modality gap between synthetic and real images, both from the same caption domain, requiring a different solution strategy. While traditional DA aligns features across different datasets, our method aligns representations across modalities (real vs. generated) within the same semantic context, using caption anchors.\n\n[a] Tzeng et al., Adversarial Discriminative Domain Adaptation, CVPR 2017\n\n[b] Ganin & Lempitsky, Unsupervised Domain Adaptation by Backpropagation, ICML 2015\n\n\n> References to Data Synthesis for Vision Tasks\n\nThank you for the suggestion! We’ve now added and discussed all three works in Related Work:\n- DatasetGAN (CVPR’21) [1]: Efficient labeled data generation via GANs and semantic segmentation.\n- SegGen (ECCV’24) [2]: Mask2Img diffusion pipelines for boosting segmentation.\n- DiffuMask (ICCV’23) [3]: Pixel-level annotations via text-to-mask-to-image synthesis.\n\nWe emphasize that GMAIL is complementary to these efforts:\n- They focus on generating high-quality labeled datasets.\n- We focus on bridging modality gaps when using synthetic data for training vision-language models.\n\n[1] DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort. CVPR 2021 \n\n[2] SegGen: Supercharging Segmentation Models with Text2Mask and Mask2Img Synthesis. ECCV 2024 \n\n[3] DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Models. ICCV 2023\n\n\n> Generator Generality\n\nWe have now conducted experiments using FLUX, which introduces a more powerful and differently parameterized generation pipeline compared to SD2. These additional results allow us to test GMAIL's robustness to shifts in generator-specific artifacts. The performance improvements remain consistent with FLUX, indicating robust alignment across varying artifact styles and photorealism levels.\n\n| Generator       | B@4 ↑ | CIDEr ↑ | SPICE ↑ |\n|:----------------:|:---------------:|:----------------:|:---------------:|\n| FLUX            | 37.2  | 117.82  | 23.4    |\n| FLUX + GMAIL (ours)    | **39.54** | **122.36**  | **24.15**   |\n\n\n> Robustness to Adversarial or Out-of-Distribution (OOD) Inputs\n\nThank you for the insightful observation. To evaluate the robustness of our alignment-based GMAIL framework, we conducted two evaluations.\n\n1) We evaluate GMAIL and baselines on ImageNet-A and ImageNet-R — well-established OOD benchmarks that test model resilience under distribution shifts. Naively training with generated images slightly reduces robustness, but GMAIL improves OOD generalization by aligning synthetic features more closely with real ones.\n\n| Model              | ImageNet-A ↑ | ImageNet-R ↑ |\n|:----------------:|:---------------:|:----------------:|\n| CLIP               | 51.2         | 66.7         |\n| CLIP + Gen. Images | 49.8         | 67.3         |\n| GMAIL (ours)       | **53.6**         | **69.1**         |\n\n2) We tested models using AutoAttack on zero-shot classification over ImageNet-1k (100-class subset). GMAIL improves clean and adversarial robustness, suggesting that alignment avoids overfitting to synthetic artifacts, a key cause of fragility in other generative training schemes.\n\n| Model        | Clean Accuracy ↑ | Robust Accuracy ↑ |\n|:----------------:|:---------------:|:----------------:|\n| CLIP         | 75.6             | 43.7              |\n| CLIP + Gen. Images   | 77.0               | 41.2              |\n| GMAIL (ours) | **78.1**             | **46.3**              |", "claims_and_evidence": "The primary claim is that treating generated images as a separate modality and aligning them with real images improves performance across various vision-language tasks. The evidence is compelling:\n\n1. In Tables 1-5, consistent improvements across all evaluated metrics for multiple tasks.\n2. In Table 6, ablation studies confirm the importance of the alignment component.\n3. In Table 7, positive scaling trends show continued improvement with larger synthetic datasets.\n4. In Table 13, quantitative similarity metrics show better alignment of real and generated image embeddings with their approach.\n5. Figure 3 shows qualitative visualizations demonstrating the modality gap and how alignment bridges it.", "methods_and_evaluation_criteria": "The methodology is well-explained and justifiable:\n\n1. They utilize Stable Diffusion v2 to generate synthetic images from captions in existing datasets\n2. A dual-encoder architecture separates generated and real image processing\n3. Low-Rank Adaptation (LoRA) is used to fine-tune efficiently\n4. Cross-modality alignment loss encourages similar embeddings for real and generated images with the same description\n\nThe evaluation is comprehensive, covering:\n\n1. Image captioning using standard metrics (BLEU, METEOR, CIDEr, SPICE, etc.)\n2. Zero-shot image retrieval on COCO and Flickr30k\n3. Zero-shot image classification across eight datasets\n4. Long caption retrieval on ShareGPT4V\n5. Visual question answering on ScienceQA", "theoretical_claims": "The theoretical foundation is sound:\n\n1. Identifying a fundamental issue (modality gap between real and generated images)\n2. Providing a clear theoretical explanation of why this gap leads to problems\n3. Presenting a theoretical framework to address it\n\nThe explanation of the modality gap and its effects is well-developed, particularly in Appendix C where they contrast single vs. dual modality approaches and explain why their cross-modality alignment is crucial.", "experimental_designs_or_analyses": "The experimental design is thorough and well-executed:\n\n1. Multiple datasets of varying scales are used\n2. The method is evaluated on diverse tasks to demonstrate generalizability\n3. Appropriate baseline comparisons are included\n4. The ablation studies isolate the contributions of different components", "relation_to_broader_scientific_literature": "The authors position their work well within the broader literature on:\n\nDiffusion models for image generation\nGenerated image learning\nVision-language models\n\nThey appropriately cite related work in each area and clearly articulate how GMAIL advances beyond previous approaches, particularly in addressing the modality gap problem.", "essential_references_not_discussed": "The paper is well-referenced, but a few additional references could strengthen it:\n\n1. Work on domain adaptation and domain gap in computer vision, as this bears similarity to the modality gap problem\n2. More recent work on synthetic data generation for computer vision tasks beyond those cited", "other_strengths_and_weaknesses": "Additional Strengths\n\n1. The GMAIL framework addresses a real-world problem that will become increasingly important as generated images become more prevalent in machine learning pipelines.\n2. The approach demonstrates effectiveness across different backbone architectures (CLIP, Long-CLIP, LLaVA, LLaMA-3), suggesting it's a general solution rather than one tied to a specific model architecture.\n\n\nAdditional Weaknesses\n\n1. The experiments rely exclusively on Stable Diffusion v2. Testing with other generative models (e.g., Stable Diffusion 3, FLUX) would better demonstrate the generality of the approach, as different generators may introduce different types of artifacts.\n2. There's no exploration of whether the alignment approach makes models more or less robust to adversarial examples or out-of-distribution inputs, which would be valuable for understanding practical limitations.", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper receives all positive ratings from the reviewers. They appreciated the technical contributions, the solid experimental validation, and the presentation quality. AC agrees with the comments of the reviewers and recommends an acceptance for this submission.
Decision: Accept (spotlight poster)